The 45-year-old's future was decided by United chief executive Omar Berrada, director of football Jason Wilcox and co-owner Sir Jim Ratcliffe after a meeting in Monaco.
It comes in the wake of Sunday's dramatic 4-4 draw with Aston Villa at Old Trafford, which extended their winless league run to eight games.
"After a disappointing start to the season, the club has come to the extremely difficult conclusion that a change is necessary in order to achieve our ambitions," United said in a statement on Thursday.
"Michael is, and always will be, a Manchester United legend. Just as he did as a player, he has conducted himself with the highest standards of dedication and professionalism throughout his time back with the club.
"His leadership and achievements last season will always be fondly remembered."
Carrick is the second boss to be fired in as many seasons by United, who are enduring their worst start domestically in 40 years.
Reports claim United States head coach Mauricio Pochettino – the former Tottenham and Chelsea boss – is the leading candidate to replace him in the dugout.
Carrick took over as head coach of his former club in January following the departure of Ruben Amorim, in a deal that initially lasted until the end of the season.
Having steered United back into the Champions League with a third-place finish, he was rewarded with a two-year deal in May.
But things have unravelled since then and Carrick's position wavered, with United winning just once in the league and three times in all competitions.
After a 4-1 loss to Como in the Champions League, a 3-0 defeat at home to Bournemouth last month looked to be the final straw.
But United opted to stick with him and won one of the next three games – Roma in the Champions League.
Carrick represented United 464 times as a player, winning five Premier League titles, one Champions League and the FA Cup in a decorated career.
After retiring, he joined the first-team coaching staff where he served under Jose Mourinho and Ole Gunnar Solskjaer before undergoing his first stint as caretaker boss when the latter was sacked.
Carrick then joined Middlesbrough for two and a half years before making his return to Old Trafford in January this year.
