Philippe Clement's side were rocked by Sam Dalby's opener for United in the 19th minute.
But Rangers midfielder Mohamed Diomande levelled after 37 minutes and Robin Propper put the visitors ahead shortly after half-time with a deflected drive.
Cyriel Dessers, who had scored a stunning equaliser in the 2-1 Europa League defeat by Manchester United on Thursday, wrapped up the points when he netted in the closing stages.
There was more drama when Diomande, returning after being suspended for the game at Old Trafford, was controversially sent off in the fifth minute of stoppage time by referee Nick Walsh for tangling with United defender Kevin Holt.
Clement's side still have little chance of catching leaders Celtic, who have played a game less after their match on Saturday against Dundee was called off due to weather damage to their stadium.
Rangers had fallen behind on seven previous occasions in the league this season, drawing three and losing four, but they finally ended that disappointing streak with a gritty fightback.
Dalby, on loan from Wrexham, pounced after the Rangers defence failed to deal with a long throw from Will Ferry, forcing the ball high past Jack Butland from 10 yards to take his tally to 11 for the season.
Rangers equalised when Diomande took a pass from James Tavernier and drilled a shot from the edge of the area past keeper Jack Walton.
United remained dangerous and Butland made a point-blank save from Dalby, while Walton made stops from Dessers and Nedim Bajrami.
Rangers took the lead four minutes after half-time when the unmarked Propper took a pass from Ianis Hagi inside the United area, turned and shot with the ball coming off Holt and past the helpless Walton.
Rangers midfielder Nicolas Raskin turned a Dalby cross against his own post before Nigeria striker Dessers drilled home from Zak Lovelace's pass in the 86th minute.
Diomande was dismissed after tangling with Holt, the official sticking with his original decision after looking at his pitchside monitor at the behest of VAR.