Celtic remain six points adrift of the Jambos, with a game in hand, but there was huge relief around Parkhead as they ended their worst run since 1978.
In a recurring theme of Nancy's matches since he arrived in Glasgow, the Frenchman could only look on aghast as his side missed a huge number of chances.
Benjamin Nygren finally broke the deadlock from Daizen Maeda's cross.
Dylan Lobban was then dismissed for denying a goalscoring opportunity when he brought down Maeda just before half-time.
Celtic hit the woodwork three times in the second period before they appeared to throw away another two points when Kenan Bilalovic broke clear to equalise 16 minutes from time.
Two of the Hoops' old guard dug them out of a hole as Kieran Tierney headed in on 88 minutes before James Forrest secured a much-needed win in stoppage time.
However, Nancy still has his work cut out to chase down Hearts.
Derek McInnes' men have won all four meetings with Celtic and Rangers this season to spark credible ambitions that they can become the first non-Glasgow side since Alex Ferguson's Aberdeen in 1985 to win the Scottish title.
The Edinburgh side have not won the top-flight since 1959/60.
Rangers had been unbeaten in eight league games since Danny Rohl took charge to rekindle their own title hopes, but were undone by two goals in four minutes just before half-time at Tynecastle.
Stuart Findlay headed in a well-worked corner before Lawrence Shankland embarrassed Jack Butland at his near post.
Hearts then saw out the second half comfortably until an error from Frankie Kent allowed Youssef Chermiti to pull a goal back with virtually the last kick of the game.
Rangers fall 12 points behind the leaders, albeit with a game in hand, and six adrift of Celtic.