Sunderland were holding onto a 2-1 lead from a first-leg triumph over Coventry, but found themselves needing another goal after Ephron Mason-Clark breached their net.
The match went into extra time and the Black Cats bagged their winner in the final moments through Daniel Ballard, who had been solid at the back.
United's path to the play-off final was vastly more straightforward as they thrashed Bristol City 6-0 on aggregate.
The Championship play-off final will take place on 24 May at Wembley.
Sunderland's hopes to return to the Premier League after eight years - which included a stint in League One - are still very much alive.
However, Regis Le Bris' side must improve after a shaky performance that nearly handed their spot in the final to Coventry.
Revived under Chelsea legend Frank Lampard, after sitting 17th in November, the Sky Blues were unlucky to see their push for promotion extinguished.
Despite losing five matches on the spin before the play-offs, Sunderland showed resilience to reach the final in remarkable style.
Following their 2-1 victory in Coventry, the Black Cats' staff were accused of moving advertising boards closer to the pitch to disrupt long throws.
Coventry almost pulled level early in the second leg when Mason-Clark tested Anthony Patterson, before Jack Rudoni missed a clear chance as half-time approached.
The second period saw Sunderland sit deep, in an attempt to protect their one-goal advantage, as Coventry pressed.
Sunderland nearly sealed it through Trai Hume, but Ben Wilson made a superb save to deny him.
Coventry finally broke the deadlock in the 76th minute to draw level in the tie, with Mason-Clark making no mistake from close-range.
Haji Wright had a golden chance to win it for Coventry in stoppage time, but his effort went wide and extra time it was.
Ballard secured Sunderland’s place in the final with a header in the 122nd minute, sparking wild celebrations.
