Arthur Okonkwo was Wrexham’s hero, saving Omari Hutchinson’s decisive fifth penalty to seal a famous victory -- their first win over a Premier League side since 1999.
Liberato Cacace and Oliver Rathbone struck in quick succession in the first half, before Dominic Hyam appeared to settle matters midway through the second period, powering in a header from a free-kick to make it 3-1 to the Dragons.
Sean Dyche, having made eight changes from Forest's Premier League win over West Ham, sought inspiration from his bench.
The introductions of Neco Williams, Nicolas Dominguez and Morgan Gibbs-White changed the tempo, and Igor Jesus pulled one back in the 64th minute after his initial header hit the post.
The fightback gathered momentum, and Callum Hudson-Odoi dragged Forest level with two goals in 13 minutes, the second arriving a minute from time to force another half-hour.
With neither side able to find a winner, the tie was settled from the spot, where Okonkwo’s intervention proved decisive.
"It was a rollercoaster," Wrexham boss Phil Parkinson told TNT Sports. "The late goal knocked the stuffing out of us. We had to rally the troops a bit; the whole ground had gone flat."
Reflecting on the shootout, he added: "I didn't watch it, I was staring at the crowd and waiting for their reaction. Arthur has produced the save and will be the hero, and rightly so."
Elsewhere, League One Wigan claimed a notable scalp with a 1-0 win at Preston. Harrison Bettoni’s deflected strike decided a scrappy contest, with goalkeeper Sam Tickle producing late saves to preserve the victory.
Port Vale edged past Fleetwood Town by the same scoreline, while Oxford United advanced after winning a penalty shoot-out against MK Dons.
Results
Wrexham 3-3 Forest (4-3 on pens)
MK Dons 1-1 Oxford Utd (3-4 on pens)