Matthias Jaissle's side were inches away from falling behind in the closing stages at The Den when Millwall striker Tairyk Arconte hit the post.
But the Magpies made the most of that escape to snatch the third-round victory in the 81st minute.
Willock met Lewis Hall's low cross with a close-range shot that trickled into the far corner after taking a deflection.
Second-tier Millwall brought on 15-year-old forward Regan Anderson to become the club's second-youngest ever player in the final minutes.
But Newcastle survived to keep alive their bid to win the League Cup for a second time in three years.
The Premier League club are unbeaten in five matches in all competitions under new boss Jaissle, who was the surprise choice to replace Eddie Howe when he stepped down in August.
Marco Rose secured his first victory as Bournemouth boss as Ryan Christie's hat-trick inspired a 4-0 win against second-tier Lincoln at the Vitality Stadium.
Scotland forward Christie netted in the seventh, 69th and 88th minutes for his second career hat-trick before Rayan wrapped up the victory in stoppage time.
Rose had overseen draws with Newcastle and Everton and a defeat at Manchester City in his first three matches in charge since replacing Andoni Iraola.
Jorgen Strand Larsen struck twice to fire Crystal Palace to a 3-0 win against Championship side Middlesbrough at Selhurst Park.
Strand Larsen nodded home in each half before substitute Daichi Kamada lobbed goalkeeper Sol Brynn for the third.
Chemsdine Talbi clinched Sunderland's 1-0 win against Hull at the Stadium of Light.
A lacklustre meeting between the two Premier League sides was settled by Talbi's chipped finish in the 71st minute.
It was Hull's first defeat this season after three unbeaten games in the Premier League and a League Cup second-round win over Stoke.
Bradford won 3-1 at fellow third-tier side Leyton Orient.
Will Swan, Stephen Humphrys and Nick Powell netted for Bradford, with Orient's reply coming through a Macaulay Gillesphey own goal.
Chelsea host Leeds in Wednesday's third-round tie, while holders Manchester City, Arsenal, Manchester United and Liverpool all play their ties next week.