Leading 3-2 from the first-leg, Arsenal won 1-0 at the Emirates thanks to Kai Havertz's injury-time winner.
Merson later said on Sky Sports: "I’m flabbergasted. I can’t believe what I’ve just watched.
"Chelsea aren’t a bottom-five team. They have World Cup winners.
"(Wesley) Fofana is crying. He should be crying because they never had a go. They’ve gone out with a whimper in a semi-final.
"It hasn’t worked. Go out in a blaze of glory, don’t go out like that."
In response, Chelsea manager Liam Rosenior said: "I’ve been a pundit. It’s easy. It’s easy in hindsight.
"If I go and attack the game, press really high and we can see two goals early, everyone says, ‘What is he doing?’
"That’s the reality of my job. The reality of my job is if you lose games, you’ll be criticised.
"If you win, you’re a genius. It’s normally somewhere in between."