Bazeley's side had been pegged back twice by Iran in their opening 2-2 draw, and they took a 1-0 lead in Vancouver before Mohamed Salah turned on the style in the second half, scoring one goal and creating another as Egypt secured a first win at the World Cup.
Egypt, who drew 1-1 with Belgium in their opener, top the group with four points while New Zealand are bottom with one. Belgium and Iran have two points after their 0-0 draw.
"It's frustrating," Bazeley said. "We played so well in the first half. We scored a great goal, created lots of chances, felt like we were dominating possession a lot of the time in the first half, and we were comfortable. We weren't really getting hurt.
"We talked well at half-time, looked at some things we can do a little bit better, we went out second half and we just weren't able to recreate the tempo and quality that we showed in the first half."
World Cup 2026
The 2026 World Cup will be held from June 11th to July 19th in the United States, Canada and Mexico. The tournament will feature 48 national teams and will be played in 16 modern stadiums.
Match schedule and times | Group tables | England at the World Cup | Scotland at the World Cup | Team rosters at the World Cup | How to watch the World Cup | Predictions and odds | Which teams could be considered dark horses?
After defender Finn Surman had put the Kiwis ahead in the 15th minute, Egypt came storming out of the blocks after the break and it was only a matter of time before they got back on level terms, with Mostafa Zico getting the equaliser.
Salah then put them ahead before Mahmoud Trezeguet got their third.
"Egypt came out and obviously came out with a different attitude and upped the tempo," said Bazeley. "I thought they played well second half, but it's frustrating to end up losing a game. That could be us celebrating on the pitch at the end.
"They've got some really good players, and Salah scores a goal that I've been watching him score for 10 years, coming in and left foot. But, yeah, overall, I think we defended players like him really well most of the game."
New Zealand striker Elijah Just told Reuters they would not underestimate group favourites Belgium despite the European side drawing their first two matches.
"You can look at it one way, where the draws are maybe not so flattering to Belgium when they're probably favourites to get out of the group," he said.
"At the same time, we know how hungry they're going to be, they need to beat us to get out of the group, so yeah, we're definitely not underestimating them because of the draws."