The Cottagers went away from the London Stadium following their game against West Ham United with no points.
Having lost 3-2 and conceded two bad goals while playing out from the back, Silva told BBC Match of the Day: "The result didn't reflect at all what happened on the pitch. The best team lost the game. The team that tried to win the game and created chances didn't win as well.
“In football, you have to be strong at both ends to be a dominant team and mistakes that we made this evening cannot happen.
"We were penalised for the mistakes and it was completely our fault we didn't win the game.
"It's happened too often for us this season. It's happened over the last few weeks when we were playing too well. The goal conceded against Bournemouth and against Ipswich as well. We have been punished by our own mistakes."