Williams had enjoyed her first victory over a top-50 opponent in four years on Monday when the former world number one beat Camila Giorgi in her Birmingham opener.
But the 43-year-old, a seven-time Grand Slam champion, was unable to make it two in a row as she succumbed to a hard-fought 6-3, 5-7, 6-3 loss to Latvia's Ostapenko on the Edgbaston grass.
Williams briefly rolled back the years when she saved a match point at 3-5 in the second set, then reeled off three successive games as second-seeded Ostapenko lost her rhythm.
After Williams moved 2-0 ahead in the deciding set, it appeared the five-time Wimbledon winner might be on course for the last eight.
But Ostapenko, a former Wimbledon semi-finalist, recomposed herself and eventually shattered Williams' hopes of a morale-boosting run at Birmingham ahead of the start of Wimbledon on July 3rd.
"Of course I could finish it a bit quicker. I got a little bit disappointed after the match point in the second set. But Venus is a great player, a great champion and it's an honour to share the court with her," former French Open champion Ostapenko said.
"She's a very dangerous player, especially on grass. She was serving really well.
"It was a little bit hard, but I'm really happy that I managed it and I was fighting until the very last point. She's an idol for a lot of people so it was very special."
Williams will hope to recover from a knee injury that dogged her in Birmingham after being awarded a wild card entry for Wimbledon, where she made her debut in 1997 and reached her most recent Grand Slam final in 2017.
In the quarter-finals, Ostapenko faces Poland's Magdalena Frech, who battled past eighth seed Sorana Cirstea 6-3, 6-7(1), 6-4.
Britain's Harriet Dart will take on fourth seed Anastasia Potapova, who defeated American Caty McNally 6-3, 2-6, 7-6(0).
Meanwhile, there will be an all-Czech quarter-final between top seed Barbora Krejcikova and 18-year-old Linda Fruhvirtova.
Fruhvirtova was already through to the last eight and former French Open champion Krejcikova joined her by seeing off Tereza Martincova 6-4, 6-4.
