It leaves world number one Aryna Sabalenka as favourite to claim her first title in Paris, though Kostyuk will be one to watch as she builds the momentum that has helped her sparkle on clay this season.
"I'm still in shock," Kostyuk said in an on-court interview.
"I feel like I've given myself more space to just create something, to challenge my opponents. I woke up in the morning and all I thought was 'what an unbelievable day I have to live today ... there's nothing I could do other than this'.
"I try not to focus at all on winning and losing, because I'm not playing tennis to win. I'm playing tennis because I love it. I want to connect to people, I want to feel this energy ... make people happy and unite people."
There was little to separate the duo in the early exchanges as they twice traded breaks, before Kostyuk came up with a tight hold in the 11th game and raised her level at the end to claim the opening set with a backhand crosscourt winner.
It was the first time that Kostyuk had taken a set off the third-seeded Pole after three straight defeats in their previous meetings, and she sensed a big upset when she went ahead 3-1 in the next set after a battling effort.
The Rouen and Madrid champion held her nerve from there to take her record on clay this season to 15-0, and book a meeting with seventh-seeded Ukrainian Elina Svitolina or 11th-seeded Swiss Belinda Bencic in the next round of Roland Garros.