Muchova, the seventh seed, overcame 50 unforced errors across the two-hour-and-52-minute match to beat Osaka 7-5, 1-6, 7-6(7).
Osaka, the current world number 17, surrendered a three-game lead in the first set after a series of misfired shots in the seventh game.
The 28-year-old who represents Japan dropped just one game to win the second set and saved eight match points in the third but could not get past Muchova's match-winning volley.
"I think she played always great and steady," the 30-year-old Czech player told the crowd.
"I was shaking. I'm happy I made the last volley... Naomi could be standing here. It's me, and I'm really grateful for it."
She next faces fellow compatriot Nikola Bartunkova.
Earlier, Russia's Mirra Andreeva beat Ukraine's Daria Snigur 6-2, 6-2 to book her quarter-final spot against fellow Russian Ekaterina Alexandrova.
Alexandrova, ranked 28th, upset Wimbledon champion Linda Noskova 6-2, 6-2, ending the Czech player's bid to return to the Beijing final that she reached last year.