The hosts had led 13-0 at the break before an impressive comeback from the French side, who had a 22-16 advantage before Libbok failed with his kick following Suleiman Hartzenberg's try.
La Rochelle, bidding for a third successive Champions Cup title, will face the winner of the game between four-time winners Leinster and Leicester.
Ronan O'Gara's side were handed a pre-match boost with giant Australia lock Will Skelton returning after two months out with injury for one of five repeat fixtures from the group stage in the last 16.
The Stormers had four Rugby World Cup winners in their matchday squad, including fly-half Libbok who kicked the winning conversion when the sides met earlier in the season.
The hosts led 13-0 before the half-hour mark as Libbok slotted eight points and scrum-half Herschel Jantjies crossed for a try.
Stormers coach John Dobson had predicted a Test match intensity to the fixture against La Rochelle's sizeable pack and he lost one of his three starting back-rowers after half an hour as flanker Ben-Jason Dixon was forced off for Hacjivah Dayimani.
Things worsened just before the break for the Stormers as Deon Fourie followed fellow flanker Dixon off the field, but the home side remained in control.
Flanker injuries
Despite the injuries, the Stormers stretched their lead to 16-0 with a third Libbok penalty, but they then lost a third back-rower as Dayimani was replaced by usual second-row Adre Smith.
That blow sparked the La Rochelle comeback.
Prop Louis Penverne and No 8 Gregory Alldritt crossed either side of a penalty from fly-half Antoine Hastoy to put La Rochelle 17-16 up with 18 minutes to play.
La Rochelle extended their advantage with a 68th-minute touchdown from Argentina prop Joel Sclavi but Hastoy missed the conversion, setting up a tense finish with the Stormers within a converted try from a quarter-final spot.
The drama increased with five minutes left as La Rochelle winger Dillyn Leyds, playing against his former side, was shown a yellow card near the hosts' line after a break from Damian Willemse.
The South Africans had a scrum five metres out before winger Hartzenberg crossed with 35 seconds to play from a delicate Libbok cross-kick, setting the latter up for a match-winning conversion with the clock in the red.
Libbook stepped up, with Willemse holding the ball on the tee due to windy conditions in Cape Town, but his effort drifted wide as O'Gara's champions survived.
Elsewhere, Exeter downed English rivals Bath 21-15 and will meet either Toulouse or Racing 92 in the next round, with the French clubs meeting on Sunday.
Bordeaux-Begles saw off Saracens 45-12, having thumped the three-time former champions 55-15 in the pool stage, thanks to doubles from France winger Louis Bielle-Biarrey, centre Nicolas Depoortere and fly-half Mateo Garcia.
The French club will host Harlequins in the last eight after the English outfit beat Glasgow 28-24 on Friday.
"We had to be patient and trust our systems and if we held the ball, build pressure, we were sure it would give at one point," Bordeaux-Begles's prop Ben Tameifuna told TNT Sports.
"We're one step closer to what we want to achieve as a group."
There is also repeat on Sunday of the 2000 final as Munster head to English Premiership leaders Northampton for a chance to meet the Bulls, who hammered a second-string Lyon 59-19.
