Rangers had won their first leg against Czech side Jablonec but Vakhtang Tchanturishvili scored the only goal of the game to level the tie on aggregate.
Findlay Curtis and James Penrice missed in the penalty shoot-out for Rangers as the Czechs moved into the competition's league phase.
They were joined by Brighton, who thumped Norway's Tromso 4-0 to progress by the same score on aggregate.
Charalampos Kostoulas, Maxim De Cuyper and Mats Wieffer scored first-half goals to earn the Premier League side a comfortable advantage.
Olivier Boscagli sealed the win late on.
Nigerian Toluwalase Arokodare scored a brace as Ajax stormed into the league phase with a 5-3 victory over Sion, whose forward Winsley Boteli bagged a hat-trick, for a 9-5 aggregate success.
Italy's Atalanta avoided a potential shock as Gianluca Scamacca scored the only goal of the tie away to Hapoel Tel-Aviv.
Germans Freiburg, who lost in the Europa League final to Aston Villa in May, French side Monaco and Spaniards Getafe also progressed.
In the Europa League, Egyptian star Mohamed Salah's Trabzonspor were thumped 4-0 by Ferencvaros and dumped out of the competition 5-0 on aggregate, although they were hampered by two red cards.
Benfica and Anderlecht, two sides who have won European competitions in the past, both eased into the league phase, easily eliminating Aarhus and Kairat Almaty respectively.
Transfers, news and interviews: Turn on notifications for articles about your favourite players and teams.
How to turn on these notifications?
