"The message has been about continuing to improve as a side. We know we're in the quarter-finals, but we want to finish top of the group and win four from four," said Gatland.
Wales have won their three opening games in Pool C and are already qualified for the quarter-finals, having started with a 32-26 victory over Fiji before seeing off Portugal 28-8 and sailing past Australia 40-6.
The Georgians are yet to win in France after a 35-15 loss to Australia, an 18-18 draw with Portugal and a 17-12 defeat by Fiji.
It was only in November last year, however, that Georgia stunned Wales with a 13-12 victory in Cardiff.
There are 15 survivors in the Georgian match-day 23 from that win 11 months ago, including 11 in the starting XV.
But Gatland insisted that talk had not been about that loss.
"I only watched the first half. I was on my way to Twickenham," said Gatland, whose team will play the winners of Sunday's Pool D match between Argentina and Japan in the quarter-finals.
"Georgia are a tough team to put away. I was confident Wales would win. It's a learning process. We're not looking back. We're only looking forward. We're pleased with the progress we've made. We're excited about where we are."
Gatland is back at the helm for a second tenure with Wales, having overseen three Six Nations Grand Slams and two World Cup semi-final appearances in his first between 2007-19.
Tough team to beat
He took over from Wayne Pivac with Welsh rugby riven by problems, but the Kiwi insisted those woes were behind the team.
"I think there were a lot of things happening off the field and they obviously had a significant impact that I probably didn't realise at the time," he said.
"There is a line in the sand under that and the focus in this group has been about the rugby and building confidence.
"They have worked incredibly hard and I couldn't have asked for more in terms of what this group has put in so I think making a comment or statement like that is about the confidence from not only what I say but from how hard they have worked.
"If we go in there and implement the game plan and have self-belief then we know we are a tough team to beat.
"If you're a tough team to beat, often the performance and results take care of themselves. You get a group of players who play for each other."
Georgia coach Levan Maisashvili said his team had been searching for a more balanced game.
"In this World Cup against every team we try our best. We have been disappointed (with no wins) but definitely against Wales a good result for us will only be winning the game," he said.
"If you watch our game, not only at the World Cup but in the last year or two years, we started to play a more balanced game because everyone understands if you want to go and win big games against tier-one countries you cannot play only this heavy game with just scrummaging or mauling."
Georgia skipper Merab Sharikadze, back at centre in one of five changes from the loss to Fiji, said harking back to the victory in Cardiff helped no one.
"They are playing much, much better than they did last year," he said.
Sharikadze will be lining up against Nick Tompkins, one of the stand-out players for a resurgent Wales.
"We want no dip in our performance," Tompkins warned.
"We want to get better, we don't want to take our foot off the pedal - that's the last thing we want. We want confidence and momentum and keep on going."
