The 29-year-old Dane, who was sitting second overall, went down hard on a bend as his Visma-Lease a Bike team were setting a fierce pace to chase down the day's breakaway.
Vingegaard was visibly in pain following the crash, which caused another four riders to also fall.
He was then helped into a medical vehicle with his arm in a sling.
This was the first time that Vingegaard had been unable to finish a Grand Tour.
In his five previous participations at the Tour, he won twice and finished second in the other three.
Race leader and reigning champion Tadej Pogacar's teammate Isaac Del Toro, who was seventh overall, was one of the other fallers, although he was able to set off again soon afterwards to rejoin the peloton.
Vingegaard's American teammate Sepp Kuss was another to tumble and he took a lot longer to get going again.
When he did, he was a long way behind the group of leaders.
Vingegaard's crash came just hours after he and Pogacar were amongst a number of riders to have been woken up at their hotels in the night by anti-doping controllers.
Vingegaard was woken up at 02:00 local time, while Pogacar was disturbed three hours later.