The 27-year-old, who won three medals at the 2024 Paris Olympics for Australia before switching allegiance to Britain, had been preparing for the start of the championships on Wednesday.
"British Cycling can confirm that Matt Richardson has been withdrawn from the Great Britain cycling team squad for the upcoming 2026 track cycling world championships in Shanghai, China," British Cycling said in a statement on Monday, without giving any reason for the decision.
The Times reported allegations that Richardson had made discriminatory remarks about ethnic minorities and joked "about his use of the N-word".
Richardson, whose partner is fellow British track star Emma Finucane, was expected to battle with Dutch great Harrie Lavreysen for gold in the sprint events in Shanghai.