Given the fact that most of them get on their bikes again and continu the race at least pretends to show that their self-confidence isn’t affected much. Perhaps because of a combination of experience and cycling attitude (don’t complain, ride!).
On the other hand, regular falls must drive you crazy and will almost certainly affect your SC. Effects might be seen during sprints at mass arrivals and going down hill: with lowered SC, one might just not take that extra bit of risk.