When modern-ish helmets first came about, you were considered a pussy for wearing one unless you had a head injury before. It might have been more acceptable if you were a pure finesse player, but back then, everyone was expected to fight and to be able to handle themselves. That part of the game has gone away since helmets became mandatory, it became more of the domain of the goon who could destroy his hands on helmets and not worry about his career.
Its different down in the CHL. In the NHL, if you play a certain way, its expected that you shouldn't wear a visor. Its fine for a guy like Iginla to play a hard, clean style and wear one, but it would be completely unacceptable for a guy like Lucic to. If you cross the line once in a while you have to back it up, and you don't hide behind visors, thats just part of the culture. A guy like Evgeny Artyukhin who hides behind the huge visor is not a respectable player at all, thats not where I want the league to go.
You'd have tons of guys breaking their hands on visors, like Mark Stuart last year when that faux tough guy Simmonds jumped him. Fights already aren't what they used to be, if everyone had visors it would be a complete joke, it would be wrestling with an uppercut here or there.
Plus there is the risk of getting sticks caught in between your visor and your face, bouncing around your eyes. Tons of guys already get cut by visors, and I think they contribute to the sense of invincibility and lack of respect that many younger players have. They don't keep their sticks down anymore, they don't hesitate to hurt people, and they also don't protect themselves the way they used to.