FWIW, regarding ear damage, it is mainly the SPL - pressure level - at the ear canal which causes damage. The scale we use for that measurement is in dB. The issue with headphones isn't that they alone cause more damage, but that the way people usually use headphones, they get cranked well above 85dB *at the ear canal*.
Now, this problem can be compounded when you use sealed headphones like canal phones or DJ/studio heaphones designed to provide a tight seal around your ear and block out background. If your sinuses are clear and you don't crank them, it's not a big deal, but if you do, you are putting your ears under a lot of additional stress because this increases the overall pressure level (see the mythbusters about what happens with a massive stereo system in a sealed car? the dB goes up and the pressure takes the path of least resistance to get out). This is probably what your audiologist was getting at. But if you listen to sealed cans at a lower volume, they can actually be better on the bus and such than normal headphones because the overall dB level is lower at your ear without all the background noise.
Regardless, the simple answer is don't crank your headphones and it won't be a problem. Most people listen well over what they really should. If you have a day when you just have to jam loud, make sure to give your ears at least 24 hours of no loud noise to recover, just like when you go to a concert, and you should be fine.