Wow, I have to say that I'm really impressed with the (for the most part) level-headed responses in this thread. Kudos, CAGs!
Anyway, I'm admittedly not a lawyer or legal scholar of any sort so I can't give any factually-based opinion on whether the guy was trespassing or not by not leaving...