I don't know where you live, but where I live, homeless people are not friendly folk just lookin' for some conversation. They are usually drugged up, or drunk & if you don't give them money can get quite stabby. I will always lean on the side of caution. Doesn't make me a douche, just a realist.
Or they follow you your whole trip bothering you and get aggressive when you say no.
Also, giving them anything makes more people show up and ask for money.