Component video is capable of handling 1080p, and the video quality of HD video over component cable is generally indistinguishable from HDMI. Where it gets sticky is the fact that most TVs employ HDCP, a copy protection scheme that will only allow 1080p over HDMI. So if you had an older HDTV that would allow 1080p over component, then you're good.
Digital sound from optical cable on the 360 is also about as good as you're going to get on the 360, too. I don't think there's a noticeable difference in quality there, either.
However, as mentioned above, few games run in 1080p, and most people can't really tell the difference between 720p and 1080p, even on a 55" or 60" TV.
If you were talking about a PS3 (or a blu-ray or HD-DVD player...if anyone has those anymore), there'd be more reasons to care about HDMI, since HD audio formats do require HDMI.
I'm not sure how many people can tell the difference between Dolby TrueHD or regular old Dolby 5.1, though, so that's also not exactly a dealbreaker for the majority of folks. At that point, you're talking about having a newer receiver that can decode HD audio streams, and chances are the HDMI ports on that receiver would work.