Yes, you can do that. I have my X41's plugged into my tv's USB port. Just keep them plugged into your TV instead of the 360 and you can use them for every component you have hooked up to it (dvd, Wii, PS3, av receiver, dvr, etc)
For those on the fence about getting these, the X41's really are...