I always find rhythm games hard. It seems like sometimes I'm a hair off the beat, and I always feel like it's more a problem with the game than myself, although I'm probably just kidding myself.
Rhythm Heaven is a quality title, but if you have trouble with rhythm games, like myself, you'll get frustrated not too long in.