I'm personally biased in that I dislike a good majority of the XBLA releases, but I've bought (mostly on sale):
Toy Soldiers
Duke Nukem 3D (classic!)
Shadow Complex (I regret this one, not a fan, though it is a high quality game.)
Serious Sam: The First Encounter
RISK Factions
Torchlight also comes out on the 9th (tomorrow). That will probably be one of the top XBLA games for a while. Its key value comes from massive replay value.
Castle Crashers is apparently a very popular game, as well, though I never bought it. Worth looking into. Most people I've met on XBL seem to own it.
Several people I know adore Castlevania HD, but it's not for everyone.
Other notable games:
DOOM
Peggle
Puzzle Quest 1 or 2
Defense Grid
Magic the Gathering
Monday Night Combat
Lara Croft: Guardian of Light
Also worth trying trials of anything that looks neat.