You're not really able to do a whole lot with low profile cards and slim designs.
The best video card you can really do,
http://www.newegg.com/Product/Product.aspx?Item=N82E16814102882
Radeon 5450. The *450 series is about as high as you can go. There are more powerful low profile cards after the *450 series, but the price jumps from $20-$30 to about $150 with not a huge diff in performance.
http://www.newegg.com/Product/Product.aspx?Item=N82E16814121439
Is the 6450, which is slightly newer, but you won't notice much, if any difference between the two. And the 6450 will be $10-$15 more.
Get the 5450. It's cheap and will let you play games on low-mid. (considering your PC is old, and your CPU was budget when you bought it)
Also, optimize your PC. Ideally, reinstall Windows if you want the best gaming performance. Adding a video card to a machine bogged down with years of crap isn't going to do much for you.
You won't need a new power supply.
Keep in mind this is pretty much the best thing you can do to extend the life of this machine, but it's still a cheap and temporary fix. You can probably get another year or two of light gaming with this solution.