I own and have played both. I've logged about 2 hours into Demon's Souls, fighting to the first boss only to die (after dying many times before that). I've also logged about 80 hours or so into Dragon Age on the PC, completing the game and seeing most of the game's content. I will try to add my perspective into the mix and hopefully you will find it helpful in making your decision...
I'll start with Demon's Souls. As others have mentioned, this game is difficult. Even more so it is unforgiving. It will take you on, chew you up, and spit you out. However, if you can endure, you will be stronger for it. For example, in my brief experience with the game, I died countless times due to hidden traps, enemy tactics, etc. From this failure, I began to learn where the traps are located and how to best avoid them, how to properly take on various enemies and so on making me a more effective/efficient player. Initially, this game can be very frustrating, but if you find it interesting and do not give up, I am sure it will pay off, providing one of the most rewarding experiences a game can offer.
Moving on, Dragon Age: Origins is an entirely different beast. If you have played any of Bioware's previous RPGs (Star Wars: Knights of the Old Republic, Jade Empire), you should feel right at home with Dragon Age. As you may know, you first create your primary character, determining the race and class you wish to play as. From your selection, you get to play through one of six origin stories. Upon completion of your origin, you are seamlessly introduced to the main story arc of the game. Combat sequences often require you to pause the game and choose actions for your party members.
To wrap things up, if you are more interested in an incredibly unforgiving yet rewarding real-time action RPG, go with Demon's Souls. If you are more interested in semi turn-based strategic RPG that places a much greater emphasis on story and character development, go with Dragon Age: Origins. All in all, I think both games are great and you will not be disappointed.
*Disclaimer: As mentioned, I played Dragon Age on the PC, which is, from what I hear, superior to the PS3 version. Not sure if this makes much of a difference.