If you loved Shadow of the Colossus, you will probably love the Prince of Persia series. Sands of Time feels the most like playing a fairy-tale. Warrior Within is similar to GOW (actually it is more correct to say that GOW is similar to WW).
Play God of War since you already have it before you decide whether to buy GOW2. You've waited this long, GOW2 will undoubetdly make a drop to $10 or under anytime now so you might as well wait for that.
MGS3 is a prequel to the others so go ahead and play it even though you haven't finished 1 or 2. If you like it go for the Metal Gear Collection. Metal Gear is definitely a must have series IMO.
If you are interested in Persona 3, you might as well get Persona 3 FES instead since it gives you more bang for your buck.
Since you loved SOTC you might as well get ICO if you can find a good deal on it. There was a recent thread here about ICO that discusses its pros and cons that you can check out.
Resident Evil 4 and Okami are must have games.
Since you seem to like action/adventure games I would say that another must have game that is not on your list is Devil May Cry 3:Special Edition.
There are so many great action/adventure games on the PS2 its hard to pick just a few. I'll second the recommendation on the Onimusha series though I'm not sure it qualifies as a must play. My personal favorite is Demon Seige but if I remember correctly a lot of people in general didn't like it. I also liked the 2nd Onimusha a lot, but it definitely has clunky, dated controls.
A lot of people say Silent Hill 3 is a must have. I personally thought it was boring and the controls made it a chore to play. I think the people who love it so much mostly played it when it came out. IMO it is very dated.
If you want a horror game that is great (but is only a must play for horror fans) check out Haunting Ground.
If you like platformers then one of the games from the Ratchet & Clank series (except Deadlocked) is a must have. The Jak & Daxters and the Sly Coopers are also great, but Ratchet is the best. Actually PS2 has a lot of great platformers. Psychonauts and Kya Dark Lineage are some other great ones. If you like action/adventure type games you will probably like these platformers.
I'll second the recommendation for the Disgaeas -- but only if you are the kind of person who likes SRPGs.
There are plenty of great RPGs on the PS2. I'm going to have to say that FFX would be the top must-have one, if only because it is the one PS2 RPG that everyone likes to argue about the most. You need to play it so that you can join the many threads about its merits or lack thereof.

My personal favorite RPG is Shadow Hearts:Covenant.
There are too many must-have games for the PS2. It is a great system.