I just finished III yesterday, I have a lot to say about it. As others mentioned, the voices are back to good in III.
The problem with II and the beginning of III is the ambiguous story that tries so hard to be massive but does not do it in an easy fashion. If you wish to follow everything in the purposely hidden manner they want you to, you have to look up a lot of information that was left out or put into other games that did not come here.
The story of III, especially from the middle of the game on, really comes together. It leaves things open to quite a bit of interpretation on your own, allowing you to comprehend the story your own way. It is refreshing that every single character in the series has a true motive, it isn't just someone going mad or someone trying to be good. If the series could have done this story in an easier more relatable way throughout as they finally did at the end of III, it would easily be the best and most epic story of any game to date.
Xeno I and Xeno III had two of the best minigames ever made in my opinion.
The bad thing about all of Xenosaga...the battle system. Xeno I had a mediocre system, Xeno II was a little worse, and Xeno III...I didn't even bother with once I realized that every battle could be won with a single attack and you basically had unlimited times to do this. Xeno got caught up in the big numbers game, making enemies that had a few hundred million hitpoints so regular attacks that would do between 400-3000 damage were worthless, only super attacks were worth anything. A good game knows how to balance the number system, so that yes super attacks are more damaging, but regular attacks are also not obsolete and have their place.
So, ultimately the reason why Xeno did not do as good as it should have is that the developers spent their entire focus on storyline, crafting an immensely interesting backdrop and religious allegory, but did not know how to balance this story with comparable gameplay. Weak gameplay, excellent but too complexly shown story, excellent minigames. If the gameplay had been done much different, say in a Shin Megami type battle system, Xenosaga could have trumped Final Fantasy's series.