Agreed. Most people with money, rich or not-so-rich, eventually spend that money. If that money gets spent, it gets taxed.
I think there would probably be some organization that does the actually 'collection' and accounting, but the problem with the IRS is that it's just such a huge powerful inefficient behemoth with virtual immunity [Something like 46% of the tax questions the IRS is asked, are answered wrong--and of course it's the taxpayers fault if they follow the wrong instructions.
I would love to see the IRS go away and the tax code vastly simplified; I don't think it'll happen, unfortunately. Most of a politician's job consists of one of three things:
* promising things to his constituents, or certain parts of his constituency
* running for re-election
* being schmoozed by lobbyists and smaller governments to get a 'tax break' of some sort for their specific industry/organization/city.
Many people would lose their jobs--IRS agents, accountants, lobbyists--but many of those jobs are parasitic anyway. And many of those skills could easily be transferred to benefit private industry.