The case for debt as a bad for taxpayers is easy enough. We spent some $200 billion every year on net interest on debt, or roughly $2 trillion a decade -- for which taxpayers receive absolutely nothing at all in return. This interest expense wouldn't exist in a world of saner budgeting in which expenses equalled income over reasonable periods of time.
Then, there's the foregone capital gains on that as well. Gilts are tax-free. If we had no debt and investors held the same amount of debt in the form of private debt, the IRS would be receiving a sizable chunk of revenue a year.
In theory, debt financing makes sense if an entity is making some fundamental investment, for which future cash flows will finance future interest payments. But that's not how the government works -- we borrow from tomorrow to finance consumption today. We're paying billions of dollars extra because the folks in 2003 were too short-sighted to finance their consumption in 2003 rather than later. That's no good reason to pay billions in the form of interest payments and foregone capital gains income. Think of how much easier it would be to handle budget problems with that extra buffer. Instead, we're taking those interest payments and throwing those back on the credit card. We'll end up paying substantially more than one dollar in future taxes to finance one dollar of consumption.
Does getting rid of the debt altogether sound crazy? Australia has typically held basically no public debt over the business cycle. They have a little more today due to the financial crisis, but will in all probability wind that down back to basically zero.
Then, there are the warping knock-on effects that "safe" sovereign debt has on the entire financial system. As Perry Mehlring outlined in The New Lombard Street, the Fed was once optimized for a world where the entire system of payments and debt revolved around private debt. What we dub "quantitative easing" was once done sort of more routinely as the Fed manipulated the prices of privately issued debt in order to determine country-wide credit and money ability.
FDR's enormous deficits -- and the resulting stock of public debt -- changed that. The Fed altered its mandate to only mess with public debt, only recovering that old role during the financial crisis, when a zero-rate bound on short-term Treasury debt and broader financial problems brought about financial interventions into private debt. But had FDR not bothered with basically useless fiscal stimulus programs (ie, decided to follow his campaign pledges), and instead stuck with the monetary interventions that actually worked -- we could imagine an alternate central banking world in which shaping interest rate expectations on private debt would constitute the totality of Fed operations. That would have been a much more stable world to live in.
In particular, you don't have the nonsense of "safe" debts underpinning the architecture of the entire financial world. Gary Gorton talks about the creation of structured finance as a way to meet some "shortage" of safe assets provided by the government. I would instead say that the fallacy of assuming that safe assets exist is invariably the cause of financial crises. Rather than worrying about what a debt downgrade will do to the financial system - the goal should be to build a system with is tolerant to (normal! expected!) downgrades of certain types of debt. Rather than pricing everything on the basis of Treasuries, we could have a world where everything has an assumed credit risk, and people bear enough capital to handle expected credit shocks.
This doesn't say anything about the mechanism by which debt goes to zero - whether we get there by higher taxes or lower spending. Obviously, right now, that's hard to imagine. But Australia seems to have figured something out, and even countries like Sweden and Canada have made important steps in that direction. At least in terms of how to imagine debt, I think it's important to say, "piling up this much debt was a really bad idea we should reverse as soon as possible," rather than, "interest rates are low, so let's pile on as much more of this as possible." The price of debt tells us nothing about it's value.