One way to think about U.S. policy toward the Persian Gulf since the beginning of the Iran-Iraq War is that we’ve been struggling to create a situation in which Iraq is strong enough to avoid being dominated by Iran while not being strong enough to dominate Kuwait and Saudi Arabia. Basically, right now there are four oil rich countries all very close to each other and we want to make sure that they remain four separate countries so that none of them becomes too strong. The situation is one of enhanced concern because not only are these countries oil rich, but they share important cultural affinities that make successful domination seem more plausible.
This seems about right. The United States doesn't want to tolerate political monopoly of oil (the only country to worry about that; everyone else just buys it like any other commodity) so goes to extraordinary lengths to avoid political unification of areas controlling oil. One interesting part about this is that in Iran, Iraq (before the invasion), and Saudi Arabia the oil is on ground occupied by politically marginal groups (Shias in Saudi and Iraq, Sunnis in Iran). This is a great political solution--keeping both the oil politically divided, and each country riveted as well, led by unpopular autocrats dependent on the US. The three of these, plus Iran, have all been supported by America in a bid to maintain 1984-like political jamlock.
From this angle, it's pretty clear that the costs have been pretty disproportionate. A more unified Gulf might be able to push harder than OPEC for higher oil prices; but Saudi Arabia--the oil producing state that thinks most like a monopolist--has always been the one pushing for lower oil prices to prevent the rise of alternative fuels. If the oil consumers really wanted lower oil prices, they could just form a cartel of their own to force down prices. Spending on new technology would be nice, but as long as America's allies depend on Middle Eastern oil, America will probably care too. Maybe the 'oil companies' push for this policy, but really they would be better off with higher oil prices.
The big wrinke in this political dynamic is the rise of Shiites and assymetrical warfare. Iran-Iraq of course get closer, but Hezbollah and Sadr are also interesting. Both represent the power of mobilizing a young and growing population through social services and launching a bottom-up Maoist campaign. Both are smart enough to use military force to the degree necessary to maintain control without requiring military occupation, which is very hard these days. The dilution of power represented by RPGs, cruise missiles, rockets, SAMs, and assault rifles, and terror attacks--especially in crowded areas where an occupier can't strike back--means that even the strongest conventional power like Israel is hard-pressed to launch any sort of offensive attack. There are parallels to the Pastuns, which is why the Pakistani Army is so desparate to move troops back to the Indian border.
In the interests of continued Middle Eastern anarchy, you would try to forestall a realignment by hoping that the Shiite forced simply get spilt in bloody civil wars within mixed states. That hasn't happened, as Lebanon and Iraq move much more Shiite, and Iran picks up nuclear weapons.