One thing I have noticed (I am sure this has been pointed out by much smarter people than me.) is that the one thing you can count on from al-Qaida (and other similar groups) is that they will run when confronted. We invade a country they head for the hills; we let them have a town and when we go to take it back, they just float away. When they attack, they attack like horny prison inmates, attacking from behind.
Is it possible to beat a enemy that will not stand up and fight? Is it possible to beat and enemy that will only attack on their terms.? I believe it is possible, but not through a centralized campaign like a war. Though a centralized campaign might be part of the solution, it cannot be the whole solution.
The only war I can compare the war on terror is to the war on drugs. Both wars are wide ranging and not bound within borders. Both wars are fought against groups that live on the edges of society. Terrorist and drug dealers both are decentralized groups with different goals and preferred tactics. In some cases, the groups overlap, with the terrorist making money selling drugs.
There are some main differences. I would think the drug war would be easier to or at least make progress on, because the drug dealers, manufactures, and distributors are just in it for the money. You take away the profit incentive, though controlling the market forces (legalizing drugs for example), and they will find other lines of work. Whereas the terrorist, will keep plugging along, although less effectively. Drug dealers also have a clearer purpose and mission compared to terrorists. Selling drugs compared to blowing anything up. They also have the problem of having to maintain a certain amount of stability to grow, process, and distribute their product, where terrorists only need time to gather supplies to execute their plan.
I do not know how much real progress has been made in the drug war or the war on terror, but we have been fighting the drug war for sometime now. I doubt the war on terror is going to go any better for the foreseeable future.
Some Tactics for fighting the shadows
1) Shed light on their movements. Control the land, sea, air, and information environments. To succeed in the wars in the shadow, we must shed light on the roaches. Transporting illegal goods must be nearly impossible. Transporting illegal money, must be even harder. We must be able to intercept their communications to find supplies, attack them in their sleep, and disrupt their carefully laid plans. How this can be done in without giving up our rights, I have no idea.
2) Take out their leadership. Even as decentralized as their groups are, they have a leader at some level. Those leaders must be found. While some of those leaders will be replaced, they will be replaced with people that are, more than likely, less capable. These men (or women) did not become leaders because they "knew" someone. No, they found their way into leadership rolls because of their loyalty, skills, or dedication. I doubt the groups reward failure very well.
3) Take away their markets and bring market forces to bear. If there is no product to sell, export, or deliver there is no work to do. This applies more to the drug dealers than the terrorists, but even terrorists are not above market forces. If they have to pay more for guns and ammo, they will have less money to spend else were. If there is no one who wants to blow their selves up, the will have to find other less effective was of getting the bombs to their victims.
4) Fight them from dusk till dawn, from their homeland to or doorstep. On checkpoint, can be skipped, tricked, or bought off. A thousand checkpoints cannot be avoided. A decentralized threat needs a decentralized response.