In 1948, three years after the second world war, and some years before I was born, the school I was educated at celebrated its 1000th anniversary. Yes, that's right, my school was founded in the year 948. Even then it was only the 5th oldest scool in England. So I hope you won't object to me taking a slightly long view, let's say 10 years in the future.
I was part of the team that ended up producing the programming language Python. In the shadow of Moore's Law we were looking down the road and thinking about how we might solve today's problems in the future. Our plan wasn't to get rid of C, or anything like that (in fact I wrote part of the Gnu C compiler gcc). But we got a lot of pushback from people for the work we were doing, because it didn't solve the problems they had then, and what we were doing ran really slowly -- it was only the beginning of the personal computer age, PC's only had floppy disks. My current mobile phone is 700 times more powerful than the mainframe computer we were using then to develop on. In fact our project got shut down by management, and it is only because Guido van Rossum was willing to flee elsewhere and carry on the good work that we have Python today; well that and a bunch of guys in Amsterdam in the 1980's who thought it was an interesting thing to investigate.
So pitting HTML5 against XHTML2 is the wrong way to look at things. HTML is the assembly language of the Web. The suggestion isn't that it should go away. If the browser manufacturers can get together and make producing interoperable Web pages easier, that can only be a good thing for all of us. But XHTML2 tries to step back and take a longer view. What are the problems that we are trying to solve, and how can we work towards easier ways of solving them. Problems like:
We want accessibility not only so that Raman can also be a first-class citizen on the Web, but also because we all will one day be partially sighted, and we will even then still want to be able to browse the web without having to educate each new generation of 20 year olds with perfect sight who are building the web sites we use. We want accessibility out of the box, so that web sites will automatically be accessible, without having to follow long lists of guidelines.
We don't want to have to write our web sites 10 times, once for each sort of device that is on the market. We want to write it once and let it adapt to the device.
Less scripting
The web world already largely understands the advantage of style sheets: xxx
But this should be possible for more parts of web construction: we want domain experts to be able to concentrate on their part of the website, database people for the data, graphics designers for the look, user interface people for the user experience, and so on, and to only need to communicate through a thin interface, so that they can get on with their work.
and so on.
So to take just one example, XForms, which is a part of XHTML2. People have demonstrated the same form running on a desktop, a cell phone, over a voice browser, even as an IM buddy; a company that produces large machines with complicated user interfaces, who do it regularly enough to know that it takes 30 people working 5 years to produce the user interface, did a pilot study using XForms instead. It took 10 people 1 year. Do the maths: 10 person years instead of 150. How much does a person year cost? Let's be conservative and say $100,000. So it cost them 1 million dollars instead of 15 million dollars. That alone covered the cost of their W3C membership for the next thousand years or so.
XHTML2 is popping up here and then on the Web even now. Ebay UK uses it for their mobile site. You won't see that it is XHTML2 any more than you can see that Mozilla is written in C. But it is solving problems that people have.