For about ten years the web was going to learn to read itself. Pages would carry labels describing what they meant, machines would reason over the labels, and software would do your errands. It did not happen, and the reason it did not happen is more useful than the idea was.
October 2026
Google does not use the keywords meta tag in web ranking.
Google Webmaster Central, September 2009
That sentence is the whole story in miniature, and it was published as a flat administrative notice. The keywords meta tag was the simplest possible version of the big idea: a place where the author writes down what the page is about, so that a machine does not have to work it out. It was abused so consistently, by so many people, for so long, that the machines stopped reading it. Nine years earlier that tag had been the future.
I was taking notes through that whole period — at user experience and information architecture conferences, and in the groups that formed around them, where this was the subject that would not go away. Reading those notes back, what stands out is not the content but the confidence: Dublin Core, RDF, PICS, GILS, the Encoded Archival Description, a dozen committees, and an unmistakable sense that one of them was about to win.
What was actually promised
In May 2001 three authors, one of them Tim Burnes-Lee — the inventor of the web, set it out in Scientific American: a form of web content meaningful to computers, which would unleash a revolution of new possibilities. The examples in that article are still the examples people reach for. Your software finds a specialist who takes your insurance, checks their free slots against your calendar, and books the appointment — because the surgery’s page says, in a way a machine can act on, that this is an appointment and that is a postcode.
Nothing about the plan was silly. It needed three things: a shared vocabulary for saying what things are, a syntax to carry it, and authors willing to label their pages honestly. The first two were built, exhaustively. The third was the problem.
Three reasons it did not arrive
Nobody labels their own page honestly. Metadata written by the party who benefits from how it is read is advertising. The keywords tag proved this inside a few years, and no amount of better syntax fixes it — the failure is in the incentive, not the format. Search went the other way instead: ignore what the page claims about itself, measure what other pages say about it.
The standards were built before the uses. Committees produced abstract, elegant specifications and then looked for applications. Meanwhile a working developer compared RDF and ontologies with a database and some JSON, found the simpler thing entirely sufficient, and went home. That is not laziness. A standard that is harder than the alternative has to be much better than the alternative, and it never was.
It assumed everybody would publish their own data. The architecture needed a decentralised web of sites describing themselves. What happened instead was five companies, and the data went into them, where it is not described for your benefit and cannot be reasoned over by your software at all.
What we got instead
On 2 June 2011 three search engines — Bing, Yahoo and Google, with Yandex joining that November — launched a shared vocabulary of their own at schema.org. It is the Semantic Web, with the ambition removed and the incentive inverted. You do not label your page so that other people’s software can reason about it. You label your page so that the search engine will draw your opening hours, your price or your review stars directly into its results.
And it worked, on a scale the committees never approached: tens of millions of domains carry it. The honesty problem was solved the only way it could be, which was by one party being big enough to check, and to stop showing you if you lie. That is not a web that understands itself. It is a web that fills in forms for a search engine.
That is the thinking. Here it is as decisions for an ordinary site:
What to do about metadata now
- Leave the keywords tag out. It has done nothing since 2009 and its presence tells a reader of your source what decade your template is from. Remove it when you find it.
- Write the description for a human, once, per page. It is not a ranking signal, it is the sentence under your link in the results. A generated one, or the same one site-wide, wastes the only piece of copy a stranger reads before deciding.
- Add structured data only where it changes the search listing. It is a second, invisible copy of a few facts, written for the search engine rather than for your reader. It pays for itself when the engine prints one of those facts in the results: the opening hours under a shop, the date under an event, the stars under a recipe. If you cannot say which line you expect to appear there, you are keeping a second copy of your data up to date for nothing.
- Never let it disagree with the page. Structured data that says something the visible page does not is both a penalty risk and a lie. If they can drift apart, generate them from one source or do not add it.
- Prefer real markup to extra markup. A heading that is a heading, a time that is a
timeelement, a list that is a list. Honest HTML is machine-readable metadata that cannot fall out of step with what the reader sees, because it is what the reader sees. - Keep the labels that serve you, not the search engine. Categories, dates and authorship that make your own archive navigable are worth maintaining for their own sake. Those were always the metadata that paid.
And finally, what happened to the Semantic Web
It was not defeated. It was answered, from a direction nobody in the debate was watching.
Look at what the question actually was. Could meaning be written down separately from the words — set in a label, beside the sentence, so that a machine would never have to understand the sentence itself? Everything that followed from that, the schemas and the syntaxes and the committees, was an attempt to answer yes. Two decades later, machines read the sentence.
That is why the labels were never going to be enough, and it is a deeper reason than dishonest authors or difficult standards. The project assumed that meaning is something a page has, which can be lifted out, stored next to it and kept in step with it. But meaning is not kept in a page. It is what happens when somebody reads one. The schemas were a careful, decades-long effort to write down the part that cannot be detached.
So the ending is stranger than failure. The promise was kept and the architecture was ignored. The worked example from 2001 — software that reads a page, works out that this is an appointment and that is an address, and acts on it — is ordinary now. Not because anyone labelled the page. Because the machine learned to read it the way a person does, with all the unreliability that implies.
Which leaves the smallest possible instruction for anybody making a website, and it is the same one it was forty years ago. Write the page clearly. Mark it up honestly. Say what you mean in the words, not in a field beside them. Whatever you will not keep true, do not mention.
The web just learned to read what we had been writing all along.
Read the originals
The first is the promise, in the authors’ own copy. The second is the best account of why it did not land.
- Tim Berners-Lee, James Hendler and Ora Lassila, The Semantic Web — Scientific American, May 2001. Read the worked example and then look at your own address page.
- Whatever Happened to the Semantic Web? — Two-Bit History, 27 May 2018. The retrospective: dishonest metadata, standards built ahead of their uses, and a centralised web that the architecture had not allowed for.
- Google Webmaster Central, Google does not use the keywords meta tag in web ranking — September 2009. Four paragraphs, and the end of an era announced like a change to opening hours.
- Schema.org — what the search engines built in 2011 instead, and how widely it is now used.