Last week I was in Lock’s, the old hat-maker’s in St James’s. I was there with an old rock star. Old in rock years. Middle-aged in human years. He was having a bespoke cloth cap made: a proper Jarrow march cap.

On the other side of the shop there was a man being fitted with a bowler. He looked a prat. He was Spanish. He looked at himself in the mirror with an assumed English gent’s patrician haughtiness donated to him solely by the cloth on his head.

Then who should walk in but Norman St John Stevas, which was a surprise because I thought the great genuflecting courtier of heredity had kicked the monogrammed bucket years ago. Apparently he’d come in to have the sweatband of his American cowboy fedora thing changed.

And it struck me that the time when men wore their class on their heads had long gone. Here was a multi-millionaire looking like a DH Lawrence character, a foreigner imitating one of Macmillan’s civil servants and the hand-kissing lord dressed up like an extra from Brokeback Mountain.

Class has raised its ugly head again, though. The top hats of class war were being superimposed on the Conservative candidate at Crewe and Nantwich and were also in evidence in the London mayoral campaign. Toffistry has been put back into the bleak, snobbish heart of British politics, and it’s been put back by those who will be the most damaged by it - and have the most to lose.

Of all the changes in British life in the past generation, the slow-fading of the class system is the most dramatic, the most encouraging and rewarding. It was smothered by the demise of the old industrial, rust-and-dust communities, the shrinking unions, the natural withering and profligacy of the hereditary aristocracy and the rise of a meritocratic and well-educated middle class.

Most of all, however, it went because we all decided, collectively, that it was time for it to go. Tom Paine said that the best things about politics, societies decided for themselves. The old class system didn’t fit with the lives we wanted for ourselves and for our children. We didn’t want to be pigeonholed by what our fathers did or by the hat we wore.

The rise in immigration only highlighted what an embarrassing, parochial pastime class had become, like morris dancing. It survived in a sort of arch aspic in films and on television - as a shorthand for character - and in the columns of some papers.

However, that famous Frost Report comedy turn in which John Cleese and Ronnies Barker and Corbett are upper, middle and lower class - Cleese: “I look down on him because I am upper class.” Barker: “I look up to him because he is upper class, but I look down on him because he is lower class.” Corbett: “I know my place” - would be inexplicable to the viewers of The Mighty Boosh. And even at the time, in 1966, it was funny principally because it mocked what was already ridiculous.

The rude reinvention of class by the Labour party is cynical and indefensible. If you and I were to sit down for lunch and consume a bottle of claret or a couple of pints and start an argument about class, we’d soon be bogged down in the orthodoxy of definitions: how many lords could dance on the head of a pin? How do we define class now? Is it what you do; is it what your parents did; is it income; is it where your house is? Is it that it’s your house?

Perhaps it’s what you sound like: as the Irishman George Bernard Shaw had it, an Englishman has only to open his mouth to make another Englishman despise him. It’s instructive and amusing to listen to early broadcasts by the Queen, with her accent of absurdly cut glass, full of ripe plums, like a young Celia Johnson. And then listen to her last speech at the opening of parliament: the voice, if not her hats, has glissandoed down the social scale. Her grandsons speak in modified army estuary, the flattened-out lingua franca of the young. It may dismay grammarians and drama teachers, but it does mean that they’re being judged by what they say and not how they say it.

Class in politics is by its very nature divisive; and we really don’t need any more manufactured division. The people who relish it, who welcome its reemergence, are the ones who miss the stasis and waste of trench politics. It’s not so much about exposing Tories for what they’ve always been as about dragging the left back to what it always was: a fondly imagined embattled, hobnailed past of martyrs and coughed-up blood.

However, these very same people who happily accuse their opponents of being “toffs” are also the ones who are scrupulous about political correctness and the nomenclature of race, creed, colour and disability.

The prejudice of ascribing character and ability to class is so obvious it barely needs arguing; yet even Polly Toynbee in The Guardian will effortlessly refer to “toffs” as if her own privileged background and education had been whitewashed. She would never dream, though, of referring to didicoys or Paddies or yids.

To discard anything an Old Etonian has to say or may wish to do because of where he went to school is politically cretinous – and anyway, are we talking about Old Etonians such as Jonathon Porritt, George Orwell and Hugh Laurie? The same goes for Oxbridge. The electorate is being asked to discount educated candidates because they’re educated.

The most depressing element of this is not its intellectual paucity, its social divisiveness or its thuggish prejudice; it’s a fact that the people it will harm the most are the people least able to defend themselves. The people Labour was entrusted to protect.

The socialist movement grew out of Chartism, nonconformist religion and the unions. Not to enshrine the class system, but to tear it down; and over the course of 100 years it and other people and factors did just that. We don’t live in a wholly meritocratic society but it is a lot better than it was, isn’t it? By common and universal consent we’ve agreed that it probably is.

Wishing for class war again is like wishing for outside privvies, black lung, bread and dripping and hopelessness. The arguments and assumptions of class obscure all the real issues and problems. They chain politics to fixed positions.

The unfairness in society has nothing to do with class, but everything to do with access, resources and understanding. The Labour party isn’t what it was in 1946; nor are the Tories; nor are the rest of us. The idea that in a democracy you cast your vote dependent on the accident of your birth is abhorrent.

I interviewed Tony Benn when he retired from the Commons; we drank tea. I pointed out that the class system he’d spent his life fighting against no longer existed. He chuckled and said that, on the contrary, lots and lots of people who never knew it before were finding that they were working class.

“Many bankers and captains of industry only realise they’re working class when they get fired. If you work for someone else, you’re a worker. If you can be sacked by a boss or a board or the shareholders, you’re working class.”

Well, doesn’t that sort of include everyone? I mean, even the Queen’s employed by us. He puffed on his pipe and beamed, and you could hear Keir Hardie, the Labour party’s founder, chuckling.

Times recommends

• The Boxing Day tsunami: five years on
• What took place in 2010...
• Can you really cure a hangover?