There is no more ridiculed literary genre than the self-help book. Admit that you regularly turn to such titles to help you cope with existence and you are liable to attract the scorn and suspicion of all who aspire to look well educated and well bred. As if on a mission to deny

the category even a shred of respectability, the publishers of self-help books deck them out with lurid covers, while booksellers entomb them near the mind, body and spirit section, where they blur into an indistinguishable mass of sickly pink and purple spines.

It wasn’t always like this. For 2,000 years in the history of the West, the self-help book stood as a pinnacle of literary achievement. The ancients were particularly adept practitioners. Epicurus wrote some 300 self-help books on almost every topic, including On Love, On Justice and On Human Life. The Stoic philosopher Seneca wrote volumes advising his fellow Romans how to cope with anger (On Anger), how to deal with a child’s death (Consolation to Marcia), and how to overcome political and financial disgrace (Letter to Lucilius). It is no injustice to describe Marcus Aurelius’s Meditations as one of the finest works of self-help ever written, as relevant to someone facing a financial meltdown as the disintegration of an empire.

Christianity continued in this vein. The Benedictines and Jesuits poured out handbooks to help one navigate the perils of earthly life. In his medieval bestseller, The Imitation of Christ, the theologian Thomas à Kempis recommended that one note down sentences from the text, learn them by heart and repeat them at times of crisis.

Great self-help writers were still dispensing advice down to the early 19th century. Consider that master of pithy and useful phrases, Arthur Schopenhauer, author of The Wisdom of Life, who advised in 1823: “A man must swallow a toad every morning to be sure of not meeting with anything more disgusting in the day ahead.”

The assumption behind this long tradition was that the words of others can benefit us not only by giving us practical advice, but also — and more subtly — by recasting our private confusions and griefs into eloquent communal sentences. We feel at once less alone and less afraid.

So what explains the gradual decline in the prestige of self-help books? A key catalyst was the development of the modern university system, which in the mid-19th century became the main employer for philosophers and intellectuals and started to reward them not for being useful or consoling, but for getting facts right. There began an obsession with accuracy and a corresponding neglect of utility. The idea of turning to a historian or philosopher in order to become wise (an entirely natural assumption for our ancestors) started to seem laughably idealistic and adolescent.

Alongside this came a growing secularisation of society, which emphasised that the modern human being could do the business of living and dying by relying on sheer common sense, a good accountant, a sympathetic doctor and hearty doses of faith in science. The citizens of the future weren’t supposed to need lectures on how to stay calm or free of anxiety.

Universities, though often described as the new churches, ceased to offer what the churches had always focused on: the saving of your soul. Instead, they merely presented us with a diet of information. Go to a university today in the hope of finding answers to life’s great dilemmas and the academics will laugh — or call for an ambulance.

And so the self-help field was abandoned to the many curious types who thrive in it today: people who are reclothing the Christian message so as to promise us financial heaven if we believe in ourselves, have faith, work hard and don’t despair. Or else those with a passing acquaintance with Buddhism, psychoanalysis or Daoism.

What unites modern practitioners is their fierce optimism. They make the grave assumption that the best way to cheer someone up is to tell them all will be well. They are utterly cut off from the spirit of their more noble predecessors, who knew that the fastest way to make someone feel well is to tell her that things are as bad as, and possibly much worse than, she could ever have thought. Or, as Seneca put it so well, “What need is there to weep over parts of life? The whole of it calls for tears.”

We need self-help books like never before, so it seems especially sad that our most serious writers are unalive to the possibilities of the genre and that the very idea of saying something “useful” to a reader has become synonymous with banality.

“Twenty tips from Othello on relationships” might seem like a stupid idea for a book, but there is nothing intrinsically wrong with the idea. Imagine what this could have been if Carlyle, Emerson or Virginia Woolf had had a shot? In our current moral and practical confusions, the self-help book is crying out to be reborn and rehabilitated.

Times recommends

• How to solve the big style questions
• Aids in Africa: ‘It is the staff who really inspire you’
• Video: How to do Beyonce's 'Single Ladies' dance