Before Willem de Vlamingh explored the west coast of Australia in 1697 Europeans assumed that all swans were white. Countless observations had suggested the whiteness of swans to be a general rule and there was no evidence to the contrary. De Vlamingh’s reports proved otherwise, and proved an important point of logic. As John Stuart Mill explained: “No amount of observations of white swans can allow the inference that all swans are white, but the observation of a single black swan is sufficient to refute that conclusion.”

In his book The Black Swan, published this week, Nassim Nicholas Taleb uses the birds as a metaphor for uncertainty. A black swan event is rare and cannot be predicted. It has an extreme impact, such as overturning a widely held belief. After the event it can be rationalised to the point of looking predictable. We ask ourselves why we did not see it coming.

September 11, for Taleb, was such an event: impossible to anticipate, huge in its effects and apparently foreseeable in hindsight. So was the stock market crash of 1987. Such events can be positive, too: technological advances such as the internet; artistic success such as JK Rowling’s. Rarely, though, do we recognise these as random events that defy prediction. Instead, we construct dubious explanatory narratives to help us to avoid black swans, or to encourage them, next time.

Taleb’s main subject is the financial markets – he is a former Wall Street trader – but his insights are also relevant to our health. For the individual, black swan events can explain many misjudgments about illness.

And they hold important lessons for medical research.

A diagnosis of autism, for example, is often a black swan. Though it is influenced by genetics, there is no way of predicting whether any individual child will be autistic. When it does develop, there is a tendency to spot “obvious” environmental causes that might have been foreseen, such as the MMR vaccine or an emotionally distant mother. That there may be no evidence of cause and effect does not matter. Parents blame themselves for failing to predict the unpredictable.

Something similar follows apparently miraculous recoveries from incurable disease, particularly when alternative medicine has been involved. Cancer occasionally, and unpredictably, goes into remission with no obvious cause. Yet the great temptation is to seek one in an unproven treatment regimen the patient has followed. This doesn’t mean that there is no point in trying to establish what causes diseases, or helps recovery. But the black swan effect shows why anecdotes are no substitute for robust data.

Some medical research benefits from black swans. Many breakthroughs are not made by deliberate study in pursuit of a goal, but are serendipitous accidents. Penicillin was the result of a contaminated experiment. Viagra was designed to treat hypertension and angina. As Taleb says, research must be structured so that it can take advantage of the unexpected. That means allowing scientists to pursue their interests, without having to predict unpredictable applications before they can get a grant.

This is something the Government should remember as it merges the research budgets of the Medical Research Council and the Department of Health, announced last year. Some scientists are concerned that the new body could be too dogmatic about setting priorities, directing money towards applied research that has ends such as cancer therapy in mind.

There is a place for this, but basic, “blue skies” research must not lose out. It might appear unlikely to bring immediate benefits, but the black swan effect suggests that chance spin-offs of such work are most likely to transform health.

The Black Swan (Penguin, £20); Mark Henderson is Science Editor of The Times

Times recommends

• Live chat: Thrifty ways to update your winter wardrobe
• Is anybody out there really listening?
• Nutrition Q&A: Will eating late make me fat?