What counts as evidence

What it is

Evidence is whatever should move your confidence in a claim, and by how much.

That second half is the part people skip. Arguments about the spiritual and metaphysical world are usually conducted as though evidence were binary — you have proof or you have nothing. Almost nothing works that way. Evidence comes in strengths, and the useful question is never "is this proof?" but "how much should this shift me, and from what starting point?"

A few distinctions do most of the work:

• Evidence is not proof. Outside mathematics there is essentially no proof. There is only evidence strong enough that withholding assent becomes perverse.
• Testimony is evidence. It is how you know almost everything you know — that Rome exists, that your grandmother was born when she was. Treating testimony as worthless is not scepticism, it is incoherence. But testimony's strength varies enormously with what is being claimed and who is claiming it.
• Anecdote is data with no denominator. The problem with "it worked for me" is not that it is a lie. It is that you cannot see the people it did not work for, and neither can the person telling you.
• The prior matters as much as the evidence. The same test result means different things depending on how likely the claim was beforehand. This is not a trick to dismiss things you dislike — it is arithmetic, and it applies symmetrically.

Where it came from

David Hume's essay "Of Miracles" (1748) set the shape of the modern argument. His move was not to declare miracles impossible but to weigh two improbabilities against each other: how unlikely is the event, versus how unlikely is it that this testimony is mistaken or false? Believe the miracle, he argued, only when the falsehood of the testimony would be more miraculous than the event. The argument has been attacked for nearly three centuries and it has not gone away.

Thomas Bayes' theorem, published posthumously in 1763 and developed by Laplace, gave the weighing a formal structure: how a prior probability should be updated by new evidence. It is now the standard framework for thinking about evidence strength, in philosophy and increasingly in science.

Karl Popper contributed falsifiability in the 1930s — the proposal that what makes a theory scientific is that it forbids something, that there is an observation which would show it false. A theory compatible with any outcome tells you nothing.

The slogan most people know arrived much later, and its history is worth knowing. "Extraordinary claims require extraordinary proof" was coined by the sociologist Marcello Truzzi in 1978, and popularised in Carl Sagan's "evidence" wording. Truzzi was a founding member of the sceptics' organisation CSICOP and later left it, publicly critical of how the phrase was being used — as a conversation-ender rather than a standard, and applied to claims the user already disliked while their own assumptions went unexamined. The most quoted line in scepticism was disowned by the man who wrote it.

How it is used

Both sides of every dispute in this territory reach for the same tools, and both sometimes misuse them.

Falsifiability is the most abused. It is invoked constantly by people who have not encountered the objections to it, of which there are serious ones. The Duhem-Quine thesis points out that no hypothesis is ever tested alone — it is always tested along with a bundle of assumptions about instruments, conditions and background theory. When a prediction fails you can always adjust one of those instead of abandoning the hypothesis, and working scientists do this routinely and legitimately. Popper's criterion describes an ideal rather than what actually happens, and Kuhn and Lakatos built substantially on that gap.

"Absence of evidence is not evidence of absence" is the mirror-image abuse. It is false as stated. Absence of evidence is evidence of absence, precisely when evidence would be expected if the claim were true. If a hundred careful studies find no effect, that is evidence there is no effect. What the slogan correctly protects against is inferring absence where nobody has looked.

What we can and cannot say

We can say there is no algorithm. Nobody has produced a rule that mechanically sorts claims into true and false, and the philosophy of science spent the twentieth century discovering how hard the problem is rather than solving it. Anyone who tells you the matter is simple — in either direction — is telling you about their confidence, not about the evidence.

We can say some things are nevertheless reliably better than others. Controlled trials beat uncontrolled ones because they account for the fact that most conditions improve on their own and most people want to please the person asking. Replication beats a single striking result. Pre-registration beats analysis after the fact. Large samples beat small ones. These are not arbitrary conventions — each exists because a specific way of being fooled was identified and then guarded against.

We can say the honest position is often "we don't know", and that this is a real answer rather than a diplomatic evasion. A great deal of what this site covers sits there, and articles that say so are not fence-sitting.

What we cannot do is pretend this is neutral ground. Choosing a standard of evidence is itself a choice, and it is one people make partly for reasons that are not themselves evidential. Naming that honestly is better than pretending to a view from nowhere.

Further reading

• David Hume, "Of Miracles" — Section X of An Enquiry Concerning Human Understanding (1748). Ten pages, and still the centre of the argument.
• Karl Popper, The Logic of Scientific Discovery (1934/1959) — falsifiability from the source, including the qualifications the slogan drops.
• Marcello Truzzi, "On the Extraordinary: An Attempt at Clarification" (1978) — where the famous phrase comes from, by someone more careful with it than his popularisers.