Signal and Noise: A Seismologist's View of AI Credibility

A seismologist’s daily work is finding signal in noise. In a continuous record, almost everything is the background hum stirred up by wind, traffic, and ocean waves; real events come rarely, and often lie deeply hidden. Do this long enough and you grow an almost reflexive skepticism: shown a beautiful waveform, your first reaction is not excitement but a question — what makes it real?

Over the past two years of working closely with AI, I slowly realized that this discipline of doubt transfers almost unchanged.

The seismologist’s credibility toolkit

Faced with a “candidate event,” we do roughly four things:

1. Check the noise floor first. Whether a peak is significant depends not on how tall it is, but on how far it rises above the background. Signal-to-noise ratio comes before every other judgment.
2. Corroborate across stations. A beautiful spike on a single station may just be a passing truck. A real event shows up on multiple stations with physically consistent time delays — if the travel times don’t fit, no amount of beauty counts.
3. Reproduce. Re-process with different parameters, a different time window: is the conclusion still there?
4. Actively try to refute. Ask in earnest: is there some noise mechanism that could explain all of this just as well?

Substitute “AI answer” for “waveform”

Swap the waveform for an AI’s answer, and the four rules barely need editing.

Fluency is the noise floor of AI output. Everything a language model produces comes with a built-in sheen of looking right, so “it reads as credible” has zero discriminating power — just as “beautiful” never proved a waveform was an event. Judgment can only come from external evidence: did the code actually run? Where are the command and the output? That is a rule I hold myself to — “verified = command + output” — the equivalent of checking travel times.

A single answer is a single-station record. For conclusions that matter, I have a second AI with a fresh context review the work independently, or I re-derive it along a different route — only when two independent paths reach the same conclusion does credibility genuinely rise.

And finally, refutation: rather than asking “is it right?” over and over, try to break it — deliberately violate one premise and watch whether the conclusion moves. A “conclusion” that never budges is usually not robust; it is not listening.

The same lesson twice

What seismology taught me, AI taught me again: the more beautiful the result, the cooler the head required. Credibility is not a feeling; it is a chain of evidence.

Turned around, the same sentence is gentler than it sounds — you don’t need to distrust your AI colleague, any more than you need to distrust your seismometer. You only need to build mutual verification into the way you work together, and then spend your attention, with an easy mind, on the genuinely interesting problems.