Correlation Isn't Causation. Now What?
The pattern behind every failed prediction, from 2008 crash to COVID pandemic
The Convergence - From Correlation to Causation
Last week I wrote about AI prediction’s blind spot: the gap between pattern matching and genuine causal understanding. The response surprised me. Engineers kept asking the same question: if ML can predict who will churn, why can’t it tell us why?
I spent the last week digging into this, reading Judea Pearl’s work on causal inference alongside Buddhist texts on dependent origination. I don’t have a neat conclusion. But what I found is worth sharing, because two very different traditions seem to be wrestling with the same problem.
Judea Pearl, the Turing Award-winning computer scientist, built what he calls the Ladder of Causation. Three rungs. The first is association: seeing patterns in data. This is where virtually all machine learning lives. Feed it historical data, it finds statistical regularities. Impressive, but limited. Pearl himself puts it bluntly: “deep learning can give you answers to a very limited class of questions.”
The second rung is intervention: what happens if I do X? Not “what happened when X occurred,” but “what would happen if I made X happen right now?” This is where ML starts breaking. A model trained on hospital data might discover that patients who receive a certain drug die more often. But the drug was given to the sickest patients. The correlation is real. The causal inference is backwards.
The third rung is counterfactual: what would have happened if I had done something differently? This requires imagining alternative realities. Pearl argues that no amount of data can get you here without a causal model.
Here’s where it gets interesting. The Buddhist doctrine of dependent origination, laid out in the Samyutta Nikaya (SN 12.1, ~5th century BCE, Shravakayana), describes a 12-link causal chain: from ignorance arises mental formations, from mental formations arises consciousness, all the way through to suffering. The formula is explicit: “When this exists, that comes to be. With the arising of this, that arises. When this does not exist, that does not come to be. With the cessation of this, that ceases.”
Read that again. The second half looks a lot like counterfactual reasoning to me. If this ceases, that ceases. I’m not claiming the Buddha was doing formal logic. But the structure maps surprisingly well onto rung three of Pearl’s ladder, articulated roughly 2,400 years earlier.
And the 12-link chain itself is a causal graph. Not a list of correlations. A directed sequence where each link produces the next through specific conditions. Remove a link (say, craving), and the downstream chain (clinging, becoming, birth, suffering) collapses. That’s an intervention. Rung two.
As a recent article in Towards Data Science put it, “causal inference is eating machine learning.” A review of immunotherapy studies found that 72% used traditional ML with zero causal inference, and propensity score methods were misapplied in 72% of cases. These are medical journals, where people make life-and-death decisions based on the results.
What strikes me about the Buddhist framework is that it seems to avoid this confusion entirely. Dependent origination doesn’t say suffering correlates with craving. It says craving produces clinging, which produces becoming, which produces birth, which produces suffering. Change the conditions, change the outcome. Whether you call that philosophy or a causal model probably depends on your starting point. But the structural similarity is hard to ignore.
Pearl is essentially arguing that AI needs to climb from seeing to doing to imagining. I think the Buddhist path moves through similar territory: from observing mental phenomena (mindfulness), to intervening in habitual patterns (ethical training), to understanding what would happen under different conditions (wisdom). Whether it’s truly the same ladder with different vocabulary, or just a superficial resemblance, I’m honestly not sure. But the more I research both, the harder it gets to dismiss as coincidence.
I’d love to hear what you think. Are these genuinely parallel frameworks, or am I seeing patterns where there aren’t any?
The Intervention Gap - Why “What If” Changes Everything
There’s a concept in debugging that every engineer knows intuitively: you can log every metric in your system and still not understand why it crashed. Logs tell you what happened. Root cause analysis tells you why. And the fix requires imagining what would have happened under different conditions.
This is exactly the gap Pearl identified in machine learning. And it’s the same gap the Buddha addressed in the first teaching on dependent origination.
Consider the hormone replacement therapy disaster. For decades, observational studies showed that women on HRT had lower rates of heart disease. Doctors prescribed it widely. Then randomized controlled trials revealed the truth: HRT actually increased heart disease risk. The correlation was real but confounded. Healthier, wealthier women were more likely to both choose HRT and have naturally lower heart disease rates. The data saw a pattern. It missed the cause.
This isn’t just a medical problem. In 2008, ML models trained on historical market data couldn’t predict the financial crisis because a crash caused by cascading subprime mortgage failures had never appeared in the training set. The correlations were there (housing prices always go up, CDOs are diversified) but the causal chain (loose lending → bundled risk → systemic contagion) was invisible to pattern-matching systems. The few people who saw it coming, like Michael Burry, were doing causal analysis: tracing conditions, not fitting curves.
COVID told the same story. A 2020 review in the International Journal of Forecasting put it bluntly: “forecasting for COVID-19 has failed.” One model predicted over 23,000 deaths within a month of Georgia reopening. The actual number was 896. The models broke because they treated transmission as a statistical regularity rather than a conditional process. Change the conditions (behavior, policy, population density, immunity), and the pattern changes. As the New England Journal of Medicine noted, models are “constrained by what we know and what we assume.” When those assumptions miss the causal mechanics of transmission, the predictions fall apart.
I think Buddhist analysis is less likely to make this kind of error, because dependent origination explicitly models the conditions, not just the outcomes. The tradition asks: what are the specific conditions producing this result? Remove each condition one at a time. Which removal changes the outcome? That looks a lot like systematic causal analysis to me, though I’ll admit the contexts are very different.
Nagarjuna, the second-century Madhyamaka philosopher, pushed this further. In the Mulamadhyamakakarika, he argued that nothing possesses inherent, independent causation. Effects don’t live inside their causes waiting to emerge. Causes don’t independently produce effects. The relationship is conditional, context-dependent, and empty of fixed essence. I find this remarkably close to what Pearl calls the “do-operator”: you can’t just observe a system and infer causation. You have to intervene, change conditions, and observe what shifts. Maybe the resemblance is coincidental. But it keeps showing up.
We wrote about a related pattern in Every Bubble Believes It’s Different. Bubbles persist because people mistake correlation (prices went up every quarter) for causation (prices will always go up). The moment you ask “what would happen if conditions changed?” the bubble logic collapses. That’s counterfactual reasoning. Rung three.
The gap between prediction and understanding isn’t just a technical limitation. It might be an epistemological one. And from what I can tell, both Pearl and the Buddhist tradition are pointing in the same direction: you can’t understand a system by watching it. You have to engage with its conditions. Whether that insight transfers cleanly across 2,400 years and two radically different contexts, I’m still figuring out.
Signal & Noise
Causal Inference Is Eating Machine Learning — Kaushik Rajan makes the case that correlation-based ML is hitting a wall. When prediction works but decisions fail, the missing piece is always causation.
Judea Pearl on LLMs and the Need for Causal Reasoning — Pearl argues that LLMs are stuck on rung one of his ladder. Related: we explored a similar limitation in Can AI See Itself Clearly?
Biology, Buddhism, and AI: Care as the Driver of Intelligence — A PMC paper arguing that Buddhist dependent origination offers a framework for understanding AI-human co-evolution. Dense but worth the read.
Causality for Machine Learning — Bernhard Scholkopf’s foundational paper on why causal reasoning is essential for robust ML. If you read one technical paper this month, make it this one.
Glossary
Dependent origination — Skt: pratityasamutpada / Pali: paticcasamuppada. The principle that all phenomena arise from specific conditions and cease when those conditions change. Articulated as a 12-link causal chain in the Samyutta Nikaya (SN 12.1).
Causal inference — A statistical and philosophical framework for determining cause-and-effect relationships, as distinct from mere correlation. Formalized by Judea Pearl’s do-calculus and ladder of causation.
Counterfactual — A statement about what would have happened under different conditions. The third rung of Pearl’s ladder and implicit in the Buddhist formula: “when this ceases, that ceases.”