Every few months a startup deck promises to predict the next bank failure from news data. I wanted to know whether that’s buildable, so one night I sat up past midnight waiting for the last of seven tests to finish. The question I’d chased all winter: can you tell which Indian bank is about to get hit, using nothing but public information? The number came in a little after one in the morning. It landed below blind guessing.
So had the other six, a couple slightly worse than random, which takes a certain talent.
The setup
Every night a monitor pulls public documents about 25 Indian banks, big and small, from every corner where bank trouble becomes public: penalties, circulars, market disclosures, court and enforcement records, cyber advisories, and news. Around 4,600 documents, each tied to its bank and linked to the original. The question: which bank gets an RBI penalty, a SEBI order, or a cyber incident in the next 14 days? Ground truth: 86 incidents (35 penalties, 16 orders, 21 cyber, plus fraud and court events). That scarcity is itself a finding.
Seven ways to ask one question
Seven ways of asking, all on the same locked split:
- Counting: the documents, penalties, and stories trailing each bank, and whether they’re rising or falling.
- The same counts by source.
- The same counts by kind of event.
- Meaning: a small open language model summarizes what the words say; simple statistics sort the summaries.
- Word counting: which words show up, and how often.
- Coaching: the same model, taught from a handful of worked examples.
- Reading and judging: an AI reads the bank’s recent documents and must answer yes or no.
The pass mark isn’t fifty-fifty; for rare events it’s what a coin weighted by rarity would score, so blind guessing already banks 0.2591. My stop rule, set in advance: beat random by more than 0.05, or abandon prediction entirely. Pre-committing a kill criterion is the only defense against noise at two in the morning.
The results
| Method | Score | Random-guessing bar | Difference |
|---|---|---|---|
| Number-crunching the paper trail | 0.2433 | 0.2591 | −0.0158 |
| Reading the meaning of the words | 0.2597 | 0.2591 | +0.0006 |
| Counting the words | 0.2587 | 0.2591 | −0.0004 |
| Coached on worked examples | 0.2448 | 0.2591 | −0.0142 |
| An AI reading the documents | 0.2461 | 0.2591 | −0.0130 |
Frozen test split, 14-day head start; the bar is random guessing on the same data.
Every entry sits at or below random, bunched within two hundredths of a point. The best posted +0.0006 over random. That’s static. The best single probe managed +0.0418 (word counting, on SEBI orders), short of the 0.05 gate. The gate didn’t trip. I stopped.
Even a cheating version fails: hand a method everything said on the day the bad news broke, and it still can’t pick out the banks that got hit. A bigger model wouldn’t have helped either. Buckmann and Hill (two DeepMind researchers, arXiv 2408.03414) found simple statistics on a small model’s reading usually beat a giant model guessing from a few examples, my exact case.
Why I believe the null
Weak-signal hunts usually end with a story by slide forty. I inverted the incentives: split, yardstick, pass mark, and stop rule, all written down and locked before the final tests ran. Even my tooling refuses to publish site copy that claims prediction, a guardrail whose only job is stopping me oversell my own data. Quietly satisfying. Honest limits: 25 banks, one market, eighteen months, 86 incidents. What I can say, with receipts:
Everything a well-equipped outsider can assemble from public sources does not forecast these outcomes at a 14-day lead, and the failure is not for lack of architecture.
What I built instead
The monitor exists: a continuously updated, fully-referenced record of what’s publicly happening to each bank, free at banks.rajagrawal.com. No account, no gate. Each bank gets an attention score: how much public discussion is swirling around it right now. It ranks conversation, nothing more, and the site says so wherever the number appears. There’s a heat map, every number clicks through to its sources, and thin data gets “insufficient data” instead of a guess.
If you follow Indian banks, as an analyst, journalist, or depositor, that record is the product. If you think I’m wrong, good. The standing challenge: break the null. Find an approach, a sharper definition of trouble, or a licensed dataset that clears the bar under rules locked in advance. I want to hear about it.
Leave a Reply