Someone Built A Framework Just To Audit Their Own Experiments
A strange kind of audit showed up on r/PromptEngineering this week, and it’s not the kind that checks your code. It checks your old conclusions. This Redditor, u/No_Understanding6388, has been building a personal reasoning system called “Fluid Relational Reasoning” (FRR) for a while now, and the newest release, v0.5, came straight out of that audit.
What’s New
FRR started as a way to let intuitions move freely across math, code, dynamics, genealogy, structure, and evidence, without forcing them into one fixed pipeline before they’re ready. The original poster describes it as a reasoning environment rather than a checklist: exploration, formalizing, testing, and compressing can all happen in a different order depending on what the idea actually needs. That’s a deliberate break from the usual “define hypothesis, run test, report result” template most people default to when they’re reasoning through a problem with an AI in the loop.
Version 0.5 adds the piece that was missing: a way to check whether an experiment’s “success” was actually earned, or whether the answer was quietly baked into the labels, the synthetic data, or the scoring rule from the start. Anyone who has built an eval set knows how easy this is to get wrong. You write the test, you write the labels, and if you’re not careful you write both from the same mental model, so the test can only ever agree with you. FRR v0.5 treats that as a first class failure mode instead of an edge case to shrug off.
That’s the headline. It’s not the twist.
The Twist
Here’s the part that stands out. The biggest addition in v0.5 isn’t a stricter success metric. It’s a rule about what happens after you correct a claim.
The creator calls it the “documentation wake.” Once an early conclusion lands in a README, a paper draft, a variable name, or a prompt fed to another model, fixing the original experiment doesn’t erase that claim from everywhere it already spread. The outdated version keeps circulating downstream on its own, completely detached from the correction that killed it. Think about how many times a variable named `is_validated` or a comment saying “confirmed: works” survives three refactors after the thing it describes stopped being true. Nobody goes back and hunts those down, because nothing in the workflow tells you to.
So the real innovation here isn’t “test more carefully.” It’s “track what your old, wrong answer already contaminated, and go fix that too.” That’s a much harder problem than writing a better test, because it means treating your own past outputs as a liability that needs an inventory, not just a paper trail you can ignore once the correction is made.
The Mini-Workflow
The author’s core loop, the “Intuition-Artifact-Pressure Loop,” distills into something you can run on your own projects:
- 🧭 Start with the intuition and let it stay loose. Don’t force it into math or code before it’s ready. If you formalize too early, you end up defending the formalization instead of testing the idea underneath it.
- ⚙️ Build an artifact, a script, dataset, metric, or prompt, that’s actually capable of proving the intuition wrong. If you can’t picture what a failed run looks like before you hit go, the artifact isn’t doing its job yet.
- 🔍 Apply real pressure: counterexamples, a simpler baseline, a rival explanation, anything that could genuinely change the result. A baseline that’s dumber than your approach but scores almost as well is one of the fastest ways to catch a hollow win.
- 🧵 Ask the counterfactual question the author leans on hardest: if the proposed mechanism were absent, could this exact setup still produce the same apparent win?
- Name what actually survived. Executing correctly, predicting unseen cases, and explaining the true mechanism are three different kinds of success, not one word called “validated.” A model that runs clean, predicts well on new data, but gets the mechanism wrong is still only two thirds of the way there, and calling it “done” hides that gap from everyone downstream.
Pro Tips
The sharpest line in the whole post doubles as the best pro tip: “a test that cannot meaningfully fail cannot meaningfully confirm.” If your labels, your synthetic corpus, or your prompt already hands the model the answer, the win doesn’t count for anything. Before you trust a passing eval, ask what a failing run would have had to look like, and if you can’t answer that, the eval was never really testing anything.
Second one: after any real correction, go back through your own docs, prompts, and old threads for the claim you just walked back. That’s the step almost everyone skips, and it’s the one this whole framework got built around. A quick habit that helps: whenever you retract a claim, do a plain text search across your notes, prompts, and commit messages for the old phrasing before you move on. It takes five minutes and it’s the difference between a correction that actually sticks and one that just adds a second, contradictory version of the truth to your notes.
Go Check the Thread
The full writeup digs into effective states, boundary conditions, and how to separate a failed argument from a genuinely contradicted claim, more than fits in one post here. Worth the read if you test anything with AI reasoning loops. 🚀
Frequently Asked Questions
Q: What’s the “documentation wake” and why does it matter?
The documentation wake is when outdated conclusions sneak into your README, slides, comments, and external copies, even after the main work is corrected. FRR helps you trace where these old conclusions keep influencing others, so you can catch and fix the propagation before it spreads further.
Q: What does “path count is not automatically a probability” mean?
It’s a subtle mistake where counting the number of paths to a conclusion gets confused with the probability it’s actually correct. FRR catches these hidden logical errors, ensuring your confidence actually reflects your evidence instead of just the number of ways you could reach that answer.
Q: How does FRR distinguish real validation from luck?
By separating intuition from artifacts that can genuinely fail, FRR prevents success that only happened because the right answer was accidentally baked into your data, labels, or scoring rules. It distinguishes an idea that survived testing from an idea that was never actually tested, a critical difference most experiment audits miss.