Published
The demo is not the test
Across the systems we build and the systems we cover — entity resolution matchers, multi-agent orchestration, ontology-driven intelligence — the same failure pattern shows up. A system looks solved because it was evaluated on inputs that were already close to correct. The interesting failures live in the messy cases: the near-duplicate that should not merge, the agent route that should have adapted but did not, the signal that needs structure it was never given.
A demo that only shows the clean path is not evidence the system works. It is evidence the demo was curated.
Honest evaluation needs production-shaped inputs
This is the thread running through generate-data.com: matching and scoring systems need test data with a known duplicate rate, fuzzy variants, and ground truth you can score against — not clean, unique rows that never stress the decision boundary. Master ID and Duplicate Type columns exist so a matcher's output has something real to be scored against, not just eyeballed.
The same discipline applies past record matching. Any system that makes a judgment call — merge or don't, route or don't, act or don't — needs a test set built to expose that judgment, not to avoid it.
The lab's approach
That is why evaluation rigor is not a side note for us — it is how we approach the systems we build and the research we publish. Test data that encodes the hard cases is the starting point, whether the system under test is a matcher, a multi-agent router, or something we have not built yet.
If you are building or evaluating a system that has to make judgment calls under mess, start with data that has some in it.
Related: generate-data.com, Generate-Data project overview and how we approach AI and technology research.