B Ben Moataz
Answer Page
All answers an entity resolution systemfor investigationshow to design entity resolution system

How to design an entity resolution system for investigations

A reference page for teams asking how to design an entity resolution system for investigations without letting the workflow collapse under scale or ambiguity.

A direct answer to: how to design an entity resolution system for investigations. Last reviewed Aug 18, 2026.

5

in-depth sections in this hand-written answer

4

follow-up questions answered on the same page

In depth

written as real guidance, not a templated summary

Aug 18, 2026

last reviewed

The short answer

Design entity resolution as a probabilistic scoring problem, not a matching problem. The moment you treat 'is this the same person?' as a yes/no lookup, you've built a system that both floods reviewers with false positives and quietly misses real matches — the two failure modes that matter most in an investigation. Everything else follows from scoring linkage instead of asserting it.

Why exact matching fails in both directions

Deterministic, exact-match resolution fails coming and going. On common names it produces a flood of false positives — every 'John Smith' collides — and reviewers burn their hours clearing noise. At the same time it misses real matches hiding behind a transliteration, an alias, a nickname, a fragmented record, or a typo in a source system.

Worse, it presents a guess as a decision: the output looks certain, and there's no visible signal that the match was actually marginal. In an investigation that hidden certainty is the dangerous part. A resolution the analyst can't interrogate is one they'll either over-trust or throw out entirely — and both are expensive mistakes. The fix isn't a better exact-match rule; it's abandoning the yes/no frame and modelling identity as something you accumulate evidence toward, with a score you can show.

Score linkage, stack weak signals

I build resolution to link records with an explicit confidence score, assembled from many weak signals rather than one strong key. A shared phone fragment, an overlapping address history, a matching handle across platforms, a co-occurrence in the same document — individually weak, collectively defensible. The system's job is to combine them into a linkage strength the reviewer can reason about.

That approach directly addresses the common-name problem. 'John Smith' plus a matching date of birth plus a shared employer plus a recurring email handle is a very different claim than 'John Smith' alone, and the score reflects it. Fragmented records that each hold one piece of the picture get pulled together into a single entity, while coincidental name collisions stay separated because the supporting signals never accumulate.

Make every link explainable

The score is necessary but not sufficient — the reasoning behind it has to be legible. Every link should carry the evidence that produced it, so a reviewer (or later, an auditor) can see exactly why two records were joined and can un-join them if the basis turns out to be wrong. A resolution system that can't show its work is a black box, and in a defensible investigation a black box is worthless.

This is also what makes the system safe to correct. Because linkage is transitive, an early wrong merge can quietly poison everything downstream. When each link is explainable and reversible, you can walk the chain back, remove the bad join, and let the affected conclusions recompute — instead of discovering months later that the whole graph is built on one bad inference no one can find.

Tune thresholds to the review pressure

There's no universal 'correct' confidence threshold — it's a function of the team's review capacity and the cost of the two error types for their specific work. Auto-merge above a high bar, queue the ambiguous middle for human review, and suppress below a floor. Where you set those lines depends on whether a missed match or a false merge is more expensive in that investigation context, and how much reviewer time exists to work the middle band.

So I make the thresholds explicit, adjustable, and observable rather than buried in code. The team should be able to see how much volume each band is producing and move the lines as they learn. Entity resolution isn't a one-time model you ship; it's an operating surface the team tunes against real review pressure, which is exactly why the scores and the reasoning have to be visible in the first place.

Treat resolution as living infrastructure

Entities drift. People change names, companies restructure, new records arrive that strengthen or weaken an existing link, and a match that was correct last quarter can become wrong. A resolution system that computes links once and freezes them slowly rots, because the world it modelled has moved on. So I build it to re-evaluate: new evidence should be able to raise or lower a link's score and, where warranted, split or merge entities that were previously resolved differently.

That's also where feedback belongs. When an analyst overturns a resolution, that correction is signal — it should inform the scoring, not vanish into a one-off manual override. Over time the system should get measurably better at the specific entity landscape the team works in, because it's learning from the reviews. Resolution treated as living infrastructure keeps a knowledge graph trustworthy; resolution treated as a batch job you ran once is a graph that's quietly drifting away from reality.

One decision I always force into the open is the precision-recall trade-off, because it's a business call disguised as a technical one. Tuning for precision means fewer, higher-confidence links and more misses; tuning for recall means catching more real matches at the cost of more false ones to review. The right point isn't an engineering preference — it depends on whether a missed connection or a wrong one is more damaging to the specific work, and how much analyst time exists to adjudicate the middle. I make that trade explicit and adjustable rather than hard-coding a threshold that silently encodes an assumption nobody signed off on, because the same resolution engine tuned two different ways is effectively two different products.

Related Context

Capabilities, systems, and essays that support the same answer.

More Answers

Adjacent questions in the same search-oriented reference archive.

Answer page

How to build an OSINT pipeline for investigations

A reference page for teams asking how to build an OSINT pipeline for investigations without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to design an evidence capture workflow for investigations

A reference page for teams asking how to design an evidence capture workflow for investigations without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to build a hybrid search stack for investigations

A reference page for teams asking how to build a hybrid search stack for investigations without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to design a monitoring and alerting system for investigations

A reference page for teams asking how to design a monitoring and alerting system for investigations without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to build an adverse media monitoring stack for due diligence

A reference page for teams asking how to build an adverse media monitoring stack for due diligence without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to design a worker orchestration system for investigations

A reference page for teams asking how to design a worker orchestration system for investigations without letting the workflow collapse under scale or ambiguity.

Open answer
Answer page

How to build an investigation platform for due diligence

A reference page for teams asking how to build an investigation platform for due diligence without letting the workflow collapse under scale or ambiguity.

Open answer
FAQ

Follow-up questions answered on the same page.

Why not just use exact matching for entity resolution?

Because it fails in both directions: it floods you with false positives on common names and misses real matches hidden behind aliases, transliterations, and fragmented records. And it hides its own uncertainty, presenting a marginal guess as a firm decision — which is exactly what an auditor will probe.

How do you stop one wrong match from corrupting the whole graph?

Make every link explainable and reversible. Because linkage is transitive, a single bad merge poisons everything downstream. If each link carries the evidence that produced it, you can walk the chain back, remove the bad join, and let the affected conclusions recompute instead of silently trusting a corrupted graph.

What confidence threshold should an entity resolution system use?

There isn't a universal one. Set an auto-merge bar, a human-review band, and a suppression floor, and place them based on the team's review capacity and whether a missed match or a false merge costs more in that context. Keep the thresholds explicit and observable so the team can tune them as they learn.

Does entity resolution need to be re-run as new data arrives?

Yes. Entities drift — names change, records arrive, and a correct match can become wrong. Resolution should re-evaluate links as new evidence lands, split or merge where warranted, and fold analyst corrections back into the scoring so the system keeps improving on the specific entity landscape the team works in. A resolution system that only ever runs forward, never revisiting old links as context accumulates, slowly encodes yesterday's mistakes as today's ground truth — which is exactly the kind of quiet corruption that makes a knowledge graph untrustworthy over time.

Work with me

Building or fixing a system like this?

This is exactly the kind of work I get brought in for. Teams unsure whether a system, architecture, or workflow will hold up under real load and scrutiny.

System Audit Start here · fixed scope
  • A focused review of the system, architecture, or codebase in question.
  • A clear map of the risks, bottlenecks, and failure modes that matter.
  • A prioritized roadmap — what to fix first, and what to leave alone.