The short answer
Build the pipeline as four separable layers — collection, resolution, evidence, and review — each of which can fail, degrade, and recover on its own. The mistake that kills most OSINT pipelines is treating collection as the whole product; the value for an investigation team is in what survives after collection, when the data is dirty and the source has already changed. Everything below is about designing for that reality instead of the demo.
Start from the investigation, not the sources
Before you write a scraper, get specific about the decision the team is trying to make. An investigation pipeline that supports a fraud case, a due-diligence file, and a trust-and-safety escalation are three different systems wearing the same dashboard. The decision determines what you have to collect, how confident you need to be before you surface something, and — critically — what has to be preserved as evidence versus what can stay a transient signal.
I map that first: what enters the case, what a reviewer does with it, and what the output has to withstand. If the output is an internal lead, the bar is speed. If it might end up in front of a regulator, a client, or opposing counsel, the bar is defensibility, and that changes the architecture from the collection layer up.
Skipping this step is why so many pipelines collect enormous volume and still can't answer the one question the investigator actually asked. Volume is easy; relevance and defensibility are the hard parts, and they're decided before any code gets written.
Collection that assumes the internet is adversarial
The open web is non-deterministic and actively hostile to automated collection. Sites change structure without warning, rate-limit aggressively, and fingerprint the rhythm of your requests. A scraper tuned in a demo works against ten targets and quietly falls over at ten thousand — usually silently, so nobody notices the coverage gap until it matters in a case.
So I build collection defensively: worker fleets with retry and backoff, jittered timing so request patterns don't look robotic, and per-source health monitoring that watches the shape of the output, not just the HTTP status. If the average number of records per scrape drops by half in an hour, that's a signal the target changed — and the system should raise it, not swallow it.
The goal is a pipeline that degrades loudly and recovers on its own, instead of one that reports success while returning nothing. The failure you can see is a nuisance; the failure that reports 200 OK while silently collecting nothing is the one that burns an investigation.
Resolution and scoring make raw collection usable
Raw collection is noise until you resolve entities and score confidence. Names collide, records fragment across platforms and jurisdictions, and a deterministic 'match' is frequently a coin flip wearing a confidence label. If you don't model that uncertainty explicitly, it compounds: link entity A to B on a weak signal, link B to C tomorrow, and the knowledge graph is quietly corrupted in a way no one can audit.
I treat identity as probabilistic — linking records with explicit confidence scores rather than brittle exact matches, and stacking weak signals into defensible linkage. That suppresses the common-name noise that floods analysts with false positives while keeping genuine-but-fragmented matches from slipping through. Just as important, the score is visible to the reviewer, so ambiguity is something they can see and act on instead of a hidden assumption baked into the output.
Evidence and review are product requirements, not cleanup
In an investigation, a conclusion without its evidence is a liability. If the system says a person appears on a given platform, it needs to have preserved the underlying artifact — the captured page state, hashed and timestamped — so the finding can be reconstructed months later when it's challenged. Store conclusions without the raw asset and you've built something that looks authoritative and can't be defended.
The last layer is review speed. If an analyst has to independently verify every output, they'll stop trusting the tool and go back to a manual search. So I design the review surface as a first-class part of the system: grouping related signals into cases, ranking by severity, and attaching the evidence to the finding so the reviewer opens a case, not a hundred tabs.
Design for the operator, and for scale from day one
The pipeline that survives is built for the operator — the professional who needs the signal to be true under pressure — not the tourist clicking through a demo. That framing changes priorities: reliability, provenance, and clear confidence beat a slicker graph animation every time, because the operator is betting real decisions on the output.
Scale is the other thing you design in from the start, because retrofitting it is brutal. The gap between ten targets and ten thousand isn't linear; it's where hand-tuned logic, unbounded retries, and analyst-in-the-loop cleanup all break at once. Build the layers to be independently observable and recoverable early, and the move from prototype to production is an increase in volume rather than a rewrite. Get that right and you have an intelligence pipeline; get it wrong and you have an expensive demo with a team of analysts hidden behind it.
The most expensive version of getting it wrong is subtle: the pipeline appears to work, so the organization scales its commitments around it, and the hidden manual cleanup grows quietly in the background until it's a department. By then the fix is no longer a technical change; it's an operational unwind. That's why I push to design the failure modes and the recovery paths in early, while the system is small — the cost of building for reality is front-loaded and finite, whereas the cost of hiding an unreliable pipeline behind human labor compounds for as long as the product exists.