How to design a worker orchestration system for investigations

Make jobs idempotent and retryable

The foundation is idempotency: running a job twice must produce the same result as running it once. Against an adversarial, non-deterministic web, jobs fail partway through constantly — a timeout, a rate-limit, a worker dying mid-task — and the only safe response is to retry. If retrying a half-finished job double-counts data or corrupts state, you can't retry safely, and without safe retries you can't survive at scale.

So I design jobs to be self-contained and repeatable, with retries bounded by backoff and a dead-letter path for the ones that keep failing. A job that fails ten times shouldn't retry forever, hammering a source and hiding the problem — it should land somewhere visible for a human to look at. The combination of idempotency and bounded retries is what lets the fleet absorb constant small failures without either losing data or spiraling.

Isolate sources so one failure doesn't cascade

Different sources fail differently, and a good orchestration layer keeps those failures contained. If one target starts rate-limiting hard or goes down, the workers hitting it shouldn't starve the workers handling everything else, and its backlog shouldn't stall the whole pipeline. Isolation — separate queues, separate concurrency limits, separate backoff per source — is what keeps one bad source from becoming a system-wide outage.

This also lets you tune behavior per source honestly. A fragile target gets gentle concurrency and long backoff; a robust one gets pushed harder. Treating the fleet as one undifferentiated pool means you either go too slow everywhere to protect the weakest source or too fast and get the fragile ones blocked. Per-source isolation is what makes the fleet both fast and well-behaved at the same time.

Manage rate, concurrency, and backpressure

Orchestration is fundamentally flow control. Too much concurrency and you flag proxy pools, trip rate limits, and get blocked; too little and you can't keep up with the collection the investigation needs. So the system has to manage rate and concurrency deliberately, with jittered timing so request patterns don't look mechanical, and backpressure so a slow downstream stage doesn't cause work to pile up unboundedly upstream.

The failure mode to avoid is a fleet that looks busy and is actually counterproductive — rhythmic, aggressive request patterns that get the whole proxy pool flagged, so throughput collapses right when you scaled up. Good orchestration paces the work to what the sources and the infrastructure can actually sustain, which usually means going a bit slower on purpose to stay unblocked and net far more data over time.

Schedule and prioritize deliberately

Not all work is equally urgent, and orchestration is where that gets decided. A fresh lead in an active case, a routine re-crawl of a monitored source, and a bulk backfill are three different priorities competing for the same finite worker and proxy capacity. If everything runs first-come-first-served, an urgent collection sits behind a giant backfill, and the system feels slow exactly when it matters most.

So I make prioritization and scheduling explicit: interactive, investigation-driven work preempts background jobs, recurring collection runs on a cadence tuned to how fast each source actually changes, and expensive bulk operations yield to the queue rather than monopolizing it. The point is that capacity is always scarce, and letting the orchestration layer allocate it by priority — instead of by arrival order — is what keeps the fleet responsive to the investigation rather than to whatever happened to be queued first.

Observe the fleet, not just the jobs

You can't operate what you can't see. Beyond individual job status, the orchestration layer needs fleet-level observability: queue depths, throughput per source, retry and failure rates, how fresh the collected data is. Those are the signals that tell you the system is healthy or quietly degrading — a rising retry rate on one source is an early warning that it changed or started blocking you, long before the coverage gap shows up in a case.

This ties orchestration back to the investigation. A fleet that reports its own health lets the team trust the coverage, because they can see it's actually running rather than assume it. Silent degradation — the fleet 'up' but a source quietly returning nothing — is the dangerous state, and the orchestration layer's observability is what turns that from an invisible gap into a flagged, actionable signal.

Underneath all of this sits session and proxy management, which is where a lot of orchestration reality actually lives. Against defended targets, the identity a request goes out under — its proxy, its session, its fingerprint — is as important as the request itself, and treating that pool as a shared, exhaustible resource changes how you schedule. Burn a proxy range with an aggressive job and you've degraded every other job that depended on it, so the orchestration layer has to manage those identities deliberately: rotating them, resting flagged ones, and pacing work to keep the pool healthy rather than spending it all at once.

The last thing I keep visible is cost, because a worker fleet is a real spend and it's easy to scale it into waste. Proxies, compute, and storage all grow with the fleet, and doubling concurrency to get blocked twice as fast is negative return on real money. So I instrument throughput per unit of cost, not just raw throughput, which turns capacity decisions into something you can reason about — is this source worth the collection budget it's consuming — instead of a number that only goes up. An orchestration layer that's blind to cost tends to grow until someone notices the bill, which is the wrong time to start thinking about efficiency.