In the vacuum of local development, code is logical. A function receives an input, performs an operation, and produces an output. If it fails, it throws an error that you can catch and debug.
But in the distributed reality of high-scale intelligence systems, logic is secondary to physics. When you deploy a “Worker Fleet” to ingest data from the public web, you are entering an environment where the network is unreliable, the targets are adversarial, and the “infrastructure” is a shifting patchwork of third-party APIs and proxy nodes.
In this world, reliability is not the absence of failure; it is the management of failure.
To build a worker fleet that doesn’t collapse under its own complexity, you must move beyond the “Try/Catch” mindset and into the world of retries, idempotency, and failure taxonomies.
1. What a Worker Really Is
A common mistake is to view a worker as a “small program.” It isn’t. An operator-grade worker is an atomic unit of state transition.
In a System-Heavy architecture, a worker exists to take a system from State A to State B. Its identity is defined not by its code, but by its Boundary.
- What are the inputs? (A secure URL, a session token, a set of extraction rules).
- What is the side effect? (A file in S3, a row in a database, a message in a queue).
- What is the proof of completion? (An ack to the orchestrator).
If your worker does more than one thing—for example, if it fetches data and processes it and updates the identity graph—it is impossible to manage. When it fails (and it will), you won’t know which state transition was successful. You have created an ambiguous system.
Reliability begins with Atomicity. One worker, one transition.
2. Why Naïve Retries Fail
The most instinctive response to a worker failure is the “Loop Retry”:
// DON'T DO THIS
for (let i = 0; i < 3; i++) {
try {
return await scrape(target);
} catch (e) {
console.error("Failed, retrying...");
}
}This is dangerous at scale for three reasons:
- The Thundering Herd: If a target site is down for 10 seconds, and you have 1,000 workers retrying every 1 second, you are effectively DDOSing the target (and your own proxy pool).
- Synchronous Blocking: The worker stays alive and consumes resources while waiting for the retry. Multiply this by 10,000 workers and your orchestration layer will run out of memory.
- Context Blindness: The worker doesn’t know why it failed. It retries a “403 Forbidden” (which requires a proxy change) the same way it retries a “500 Internal Server Error” (which requires waiting).
The professional approach is Durable, Asynchronous Retries. When a worker fails, it reports the failure to the orchestrator (like AWS Step Functions or a RabbitMQ DLX). The orchestrator then schedules a new worker instance to attempt the task later, using Exponential Backoff with Jitter.
3. Idempotent Execution Design
If you are going to retry tasks, you must accept that some tasks will run twice.
Imagine a worker that extracts a record and increments a “Total Captures” counter. If the extraction succeeds but the worker crashes before reporting success, the orchestrator will trigger a retry. If your system isn’t Idempotent, that record will be duplicated and the counter will be incremented twice. Your data is now a lie.
Idempotency is the ability to run the same operation multiple times with the same result as running it once.
In intelligence engineering, we achieve this through:
- Natural Keys: Instead of auto-incrementing IDs, we use deterministic hashes of the content (e.g.,
SHA256(source_url + timestamp)). - Conditional Puts: We use database operations that only write if the record doesn’t exist or is in an “In-Progress” state.
- Side-Effect Isolation: We store raw assets in S3 with predictable keys. If the worker runs again, it simply overwrites the same file in S3, resulting in zero net change.
Idempotency is the “get out of jail free” card of distributed systems. Without it, you are constantly cleaning up your own mess.
4. Failure Taxonomies: Classifying the Chaos
A system that treats all errors as “Error” is a system built by a tourist. To an operator, failures are categorized into Taxonomies that dictate the response.
Type A: Transient Network Failures (Wait & Retry)
DNS timeouts, connection resets, 503 errors. These are the “noise” of the internet. The response is a standard exponential backoff.
Type B: Environmental Constraints (Rotate & Retry)
403 Forbidden, 429 Too Many Requests, CAPTCHA walls. These are signals that your Environmental Signature (IP, Browser Fingerprint, Behavior) has been flagged. The response is not to just “wait,” but to switch proxies and jitter the behavior before retrying.
Type C: Semantic Drift (Alert & Pause)
The page loads (200 OK), but the extraction logic finds zero fields. This is a Structural Failure. The site has changed. Retrying 100 times won’t fix this. The system must “Circuit Break” this specific target and alert an engineer.
Type D: Hard Failures (Discard & Log)
404 Not Found, Invalid Credentials (that were previously working). The target is gone or the access is revoked. Continuing to retry is a waste of resources.
By classifying failures, you build a “Calm System.” The orchestrator handles Type A and B silently, Type C alerts the right human, and Type D preserves the evidence of the target’s disappearance.
5. Conclusion: Calm Systems Under Stress
The goal of reliability engineering in intelligence systems isn’t to build a system that never breaks. It is to build a system where breakage is part of the design.
When you look at the dashboard of a System-Heavy fleet, you don’t see 100% green. You see a sea of “Retrying,” “Rotating,” and “Recovering.” You see the machine actively managing its own survival in a hostile environment.
This is what it means to build for the real world. You move from the fragility of scripts to the robustness of worker fleets. You stop being surprised by failure and start being prepared for it.
In the next post, we will look at Designing for Disruption—how to handle the total collapse of external dependencies without losing your operational footing.