Durable file transitions
Small job and event documents are written to a temporary file in the destination directory, flushed, and atomically replaced so readers do not intentionally observe half-written JSON.
Reliability
Designed to fail without inventing memory.
Substrate uses crash-conscious writes, leased durable jobs, transactional source generations, and recoverable projections. These mechanisms protect specific state transitions; they are not an uptime guarantee or proof that every failure mode has been eliminated.
Implemented mechanisms
Durability is a chain of explicit transitions.
The current system favors small, inspectable state machines over pretending a workflow is reliable because one request returned successfully.
Recoverable jobs
The durable queue records pending, running, succeeded, and failed states. On startup, expired running leases can be recovered for another bounded attempt rather than remaining permanently stuck.
Transactional source generations
New source contributions remain in staging until an atomic ledger transaction activates the new generation and supersedes the earlier successful one.
Projection repair
Canonical ledger state and published projections are separated. Projection work is queued, can be retried, and can be rebuilt or deleted after a crash without treating the stale page as authority.
Atomic replacement and transactions reduce partial-state exposure on the configured filesystem and database. They do not protect against every storage defect, operator error, provider outage, filesystem misconfiguration, or loss outside the durability assumptions of the deployment.
Crash-safe contributions
Keep the last good source active until the next one is complete.
A source is represented by replaceable generations. This prevents an interrupted re-ingestion from silently mixing old and new contributions or displacing the last successful version with incomplete work.
Stage work privately
Extraction and derivation are recorded in a staging generation. Staging and failed work is not active memory and can be cleaned before retry.
Commit one source generation
Activation and supersession happen together in the ledger. A crash before commit leaves the prior successful generation active; it does not intentionally publish a partial replacement.
Project from canonical state
Search pages and memory cards are written as derived projections. The queue tracks which entities need a write, rebuild, or deletion.
Acknowledge after durable projection
Projection completion is acknowledged only after the corresponding write or deletion transaction succeeds, enabling later recovery of unfinished projection work.
Operational state
Failures remain visible and retryable by policy.
Durable jobs expose explicit lifecycle state and bounded diagnostics. Known transient failures can be retried with a delay; permanent policy or content failures terminate instead of being repeated indefinitely. Unexpected failures remain generic at the customer boundary and are investigated in server-side operations.
The exact retry count, delay, lease duration, provider timeout, retention, and alerting route are deployment settings—not universal guarantees.
Job state
Pending, running, succeeded, and failed states are persisted, with attempts and lease/heartbeat timestamps used for recovery.
Bounded retries
Retry behavior is selective and bounded. A durable queue does not mean every failed input will eventually succeed.
Post-write checks
Projection and quality paths can verify expected canonical and published state and leave repair work queued when completion cannot be acknowledged.
Current limitations
No reliability adjective replaces an operating boundary.
The first deployments are deliberately small and managed. Wider service objectives should be stated only after the topology and operations that support them are independently tested.
No SLA or uptime claim
The service does not currently publish an availability percentage, support-response target, RPO, RTO, or maintenance-window commitment.
Single environment
Isolation simplifies blast radius but does not imply automatic regional redundancy, hot failover, or continuous availability.
Tested paths are bounded
Durable-state, recovery, projection, and deletion paths have focused validation. That does not establish correctness for every input, integration, or compound infrastructure failure.
Operate for recovery
Make incomplete work inspectable, not invisible.
Beta integrations should define what can be retried, what requires a human, and which source remains authoritative when a provider or projection fails.