Coordinated snapshots of agent state

Alexander BelangerCo-FounderHatchet

Experts tell me that if another sandbox provider launches this week, the gravity well will be so severe that Silicon Valley might just collapse in on itself, never again to be seen by the outside world. I mean, where else do you see such a high density of compute resellers?

And sandboxes are just one part of the equation! In the last 6 months, we’ve seen the release of celld, Temporal’s massive Series E, and the actor model gaining traction like it’s the 1970s. And as much as software engineers-turned-CEOs would like to convince you that they’re just BEAM and Erlang people at heart, that’s not the reason the actor model is infecting the hive mind. Like everything these days, the reason is AI. Specifically, these technologies all solve the same problem of AI agent state.

We’ve written a lot about agent state, and we have some opinions to share. There’s a correct architecture for designing state that’s scalable and resilient. But fair warning, this involves a few different concepts. We’ll introduce these and arrive at our recommendation, that of the coordinated snapshot.

Background on AI agent statefulness

AI agents are stateful by nature. When you start taking apart a coding agent like Claude Code, you’ll notice at least 3 aspects of state:

• The filesystem: Claude Code operates on files. Its goal is to edit the filesystem to produce a diff.
• The context: the sequence of tokens which have been fed to the LLM.
• The session: the history of individual tool calls along with session state (for example, whether the session is paused or resumed).

When you run a coding agent locally, these three bits of state are also local. You can inspect their contents:

--- a/src/api/users.ts
+++ b/src/api/users.ts
@@ -3,6 +3,9 @@ export async function getUser(id: string) {
   const user = await db.users.findUnique({ where: { id } });
-  if (!user) throw new Error("not found");
+  if (!user) {
+    throw new NotFoundError(`user ${id}`);
+  }
   return user;
 }
+
+export async function listUsers(limit = 50) {
+  return db.users.findMany({ take: limit, orderBy: { createdAt: "desc" } });
+}

 {
   "model": "claude-fable-5-1",
   "system": "You are Claude Code, ...",
   "messages": [
     { "role": "user", "content": "add listUsers and a NotFoundError" },
     { "role": "assistant", "content": [
       { "type": "tool_use", "name": "Read", "input": { "file_path": "src/api/users.ts" } }
     ] },
+    { "role": "user", "content": [
+      { "type": "tool_result", "content": "import { db } from \"../db\"; ..." }
+    ] },
+    { "role": "assistant", "content": [
+      { "type": "tool_use", "name": "Edit", "input": { "file_path": "src/api/users.ts", ... } }
+    ] }
   ],
   "tools": [ { "name": "Read" }, { "name": "Edit" }, { "name": "Write" }, { "name": "Bash" } ]
 }

 {"type":"user","uuid":"b1","parentUuid":null,"message":{"role":"user","content":"add listUsers and a NotFoundError"}}
 {"type":"assistant","uuid":"b2","parentUuid":"b1","message":{"content":[{"type":"tool_use","name":"Read","input":{"file_path":"src/api/users.ts"}}]}}
+{"type":"user","uuid":"b3","parentUuid":"b2","message":{"content":[{"type":"tool_result","content":"import { db } from ..."}]}}
+{"type":"assistant","uuid":"b4","parentUuid":"b3","message":{"content":[{"type":"tool_use","name":"Edit","input":{"file_path":"src/api/users.ts"}}]}}

As we can see, local state is easy to manage on a local disk. It’s a solved problem.

The march towards cloud agents

Engineers who are building agentic products aren’t usually building local agents for long. They’re building agents which run as part of a software service, hosted somewhere on the cloud. The most popular local coding agents also have a cloud agents or background agents product.

Cloud agents need to run on a machine somewhere. But machines in the cloud aren’t like local machines. They sit behind many, many layers of virtualization, and the past 15 years of DevOps have taught us to treat our cloud machines like cattle, not pets. So we expect to run a fleet of machines, which should be resilient to random failures or interrupts and all manner of distributed systems problems.

State makes this really difficult. Treating machines like cattle is much easier when it’s a fleet of web servers with short-lived requests. Deploying agents is less like running a fleet of web servers, and more like running a large fleet of databases, which may or may not need to coordinate with each other.

Worse yet, compute is really expensive. As a result, we need to be able to pause and resume these mini databases at any time! There is precedent for this in the database world, but it took over 20 years of Postgres development before we got to Neon (autoscaling Postgres which can scale to zero)—and we’re just going to build the same for agents?

To make the requirements more concrete, we need:

1. To make each individual component of agent state—the filesystem, the context, and the session—easy to migrate between different machines.

2. Ensure the state’s durability based on a desired checkpoint frequency (an important concept we’ll get to a bit later).

3. As we scale, we’d like to keep this as low-latency and low-cost as possible.

Enter Sandbox

Remember when I said that cloud machines aren’t like local machines? This might be met with some objection, because the clunky, naive way to solve these problems is just spinning up a persistent VM with a disk attached. Each agent session just creates a new VM, and you leave it running there while it does its job. Sure, it’s expensive, but you just raised a boatload of money from the hottest investors in town, so who cares?

Now, we could just call these VMs, but as cloud coding agents were taking off, we suddenly found ourselves needing to run untrusted user code on these VMs, and we came up with the word sandbox to represent like a mini virtualized machine, a new concept, like a micro VM… shit, naming is hard.

The point is, coding agents aren’t the only agents that should be able to run code and scripts. All kinds of agents are more powerful when they’re able to execute code, because it gives them tools akin to a human working on a local laptop. They can edit files, pull data and transform it, run searches across that data, etc. In other words, sandboxes are a way for a cloud machine to get the toolset required to act like a local machine.

Filesystems

So, we’ve established that sandboxes are a really useful tool for agents because they can execute untrusted code. But I think there’s an equally powerful primitive in the sandbox providers: the ability to snapshot and restore a filesystem.

As the agent makes progress, it’s often doing a bunch of writes to a local disk. When the agent is done, it usually bundles up those writes and presents them to the user in the form of a diff, a new file, a PowerPoint, a graphic, etc. While the agent is waiting for feedback, it’s important that we preserve the state of that disk, but we pause the compute that the agent is running on. Otherwise, you’re going to rack up massive infra bills.

So how do we actually preserve the filesystem of an agent? If you were implementing this from scratch, you would think about creating incremental backups of disk as the agent makes progress; disk snapshots of your favorite persistent disk provider:

This is indeed a rather natural solution, but remember our requirement that we need to keep this cost-effective as we scale. Mounting a separate disk for each agent and snapshotting is going to get very expensive, very quickly, if you use default block storage providers. The disks also need to be networked to move quickly between machines, which can impact performance quite a bit.

If you’re thinking something like “I need an infinitely scalable, cheap disk in the cloud as a service,” congrats, you invented S3. Instead of dealing with snapshots of disk directly, you can snapshot and archive the current state of the relevant filesystem after each agent turn. There are a few ways of doing this:

1. Just use git. Each session turn is responsible for producing a commit, and resuming a session is just a git clone away.
2. If you don’t want git, just run tar and upload the archive to S3:

tar -C /workspace -czf - . \
  | aws s3 cp - "s3://$BUCKET/sessions/$SESSION_ID/turn-$TURN.tar.gz"

The problem with these two approaches is that they’re slow and expensive, and there are tons of edge cases in rolling this on your own. If you’re just getting started, and trying to get your MVP to production, these aren’t unreasonable approaches. They just get hard at scale, as you start dealing with agent swarms, forking requirements, and the agent restore times start taking tens of seconds.

This is where I see enormous value in sandbox providers like Archil, E2B, and Freestyle. For example, in the case of Archil, they mount an “infinite,” S3-backed filesystem with RW consistency guarantees. But instead of each write going directly to S3, they put a bunch of fast disks close to your agents to make performance tractable.

So not only do sandbox providers help you execute untrusted user code, but they offer good primitives for filesystem durability. Note that there’s no reason we need to couple the isolated code execution feature with the filesystem durability feature; for example, with Archil you can mount their filesystem across other sandbox providers as well. But compute and disk are often coupled for a reason.

Now, this has been about the state of the filesystem for an agent. But we’ve sort of glossed over how the pause and resume mechanism should actually work. Intuitively, we know that we’d like to resume the sandbox when a new event comes into the system, and we’d like to pause it when an agent turn is finished. What we need is some way to manage the session state: are we waiting for user feedback, currently executing a turn, cancelling a turn, etc? This is where durable workflows become essential.

Durable workflows

Let’s go back in time a bit.

The early days of (non-coding) agents didn’t need access to a filesystem. They operated solely on tool calls and context. The extent of the state was the history of tool calls and context fed into the model. It was often helpful to model an AI agent as a stateless reducer: it would take in an input (the context plus a message) and produce an output (which was often sent to the user). The next iteration loop would concatenate that output with the next message from the user.

From an infrastructure perspective, I miss those days. You could just run this agent process in-memory on a worker somewhere. You simply had to restrict the tool calls that the agents made to be properly scoped to the user’s session. Treating agents this way felt more like building a request handler on a web server than a true “agent” working on a computer.

When these agents became long-running—user sessions which lasted for hours or even days—the history of tool calls corresponded wonderfully to a paradigm known as durable workflows or durable execution. In the agent-as-durable-workflow model, every tool call made by the agent is persisted to a durable event log, which can be replayed on a new machine when the old machine dies. And platforms that support durable execution—like Hatchet or Temporal—support evicting durable workflows from machines after a period of inactivity or based on a local cache, which can make running thousands or hundreds of thousands of these types of agents pretty efficient from a compute perspective.

Now this is still its own class of agent: simple patterns which classify user requests and make a few tool calls don’t need filesystems at all. Being able to reconstruct the session is enough. But the primary tradeoff is indeed that the agent is modeled as a stateless reducer (from the application’s perspective; the state is stored implicitly through the guarantees of durable execution).

As I mentioned above, after the release of Claude Code and the proliferation of agent harnesses, it’s become very clear that lots of effective agents aren’t stateless reducers—they have a clear side effect. They need to write to disk and execute untrusted code.

Where durable execution is useful is to model session state. These workflows provide durability, they’re very often event-driven (in the case of Hatchet), have the capability to pause and resume, and are very efficient from a compute perspective. If you have millions of agents in-progress, paused and waiting for feedback, you should model them as durable workflows.

Coordinated snapshots

Until now, I’ve hopefully convinced you that durable workflows are a great way to model session state and that sandbox providers are developing unique and efficient ways to manage filesystem state. Each type of state has its own snapshot (of sorts): the durable workflow has its event log, and the filesystem has a disk snapshot.

At Hatchet, we believe these two systems fit together very nicely. The one issue is that we have state in multiple places, so we need to coordinate how we snapshot that state. The durable workflow waits for user events and invokes the agent, which runs code / writes to disk in the sandbox. When the code running in the sandbox completes, we have a new disk snapshot which needs to be reflected in the durable workflow.

This is what I mean by the coordinated snapshot. The outer durable workflow needs to be aware of the inner filesystem snapshot:

This architecture isn’t really opinionated about whether you run your agent harness inside the sandbox or outside the sandbox. If you run it inside the sandbox, your session workflow lives elsewhere, and is responsible for invoking the sandbox provider to run a turn of the agent. If you run it outside of the sandbox, the session workflow and the agent loop are one and the same, and each of your tool calls is a subtask of the durable workflow.

There’s also a third option: the dual harness approach. You might have an orchestrator agent running outside of the sandbox, while you have a dedicated agent which runs inside of the sandbox. In fact, this is how a lot of the most popular cloud coding agents work. They separate the session management layer from the coding agent layer.

Checkpoint frequency

One of the benefits of the dual harness approach is that you can tune the granularity of your state storage, in other words the checkpoint frequency.

To introduce this concept, let’s start with an example that you’re hopefully familiar with, database backups. Let’s pretend that we take a database backup every day by snapshotting a disk (or using something like pg_dump in Postgres), and then storing the backup in S3. If you ever (God forbid) have to restore from this backup, you can only restore from when the backup was taken; your backup granularity is 24 hours. We’ve solved this with databases through point-in-time restores (PITRs), which typically use a write-ahead-log plus the snapshot to provide restores to roughly any checkpointed point in time. PITRs are much more expensive to store, because they store a much higher volume of data.

In the database world, the industry has defaulted towards supporting PITRs, while daily snapshots are for non-critical database workloads. This is where the analogy differs: most agents don’t produce critical writes. Their work can be easily reproduced via a new turn.

I’m not suggesting there are no important writes. The output of each turn of the agent should be preserved, either in git or in a snapshot somewhere. But the output of each incremental write should not be preserved, as the cost of a machine-level failure is actually quite low in most cases (so in a tiny fraction of cases, the user might wait twice as long for a response, which for long-horizon agents isn’t a large impact anyway).

@hatchet-dev/sandbox

To end with a bit of self-promotion: after getting feedback from Hatchet users building agents, we discovered that many of them had organically reached the same architecture, and had started building some tasks and primitives for orchestrating sandboxes and had been thinking about snapshot coordination.

That’s why we built @hatchet-dev/sandbox, which integrates with your favorite sandbox providers like E2B, Daytona, Archil, and Freestyle. It’s an important part of offering a universal orchestration layer, because sandboxes have quickly become a critical component of the modern agentic application.

It bakes in this exact architecture: each call to a sandbox provider can produce a coordinated snapshot which is stored in the durable task’s event log:

import { HatchetClient } from "@hatchet-dev/typescript-sdk/v1";
import "@hatchet-dev/sandbox"; // adds hatchet.sandbox to the client

const hatchet = HatchetClient.init();

export const agentSession = hatchet.durableTask({
  name: "agent-session",
  executionTimeout: "1h",
  fn: async (input: { conversationId: string; repo: string }) => {
    // Runs on the provider connected to your tenant. Each call is a
    // Hatchet task, and with snapshotEveryTurn on, each one that changes
    // the workspace takes a snapshot, so a crash picks up where it left off.
    const sandbox = await hatchet.sandbox.create({
      id: input.conversationId,
    });

    await sandbox.exec(`git clone ${input.repo} app`);
    const tests = await sandbox.exec("cd app && npm test");
    await sandbox.files.write("app/REPORT.md", tests.stdout);

    await sandbox.kill(); // the snapshots stay
    return { passed: tests.exitCode === 0, snapshot: tests.snapshotId };
  },
});

// Later, from anywhere with a Hatchet client: pick the session back up
// from its latest snapshot...
const again = await hatchet.sandbox.connect("conv-42");
await again.exec("cd app && git pull");

// ...or fork a new sandbox from the snapshot taken right after the tests ran.
const fork = await hatchet.sandbox.create({ from: "<snapshotId>" });

You can also tune checkpoint frequency with the snapshotEveryTurn option in the provider config (off by default, so a snapshot is only taken on kill). We think this is an important knob, because while strongly-consistent, durable filesystems are great, they also add a lot of latency and cost to your stack.