Open-sourcing our AI-Engineering Version Control System

We're pretty excited about the capabilities of Swarmfile in terms of enabling true AI Engineering on anything from a swarm of agents acting locally on a single branch, to agents acting across thousands of runners, machines and human workers in parallel. We also find that the reasoning given by Anthropic for their Scion VCS validates our design in more ways than one.

Working locally#

To start, we're Git-interoperable today across our commercial and open-source offerings, in fact we actively encourage working with Git as an offline and familiar way of working for many engineers so that the local journey has a consistent and well-known starting point.

However, if you want to take it further then you can work with the native Swarmfile client today. Our CLI and Desktop clients are free and they both work with our open-source swarmfile-server consistently.

By taking it a step further, if you choose to work from the shared drive then the swarmfile-engine tracks the version each process last read of a file. A save produces a new whole-file version - variable chunking means only the changed chunks transfer - and it is applied against the version the writer read. If another writer has moved the file since, the save is refused rather than silently clobbering it, and the writer that lost the race gets the signal.

Here's an example:

1. Agent A reads byte-range 256-512 from index.html.
2. Agent B reads the entirety of index.html.
3. Agent A writes new content to byte-range 280-640 in index.html.
4. The write from Agent A is accepted and broadcast to the peers in the swarm (if it's online).
5. Agent B writes new content to byte-range 700-760 - a range that doesn't overlap A's at all.
6. The write from Agent B is refused with ESTALE: the file has moved since B read it. The overlap is not what matters - a save is a whole-file version, so even a non-overlapping edit is refused if you're writing against a version someone else has already replaced. Re-read, re-apply, and it lands.

That is an unedited terminal recording of two real clients against a self-hosted swarmfile-server - a scrubbable, copy-pasteable cast, like every recording on the demo page.

AI Coding Assistants are already familiar with the idea of using the version you last read as the base, and we find that this mechanism works well to enforce that behavior, mostly because Swarmfile already leans into how modern operating systems already work.

This opens the door to clean offline/online multi-agent, single-branch working on a local device in an open-source ecosystem.

Swarmfile can take that further with "multi-mount" and "single-branch", or "multi-mount" and "multi-branch" into auto-resolving three-way merges for text-files - so you can chop up your agentic working in whatever pattern you like. There's a walkthrough in the coding-agents guide, Multiple Mounts covers the mechanics, and the agentic-engineering page collects the moving parts.

Yes, we "broadcast"#

Swarmfile is built on top of the iroh ecosystem - which is a sort of peer-to-peer transport mechanism with some great extensions (the LAN-first P2P post walks through how the swarm actually works). We don't force it on you, you can still work offline, and it's part of both the open-source and the commercial ecosystem. You can also choose to turn it off if you prefer traditional hub-and-spoke data-transfer.

Whilst we've covered single-machine working so far we haven't yet covered multi-machine working and how the "swarm" (that's the terminology from iroh) is helpful here.

Let's take the scenario above (steps 1 to 6) - we're now going to have another agent listening to that broadcast from step 4.

1. Another engineer is working on the same branch somewhere else (e.g. another country).
2. Agent C has already read the entirety of index.html and it's making changes in-memory before writing the file.
3. When the swarmfile-engine instance for Agent A broadcast its file change we receive a near realtime notification of a change to that file.
4. Agent C saves their changes and they receive an ESTALE notice.
5. Agent C re-reads the file, and then has to rewrite their file.

This is a new paradigm in streaming, event-driven VCS and CI.

What's this about a "mount"#

The native Swarmfile client "mounts" a repo/project as a virtual drive so that your computer can work with it just like regular files. This is how we can present terabytes of data without downloading any of it. When you work this way you can work natively from any tool whilst also getting all the advantages of a Version Control System.

This means that your Marketing team can work on the repo like it's a shared-drive and trigger CI jobs, whilst the engineering team works on the code using Git tooling, and the two teams never have to trip each other up despite their radically different ways of working in the same monorepo.

The drive behaves the same to the tool on Windows, Linux and MacOS, on the open-source tooling and the commercial offering. The tracking itself is exact per process on Windows and Linux, where the OS hands us the process id; on macOS the NFS front-end carries no pid, so the engine falls back to a per-mount signal.

But Anthropic said...#

Scion is a sound answer to Anthropic's problem, and we think they'd agree it's theirs to own: a fleet of always-online servers replacing a mirror fleet. Their first principle is clients are always online - fetch data when they need it. We take that principle one step further. If the client streams only the bytes it actually reads, then most clients never fetch a working copy at all - and the checkout storm that motivates a system like Scion (tens of millions of checkouts a day) simply doesn't arise. That is the difference between out-scaling a workload and not having it. The economics differ too: our control plane runs on lightweight edge isolates, not a fleet of always-on servers.

• clients are always online. A starting point for us too, but not a limitation. You will notice advantages if you're working online and with the native Swarmfile client but we don't force it - you always have Git available and the native Swarmfile client supports offline working too.
• Client-server asymmetry. A starting point for us too, but not a limitation. We never download any content until the client needs it, and even then it's likely to be pulled from a peer if it's closer and faster to do so (e.g. not always the hub). This improves resilience if the team is disconnected from the hub for whatever reason. You can even have a warm cache on a NAS and erasure-coding spread around your peers (we wrote about that here). Also, Git is still here so you can still pull all content as normal if you want.
• A comprehensive source control service can power many novel applications. A starting point for us too, but not a limitation. Except that metadata may travel with the content chunk, so instead of saturating one API you may instead pull metadata and data from zero or more peers - whether you're using the open-source or commercial offering, that's the swarm.
• Everything must be horizontally scalable A starting point for us too, but not a limitation. But our limit is whatever our cloud provider offers in your chosen region, or whatever your S3-compatible bucket provider has. The commercial offering is edge-native, running on lightweight isolates in the region you pick... and you can bring your own bucket. Our open-source offering can be run from anywhere and the horizontal scalability for it comes from the swarm, which is querying and pulling data from peers.
• Commits must be cheap, incremental, and lock-free A starting point for us too, but not a limitation. We're a forward-only DAG that allows partial-hydration on content-addressed chunks. Client-side execution is (as a concept) progressive and it's a strength of Git too, plus it strengthens our full-encryption promises on our commercial offerings.
• The working copy is optional, and sparse by design A starting point for us too, but not a limitation. Partial materialization of forward-only content-addressed chunks on a DAG can be instrumented by metadata spread around peers, not just from a central API. Meaning that we don't just enable sparse-checkouts, but with the Swarmfile-native tooling a user can browse a massive monorepo without ever downloading a file. And when a CI job (or self-hosted Runner) triggers then it can perform sparse hydration, or just pull the deltas where it has stale or missing content.

We're Swarmfile, we're open-source, we don't train AI on your data.#

We're a new bootstrapped startup built by engineers just like you, trying to work more efficiently in the modern age of AI. The community server, the standalone tools and the shared libraries are dual-licensed MIT/Apache-2.0, and everything we publish lives on our own hub: browse the swarmfile organization, or start at the open-source docs home. Read our launch story to see our origins, explore our documentation to see how we're built, or just ask AI if you want a quick answer.

A quick note on Git#

If part of your team is running Git, and part is running native Swarmfile, then it doesn't matter, there's no clash. For Git users branch materialization doesn't happen until the Git user pulls, so the workflow for the Git user is the same as it always has been with Git when they pull down changes. And similarly, when they push changes then events are triggered across the swarm as described above. Conflicts are resolved using the normal Git workflow - Cloning a Project with Git covers the clone and push flow (and public projects can even be cloned anonymously).

A quick note on CI#

Swarmfile doesn't have to emit events, but when it does component systems (including self-hosted runners) will receive near realtime notification of changes. Meaning that they can watch part of the repo/project for specific changes, and then act on those changes.

For example, in the case above we can have a 60s debounce on index.html for just the promotional website. When we're notified of a change in this area then we re-run CI and build and ship the promotional website, without materialising the rest of the monorepo... we just pull the content we need.

A quick note on non-text content#

Swarmfile was originally built for massive objects, but our variable content-chunking works just as well for text-content. So in the case above it might be a change to a 2TB raw 4k promotional video that requires a post-processing pipeline, or a 1KB partial edit to a text file... and everything inbetween.