Mirroring a Project Between Hubs

swarmfile-mirror copies a project - or a subtree - from one Swarmfile hub to another. The common case is moving off the hosted service onto your own community server, or keeping a second copy on a self-hosted box.

It is the project-to-project counterpart of the migration tool: swarmfile-migrate brings a local tree or an S3 bucket in, and swarmfile-mirror copies a project to another project.

Install#

curl -fsSL https://get.swarmfile.com/mirror | sh

Use#

Give it a source and a destination - each with its own hub, org, project and token:

export SWARMFILE_SOURCE_API_KEY=sf_pat_…    # source hub token
export SWARMFILE_DEST_API_KEY=sf_key_…      # destination hub token

swarmfile-mirror \
  --source-hub-url https://hub.swarmfile.com \
  --source-org-id my-org --source-project-id <id> \
  --dest-hub-url https://your-server \
  --dest-org-id my-org --dest-project-id <id> \
  --path /Projects --branch main

• --path mirrors a subtree (default: the whole project).
• --branch selects the branch to read on the source; the destination is written to its default.
• --exclude <glob> skips entries by name (repeatable, e.g. --exclude '*.tmp').
• --dry-run lists what would be copied and touches nothing.
• --state <file> records each mirrored source path → (cid, size) and resumes on a re-run: a file already mirrored with the same source content is skipped, so an interrupted copy of a large project picks up where it left off.
• --json prints a machine-readable summary (files, bytes, dirs, symlinks, resumed, skipped).
• --concurrency <n> (default 4) copies up to n files in flight; 1 is sequential. It's ignored with --state (resume bookkeeping is ordered).
• Chunking follows swarmfile-seed: fixed-size by default, --cdc or --auto-profile to change it. The provenance mode below keeps the source's chunks instead.
• --provenance-out <file> writes the origin's portable provenance manifest and, for a plaintext whole-project mirror, preserves the source's block CIDs and entry lineage so the destination's tree is identical. It requires an empty destination branch (or a matching resume state), a confirmed/quiet source, no --exclude/--path, and a credential that can read the whole tree; sync-ignored entries are included. --concurrency, --cdc and --auto-profile are ignored in this mode (the source's chunks are copied verbatim). See Provenance.

What it copies#

Files, directories and symlinks. A file is re-encoded for the destination - chunked and sealed under the destination's key - so a plaintext or managed source lands correctly on a plaintext or managed destination. (The provenance mode above copies the source's exact chunks instead, so its tree matches.) Deleted and trashed entries are not copied.

Two refusals protect you, matching the migration path:

• a same-name/different-content collision on the destination is an error, not a silent skip; and
• a symlink with an absolute target is refused (it would not resolve on another machine).

Provenance#

A mirror is not a signed replica: the destination has its own commits, and swarmfile-verify-history --mirror-* only matches a replica that preserves the origin's confirmed head commit (the same commit DAG). The verifier checks commit/tree identity, not that every block is served.

--provenance-out <file> makes a plaintext, whole-project mirror verifiable offline. The mirror verifies the origin's signed history (confirmed head, commit chain, every checkpoint, key ring, and its signed host-identity statement when served), fetches the confirmed head's canonical tree and compares the LIVE source listing to it (refusing a partial/ACL-filtered listing, ignored-entry omission or live-uncommitted write), copies the source's block CIDs/manifests/entry lineage, then compares the destination's live listing to the same tree. It also pins a resume --state to the source head, so a resumed run after the origin advanced is refused. Add --dry-run to perform the source tree/listing comparison only: it prints the result, writes no manifest and leaves the destination untouched. The destination's tree - and usually its confirmed head commit - matches:

swarmfile-mirror … --provenance-out origin.json
swarmfile-verify-history \
  --hub-url https://your-server/orgs/my-org/projects/<dest-id> \
  --org-id my-org --project-id <dest-id> \
  --header "Authorization: Bearer $SWARMFILE_DEST_API_KEY" \
  --pin-key "$PUBLIC_KEY" --provenance origin.json

The manifest is a file: its key ring is untrusted input, so --provenance requires --pin-key (the key obtained out-of-band). When the origin serves /.well-known/swarmfile.json, the manifest carries that signed statement and the verifier reports originAttested; without one, the origin NAME is only a label. A matching commit/tree proves identity, not that the mirror serves the tree object or any block.

A managed or E2E destination (or source) is re-sealed under the destination's key, so every content id changes and no tree match is possible; --provenance-out refuses it rather than writing a manifest the destination cannot satisfy. A mirror cannot preserve a managed/E2E commit DAG. On a self-hosted community server the optional swarmfile-server admin copy-project is the supported same-server exact-history copy for managed projects (see Self-hosting); it copies the DB/key/data rather than re-sealing. Cross-hub/E2E key sharing remains a separate, deliberate gap. A subtree (--path) mirror, --exclude, a non-empty destination and a copy-mode resume state are likewise refused: the manifest describes the origin branch's whole tree. An ACL-limited source key sees only a partial tree - use an owner key; the consumer's verification fails loudly rather than claiming a match.

The manifest proves the origin's confirmed head/tree and its signed checkpoints; it does not prove every block is served by the mirror. See Verifiable History.

Scale#

The listing follows the hub's cursor pages, so a directory with more than a thousand entries mirrors completely. Up to --concurrency files copy in flight at once (default 4; each still uploads its blocks through a shared 8-way pool), and --state makes a whole-project copy resumable - an interrupted run re-reads only the files it has not mirrored. A very large project is still a full byte transfer the first time.

Exit codes#

0 success; 1 a failure (including a refused collision or symlink); 2 a flag/usage error from the argument parser.