Cloning a Project with Git

A Swarmfile project is normally something you mount, not clone: the drive is live, saves sync as you make them, and there's no working copy to keep up to date. But some tools want a real git repository - a build system, a code-review or static-analysis tool, a CI job, an archive you keep offline. For those, an owner can turn on git access for a project, and anyone who can read the whole project can then:

git clone swarmfile://<org>/<project>

The result is an ordinary git repository: real commits, branches and tags, with large files as git-LFS pointers that Swarmfile serves. git fetch brings the clone up to date, and git push sends your commits back as Swarmfile commits (fast-forward only - see Pushing), and an existing repository can be pushed into a new project.

New to how git and Swarmfile fit together? Git and Swarmfile compares this with the git-LFS server and the CLI's git-style commands.

Turning on git access#

Git access is a per-project setting that covers every branch and tag. A project owner or org admin turns it on:

In the web dashboard, open Projects, use the project's menu, and choose Git access….
Read the warning, tick I understand this cannot be turned off or re-tuned, and select Enable git access.

It's a one-way switch. At the moment you enable it, Swarmfile fixes the rules it uses to turn the project into git: the conversion version, the size above which a file becomes an LFS pointer (1 MiB), and where history starts. Those rules never change afterward, because changing any of them would change every commit id a clone has already seen. Once enabled, the dialog shows the frozen settings instead of the switch.

Before you enable it, two things must be true, and the hub refuses the enable (and says which) if they aren't:

The project keeps every saved version. Git access needs the project's full history to stay restorable. Turn it on with Keep every saved version in the project's history settings, or swarmfile project set-keep-full-history true (see Trash, History & Rollback).
Every large file has a recorded id. A file at or above the LFS threshold reaches git as an LFS pointer, and a pointer needs the file's SHA-256. Files saved through the Desktop App get one as they're saved; older files may not. Run swarmfile git status to see how many are missing and swarmfile git backfill-oids to fill them in.

The enable is also refused on an end-to-end-encrypted project (git access isn't available for E2E projects yet; mount the project, or use swarmfile materialize, instead), when a branch or tag head can't be read, and when the project's history is too long to check in one pass.

Where history starts. When you enable git access, Swarmfile walks back from every branch and tag. If some older commit's content is no longer stored (for example, history from before the project kept every saved version), history is cut there: that commit appears in git as a commit with no parent. Everything from that point forward is complete.

Cloning#

git clone swarmfile://acme/website           # by org and project slug
git clone swarmfile://acme/website site      # into a directory of your choice
swarmfile git clone acme/website site        # the same, through the CLI

The forms the URL takes:

URL	Meaning
`swarmfile://<org>/<project>`	The project on the Swarmfile service your Desktop App is signed in to (or `SWARMFILE_HUB_URL`). Slugs or ids both work.
`swarmfile://<hub-host>/<org>/<project>`	The same project on a specific Swarmfile service, always over HTTPS.
`swarmfile://<project-id>`	A project by its id, in your configured organization.

swarmfile git url <project> prints the URL for a project, handy for git remote add origin "$(swarmfile git url acme/website)".

What you need installed. Cloning uses a small git remote helper, git-remote-swarmfile, which every Desktop App installer ships (on macOS the installer also links it into /usr/local/bin; on Linux it's in /usr/bin). No running drive is needed. If git says it can't find a helper for swarmfile, use swarmfile git clone, which finds the helper next to the swarmfile binary even when it isn't on your PATH. Git 2.11 or later is required.

There's no branch in the URL: a clone fetches every branch and tag, with main as the default branch. Use git's own options to narrow it, for example git clone -b design-b --single-branch swarmfile://acme/website.

Signing in#

The helper looks for a credential in this order:

Your Desktop App sign-in. On a machine where you're signed in to the Desktop App, git clone just works.
A project API key from git's credential store. On a machine with no Desktop App session (a CI box, a container), store a project API key (sf_key_…) as the password for the Swarmfile host with git credential approve, and the helper picks it up.
SWARMFILE_API_KEY. A project API key in the environment. If both this and a Desktop App session are present, the API key wins, and the helper says so.

A clone needs to read the whole project. If your access is restricted anywhere in it (a folder you're denied, say), the hub can't give you the complete tree, and the clone is refused with a message saying so rather than producing a repository with silent holes. Use an owner's or a fully-permitted member's credential, or a project API key, for clones.

What a clone contains#

Branches and tags. Every non-empty branch becomes a git branch; tags become lightweight git tags. An empty project clones as an empty repository.
Commits. Each Swarmfile commit becomes a git commit, with your message and a Swarmfile-Commit: trailer naming the Swarmfile commit it came from. A commit with no message reads (no message). Authors are recorded by account id; the helper writes a local mailmap so git log shows the names of people in your organization.
Unsaved-to-a-commit work. If a branch has saves since its last commit (autosaves that haven't been rolled into one yet), the clone ends that branch with one extra commit, authored by swarmfile with the message "swarmfile snapshot of the live tree", so what you clone matches what's on the drive.
Large files as git-LFS pointers. Files of 1 MiB or more come through as LFS pointers, with a generated .gitattributes in each folder that has them (added after any .gitattributes you already had). The clone is configured to fetch LFS content from Swarmfile automatically - see Git-LFS for the transfer agent and performance notes.
Symlinks stay symlinks. Empty folders aren't carried (git can't store them). The executable bit isn't preserved: every file arrives as a regular, non-executable file.
On Windows, a path Windows can't create stops the clone with a message naming it.

Commit ids are stable: two people who clone the same project get the same ids, and a later fetch never rewrites what you already have.

Staying up to date#

Run git fetch (or git pull) in the clone. Only new commits are converted and downloaded, so a fetch after a day's work is quick. New branches and tags appear like they would from any remote.

If a branch is being saved to while you fetch, the fetch may ask you to retry shortly - it won't build a snapshot of a tree that's still moving.

When a branch's newest commit can't be served yet. A commit enters the git view once it has been derived (the Desktop App does this after each commit, or run swarmfile git index); a provisional - derived but not yet confirmed - commit is served like any other. A head that has never been derived, or whose claim is disputed, is refused with the reason and what to do: usually swarmfile git index from a machine with the Desktop App, or, for a disputed claim, an owner clearing it from the Git access dialog.

Pushing#

git push sends commits from the clone back to the project. Each git commit becomes one Swarmfile commit, and Swarmfile keeps the commit's exact author, committer, dates, message and any signature, so its id doesn't change: anyone who clones or fetches afterwards gets the same commit ids you have. Pushing works on managed and unencrypted projects; it isn't available on end-to-end-encrypted ones.

git push origin main                 # fast-forward an existing branch
git push origin my-feature           # a new branch is created on Swarmfile
git push origin :my-feature          # archive a branch
git push origin v1.0                 # a lightweight tag becomes a Swarmfile tag
git push -f origin v1.0              # move the tag to the commit you tagged
git push origin :refs/tags/v1.0      # delete the tag
git push -v origin main              # also print each commit's planned changes
git push --dry-run origin main       # run every check, write nothing

A successful push prints a link to the project's commits in the web app (or to the merge request, below).

What a push accepts:

Fast-forwards only. Your commits must build on the branch as Swarmfile has it, one first parent after another. A force push of a branch (--force, +refspec) is refused. If someone saved to the branch since you last fetched, git reports fetch first: run git pull --rebase, then push again. Saves on the drive that haven't been rolled into a commit count too, since a fetch turns them into a snapshot commit.
New branches. Pushing a branch Swarmfile doesn't have creates it, starting from the newest commit in your history that Swarmfile already holds.
Merge commits with two parents. If the history you merged isn't on Swarmfile yet, the push sends it first, to a branch named merged/<first 12 characters of its newest commit> that starts where that history left the main line (an existing one is reused). The push lists each branch it creates, and they stay in the project so the merged commits remain visible. This works for merges nested inside merged history too. History that shares no commit with the project at all (a merge of an unrelated repository) is refused, naming the commit. Merges of three or more branches (octopus merges) aren't supported.
Protected branches don't move. Your commits go to a branch named push/<branch>/<first 12 characters of your commit>, started at the protected branch's head, and the push opens a merge request into the protected branch, or reuses the one already open from that branch. Git reports the ref as not updated, with opened merge request #<n> (push/<branch>/<commit>); the link takes you to it. The same rules apply to merged/* branches: if one is protected, the push is refused.
Deleting a branch (git push origin :<branch>) archives it, and it can be restored from the web app. main and protected branches can't be deleted this way.
Tags. A lightweight tag on a commit Swarmfile holds becomes a Swarmfile tag on that commit. To move an existing tag, retag and push with --force (git push -f origin <tag>); the commit must already be on Swarmfile. git push origin :refs/tags/<tag> deletes a tag. A tag that a release was published from can be neither moved nor deleted. Annotated tags are refused (Swarmfile tags are lightweight).
Large commits. A commit that changes more than 5,000 paths is sent in parts and applied as one commit, up to 2,000,000 paths. The branch is locked while it applies. Above 100,000 paths Swarmfile applies the commit in stages after the upload, and the push shows its progress. Anyone reading the branch during that time can see the files that have been applied so far. If the push is interrupted after the parts were sent, push again: it picks up where it left off. If someone else's change gets onto the branch while a staged commit is applying, Swarmfile stops the commit. The push then reports how far it got, and the changes already applied stay on the branch as uncommitted changes for you to review.
A commit already on another branch (for example a fast-forward of main to your feature branch's tip) is recorded again on the target branch with the same commit id, so both branches show the same history.

Push an existing repo into a new project. A repository that has never been on Swarmfile goes into a project whose main has no commits and no files yet, with its whole history:

git remote add origin swarmfile://<org>/<project>
git fetch origin                     # configures git-LFS (and its credential) for Swarmfile
git lfs push --all origin main       # if the repository uses git-LFS
git push origin main

The first commit lands with no parent, merges bring their merged history along as described above, and a fresh clone afterwards has the same commit ids. The push sends consecutive commits together, up to 100 commits or 5,000 changed paths per request. With a push budget of about 120 requests a minute per person, that is room for about 12,000 small commits a minute, so a 10,000-commit history needs about 100 requests and the budget is rarely what you wait on. Uploading file content and indexing each commit take most of the time. When the budget does run out, the push waits for it and carries on. A main that already has files but no commit can't take a root commit: fetch first, and the push builds on the snapshot.

What a pushed commit may contain. Swarmfile accepts a commit only if it is exactly what a clone of the result would produce, so git fetch never rewrites it. The push checks every commit, and asks the hub to check the first one, before uploading anything, and names the file and the fix when one doesn't fit:

No executable bit and no submodules. Clear the bit with git update-index --chmod=-x <file> and commit again.
Large files must be LFS pointers. A file of 1 MiB or more (the project's frozen threshold) must be committed as a git-LFS pointer, and its content must already be on Swarmfile. The clone is configured to send LFS content to Swarmfile with the same credential the clone uses (your signed-in Desktop App session, a stored sf_key_ key, or SWARMFILE_API_KEY; nothing to set up), so git-LFS's own pre-push step normally takes care of that; if a push says an LFS object is missing, run git lfs push origin <branch> and push again. A file under the threshold must be committed as a regular file, not as a pointer.
Each LFS file is covered in its folder's .gitattributes, either by a git lfs track pattern in that folder's own .gitattributes (for example *.psd filter=lfs diff=lfs merge=lfs -text), or by the line a clone writes for a file the patterns don't cover: /<name> filter=lfs diff=lfs merge=lfs -text, one per file, sorted by name, at the end of the file after the line # swarmfile: generated LFS attributes - view v1. Only patterns in the file's own folder count, not ones in parent folders, and a pattern with a / in it covers nothing. When a folder's lines don't match, the push prints the lines it expects.

What happens to the files on the drive:

Changed, new and deleted files become the same changes on the drive. A file renamed without changes keeps its Swarmfile history, and so does a whole folder renamed or moved without changes: it moves as one folder.
Folders you add are created, and folders your commit empties are removed. Empty folders that exist only on the drive (git can't show them) are left alone.
A push of several commits sends them in batches, and each commit is applied as a separate Swarmfile commit, in order. If one is refused, the commits before it have already landed, and the branch stays at the last of them. Fix the refused commit and push again to continue from there.
Commits you push are provisional until a second person's fetch confirms them, unless you're a project owner with self-confirmation turned on. The push says so when that's the case; provisional commits are cloned and fetched like any other.
A hub that predates pushing answers this Swarmfile hub does not support git push yet.

What's not supported#


Force pushes to a branch, annotated tags	Refused. Swarmfile history is append-only; push new commits instead. Tags can be moved with `--force` and deleted, unless a release was published from them. See Pushing for what a push accepts.
Pushing to an end-to-end-encrypted project	Not available yet.
Shallow clones	`--depth`, `--deepen` and `--unshallow` work once the commits at the cut have been indexed (`swarmfile git index`). A depth fetch also needs those boundary commit ids confirmed; on a project only one person derives, set `SWARMFILE_GIT_ACCEPT_PROVISIONAL=1` where you run the clone (or let a second derivation - a teammate's app, a CI API key - confirm them). `--shallow-since` and `--shallow-exclude` aren't supported.
Partial clones	`--filter` isn't supported. Large files are already LFS pointers, which gives most of the same benefit.
Fetching a commit by id	Only branches and tags can be fetched, not an arbitrary commit id.
HTTPS clone URLs	Not available; clone through the `swarmfile://` helper.
End-to-end-encrypted projects	Not available yet. Mount the project, or use `swarmfile materialize` to write a ref to a plain directory.
Branch or tag names git can't represent	Refused with a message naming them.

Two kinds of problem tell you how to fix them: a large file that has no recorded id (run swarmfile git backfill-oids), and a clone made under an older version of the git view (delete it and clone again).