Hashsync: Copy Nix closures efficiently via ssh

Hello Nix community,

I’d like to announce a project that I just published, hashsync: GitHub - someplaceguy/hashsync: Run a command over SSH, deduplicating its output or input against files stored on the receiving host, to save bandwidth. · GitHub

It can be used to copy Nix closures between your own machines in a far more efficient way than nix-copy-closure (or nix copy), because it doesn’t copy data that already exists in the destination Nix store, even if it’s in a different store path, a differently-named file or even if the data to be copied is at a different offset in a file you already have.

If you use your own Hydra, this can make package updates much faster and bandwidth-efficient, since most package updates usually have almost the same contents as previous versions (so why bother copying the entire package if you already have most of the data, right?).

hashsync works as a transparent pipe over ssh, so it can also be used with tar, rsync, etc. It has no special knowledge of Nix closures, NARs, etc – it works with any data stream, as it can find blocks to deduplicate at any offset (like rsync’s delta algorithm but across your entire Nix store rather than just one file).

But it does have special support for indexing Nix stores: they can be marked as “immutable roots”, so hashsync can assume that a path inside the Nix store always has the same contents, which makes updating the index far more efficient (since it doesn’t have to traverse paths that have already been traversed).

I’ve been using it every day for weeks and it works great for me. Let me know if you have any questions.

Thanks!

1 post - 1 participant

Read full topic