| Title: Mirroring sources used in nixpkgs (software preservation)
Author: Solène
Date: 03 November 2022
Tags: nix life
Description: In this text, I will show how to download locally all the
sources used to build packages in nixpkgs
# Introduction
This may appear like a very niche use case, in my quest of software
conservancy for nixpkgs I didn't encounter many people understanding
why I was doing this.
I would like to present you a project I made to easily download all the
sources files required to build packages from nixpkgs, allowing to keep
offline copies.
Why would you like to keep a local copy? If upstream disappear, you
can't get access to the sources anymore, except maybe in Hydra, but you
rely on a third party to access the sources, so it's still valuable to
have local copies of software you care about, just to make copies.
It's not that absolutely useful for everyone, but it's always important
to have such tools available.
|
|
# How to use it
You must run it on a system with `nix` installed.
After cloning and 'cd-ing' into the directory, simply run `./run.sh
some package list | ./mirror.pl`. The command `run.sh` will generate a
JSON structure containing all the dependencies used by the packages
listed as arguments, and the script `mirror.pl` will iterate over the
JSON list and use nix's fetcher to gather the sources in the nix store,
verifying the checksum on the go. This will create a directory
`distfiles` containing symlinks to the sources files stored in the
store.
The symlinks are very important as they will prevent garbage collection
from the store, and it's also used internally to quickly check if a
file is already in the store.
To delete a file from the store, remove its symlink and run the garbage
collector.
# Limitation
I still need to figure how to get a full list of all the packages, I
currently have a work in progress relying on `nix search --json` but it
doesn't work on 100% of the packages for some reasons.
It's currently not possible to easily trim distfiles that aren't useful
anymore, I plan to maybe add it someday.
# Trivia
This task is natively supported in the OpenBSD tool building packages
(dpb), it can fetch multiples files in parallel and automatic remove
files that aren't used anymore. This was really complicated to figure
how to replicate this with nixpkgs. |