Manifest Generation in Exherbo

In my own developer package repository, and in merge requests I have made recently to official Exherbo repositories, I have started to include Manifest files alongside the Exheres packaging. This post is an attempt to document how I have the tooling set up for semi-automatically generating manifests.

The cave tool includes a cave digest subcommand which inherits the manifest generation code which had originally been developed for Gentoo. It can be used with Exherbo, but it requires some custom configuration in order to fit with the recommended Exherbo contribution workflow. The two main key points are that the Manifest files need to be generated in the development checkout of the repository (which will usually be in your home directory) instead of the system checkout (under /var/db/paludis), and you will need to override some of the digest generation settings since these aren’t configured in most Exherbo repositories.

Setting up the Paludis configuration

Paludis supports loading configuration from your user home directory – see the specpath.conf documentation. Since the configuration needed for generating manifests in your development checkouts is incompatible with the Exherbo system configuration, I recommend using an environment suffix. In my case, I run the digest command using cave -E :dev, which results in it loading the configuration from ~/.paludis-dev.

There’s a few files which are required (Paludis will give an error if they are missing): licenses.conf and platforms.conf. Since this configuration isn’t being used to install packages, the contents of both of these files can be very simple: */* * (i.e. “accept all licenses/platforms on all packages”)

I use the repository_defaults.conf file to set up some options that will apply to all repositories (see the Paludis e Repository Format documentation):

format = e
distdir = /home/yourname/Downloads/distfiles
write_cache = /var/empty
manifest_hashes = SHA512
thin_manifests = true

• Setting format here just saves typing in the repository files (we will only be adding e format repos)
• distdir is kind of tricky. It needs to be a directory that your user can write to, so using the system distfiles cache dir is not really a good idea. That said, if you are tight on disk space or network bandwidth, it is possible. I do not recommend adding your user to the paludisbuild group. There are other options available to poke at the permissions. I personally use Posix ACLs to give write permissions for my user to the directory:

  setfacl -m u:yourname:rwx /var/cache/paludis/distfiles

  But this might not work properly for packages using alternative source fetching mechanisms like scm or cargo, due to the use of subdirectories.

• Setting write_cache to /var/empty disables the write cache, since it’s not needed here.
• You can adjust the manifest_hashes value, but SHA512 is the fastest and strongest hash that is currently supported by Paludis.
• Setting thin_manifests prevents the hashes of the exheres, exlibs, and supporting files from being included in the Manifest. The repositories are using git and we’re not signing the Manifest files, so hashing these files would provide no additional security.

Next, you need to create the repository configuration files. In addition to the repository that you are developing, you will also need to configure repository files for all of the parent repositories including arbor. Lets start with arbor.conf, since it’s a bit special:

location = /home/cwalton/Development/exherbo/arbor
profiles = ${location}/profiles/amd64

• If you are not developing packages in this repository, then the location setting can point to the system repository directory (e.g. /var/db/paludis/repositories/arbor). Otherwise it should be the path to your development git checkout.
• The arbor repository needs a profile set. It doesn’t matter which one, but it has to be valid.
• Do not set a sync URL, you don’t want to accidentally overwrite your changes.

The other repository files only need the location configured, like my x11.conf file:

location = /home/cwalton/Development/exherbo/x11

Generating Manifests

Once the configuration is actually set up, generating the Manifest files is straightforwards. To generate the Manifest for a package you are working on, run

cave -E :dev digest category/pn repository

Replacing category, pn, and repository with the appropriate values. Paludis will download all of the distfiles for the selected package (including optional files), generate their checksums, and create or update the Manifest file in the package directory in your git checkout. It will print out a warning that you haven’t configured a world file – you can ignore this, since you won’t be using this configuration to install packages anyways.

Now git add the Manifest file and include it in the same commit as your packaging work.

An Example Run

$ cd ~/Development/exherbo/gnome/packages/gnome-desktop/nautilus
$ ls
nautilus-43.2.exheres-0
$ cave -E :dev digest gnome-desktop/nautilus gnome
cave@1681683995: [WARNING paludis_environment.world.no_world] In thread ID '1729698':
  ... In program cave -E :dev digest gnome-desktop/nautilus gnome:
  ... When making environment from specification ':dev':
  ... When loading paludis configuration:
  ... When loading general.conf:
  ... No world file specified. You should specify 'world = /path/to/world/file' in /home/cwalton/.paludis-dev/general.conf. Any attempted updates to world will not be saved.
Fetching gnome-desktop/nautilus-43.2:0::gnome...
Trying to fetch 'https://download.gnome.org/sources/nautilus/43/nautilus-43.2.tar.xz' to 'nautilus-43.2.tar.xz'...
wget -T 30 -t 1 --continue -O /home/cwalton/Downloads/distfiles/nautilus-43.2.tar.xz.-PARTIAL- https://download.gnome.org/sources/nautilus/43/nautilus-43.2.tar.xz
--2023-04-16 18:26:36--  https://download.gnome.org/sources/nautilus/43/nautilus-43.2.tar.xz
Resolving download.gnome.org (download.gnome.org)... 151.101.193.91, 151.101.65.91, 151.101.1.91, ...
Connecting to download.gnome.org (download.gnome.org)|151.101.193.91|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://muug.ca/mirror/gnome/sources/nautilus/43/nautilus-43.2.tar.xz [following]
--2023-04-16 18:26:36--  https://muug.ca/mirror/gnome/sources/nautilus/43/nautilus-43.2.tar.xz
Resolving muug.ca (muug.ca)... 208.81.1.244, 2605:e200:3:4::244
Connecting to muug.ca (muug.ca)|208.81.1.244|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3166792 (3.0M) [application/octet-stream]
Saving to: ‘/home/cwalton/Downloads/distfiles/nautilus-43.2.tar.xz.-PARTIAL-’

/home/cwalton/Downloa 100%[==========================================================>]   3.02M   830KB/s    in 3.9s    

2023-04-16 18:26:41 (800 KB/s) - ‘/home/cwalton/Downloads/distfiles/nautilus-43.2.tar.xz.-PARTIAL-’ saved [3166792/3166792]

mv -f /home/cwalton/Downloads/distfiles/nautilus-43.2.tar.xz.-PARTIAL- /home/cwalton/Downloads/distfiles/nautilus-43.2.tar.xz
Checking 'nautilus-43.2.tar.xz'... ok

Making manifest...
$ ls
Manifest  nautilus-43.2.exheres-0
$ cat Manifest
DIST nautilus-43.2.tar.xz 3166792 SHA512 e4a12984b74614185cd7fbb9bf8c49a2572d673ae59d0f481dac92459d965ccc0eae6d9a5f545afb52ce1f24438a7feb02c0a35b8175187b5d74a9044488c13b