Yet on a small server — say, a $4/month 10 GiB Digital Ocean droplet —
that’s still a big chunk of your storage budget. It takes 100:1 overhead
just to run a 4 MiB Fossil server container? Once again, I wouldn’t
blame you if you noped right on out of here, but if you will be patient,
you will find that there are ways to run Fossil inside a container even
on entry-level cloud VPSes. These are well-suited to running Fossil; you
don’t have to resort to [raw Fossil service][srv] to succeed,
leaving the benefits of containerization to those with bigger budgets.

For the sake of simple examples in this section, we’ll assume you’re
integrating Fossil into a larger web site, such as with our [Debian +
nginx + TLS][DNT] plan. This is why all of the examples below create
the container with this option:

document, but you can find ready advice for that elsewhere. Seeing how
we do this with Fossil should help you bridge the gap in extending
this idea to the rest of your site.)

[DD]:  https://www.docker.com/products/docker-desktop/
[DE]:  https://docs.docker.com/engine/
[DNT]: ./server/debian/nginx.md
[srv]: ./server/


### 6.1 <a id="nerdctl" name="containerd"></a>Stripping Docker Engine Down

The core of Docker Engine is its [`containerd`][ctrd] daemon and the
[`runc`][runc] container runner. Add to this the out-of-core CLI program
[`nerdctl`][nerdctl] and you have enough of the engine to run Fossil

A lighter-weight alternative to either of the prior options that doesn’t
give up the image builder is [Podman]. Initially created by
Red Hat and thus popular on that family of OSes, it will run on
any flavor of Linux. It can even be made to run [on macOS via Homebrew][pmmac]
or [on Windows via WSL2][pmwin].

On Ubuntu 22.04, the installation size is about 38 MiB, roughly a
tenth the size of Docker Engine.


Although Podman [bills itself][whatis] as a drop-in replacement for the
`docker` command and everything that sits behind it, some of the tool’s
design decisions affect how our Fossil containers run, as compared to
using Docker. The most important of these is that, by default, Podman
wants to run your container “rootless,” meaning that it runs as a
regular user.  This is generally better for security, but [we dealt with

benefit: your local system might be a lot faster than your remote one,
as when the remote is a small VPS. Even with the overhead of schlepping
container images across the Internet, it can be a net win in terms of
build time.



### 6.3 <a id="nspawn"></a>`systemd-container`

If even the Podman stack is too big for you, the next-best option I’m
aware of is the `systemd-container` infrastructure on modern Linuxes,
available since version 239 or so.  Its runtime tooling requires only
about 1.4 MiB of disk space:


```
  $ sudo apt install systemd-container btrfs-tools

```

That command assumes the primary test environment for
this guide, Ubuntu 22.04 LTS with `systemd` 249.  For best
results, `/var/lib/machines` should be a btrfs volume, because
[`$REASONS`][mcfad].  (For CentOS Stream 9 and other Red Hattish
systems, you will have to make serveral adjustments, which we’ve
collected [below](#nspawn-centos) to keep these examples clear.)

The first configuration step is to convert the Docker container into
a “machine”, as systemd calls it.  The easiest method is:

```
  $ make container-run
  $ docker container export fossil-e119d5983620 |
    machinectl import-tar - myproject
```


Copy the container name from the first step to the second.  Yours will



almost certainly be named after a different Fossil commit ID.



It’s important that the name of the machine you create &mdash;
“`myproject`” in this example &mdash; matches the base name
of the nspawn configuration file you create as the next step.
Therefore, to extend the example, the following file needs to be
called `/etc/systemd/nspawn/myproject.nspawn`, and it will contain
something like:

----

```
[Exec]
WorkingDirectory=/jail
Parameters=bin/fossil server                \

    --baseurl https://example.com/myproject \
    --chroot /jail                          \
    --create                                \
    --jsmode bundled                        \
    --localhost                             \
    --port 9000                             \
    --scgi                                  \
    --user admin                            \
    museum/repo.fossil
DropCapability=          \
    CAP_AUDIT_WRITE      \
    CAP_CHOWN            \
    CAP_FSETID           \
    CAP_KILL             \
    CAP_MKNOD            \
    CAP_NET_BIND_SERVICE \
    CAP_NET_RAW          \
    CAP_SETFCAP          \
    CAP_SETPCAP
ProcessTwo=yes
LinkJournal=no
Timezone=no


[Files]






Bind=/home/fossil/museum/myproject:/jail/museum











[Network]
VirtualEthernet=no
```

----

If you recognize most of that from the `Dockerfile` discussion above,
congratulations, you’ve been paying attention. The rest should also
be clear from context.

Some of this is expected to vary.  For one, the command given in the
`Parameters` directive assumes [SCGI proxying via nginx][DNT]. For
other use cases, see our collection of [Fossil server configuration
guides][srv], then adjust the command to your local needs.
For another, you will likely have to adjust the `Bind` value to
point at the directory containing the `repo.fossil` file referenced

in the command.

We also need a generic systemd unit file called
`/etc/systemd/system/fossil@.service`, containing:






----

```
[Unit]
Description=Fossil %i Repo Service
Wants=modprobe@tun.service modprobe@loop.service
After=network.target systemd-resolved.service modprobe@tun.service modprobe@loop.service

[Service]
ExecStart=systemd-nspawn --settings=override --read-only --machine=%i bin/fossil

[Install]
WantedBy=multi-user.target
```

----

You shouldn’t have to change any of this because we’ve given the
`--setting=override` flag, meaning any setting in the nspawn file
overrides the setting passed to `systemd-nspawn`.  This arrangement
not only keeps the unit file simple, it allows multiple services to
share the base configuration, varying on a per-repo level.



Start the service in the normal way:

```
  $ sudo systemctl enable fossil@myproject
  $ sudo systemctl start  fossil@myproject
```



You should find it running on localhost port 9000 per the nspawn
configuration file above, suitable for proxying Fossil out to the
public using nginx, via SCGI. If you aren’t using a front-end proxy
and want Fossil exposed to the world, you might say this instead in
the `nspawn` file:


```
Parameters=bin/fossil server         \
    --cert /path/to/my/fullchain.pem \
    --chroot /jail                   \
    --create                         \
    --jsmode bundled                 \
    --port 443                       \
    --user admin                     \
    museum/repo.fossil
```


You would also need to un-drop the `CAP_NET_BIND_SERVICE` capability
to allow Fossil to bind to this low-numbered port.


We use systemd’s template file feature to allow multiple Fossil
servers running on a single machine, each on a different TCP port,
as when proxying them out as subdirectories of a larger site.
To add another project, you must first clone the base “machine” layer:



```
  $ sudo machinectl clone myproject otherthing
```



That will not only create a clone of `/var/lib/machines/myproject`


as `../otherthing`, it will create a matching `nspawn` file for you

as a copy of the first one.  Adjust its contents to suit, then enable
and start it as above.

[mcfad]: https://www.freedesktop.org/software/systemd/man/machinectl.html#Files%20and%20Directories


### 6.3.1 <a id="nspawn-rhel"></a>Getting It Working on a RHEL Clone


The biggest difference between doing this on OSes like CentOS versus
Ubuntu is that RHEL (thus also its clones) doesn’t ship btrfs in

its kernel, thus has no option for installing `mkfs.btrfs`, which
[`machinectl`][mctl] needs for various purposes.

Fortunately, there are workarounds.

First, the `apt install` command above becomes:

```
  $ sudo dnf install systemd-container
```

Second, you have to hack around the lack of `machinectl import-tar` so:

```






  $ rootfs=/var/lib/machines/fossil




  $ sudo mkdir -p $rootfs
  $ docker container export fossil | sudo tar -xf -C $rootfs -
```

The parent directory path in the `rootfs` variable is important,
because although we aren’t using `machinectl`, the `systemd-nspawn`
developers assume you’re using them together.  Thus, when you give
`--machine`, it assumes the `machinectl` directory scheme.  You could
instead use `--directory`, allowing you to store the rootfs whereever
you like, but why make things difficult?  It’s a perfectly sensible
default, consistent with the [LHS] rules.

The final element &mdash; the machine name &mdash; can be anything
you like so long as it matches the nspawn file’s base name.

Finally, since you can’t use `machinectl clone`, you have to make
a wasteful copy of `/var/lib/machines/myproject` when standing up
multiple Fossil repo services on a single machine.  (This is one
of the reasons `machinectl` depends on `btrfs`: cheap copy-on-write
subvolumes.)  Because we give the `--read-only` flag, you can simply
`cp -r` one machine to a new name rather than go through the
export-and-import dance you used to create the first one.



[LHS]:  https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
[mctl]: https://www.freedesktop.org/software/systemd/man/machinectl.html


### 6.3.2 <a id="nspawn-weaknesses"></a>What Am I Missing Out On?




For all the runtime size savings in this method, you may be wondering
what you’re missing out on relative to Podman, which takes up


roughly 27× more disk space.  Short answer: lots.  Long answer:

1.  **Build system.**  You’ll have to build and test your containers

    some other way.  This method is only suitable for running them
    once they’re built.

2.  **Orchestration.**  All of the higher-level things like
    “compose” files, Docker Swarm mode, and Kubernetes are
    unavailable to you at this level.  You can run multiple
    instances of Fossil, but on a single machine only and with a
    static configuration.

3.  **Image layer sharing.**  When you update an image using one of the
    above methods, Docker and Podman are smart enough to copy only
    changed layers.  Furthermore, when you base multiple containers
    on a single image, they don’t make copies of the base layers;
    they can share them, because base layers are immutable, thus
    cannot cross-contaminate.



    Because we use `sysetmd-nspawn --read-only`, we get *some*
    of this benefit, particularly when using `machinectl` with
    `/var/lib/machines` as a btrfs volume.  Even so, the disk space
    and network I/O optimizations go deeper in the Docker and Podman
    worlds.


4.  **Tooling.** Hand-creating and modifying those systemd
    files sucks compared to “`podman container create ...`”  This
    is but one of many affordances you will find in the runtimes


    aimed at daily-use devops warriors.

5.  **Network virtualization.** In the scheme above, we turn off the
    `systemd` virtual netorking support because in its default mode,




    it wants to hide the service entirely.

    Another way to put this is that `systemd-nspawn --port` does
    approximately *nothing* of what `docker create --publish` does
    despite their superficial similarities.

    For this container, it doesn’t much matter, since it exposes
    only a single port, and we do want that one port exposed, one way
    or another.  Beyond that, we get all the control we need using
    Fossil options like `--localhost`.  I point this out because in
    more complex situations, the automatic network setup features of
    the more featureful runtimes can save a lot of time and hassle.
    They aren’t doing anything you couldn’t do by hand, but why
    would you want to, given the choice?




I expect there’s a lot more I neglected to think of when creating
this list, but I think it suffices to make my case as it is. If you
can afford the space of Podman or Docker, I strongly recommend using
either of them over the much lower-level `systemd-container`



infrastructure.


(Incidentally, these are essentially the same reasons why we no longer
talk about the `crun` tool underpinning Podman in this document. It’s
even more limited, making it even more difficult to administer while
providing no runtime size advantage. The `runc` tool underpinning
Docker is even worse on this score, being scarcely easier to use than
`crun` while having a much larger footprint.)


### 6.3.3 <a id="nspawn-assumptions"></a>Violated Assumptions

The `systemd-container` infrastructure has a bunch of hard-coded


assumptions baked into it.  We papered over these problems above,
but if you’re using these tools for other purposes on the machine
you’re serving Fossil from, you may need to know which assumptions
our container violates and the resulting consequences:


1.  `systemd-nspawn` works best with `machinectl`, but if you haven’t

    got `btrfs` available, you run into [trouble](#nspawn-rhel).




2.  Our stock container starts a single static executable inside
    a stripped-to-the-bones container rather than “boot” an OS
    image, causing a bunch of commands to fail:


    *   **`machinectl poweroff`** will fail because the container
        isn’t running dbus.
    *   **`machinectl start`** will try to find an `/sbin/init`
        program in the rootfs, which we haven’t got.  We could
        rename `/jail/bin/fossil` to `/sbin/init` and then hack
        the chroot scheme to match, but ick.  (This, incidentally,
        is why we set `ProcessTwo=yes` above even though Fossil is
        perfectly capable of running as PID 1, a fact we depend on
        in the other methods above.)
    *   **`machinectl shell`** will fail because there is no login
        daemon running, which we purposefully avoided adding by
        creating a “`FROM scratch`” container. (If you need a
        shell, say: `sudo systemd-nspawn --machine=myproject /bin/sh`)
    *   **`machinectl status`** won’t give you the container logs
        because we disabled the shared journal, which was in turn
        necessary because we don’t run `systemd` *inside* the


        container, just outside.





    If these are problems for you, you may wish to build a
    fatter container using `debootstrap` or similar. ([External
    tutorial][medtut].)

3.  We disable the “private networking” feature since the whole
    point of this container is to expose a network service to the
    public, one way or another.  If you do things the way the defaults
    (and thus the official docs) expect, you must push through
    [a whole lot of complexity][ndcmp] to re-expose this single
    network port.  That complexity is justified only if your service
    is itself complex, having both private and public service ports.




[medtut]: https://medium.com/@huljar/setting-up-containers-with-systemd-nspawn-b719cff0fb8d
[ndcmp]:  https://wiki.archlinux.org/title/systemd-networkd#Usage_with_containers

<div style="height:50em" id="this-space-intentionally-left-blank"></div>