This shows us remapping the internal TCP listening port as 9999 on the
host. This feature of OCI runtimes means there’s little point to using
the “`fossil server --port`” feature inside the container. We can let
Fossil default to 8080 internally, then remap it to wherever we want it
on the host instead.

For debugging the live container while it runs, you can get an interactive
shell like so:

```
  $ docker exec -it -u fossil fossil sh
```

Our stock `Dockerfile` configures Fossil with the default feature set,
so you may wish to modify the `Dockerfile` to add configuration options,
add APK packages to support those options, and so forth.

The Fossil `Makefile` provides two convenience targets,
“`make container-image`” and “`make container-run`”. The first creates a
versioned container image, and the second does that and then launches a

### <a id="repo-inside"></a> 2.1 Storing the Repo Inside the Container

The simplest method is to stop the container if it was running, then
say:

```
  $ docker cp /path/to/my-project.fossil fossil:/jail/museum/repo.fossil
  $ docker start fossil
  $ docker exec fossil chown -R 499 /jail/museum
```

That copies the local Fossil repo into the container where the server
expects to find it, so that the “start” command causes it to serve from
that copied-in file instead. Since it lives atop the immutable base
layers, it persists as part of the container proper, surviving restarts.

destroyed, too. The solution is to replace the “run” command above with
the following:

```
  $ docker run \
    --publish 9999:8080 \
    --name fossil-bind-mount \
    --volume ~/museum:/jail/museum \
    fossil
```

Because this bind mount maps a host-side directory (`~/museum`) into the
container, you don’t need to `docker cp` the repo into the container at
all. It still expects to find the repository as `repo.fossil` under that
directory, but now both the host and the container can see that repo DB.

You might be aware that OCI containers allow mapping a single file into
the repository rather than a whole directory.  Since Fossil repositories
are specially-formatted SQLite databases, you might be wondering why we
don’t say things like:

```
  --volume ~/museum/my-project.fossil:/jail/museum/repo.fossil
```

That lets us have a convenient file name for the project outside the
container while letting the configuration inside the container refer to
the generic “`/museum/repo.fossil`” name. Why should we have to name
the repo generically on the outside merely to placate the container?

[dbcorr]: https://www.sqlite.org/howtocorrupt.html#_deleting_a_hot_journal
[wal]:    https://www.sqlite.org/wal.html


## 3. <a id="security"></a>Security

### 3.1 <a id="chroot"></a>Why Chroot?

A potentially surprising feature of this container is that it runs
Fossil as root. Since that causes [the chroot jail feature](./chroot.md)



to kick in, and a Docker container is a type of über-jail already, you


may be wondering why we bother. Instead, why not either:





*   run `fossil server --nojail` to skip the internal chroot; or


*   set “`USER fossil`” in the `Dockerfile` so it starts Fossil as
    that user instead


The reason is, although this container is quite stripped-down by today’s



standards, it’s based on the [surprisingly powerful Busybox

project](https://www.busybox.net/BusyBox.html). (This author made a

living for years in the early 1990s using Unix systems that were less

powerful than this container.) If someone ever figured out how to make a
Fossil binary execute arbitrary commands on the host or to open up a
remote shell, the power available to them at that point would make it
likely that they’d be able to island-hop from there into the rest of
your network. That power is there for you as the system administrator
alone, to let you inspect the container’s runtime behavior, change
things on the fly, and so forth. Fossil proper doesn’t need that power;


if we take it away via this cute double-jail dance, we keep any
potential attacker from making use of it should they ever get in.









Having said this, know that we deem this risk low since a) it’s never
happened, that we know of; and b) we haven’t enabled any of the risky
features of Fossil such as [TH1 docs][th1docrisk]. Nevertheless, we
believe defense-in-depth strategies are wise.








If you say something like “`docker exec fossil ps`” while the system is



idle, it’s likely to report a single `fossil` process running as `root`


even though the chroot feature is documented as causing Fossil to drop
its privileges in favor of the owner of the repository database or its



containing folder. If the repo file is owned by the in-container user

“`fossil`”, why is the server still running as root?




It’s because you’re seeing only the parent process, which assumes it’s

running on bare metal or a VM and thus may need to do rootly things like
listening on port 80 or 443 before forking off any children to handle



HTTP hits. Fossil’s chroot feature only takes effect in these child
processes. This is why you can fix broken permissions with `chown`


after the container is already running, without restarting it: each hit

reevaluates the repository file permissions when deciding what user to
become when dropping root privileges.




[th1docrisk]: https://fossil-scm.org/forum/forumpost/42e0c16544


### 3.2 <a id="caps"></a>Dropping Unnecessary Capabilities

The example commands above create the container with [a default set of
Linux kernel capabilities][defcap]. Although Docker strips away almost
all of the traditional root capabilities by default, and Fossil doesn’t
need any of those it does take away, Docker does leave some enabled that
Fossil doesn’t actually need. You can tighten the scope of capabilities
by adding “`--cap-drop`” options to your container creation commands.

*   **`CHOWN`**: The Fossil server never even calls `chown(2)`, and our
    image build process sets up all file ownership properly, to the
    extent that this is possible under the limitations of our
    automation.

    Curiously, stripping this capability doesn’t affect your ability to
    run commands like “`chown -R fossil:fossil /jail/museum`” when
    you’re using bind mounts or external volumes — as we recommend
    [above](#bind-mount) — because it’s the host OS’s kernel
    capabilities that affect the underlying `chown(2)` call in that
    case, not those of the container.

    If for some reason you did have to change file ownership of
    in-container files, it’s best to do that by changing the

    [backoffice], and then only for processes it created on earlier
    runs; it doesn’t need the ability to kill processes created by other
    users. You might wish for this ability as an administrator shelled
    into the container, but you can pass the “`docker exec --user`”
    option to run commands within your container as the legitimate owner
    of the process, removing the need for this capability.




*    **`MKNOD`**: All device nodes are created at build time and are
    never changed at run time. Realize that the virtualized device nodes
    inside the container get mapped onto real devices on the host, so if
    an attacker ever got a root shell on the container, they might be
    able to do actual damage to the host if we didn’t preemptively strip
    this capability away.

*    **`NET_BIND_SERVICE`**: With containerized deployment, Fossil never
    needs the ability to bind the server to low-numbered TCP ports, not
    even if you’re running the server in production with TLS enabled and
    want the service bound to port 443. It’s perfectly fine to let the
    Fossil instance inside the container bind to its default port (8080)
    because you can rebind it on the host with the
    “`docker create --publish 443:8080`” option. It’s the container’s
    _host_ that needs this ability, not the container itself.

    (Even the container runtime might not need that capability if you’re
    [terminating TLS with a front-end proxy](./ssl.wiki#server). You’re
    more likely to say something like “`-p localhost:12345:8080`” and then
    configure the reverse proxy to translate external HTTPS calls into
    HTTP directed at this internal port 12345.)

*    **`NET_RAW`**: Fossil itself doesn’t use raw sockets, and our build
    process leaves out all the Busybox utilities that require them.
    Although that set includes common tools like `ping`, we foresee no
    compelling reason to use that or any of these other elided utilities
    — `ether-wake`, `netstat`, `traceroute`, and `udhcp` — inside the
    container. If you need to ping something, do it on the host.

    If we did not take this hard-line stance, an attacker that broke
    into the container and gained root privileges might use raw sockets

back-to-basics nature makes static builds work the way they used to,
back in the day. If that’s all you’re after, you can do so as easily as
this:

```
  $ docker build -t fossil .
  $ docker create --name fossil-static-tmp fossil
  $ docker cp fossil-static-tmp:/jail/bin/fossil .
  $ docker container rm fossil-static-tmp
```

The resulting binary is the single largest file inside that container,
at about 6 MiB. (It’s built stripped.)

[lsl]: https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead

[ctrd]:    https://containerd.io/
[nerdctl]: https://github.com/containerd/nerdctl
[runc]:    https://github.com/opencontainers/runc


### 6.2 <a id="podman"></a>Podman

A lighter-weight alternative to either of the prior options that doesn’t
give up the image builder is [Podman]. Initially created by
Red Hat and thus popular on that family of OSes, it will run on
any flavor of Linux. It can even be made to run [on macOS via Homebrew][pmmac]
or [on Windows via WSL2][pmwin].

On Ubuntu 22.04, the installation size is about 38&nbsp;MiB, roughly a
tenth the size of Docker Engine.

Although Podman [bills itself][whatis] as a drop-in replacement for the
`docker` command and everything that sits behind it, some of the tool’s
design decisions affect how our Fossil containers run, as compared to
using Docker. The most important of these is that, by default, Podman
wants to run your container “rootless,” meaning that it runs as a
regular user.  This is generally better for security, but [we dealt with
that risk differently above](#chroot) already. Since neither choice is
unassailably correct in all conditions, we’ll document both options
here.

[pmmac]:  https://podman.io/getting-started/installation.html#macos
[pmwin]:  https://github.com/containers/podman/blob/main/docs/tutorials/podman-for-windows.md
[Podman]: https://podman.io/
[whatis]: https://podman.io/whatis.html


#### 6.2.1 <a id="podman-rootless"></a>Fossil in a Rootless Podman Container

If you build the stock Fossil container under `podman`, it will fail at
two key steps:

1.  The `mknod` calls in the second stage, which create the `/jail/dev`
    nodes. For a rootless container, we want it to use the “real” `/dev`
    tree mounted into the container’s root filesystem instead.

2. Anything that depends on the `/jail` directory and the fact that it
   becomes the file system’s root once the Fossil server is up and running.

[The changes to fix this](/file/containers/Dockerfile-nojail.patch)
aren’t complicated. Simply apply that patch to our stock `Dockerfile`
and rebuild:





```
  $ patch -p0 < containers/Dockerfile-nojail.patch
  $ podman build -t fossil:nojail .
  $ podman create \
    --name fossil-nojail \
    --publish 127.0.0.1:9999:8080 \
    --volume ~/museum:/museum \
    fossil:nojail
```

Do realize that by doing this, if an attacker ever managed to get shell
access on your container, they’d have a BusyBox installation to play
around in. That shouldn’t be enough to let them break out of the
container entirely, but they’ll have powerful tools like `wget`, and
they’ll be connected to the network the container runs on. Once the bad
guy is inside the house, he doesn’t necessarily have to go after the
residents directly to cause problems for them.


#### 6.2.2 <a id="podman-rootful"></a>Fossil in a Rootful Podman Container

##### Simple Method

Fortunately, it’s easy enough to have it both ways. Simply run your
`podman` commands as root:

```
  $ sudo podman build -t fossil --cap-add MKNOD .
  $ sudo podman create \















    --name fossil \
    --cap-drop CHOWN \
    --cap-drop FSETID \
    --cap-drop KILL \
    --cap-drop NET_BIND_SERVICE \
    --cap-drop SETFCAP \
    --cap-drop SETPCAP \
    --publish 127.0.0.1:9999:8080 \
    localhost/fossil
  $ sudo podman start fossil
```

It’s obvious why we have to start the container as root, but why create
and build it as root, too? Isn’t that a regression from the modern
practice of doing as much as possible with a normal user?

We have to do the build under `sudo` in part because we’re doing rootly
things with the file system image layers we’re building up. Just because
it’s done inside a container runtime’s build environment doesn’t mean we
can get away without root privileges to do things like create the
`/jail/dev/null` node.

The other reason we need “`sudo podman build`” is because it puts the result
into root’s Podman image registry, where the next steps look for it.

That in turn explains why we need “`sudo podman create`:” because it’s
creating a container based on an image that was created by root. If you
ran that step without `sudo`, it wouldn’t be able to find the image.

If Docker is looking better and better to you as a result of all this,
realize that it’s doing the same thing. It just hides it better by
creating the `docker` group, so that when your user gets added to that
group, you get silent root privilege escalation on your build machine.
This is why Podman defaults to rootless containers.  If you can get away
with it, it’s a better way to work.  We would not be recommending
running `podman` under `sudo` if it didn’t buy us [something we wanted
badly](#chroot).

Notice that we had to add the ability to run `mknod(8)` during the
build. [Podman sensibly denies this by default][nomknod], which lets us
leave off the corresponding `--cap-drop` option. Podman also denies
`CAP_NET_RAW` and `CAP_AUDIT_WRITE` by default, which we don’t need, so
we’ve simply removed them from the `--cap-drop` list relative to the
commands for Docker above.

[nomknod]: https://github.com/containers/podman/issues/15626


##### <a id="pm-root-workaround"></a>Building Under Docker, Running Under Podman

If you have a remote host where the Fossil instance needs to run, it’s
possible to get around this need to build the image as root on the
remote system. You still have to build as root on the local system, but
as I said above, Docker already does this. What we’re doing is shifting
the risk of running as root from the public host to the local one.

Once you have the image built on the local machine, create a “`fossil`”
repository on your container repository of choice such as [Docker
Hub](https://hub.docker.com), then say:

```
  $ docker login
  $ docker tag fossil:latest mydockername/fossil:latest
  $ docker image push mydockername/fossil:latest
```

That will push the image up to your account, so that you can then switch
to the remote machine and say:

```
  $ sudo podman create \
    --any-options-you-like \
    docker.io/mydockername/fossil
```

This round-trip through the public image registry has another side
benefit: your local system might be a lot faster than your remote one,
as when the remote is a small VPS. Even with the overhead of schlepping
container images across the Internet, it can be a net win in terms of
build time.



### 6.3 <a id="nspawn"></a>`systemd-container`

If even the Podman stack is too big for you, the next-best option I’m
aware of is the `systemd-container` infrastructure on modern Linuxes,
available since version 239 or so.  Its runtime tooling requires only

Next, create `/etc/systemd/nspawn/myproject.nspawn`:

----

```
[Exec]
WorkingDirectory=/jail
Parameters=bin/fossil server                \
    --baseurl https://example.com/myproject \
    --chroot /jail                          \
    --create                                \
    --jsmode bundled                        \
    --localhost                             \
    --port 9000                             \
    --scgi                                  \
    --user admin                            \
    museum/repo.fossil

    CAP_SETFCAP          \
    CAP_SETPCAP
ProcessTwo=yes
LinkJournal=no
Timezone=no

[Files]
Bind=/home/fossil/museum/myproject:/jail/museum

[Network]
VirtualEthernet=no
```

----

*   The command given in the `Parameters` directive assumes you’re
    setting up [SCGI proxying via nginx][DNT], but with adjustment,
    it’ll work with the other repository service methods we’ve
    [documented][srv].

*   The path in the host-side part of the `Bind` value must point at the
    directory containing the `repo.fossil` file referenced in said
    command so that `/jail/museum/repo.fossil` refers to your repo out
    on the host for the reasons given [above](#bind-mount).

That being done, we also need a generic systemd unit file called
`/etc/systemd/system/fossil@.service`, containing:

----

public using nginx via SCGI. If you aren’t using a front-end proxy
and want Fossil exposed to the world via HTTPS, you might say this instead in
the `*.nspawn` file:

```
Parameters=bin/fossil server \
    --cert /path/to/cert.pem \
    --chroot /jail           \
    --create                 \
    --jsmode bundled         \
    --port 443               \
    --user admin             \
    museum/repo.fossil
```

rather than “boot” an OS image. That causes a bunch of commands to fail:

*   **`machinectl poweroff`** will fail because the container
    isn’t running dbus.

*   **`machinectl start`** will try to find an `/sbin/init`
    program in the rootfs, which we haven’t got.  We could
    rename `/jail/bin/fossil` to `/sbin/init` and then hack
    the chroot scheme to match, but ick.  (This, incidentally,
    is why we set `ProcessTwo=yes` above even though Fossil is
    perfectly capable of running as PID 1, a fact we depend on
    in the other methods above.)

*   **`machinectl shell`** will fail because there is no login
    daemon running, which we purposefully avoided adding by