Skip to content

The feature that made 66 leave s6: the event system

With 0.9.0.0, 66 no longer links against skalibs, s6 or execline. The scanner, the supervisor, the logger, the fd holder and the privilege dropper are now native programs of the suite. Leaving the foundation a project was built on is not a decision to make lightly, so here is why. The tipping point was the event system: implementing it correctly is what made it clear that going further meant leaving s6. To see what it is for, start with a very ordinary problem that service managers usually solve with glue.

The certificate that needs a reload

nginx serves TLS. A certificate gets renewed, by certbot, by an ACME client, by whatever you run. nginx now has to reload, or it keeps serving the old certificate until someone notices. Nothing exotic; every TLS server on earth has this need. Watch how the init systems meet it.

  • systemd reaches for a second unit. You write a .path unit that watches the certificate file and activates a .service running systemctl reload nginx: two extra units, living apart from the one they serve. Or you leave it to the renewal tool, certbot's --deploy-hook for instance, and the init steps out of the loop entirely.
  • runit and OpenRC leave it to you: cron, an inotifywait loop, or a hand-written script. s6 publishes its own state changes on a fifodir, which is more than either, but nothing binds a reaction to a service: something else has to listen and decide, and that something is a script you write.
  • dinit offers an external observer, dinit-monitor; finit has state conditions. Both are real, and both live outside the service.

The common thread: in none of them does the service itself state what it reacts to. The reaction is glue, a second unit or a hook or a cron line or a watcher process, bolted on beside the service and never part of it. It works, but it leads a separate life. It can be changed, moved or forgotten with no link back to the service, and it is nowhere in the service's own definition.

What the event system does

The event system in 0.9.0.0 was built to close that gap. A reactor is an ordinary service with an [Event] section: when its trigger fires, it runs a 66 command on itself, or raises a named event for others. A source is what it listens to: another service's state, a signal, a filesystem change, a cron expression, a timer, or a user event. So nginx watches the certificate directory and reloads itself; or the renewal service emits cert-renewed and every service that cares reacts.

The reaction then becomes a property of the service. nginx carries it in its own frontend, one section: when the certificate changes, reload me. Not a second unit, not a hook in another tool. That is not where the work started; it is what a correct event system gives once it exists. Building it correctly is the part s6 could not carry.

Why that meant leaving s6

You cannot build an event system on a supervisor you merely drive from outside.

It needs two things 66 did not own: a communication channel it fully controls, and a supervisor whose announcements it can extend. s6-supervise does announce. It writes single characters into a ./event fifodir, such as u when it spawns the run script, U when the service reports itself ready, d when the service goes down. 66 consumed exactly that for years; readiness came from listening on that fifodir.

So the wall was never a missing emitter. It was that the vocabulary is closed, and a letter carries nothing beyond itself. There is no letter for a state s6 has no concept of, such as a reactor armed and waiting for its trigger. Nor does the stream say why a service went down (a commanded stop, a crash, a crash budget exhausted) or who asked for it. s6 does keep the exit code and the signal in the service's status file, but an event system needs them in the announcement itself, at the moment it happens, not in a file to read back afterwards. Adding a letter is not a patch you carry downstream; it is an edit to a contract that belongs to someone else. The channel underneath, the ftrig fifodir, was s6's and skalibs' too. To let services react, 66 needed its supervisor to say more than s6's vocabulary allows, and that vocabulary was not 66's to extend.

And once the wall showed here, it stood everywhere:

  • The state contract. Every richer thing a service needed to report (the true result of a run instead of a bare exit code, a reactor that is waiting to be triggered) meant changing the record a service keeps about itself. That record belongs to s6-supervise. You cannot evolve a contract you do not own.
  • The dependency treadmill. Standing on skalibs meant riding its release cycle; a new release could, and did, break 66. A foundation that moves on its own schedule moves what you stand on with it.
  • A different philosophy. s6 is built as a set of precise, composable tools, and that design is deliberate. 66 has another mission: to make supervision of that same quality approachable, a few readable lines with the hard, correct machinery running underneath, unshown. Two goals, two contracts. Building on s6 meant inheriting choices made for the other one.

This takes nothing away from s6. s6 is minimal, rigorous and correct; it is where 66 learned how supervision should be done, and 66-scandir and 66-supervise are its heirs. 66 simply grew a purpose s6 was never meant to serve.

Ten years of standing on it

There is a second half to this, and it is mine to tell, so this section speaks in the first person. Leaving it out would make the first half read as lighter than it was. Obarun shipped its first ISO in 2015, booting on runit. In May 2016, i wrote s6opts, a set of shell tools whose only purpose was to make s6 and s6-rc usable as the init of a real distribution: writing the service directories, driving the supervisor, handling user supervision. It existed because nothing else did. At the time, s6 as the init of a general-purpose distribution was still a rare idea.

The ISO answered the question by existing. A machine you could install and boot, whose PID 1 was s6-svscan and whose services were managed by s6-rc, is not an argument. It is a demonstration, and it was among the first. 66 itself, first released in March 2019, was built as the next step of exactly that work: not an alternative to s6 but a way to make it approachable, wrappers that turned a handful of scattered files into one readable frontend and a small set of commands. The early tags say so plainly for anyone who reads them. Artix picked 66 up for a while and later dropped it, during a stretch when my working life made it impossible to keep pace with the churn underneath: s6, s6-rc and skalibs all moving on their own schedule, and at one point moving in a way that broke Obarun outright. That is the dependency treadmill above, seen from underneath rather than described in the abstract.

It is not the reason 66 left; the reasons are the technical ones above, and they are sufficient on their own. It is here because independence in 0.9.0.0 reads very differently depending on what came before it. Ten years of building on s6 is where i learned what an init system actually is, and, more lastingly, what good software design looks like. That debt is real and this document has already said so. But a decade of writing the tooling that made a foundation usable, and of carrying the cost when it shifted, is also what makes owning the foundation the ordinary next step rather than a break. The bird leaving the nest is not a quarrel with the nest.

Taking the core

So the borrowed pieces became native: s6-svscan is now 66-scandir, s6-supervise is 66-supervise, s6-log is 66-log, along with the fd holder and the privilege dropper; skalibs gave way to oblibs, 66's own base library. execline stays at runtime, for the chainloading it is genuinely best at, but nothing links against skarnet libraries anymore. Now the supervisor belongs to 66, and what it announces is 66's to define, which was the whole point.

With the core in hand, a couple of other things fell into place almost for free. The boot now runs in memory, with no rc.init script and no fresh 66 spawned to finish the job: 66 brings up the boot group and then every enabled tree by itself. A small convenience appeared too. The status now tells you who caused a transition: the user, the boot, an event, or the service itself.

What the supervisor can learn now

Owning 66-supervise is not only about what it can announce. It means policy can move into the supervisor: both work that used to sit outside it and operations it genuinely never had. 66 has already started.

Take a service that keeps crashing. How many times may it die before the system stops trying? s6 answers this, and answers it well: s6-supervise keeps a death tally, each recent death with its timestamp and its cause, and s6-permafailon, dropped into the service's ./finish script, reads that tally and exits 125 once the process has died more than n times inside a window, counting only the exit codes or signals you name. s6-supervise sees the 125 and stops respawning. It is deliberate and precise: you name the exact exit codes and signals that count, intervals and all.

What 66 changes is not whether this can be done, but where it lives. In s6 the policy is a separate program you wire into a ./finish script: the certificate story again, glue bolted on beside the service rather than a property of it. 66 folds it into the frontend as two keys, MaxDeath and MaxDeathInterval. More than MaxDeath deaths inside a MaxDeathInterval window and the supervisor gives up on its own. A commanded stop never counts against the budget, and a fresh 66 start clears it. Because 66-supervise is now 66's to change, giving up also stops being merely stop respawning and becomes a first-class outcome in the service's own record, state failed and result crash-limit, which 66 status reports. s6 can tell the supervisor to stop; owning the supervisor is what lets the outcome become part of what the service is.

The same precision is 66's too, in a different place: because 66-supervise now emits the projected result, the exact exit code or the exact signal, a reactor can single out exited:1 or signaled:SIGSEGV and act on those alone. What s6 keeps in a ./finish script, 66 reads from the event stream the supervisor now owns.

The clearest lesson still ahead is reload. Right now, 66 reload throws a signal at the service from the side: SIGHUP, and hope. The supervisor delivers the signal but takes no part in what follows; it does not know a reload is under way, cannot give it a timeout, cannot tell you whether it worked. A real reload should be a first-class transition, the way start and stop already are: its own step in the frontend, its own state while it runs, its own readiness when it finishes. A [Reload] section that says this is how I reload, wait for me, and know when I am back.

That cannot be bolted on from the outside. It is a transition the supervisor itself has to understand and carry, and s6-supervise's answer to reload will only ever be "which signal?". It becomes possible only because the code of 66-supervise is now 66's to change. The event system forced that ownership, and a supervisor you own is one you can keep teaching.

66 did not leave s6 to be different. It left because the event system could not be built correctly on a supervisor it did not own. That a service can now declare what it reacts to is what that ownership bought.

Thanks for reading, and for your continued support of Obarun.

Eric Vidal: [email protected]