crumbles.blog

HOWTO: Unlock LUKS encrypted disks over SSH on a Raspberry Pi 4 running NixOS

2026-03-13T12:01:29+0100

Follow the instructions on the NixOS wiki. For a Raspberry Pi 4 connected over Ethernet, you need:

boot.initrd.availableKernelModules = [
  "xhci_pci"
  "usbhid"
  "uas"
  "pcie-brcmstb"
  "reset-raspberrypi"
  "genet"
  "broadcom"
  "bcm_phy_lib"
];

Also note: use cipher xchacha20,aes-adiantum-plain64 on Raspberry Pi 4 due to the lack of AES hardware instructions. The default aes-xts-plain64 is slow without these instructions; xchacha20,aes-adiantum-plain64 is over twice as fast. (Raspberry Pi 5 has AES instructions, but doesn’t support NixOS very well yet.) If you forget to set the cipher when creating the encrypted device, cryptsetup reencrypt can help, but it may take multiple days once you have any real amount of data on the disk at all.

HOWTO: Set up an IPsec VPN on NixOS and connect to it with Mac OS and iOS

2026-02-07T16:45:13Z

There are two options for IPsec VPNs on NixOS: Libreswan and Strongswan. Since Strongswan has much better NixOS configuration, we’ll use that.

Note! In the best tradition of howto guides on blogs, I’m not an expert on IPsec, Strongswan, VPN configuration, nor even really NixOS. However, these are the settings that worked for me, derived mostly from the DigitalOcean guide to setting up Strongswan on Ubuntu and adapted for Nix. Please try every other support forum you can think of before asking me personally for help, because I probably have no idea :-)

Generating the needed certificates, etc.

For its own very special reasons, Strongswan’s command-line tools for generating keys and certificates don’t work without an /etc/strongswan.conf file present, even though they don’t need it. Fortunately, an empty one is okay, so let’s create one:

$ sudo touch /etc/strongswan.conf

Now use the Strongswan command line tools in a Nix shell to generate the necessary credentials. (Use nix-shell -p strongswan if you don’t have flakes and nix-command enabled!)

$ nix shell nixpkgs#strongswan

Create directory to hold our keys and certificates and set some permissions for safety:

$ mkdir -p ~/pki/{cacerts,certs,private}
$ chmod 700 ~/pki
$ pki --gen --type rsa --size 4096 --outform pem > ~/pki/private/ca-key.pem
$ pki --self --ca --lifetime 3650 --in ~/pki/private/ca-key.pem --type rsa --dn "CN=VPN root CA" --outform pem > ~/pki/cacerts/ca-cert.pem
$ pki --gen --type rsa --size 4096 --outform pem > ~/pki/private/server-key.pem

For a private VPN, I suggest connecting only through its IP address rather than messing about with DNS. In the following, replace 12.34.56.78 with the IP of your own server. If you really want to connect through a domain name, you can delete the last --san @12.34.56.78 below and just use the other two with the IP address replaced by your domain name.

$ pki --pub --in ~/pki/private/server-key.pem --type rsa \
    | pki --issue --lifetime 1825 \
          --cacert ~/pki/cacerts/ca-cert.pem \
          --cakey ~/pki/private/ca-key.pem \
          --dn "CN=12.34.56.78" --san 12.34.56.78 --san @12.34.56.78 \
          --flag serverAuth --flag ikeIntermediate --outform pem \
        > ~/pki/certs/server-cert.pem

Now you can exit the Nix shell and copy these new keys and certificates to the right place:

$ sudo cp -r ~/pki/* /etc/ipsec.d/

Finally, delete the temporary strongswan.conf file we created; NixOS will manage all further Strongswan configuration.

$ sudo rm /etc/strongswan.conf

Configuring Strongswan in `configuration.nix`

services.strongswan = {
  enable = true;
  # Where your user authentication information will be stored:
  secrets = [ "/etc/ipsec.d/ipsec.secrets" ];

  setup = {
    # Log daemon statuses
    charondebug = "ike 1, knl 1, cfg 0";
    # Allow multiple connections
    uniqueids = "no";
  };

  connections = {
    # You can change the name of the connection from `vpn` if you
    # like; this is only used internally
    vpn = {
      auto = "add";
      compress = "no";
      type = "tunnel";
      keyexchange = "ikev2";
      fragmentation = "yes";
      forceencaps = "yes";
      # Detect and clear any hung connections
      dpdaction = "clear";
      dpddelay = "300s";
      send_cert = "always";
      rekey = "no";
      # Accept connections on any local network interface
      left = "%any";
      # Set this to your domain name (prefixed with @) or your IP address
      leftid = "12.34.56.78";
      leftcert = "server-cert.pem";
      leftsendcert = "always";
      # Tell clients to use this VPN connection for connections to
      # all other IP addresses
      leftsubnet = "0.0.0.0/0";
      # Accept connection from any remote client
      right = "%any";
      # Accept connection from any remote client ID
      rightid = "%any";
      # Authentication method `eap-mschap-v2` works on Mac OS, iOS,
      # and allegedly on Android and Windows too
      rightauth = "eap-mschapv2";
      # Give clients local IP addresses in the 10.0.0.0/1 subnet
      rightsourceip = "10.0.0.0/24";
      # Set this to your preferred DNS server
      rightdns = "1.1.1.1";
      # Clients do not need to send certificates
      rightsendcert = "never";
      # Ask clients for identification when they connect
      eap_identity="%identity";
      # Recommended ciphersuite settings for iOS and Mac; you may need
      # different ones on other platforms
      esp = "aes256-sha256-modp2048";
      ike = "aes256-sha256-modp2048-modpnone";
    };
  };
};

We also need to configure the kernel to allow IP forwarding and do some related hardening by setting the appropriate sysctls:

boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
boot.kernel.sysctl."net.ipv6.all.forwarding" = 1;
boot.kernel.sysctl."net.ipv4.ip_no_pmtu_disc" = 1;
boot.kernel.sysctl."net.ipv4.conf.all.accept_redirects" = 0;
boot.kernel.sysctl."net.ipv4.conf.all.send_redirects" = 0;
boot.kernel.sysctl."net.ipv6.conf.all.accept_redirects" = 0;
boot.kernel.sysctl."net.ipv6.conf.all.send_redirects" = 0;

Finally, we need to configure the NixOS firewall to allow connections on the IPsec ports, and also to route connections through the VPN properly. (Thanks to Erik Dombi on the Strongswan issue tracker for the information on how to set this up.)

You will need to know the name of your network interface. If you don’t use a declarative, static configuration of your IP address (which for a VPN server you probably should, unless you are using Dynamic DNS or something) you may not know it. Find it with ip route; the network interface name is the word that appears after dev. (In my case, it says default via 98.76.54.32 dev ens3 proto static, so my interface is ens3.) Here I’m using ens3. Replace ens3 everywhere in the extraCommands configuration with the name of your own interface if it’s different for you.

# UDP ports 500 and 4500 are used for IPsec connections
networking.firewall.allowedUDPPorts = [ 500 4500 ];
networking.firewall.extraCommands =
  ''
    iptables -P INPUT ACCEPT
    iptables -P FORWARD ACCEPT
    iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    iptables -A FORWARD --match policy --pol ipsec --dir in  --proto esp -s 10.0.0.0/24 -j ACCEPT
    iptables -A FORWARD --match policy --pol ipsec --dir out --proto esp -d 10.0.0.0/24 -j ACCEPT
    iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ens3 -m policy --pol ipsec --dir out -j ACCEPT
    iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ens3 -j MASQUERADE
    iptables -t mangle -A FORWARD --match policy --pol ipsec --dir in -s 10.0.0.0/24 -o ens3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360
    iptables -A INPUT -j DROP
    iptables -A FORWARD -j DROP
  '';

Setting up users

Above in configuration.nix we told the Strongswan NixOS module that our secrets will come from the file /etc/ipsec.d/ipsec.secrets, so we need to create it, tell it where to find our server private key, and add some users:

: RSA "server-key.pem"
username : EAP "password"

Replace username and password by your chosen credentials. You can add more lines of this type if you want more users.

Run it!

nixos-rebuild will start the Strongswan service, reconfigure the firewall, and you should have a working VPN.

For some reason after a system reboot, the VPN seems to take a minute or two to start; nixos-rebuild manages to bring it up immediately.

Connecting on Mac OS and iOS

The advantage of an IPsec VPN is that it’s supported natively by Mac OS and iOS. The process is pretty similar on both:

Download the ca-cert.pem file from /etc/ipsec.d/cacerts
On Mac, open it in Keychain Access and add it to the System keychain. You then need to open the System keychain, right click on the certificate and choose ‘Get Info’, unfold the ‘Trust’ disclosure triangle, and select at least ‘Always Trust’ for IPsec.

On an iOS device, after downloading the file, you will be offered to install the new ‘profile’ by going to the Settings app. Once you install it, you still need to mark the certificate as trusted. Go to General → About → Certificate Trust Settings (right at the bottom) and enable full trust for that root certificate. (There is no fine-grained trust setting on iOS, at least for manual configuration, as far as I know.)
Add a VPN in the VPN settings pane, choosing IKEv2 as the type. Set both the ‘Server address’ and ‘Remote ID’ to the IP address or domain name of your VPN server. Under ‘Authentication’, select ‘User authentication’ method ‘Username’ and enter the username and password you put in the /etc/ipsec.d/ipsec.secrets file.
Create the VPN and turn it on. Hopefully it will work!

Improvements I want to make in a future version of this guide

Use Agenix to store secrets instead of just dumping them in /etc
Understand more about what those iptables incantations do and maybe pare them down

Make the font bigger

2025-12-11T10:16:55Z

Am I the only one who often gets obsessed with stuff they discovered in really weird ways? I watch a film and some character listens to some music on their car stereo; I look it up and have a new favourite band for the next few months. That kind of thing.

Along those lines, quite a while ago I made an unfunny remark on Mastodon about the tendency of some widely-respected computer scientists to use a certain font in their presentations; just recently, prompted by referring to that post for the nth time – so much for jokes getting funnier by repetition – I decided to look into why they do this, even though that particular font is widely disliked. The answer turned out to be pretty much that they don’t care, and they wish people would focus on the content instead of the font. Fair enough.

Except someone also mentioned that that particular font might be good to use because it’s allegedly easier to read for dyslexic people.

Ah. Hmm. Hrrmph. Well, maybe I need to do something about this in future as well, then!

Cue a whole morning spent overthinking this!

The first thing to do was to find out whether that claim was really true of that font in particular. Answer: unsurprisingly, initial research pointed to ‘no’; some people say it helps, but just as for non-dyslexic people, opinions vary and you can’t generalize over ‘dyslexic people’ as a class here. For one thing, there are very different types of dyslexia. The chap who wrote that Reddit post says it only reminded him of the patronizing treatment he got as a dyslexic kid at school; again, that’s fair enough.¹

There are a couple of fonts – Dyslexie and OpenDyslexic – which claim to be designed to be helpful for dyslexic people. The idea of both seems to be the same: somehow making the stroke weight heavier at the bottom of the font maybe gives the letters more ‘gravity’, as if that would help them stay in place better?

Type designer David Jones happened upon my posts fretting about this on Mastodon and was able to point me to a handy introduction to the topic of legibility for dyslexic folk by big-shot typographer Chuck Bigelow. Bigelow looks at the actual scientific research: on this kind of bottom-heavy font in particular, it’s a wash.

Sidebar! Personal hypothesis about why these fonts might feel like they work for some dyslexic people

Tour of a pattern matcher: decision trees

2025-11-28T14:11:49Z

I’ve teased it long enough:¹ today we’re going to look at the real meat of turning declarative, something-that-looks-like-this patterns into fast, running, imperative, if-this-then-that code. The absolute final generation of working Scheme code will have to wait until our thrilling conclusion in part 4, but by the end of this episode you’ll probably at least have an idea of what that will look like.

Today we’re in the (extensible-match decision-tree) library, which takes patacts (the AST representation of a pattern match clause introduced last time) and produces an optimized decision tree. It starts with a foreboding comment:

;; This is some of the hardest code I have ever had to write.

And, indeed, not since I was a tweenaged baby hacker struggling through an online tutorial on how to implement A* pathfinding for my game² have I had so many false starts and wrong directions in implementing a known algorithm. But these were problems with working out how to translate the idea into code – the idea itself is pretty intuitive!

Motivating example

To see how we benefit from an optimizing decision tree generator, let’s look at an example not all too dissimilar from the last example we saw in the previous part.³

(define (uniq ls same?) ; remove adjacent equivalent elements from list
  (match ls
    ('()
     '())
    ((cons _ '())
     ls)
    ((cons* y x ls*)
     (if (same? y x)
         (uniq (cons x ls*) same?)
         (cons y (uniq (cons x ls*) same?))))))

For this input, a typical Lisp hacker’s first attempt at a pattern matcher implementation usually generates code that looks basically like this:

(define (uniq ls same?)
  (let ((subject ls))
    (define (clause-1)
      (if (equal? subject '())
          '()
          (clause-2)))
    (define (clause-2)
      (if (pair? subject)
          (let ((g0 (car subject)) (g1 (cdr subject)))
            (if (equal? g1 '())
                ls
                (clause-3)))
          (clause-3)))
    (define (clause-3)
      (if (pair? subject)
          (let ((g2 (car subject))
                (g3 (cdr subject)))
            (if (pair? g3)
                (let ((g4 (car g3)) (g5 (cdr g3)))
                  (let ((y g2) (x g4) (ls* g5))
                    (if (same? y x)
                        (uniq (cons x ls*) same?)
                        (cons y (uniq (cons x ls*) same?)))))
                (no-matching-clause)))
          (no-matching-clause)))
    (define (no-matching-clause)
      (assertion-violation 'match "no clause matches" subject))
    (clause-1)))

The idea is simple: compile each clause to a procedure, and tail-call the next procedure’s clause as a ‘failure continuation’ at any point where you notice that the pattern doesn’t match.

This is a completely reasonable implementation for very many, nay, probably even a majority of use cases. It’s how everyone learning to write a pattern matcher starts out, and indeed probably how every individual pattern match compiler starts out (including extensible-match!) The syntax-case pattern matcher implemented by psyntax basically works like this, because the absolute fastest matching time isn’t a priority for something that runs at expand time. Alex Shinn’s pattern matcher implementation works like this – anything more complicated would probably be hell to write in pure syntax-rules, and the sheer portability Shinn managed to achieve by writing only in syntax-rules has let his pattern matcher take over the world, inefficiencies and the occasional fudging on semantics be damned. (By contrast, SRFI 262’s dependence on identifier properties will likely hold it back from any equivalent level of world dominance for some time yet, until more Scheme implementers start to catch on to how nifty they are.)

Some of the inefficiencies in this particular code will be optimized away by the Scheme compiler. I mentioned in the first part of this series that redundant let bindings are easy to get rid of. Unfortunately, more consequential inefficiencies are harder to deal with. Most significantly, even after optimization, this code will call pair? and cdr twice every time it goes through the loop: once for the second clause, then again in the third clause. While Scheme compilers are usually clever enough to eliminate repeated type checks when they occur within a single nested tree of code, putting each check in its own little procedure messes this compiler optimization up. The exception is Guile, because Andy Wingo developed a very clever compiler pass which can recognize common prefixes across these internal procedure boundaries. Also, very simple uses – like the last example from last episode – can have the clauses inlined into one another and thus end up in a nice nested tree for the compiler to work with; but as a data point, clause-3 above is already complex enough (or complex enough and referenced from enough places) that Chez’s cp0 doesn’t bother.

This repeated calling and branching is theoretically unsatisfactory, but the real philosophical issue in these inefficiencies – the thing that really motivated me to want to do better – is specifically that this generated code is nothing like what a programmer would write in the absence of pattern matching. Nobody would ever write code which had those repeated pair? calls. The real point I want to make by spending many lines of code on trying to do something better is that pattern matching is a good idiom, and programmers should use it. To that end, pattern matching should be a usable tool even in the hottest of hot loops, even on the slowest of microcontrollers: nobody should ever feel like they have to switch back to writing out if expressions by hand for performance’s sake. The fact that pattern matching is a declarative idiom, and declarative programming has something of a reputation for being inefficient, doesn’t exactly help its reputation here.

There are mountains of research into efficient compilation of pattern matching to avoid generating silly code like this. In the usual framing, there are two approaches: the backtracking automaton and the decision tree. The backtracking automaton, as its name implies, will still sometimes run the same tests on the same subject more than once, but tries to do so less than the naïve approach shown above which re-examines everything from scratch in every clause. The decision tree, on the other hand, will never re-examine any of its inputs, but may have much larger (exponentially larger!) code size than the automaton.

In fact these two approaches aren’t so different in practice – once you run the right time-saving optimizations for generating an automaton, or the right space-saving optimizations for generating a decision tree, the results are pretty close to one another. They’ll be nearly identical on very, very many use cases; it’s in the comparatively rare cases where they fall down that they fall down in different ways.

In the Scheme and Scheme-adjacent world, Racket’s pattern matcher by Sam Tobin-Hochstadt uses an optimized backtracking automaton after the technique by Le Fessant and Maranget (2001);⁴ extensible-match uses a decision tree after the technique by Maranget (2008). Yep, same guy! Luc Maranget is actually far from the only compiler researcher to have written papers on optimizing both techniques; but he is the only one whose papers now seem to be regarded as the somewhat definitive introduction to the state of the art for each of the two techniques.

Generating a decision tree

Maranget’s 2008 paper is heavy on notation, but mostly explains it as it goes along. It’s pretty accessible as European academia goes: if you’re good at reading and understanding other people’s code, you should be able to understand Maranget’s paper. It’s worth a read (or take a look at a shorter explanation by Colin James or David Nolen) – I’ll explain it here as it’s implemented in extensible-match, noting some (but not all!) differences to Maranget’s original presentation.

So, our input is a list of patterns and their associated actions, one for each clause. In fact, our input is always a list of rows of patterns – remember, we’re in %core-match-lambda which matches multiple patterns against multiple values, using the row-pattern AST type to group them together. Each AST pattern (and subpattern, but we’ll get to that) represents something we can do to get closer to picking a clause which matches the subject – meaning that with the row-pattern, we have a choice of multiple potential patterns to choose from. The ?-pattern, for example, represents that we could test its subject against a predicate; the var-pattern represents renaming an internal subject variable to a pattern variable visibly bound in the clause.

Let’s say we pick one pattern from the first row, since that’s the one we have to test first. (How we pick one is a topic we’ll get to later.) We’re going to act on this pattern – let’s say a ?-pattern testing pair? against our input x. What do we need to do?

What we ultimately want to do is generate (if (pair? x) ); this gives us a hint that we also need to work out how to fill in those two slots. Well, what goes in those two slots?

The clause is easier to understand first, because if (pair? x) is false, the whole row can’t match any more: a row is an and combination of patterns, and one of the anded subpatterns just threw up false. So we can throw out that entire row and fill in with the code generated by repeating this process on that smaller list of rows with their actions.

But wait! What if the next row also contains a ?-pattern testing pair? on x – there’s no point testing that again, because it will be false a second time as well! So apart from throwing away the first row, we also walk through the remaining rows and throw away any of them which also depend on the same test before we repeat this process to generate the .

The is generated by a similar walk over the rows and recursion on the result. We don’t throw away the first row because in this case, that part of it did match. Instead, we remove it from the row and from any subsequent rows which make exactly the same test, preventing generating the same test again in the case where we know it succeeded as well as the case where we know that it failed.

That’s basically the core of Maranget’s algorithm. Maranget calls generating the new rows for specializing the patterns, and generating the new rows for defaulting the patterns – both with respect to a particular ‘constructor’, but because Scheme is different (more below), extensible-match calls these specializers. There are two base cases of the recursion: one when there are no rows left (no patterns matched; generate the code to signal a matching failure) and one when the top row only consists of irrefutable patterns (ones like _ wildcard patterns and var-patterns, which don’t have failure cases: in this case that row matched; generate the code to activate the corresponding action).

However, there’s one type of pattern in our AST which doesn’t have a failure case, but hides more subpatterns which do: the apply-pattern which calls some procedure on its subject variable and creates more subject variable(s) for the value(s) returned. Those subject variables are then available to be the subject of a subpattern of the apply-pattern.

For his equivalent of an apply-pattern Maranget makes a whole separate set of patterns when specializing, just containing the subpatterns. This probably works better for his compiler of ML patterns than it did when I tried in my compiler of Scheme patterns: for one thing, testing type (our ?-pattern) and deconstructing into values for subpatterns (our apply-pattern) are the same thing in his implementation; since ML is statically typed and pattern matching is the primitive means of deconstruction, the compiler has a lot more knowledge about the potential range of input values and what is and isn’t possible after each deconstruction step than extensible-match can get as a Scheme macro. In particular, Maranget’s patterns after each deconstruction step always form a matrix with columns for each value and rows for each clause. That’s not guaranteed by the structure of this AST (but explains why the term ‘row’ is used in extensible-match).

So instead, when an apply-pattern has been chosen as the specializer, each specialized row simply has the subpattern of that apply-pattern spliced into it. This means that each row-pattern can have a different length, depending on which apply-patterns have been specialized and had their subpatterns spliced in!

As one last point, in order to simplify the job of the procedures which pick a specializer and specialize and default the rows on it, after this splicing takes place (or more accurately before each recursive step), any rows nested within the rows are flattened out. If there were any or patterns, they’re also expanded into multiple copies of the entire patact, each with one of the or clauses, so the decision tree generator proper doesn’t need to look for specializers specially inside of or patterns either.

Representing a decision tree

For reasons that will become more apparent below and in the next episode, a decision tree is actually another kind of intermediate representation: we don’t represent it directly as Scheme code. Here’s its definition, slightly simplified:

(define-record-type dt-node
  (fields success-branch failure-branch))
(define-record-type dt-test
  (fields proc var)
  (parent dt-node))
(define-record-type dt-apply
  (fields proc var vars)
  (parent dt-node))
(define-record-type dt-equal
  (fields val var)
  (parent dt-node))
(define-record-type dt-rename
  (fields internal external)
  (parent dt-node))

Immediately it’s clear how much simpler this is than the AST we got as input: only four concrete node types vs nine pattern types in the AST.⁵ The action type from the AST also appears in decision trees, as the leaf nodes, where a matching clause has been chosen or the match failed.

?-patterns, when chosen, become dt-test nodes; apply-patterns become dt-apply nodes; quote-patterns become dt-equal nodes; var-patterns become dt-rename nodes. and-patterns, or-patterns, row-patterns, and not-patterns are reflected in the structure of the tree rather than as nodes themselves. wildcard-patterns are no-ops, as far as the pattern match compiler is concerned, and don’t show up in the tree at all.

Step by step

Here’s a worked example. Let’s take the last example from the previous episode:

(define (last ls) ; returns the last element in a list
  (match ls
    ((cons x '())  x)
    ((cons _ ls*)  (last ls*))))

Our AST for this pattern match expression looks like this (in an entirely hypothetical notation):

((patact (row (and (? #:subject ls #:predicate pair?)
                   (row (apply #:subject    ls
                               #:procedure  car
                               #:vars       (ls.car)
                               #:subpattern (row (var #:subject ls.car
                                                      #:name x)))
                        (apply #:subject    ls
                               #:procedure  cdr
                               #:vars       (ls.cdr)
                               #:subpattern (row (quote #:subject ls.cdr
                                                        #:datum ()))))))
         (action return x))
 (patact (row (and (? #:subject ls #:predicate pair?)
                   (row (apply #:subject    ls
                               #:procedure  car
                               #:vars       (ls.car)
                               #:subpattern (row (wildcard #:subject ls.car)))
                        (apply #:subject    ls
                               #:procedure  cdr
                               #:vars       (ls.cdr)
                               #:subpattern (row (var #:subject ls.cdr
                                                      #:name ls*))))))
         (action do-last ls*)))

We look at the first row, which only has one subpattern and thus only one potential specializer to pick: (? #:subject ls #:predicate pair?). So we generate a dt-test node which checks if ls is a pair.

In the success branch of this dt-test node, we put the result of re-running this algorithm on all of the patacts with this specializer applied. We don’t need the pair? any more in either row, so it will disappear and be replaced by the right-hand side of the and pattern which contains it. This in turn is a row in both cases; we splice it into the main row of each patact. Our specialized patacts now look like this:

((patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (var #:subject ls.car
                                            #:name x)))
              (apply #:subject    ls
                     #:procedure  cdr
                     #:vars       (ls.cdr)
                     #:subpattern (row (quote #:subject ls.cdr
                                              #:datum ()))))
         (action return x))
 (patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (wildcard #:subject ls.car)))
              (apply #:subject    ls
                     #:procedure  cdr
                     #:vars       (ls.cdr)
                     #:subpattern (row (var #:subject ls.cdr
                                            #:name ls*))))
         (action do-last ls*)))

As for the failure branch of dt-test: we know that the first patact can’t match any more just because this single test failed. We have to default on the second patact only – and that also required (pair? ls) to be true. So our defaulted patacts in this case are just the empty list, and the recursion on that will generate the leaf node to signal an exception because no clause matched that input.

Back on the success branch, we still have these two patacts, but this time the first row gives us a choice of what to do. We could pick the leftmost one – the car – but actually the car isn’t very interesting because it’s not refutable. The cdr, on the other hand, can fail, so let’s prioritize it. We generate a dt-apply binding the cdr of ls to ls.cdr; it only needs a success branch since we assume this can’t fail. So we specialize again, and the specialized rows, after splicing, look like this:

((patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (var #:subject ls.car
                                            #:name x)))
              (quote #:subject ls.cdr
                     #:datum ()))
         (action return x))
 (patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (wildcard #:subject ls.car)))
              (var #:subject ls.cdr
                   #:name ls*))
         (action do-last ls*)))

Still two patterns in the top row, and again, the quote pattern looking at ls.cdr is still considerably more interesting than the apply pattern, so we pick that as the specializer again. We generate a dt-equal node checking if ls.cdr is the empty list. This one does need success and failure branches.

On the success branch, the specialized patterns look like this:

((patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (var #:subject ls.car
                                            #:name x))))
         (action return x))
 (patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (wildcard #:subject ls.car)))
              (var #:subject ls.cdr
                   #:name ls*))
         (action do-last ls*)))

We have completely eliminated the cdr patterns from the top row! Only the irrefutable car pattern is left, and when the first row is irrefutable, the pattern has matched – so we generate the dt-apply node and its dt-rename child and have it point to the action which returns x. Done!

On the failure branch of the check for a null ls.cdr, we’ve thrown away the top row again and we end up with this single patact:

((patact (row (apply #:subject    ls
                     #:procedure  car
                     #:vars       (ls.car)
                     #:subpattern (row (wildcard #:subject ls.car)))
              (var #:subject ls.cdr
                   #:name ls*))
         (action do-last ls*)))

Once again, this is irrefutable, so we again generate the dt-apply node for ls.car, the dt-rename node for the ls* variable, and then finish at the action.

Picking a pattern

Okay so, at each step of the pattern matching process, we pick a specializer from the first row’s patterns, generate a node from it, and specialize/default the remainder of the patterns on it. But how do we pick which pattern to choose from the row?

We have to work out which pattern is going to help us find answers the fastest, probably by comparing against the patterns in different rows somehow. We need a heuristic – a scoring function to guess which pattern is most likely to help find the matching clause in the fewest nodes possible.

Maranget considers the question of which heuristics to use, assigning letters to each of a number of potential scoring functions and measuring the impact of many potential combinations of them. In Maranget’s terms, extensible-match uses heuristic fnr: the specializer picked has to be in the first row;⁶ the specializer is the one which will specialize the most rows (most needed); if there’s a tie for the most needed specializer, we pick the pattern with the smallest combined size of the specialized and defaulted rows. (If even that goes to a tie, just pick the leftmost tied specializer.) I didn’t benchmark this particularly: n seemed like an intuitively sensible heuristic; based the suggestion in Maranget’s paper, I later added r as an additional tie-breaker.

Some of the heuristics Maranget considers aren’t really applicable to Scheme. His ML version can generate nodes with more outward branches than just success and failure, because the ML compiler can generate a single switch over many types at once; Scheme detects different types by calling individual predicate procedures for each individual type, so that’s not possible, and neither is the heuristic b which picks the smallest switch to build, for example – so extensible-match also can’t use the combination of heuristics Maranget personally recommends, pba.

So why was it so hard/why didn’t you do it slightly differently

As always, the hard thing was not the actual writing of the code, but working out what the right code to write was.

For example, you might ask why I wasn’t tempted to skip the step of representing decision trees as their own, reified type and simply recurse by generating new match expressions nested within the branches of an if or within the body of a let for the specializer we’ve chosen, for example.

This is, indeed, very tempting. If you’re implementing a compiler based on decision trees, you might even find some literature suggesting that you could do exactly this. Surely, you’ll think, just like those Swap nodes in Maranget’s paper aren’t actually part of how it’s really compiled – surely you can avoid generating the other nodes per se too, and just go straight to generating the code which those nodes will eventually be turned into?

So you try it, and it doesn’t work, and you regret it, and you commit it to the repository anyway as a warning to your future self and to others. Actually there were other things that were wrong with that first attempt – trying to insist that patterns form a matrix-like grid turned out to be the wrong thing for this kind of pattern input AST. My idea was that I really wanted to be able to property test the decision tree generator, ensuring that for any generated patterns and terms it always had the same result as the naïve style of generation shown above. I was thinking of decision tree generation as an optional optimization pass, rather than a fundamental transformation pass. This is the wrong mental model to have.

Anyway, the real reason that would have been the wrong approach even if it had worked is that it wouldn’t have allowed any optimization for code size or compilation speed. Wait whaaaaaat??

Optimizing space and time

Yeah, remember how I mentioned about the potentially exponential code size if you choose decision trees, but only in cases where the tricks to avoid it start to fail? It’s time to talk about those tricks.

The most important trick is sharing of nodes. When two different branches in the tree end up doing the same things, they should actually re-converge and become the same branch again. This makes the decision tree actually a decision dag (directed acyclic graph).

To achieve this sharing of nodes between different code paths, extensible-match uses two tricks. The most important one is hash consing, a trick more generally applicable in functional programming. When you have immutable structures which might be created many times over with the same contents, you can maybe save memory by storing all instances of structures of that type in a hash table. When someone asks to create a new instance of the type, first check if there’s already one existing which has the contents they’re interested in. If so, just return that existing one instead of creating a new one. (But if not, be sure to put the newly-created object into the table before returning it, in case someone asks for another one of the same in future.)

Neither R6RS’s record system nor its hash tables were really meant to support doing this, and the (extensible-match decision-tree) library has to do some pretty undignified nonsense in order to make it work. But it does work, and saves both time and space: space obviously, but also time because it stresses the garbage collector less to have fewer objects sitting around.

There’s a second layer of caching which is similar, one I mentioned I was considering using in an earlier post: the recursive procedure which generates tree nodes from a (specialized or defaulted) set of rows and actions is memoized. This has a similar effect to hash consing but, since hash consing already guarantees sharing, it optimizes compilation time and not decision tree size. Memoization avoids repeating an exponential recursion entirely in many cases. I measured how many cache hits the memo table got for some complex patterns, and how much it sped things up, and it was totally worth it. The 14 cases over three list variables which used to take 9.3 seconds to compile on Guile now comes back almost immediately.

Visualization

What’s the most common target language for compilers? x86? The JVM? C? JavaScript? If I were a betting lass, I might be tempted to put money on the answer being DOT, the GraphViz file format. If you want to understand what your compiler is doing, the easiest way to see it is often to actually have it draw your control-flow graph, automaton, or decision dag in two dimensions. Outsourcing the actual drawing task to GraphViz is a nice sweet spot in the trade-off between getting it working and making it nice.⁷

Initially I tried to see what was going on with a very silly (non-hygienic) code generator outputting Scheme code, but this quickly got tiring to read, especially because it couldn’t show sharing in the dag. (How the real code generator represents shared nodes is a topic for next time.) With a 2D graph, it’s relatively easy to get a rough impression on first glance of how well the decision tree generator handles certain cases; by following the arrows, you can fairly easily check for anything it’s doing wrong.

I’ve kept an archive of older DOT files showing how the pattern match compiler has got more capable over time, going back to some of the first decision trees this code ever generated, before it was even hooked up to a code generator. (This DOT file is dated 2 December 2024, and I didn’t get the code generator working until Candlemas.) This lets me see how certain changes improved or disimproved the generated trees over time.

Originally, to review these DOT files, I had to manually copy the generated DOT source into a file and invoke the dot command line tool. But recently I thought, hey, I have a terminal emulator which can display inline graphics, so why not just show them directly in the REPL? So I wrote a little library to do just that.

Source code of the library which plops a drawing of a graph straight into my terminal

Tour of a pattern matcher: core patterns and the AST

2025-11-15T13:35:12Z

Alright! Last time we looked at the mouth of extensible-match and how it chews up patterns in order to make them digestible. Before we take a look into the stomach,¹ we should look at what that chewed-up form of patterns looks like. This is an episode with a bit less how and a lot more what than the last one. I’ll explain and justify the core pattern structure we just got out of the expansion process, how and why it’s different from the AST that we eventually like to work on, but also one further transformation on that AST that makes it easier for the rest of the system to digest.

We’re still in the (extensible-match match) library, which we’ll keep dipping back up into because it contains this long chain of procedure calls which actually co-ordinates the whole of the rest of the process of compilation.

Subject variables vs pattern variables

The idea of a pattern variable is simple, and is a concept exposed to the user. In a pattern like (cons x '()), x is a pattern variable. This pattern deconstructs a pair which could previously have been constructed by the same code used as an expression: the expression form takes a variable and makes it the only element of a 1-list; the pattern form takes a 1-list and puts its element in a variable. This is the essence of the equational reasoning property which makes pattern matching such a nice style to work with and think in.

Subject variables are pattern variables’ twin from the dark side. In the above example, there’ll be three subject variables: one for the pair itself; one for its car, and one for its cdr. The upshot is that the pattern matcher doesn’t generate code like (if (equal? (cdr p) '()) ...); it will always extract the thing it’s interested in to a variable first: (let ((g1234 (cdr p))) (if (equal? g1234 '()) ...)). This simplifies things because it means that the kind of thing we’re testing against is always a simple variable expression no matter what the nesting level of patterns is.

(Obiter dictum: this doesn’t affect the efficiency of the final generated code; it anticipates the Scheme compiler’s own pass which will un-nest expressions, namely CPS or ANF conversion.)

Subject variables are always generated identifiers; their precise names are only interesting to the internals of the pattern matcher. I mention them here because they’re important for understanding the structure of expanded patterns and the first few stages of compilation we’ll look at today. The first thing that %core-match-lambda does to patterns, in fact, is to generate subject variables for each of the values it will be examining. (Recall that %core-match-lambda has multiple sets of patterns, but each such set has to match the same number of values. The values for all clauses get the same subject variables.) Although these subject variables are the entry-point for the pattern match, this is in fact the last set of new subject variables generated, because all of the others (in the example above, the ones for the car and cdr of the pair) are generated during expansion.

Core patterns

Core patterns are defined in (extensible-match core-pattern).

Behold, a grammar:

⟨core pattern⟩	::=	`(core:wildcard)`
	\|	`(core:var` ⟨identifier⟩`)`
	\|	`(core:quote` ⟨datum⟩`)`
	\|	`(core:and` ⟨core pattern₁⟩ ⟨core pattern₂⟩`)`
	\|	`(core:or` ⟨core pattern₁⟩ ⟨core pattern₂⟩`)`
	\|	`(core:not` ⟨core pattern⟩`)`
	\|	`(core:row` ⟨core pattern⟩ …`)`
	\|	`(core:?` ⟨expression⟩`)`
	\|	`(core:apply` ⟨expression⟩ `(`⟨identifier⟩ …`)` ⟨core pattern⟩`)`
	\|	`(core:subject` ⟨identifier⟩ ⟨core pattern⟩`)`
	\|	`(core:seq` ⟨don’t worry about it⟩`)`

I won’t dwell on the first six rows (core:wildcard, core:var, core:quote, core:and, core:or, core:not) because they’re not very interesting. Core:wildcard is spelled _ in the high-level pattern language, core:var is just written as a variable name; core:and and core:or are now restricted to two subpatterns each. Otherwise, you can pretty much find everything you need to know about these from the documentation of the high-level forms in SRFI 262.

Core:row is where things start to get slightly more interesting. This pattern isn’t directly exposed to high-level forms, but it’s used for the patterns which can come after the procedure expression in a high-level ? pattern and the value patterns of a => pattern. They’re also used for each of the individual values at the top-level of a multiple-pattern %core-match-lambda form.

In a row pattern, all the subpatterns have to match, but the compiler is allowed to test them in any order. While a full rationale will have to wait until the next episode, in general the order in which it makes most sense to test subpatterns is not always the left-to-right order in which they appear in the source code. In the (cons x '()) example, we’re far more interested in the cdr of this pair than we are in the car, because the cdr can tell us whether the pattern matches or not; only once we know that the cdr is the empty list should we bother trying to load the car.

Core:apply corresponds directly to the high-level => pattern (which was called apply in earlier drafts of the SRFI) but has a different structure, to do with the need to give each value its own subject variable. The ⟨expression⟩ gives the procedure which is to be applied to the pattern’s subject; the ⟨identifier⟩s are subject variables to which each respective returned value will be bound; finally, the ⟨core pattern⟩ will be matched in an environment which includes those subject variables.

Core:subject is the last piece of the puzzle; it simply says which subject variable names the subject of its own subpattern.

How this all fits together is probably best illustrated by example. Our (cons x '()) example expands, within the high-level pattern matching language, to this:

(? pair?
   (=> car x)
   (=> cdr '()))

which in core patterns expands to this:

(core:and (core:? pair?)
          (core:row
           (core:apply car (g1) (core:row (core:subject g1 (core:var x))))
           (core:apply cdr (g2) (core:row (core:subject g2 (core:quote ()))))))

A single-subpattern core:row isn’t very useful, but an equivalent expansion using SRFI 1’s car+cdr² would be:

(? pair?
   (=> car+cdr x '()))

which in core patterns is:

(core:and (core:? pair?)
          (core:row
           (core:apply car+cdr (g1 g2)
            (core:row
             (core:subject g1 (core:var x))
             (core:subject g2 (core:quote ()))))))

Something to note here is what core:var actually, formally does: it takes its subject variable and turns it into a pattern variable. g1 is a hidden, generated variable; x is bound within its clause to the exact same value.

That’s more or less it for the core pattern language; I hope it’s more-or-less clear how it works. I’ve omitted core:seq from this description. If this series ever covers core:seq and the implementation of the pattern matcher’s functionality for running simple regular expressions over sequences of Scheme values, it will be in a bonus episode, possibly quite a bit later. I’m leaving it out here because the implementation is still an unhappy mess, much more so than any other part; also, Wolfgang Corcoran-Mathe has pointed out that the current API mixes idioms in a confusing way, and fixing that might involve some other deep changes to how it’s implemented.

The Abstract Syntax Tree (AST)

As covered in a previous missive, core patterns are pseudo-record syntax objects; accessing fields from such syntax objects is slow; real records are much faster; so to make decision tree generation go faster, there’s an additional layer, called the abstract syntax tree, implemented as real Scheme records. This is in (extensible-match ast), along with the code to convert core patterns to the AST. (That’s invoked by the long chain of procedure calls that drives the whole compilation process in %core-match-lambda, but defined here.)

Actually, though, there’s (at least) one more reason besides speed that the AST is better to work with going forward – see if you can spot it in the definition:

(define-record-type primitive-pattern)
(define-record-type pattern
  (fields subject)
  (parent primitive-pattern))
(define-record-type wildcard-pattern
  (parent pattern))
(define-record-type var-pattern
  (fields name)
  (parent pattern))
(define-record-type quote-pattern
  (fields datum)
  (parent pattern))
(define-record-type and-pattern
  (fields subpat_1 subpat_2)
  (parent primitive-pattern))
(define-record-type or-pattern
  (fields subpat_1 subpat_2)
  (parent primitive-pattern))
(define-record-type row-pattern
  (fields subpats)
  (parent primitive-pattern))
(define-record-type not-pattern
  (fields subpat)
  (parent primitive-pattern))
(define-record-type ?-pattern
  (fields predicate-id predicate-expr)
  (parent pattern))
(define-record-type apply-pattern
  (fields procedure-id procedure-expr vars subpat)
  (parent pattern))

I’ve omitted the AST equivalent of core:seq for simplicity.

There are a couple of improvements here. I was so determined to kill free-identifier=? from the decision tree generator entirely that the expressions naming the procedures for ? and =>/apply patterns are now interned, giving them each a unique integer for the decision tree generator to use as a proxy for whether they are the same identifier. (This was probably a premature optimization, in hindsight.)

But the big improvement is in the (not very well-named) distinction between a pattern and a primitive-pattern. Notice that core:subject is gone from the AST! In its place, every pattern that needs to know the name of its subject variable has it stored directly within it.

In the core pattern language, you don’t actually know what (core:quote ()) means without looking for a lexically enclosing core:subject: in our (cons x '()) example, (core:quote ()) alone could be looking at the pair (no match), the car (not intended, but might incorrectly match), or the cdr (as intended).

Making this change made the decision tree generator much simpler, apart from the speed boost from using records instead of syntax objects.³ It no longer has to track a lexically implicit subject for each pattern.

You might wonder why the code doesn’t do this already during pattern expansion, and why core patterns don’t include their subjects in the same way the AST does. Well, maybe ideally it would do this, but in the sense of having everything in its right place at the right time, it’s probably the right thing for simplicity. Doing it this way lets the expansion of patterns happen completely independently of any context; we add that context to each individual subpattern later.

Tracking just the state of each expansion in continuation-passing style is tricky enough; adding the current lexical subject variable as extra state to pass around would make it more complicated. Moreover, continuation-passing expansions imply that the expander’s final result has to be a syntax object anyway – it’s not safe to go straight from unexpanded patterns to AST records – so let’s take the opportunity afforded by the need to have a slightly redundant layer in order to at least make the process of creating that redundant layer easier. That said, if Scheme does one day grow a way to call a transformer and apply marks independently of the expander’s main loop, it will probably then be time to get rid of core patterns and track the current subject variable implicitly while expanding directly into AST records.⁴

The other half of the clause

We’re not quite done with the AST. We’ve talked a lot about the left-hand side of each pattern matching clause – the pattern – and we’ll continue to do so. But there is another side, which is what happens when the pattern of a clause is known to have matched.

(define-record-type action
  (fields procedure args))
(define failure-action (make-action #'fail '()))

(define-record-type patact
  (fields pattern action))

There’s not too much to say here. When a top-level pattern matches, its corresponding action is called. It’s a procedure, named by a generated identifier for each clause, plus a list of the variables bound by the corresponding pattern (which become arguments of the procedure). If no pattern matches, we call a special action called fail which just raises an exception.

A pattern and an action together make a clause, but here we call it a patact.

Unifying subject variables

That’s all the what; let’s finish up with a quick look at another bit of how. We’ll talk a lot more about what it takes to generate optimal code for a pattern match next time, but there’s one step that takes place to get the patterns in a form that’s easier to optimize.

Recall that each (sub)pattern expansion happens independently, without knowledge of the context from other expansions; and also that the expansion of => patterns into core:apply patterns. What this means is that equivalent applications generate distinct subject variables; we really want equivalent applications to create equivalent subject variables!

To see how, let’s put the example we’ve used so far into context:

(define (last ls) ; returns the last element in a list
  (match ls
    ((cons x '())  x)
    ((cons _ ls*)  (last ls*))))

These two patterns will expand, respectively, into the same core pattern we saw above and another very similar one:

(core:subject ls
 (core:and (core:? pair?)
           (core:row
            (core:apply car (g1) (core:row (core:subject g1 (core:var x))))
            (core:apply cdr (g2) (core:row (core:subject g2 (core:quote ())))))))
;; action: (g3 x)

(core:subject ls
 (core:and (core:? pair?)
           (core:row
            (core:apply car (g4) (core:row (core:subject g4 (core:wildcard))))
            (core:apply cdr (g5) (core:row (core:subject g5 (core:var ls*)))))))
;; action: (g6 ls*)

We applied car and cdr to ls in each pattern, but the result was called g1 and g2 in the first pattern and g4 and g5 in the second one. We only want to call each procedure once – if at all – in the generated code, so it would be nice to only generate one variable. This is what the pattern subject unification pass does: for every application of the same procedure to the same subject, the created subject variables should have the same name. The result is that the first pattern will still look the same, but the second will be (the AST-level equivalent of) this:

(core:subject ls
 (core:and (core:? pair?)
           (core:row
            (core:apply car (g1) (core:row (core:subject g1 (core:wildcard))))
            (core:apply cdr (g2) (core:row (core:subject g2 (core:var ls*)))))))
;; action: (g6 ls*)

It does this in a recursive pass which looks for combinations of subject, procedure, and number of output values⁵ it’s not seen before; when it finds one, it puts the variable names that were generated for that pattern’s version into a hash table. When it sees the same combination of subject, procedure, and co-arity again, it replaces that AST apply pattern with a version whose subject variables are renamed, and recursively traverses the apply pattern’s subpattern to change instances of those subject variables appearing within quote-patterns, var-patterns, other apply-patterns etc. to the unified name.

This is also something that could be done during expansion, but it’s just simpler to do it as an extra pass after expansion because it makes expansion independent of external context.

Potential future AST transformations

Subject variable unification is the currently the only optimization pass that’s done on the AST of the patterns before handing over to the decision tree generator, which is responsible for most of the work of finding an efficient way to match a set of patterns. But there are a couple of other passes the pattern match compiler could make at this point to improve the quality of generated code, or make the decision tree generator simpler.

Rewrite complex not patterns

At the moment, using a not pattern will completely frustrate the optimizer. It’s not too bad if you only put not around a datum or a predicate – but if you put it around an (implicit) and, the optimizer will only know how to compile it and its entire subpattern at once, and be unable to avoid potentially repeating equivalent computations for multiple similar patterns. Being able to mix in patterns which do match when they don’t match anywhere arbitrarily would just add too much complexity to the decision tree generator.

Fortunately, in 1846, compiler engineer Augustus De Morgan formalized an intuition about the relationship between negations, conjunctions, and disjunctions, which could eventually be applied here to ensure that the not pattern never appears outside of an and or or pattern. Ensuring it also never appears outside of row would be ideal, but the obvious solution – turning e.g. (not (row pat1 pat2)) into (or (not pat1) (not pat2)) – may be an accidental pessimization because or has an order-of-evaluation restriction which row doesn’t.

This AST transformation alone would be an improvement for the decision tree generator, but the real improvement would come with recognizing common not operations and being able to fully optimize them. Another benefit of this transformation would be the ability, in theory, to (sometimes) issue compile-time warnings about patterns like (not _) which will never match.

Remove quote patterns from the AST

A (core:quote val) pattern can be rewritten as (core:? (lambda (x) (equal? x 'val))), but I don’t do this during expansion because it makes it harder to recognize that two quote patterns are equivalent. Since conversion from core patterns to the AST includes the recognition of equivalent expressions in order to give them unique integer IDs, it could generate an appropriate lambda expression for each mentioned datum then.

This would be good to do together with the previous one, because adding support for explicitly recognizing not patterns in the decision tree generator would add complexity there; removing quote patterns from the AST would remove some related complexity to balance that out. The cp0-equivalent pass of the Scheme compiler will then get rid of the lambda expression again.

Re-chain and patterns to only be nested on the right-hand side

Another way to frustrate the optimizer at the moment is to write something like (and (and a b) c) instead of (and a (and b c)). The optimizer will look for things it can do in the left-hand side of an and, but if it sees another and it isn’t smart enough to be able to look again at that and’s left-hand side, mostly because reconstructing the leftwardly-nested and after deciding to take action on its contents would be too much of a pain. The ordering relationship for these two nesting structures is the same, so an AST transformation should rewrite them all into a form that’s friendlier for the decision tree generator.

Or patterns, on the other hand, don’t have this problem, because the decision tree generator will flatten those out into entirely separate patterns while it’s working, and that process can handle any nesting structure it finds.

Next time

The decision tree generator! This will probably be the third of a four-part series, before the final part – a short coda on turning decision trees into executable code.

Updated 26 November 2025 to reflect some improvements in the structure of the AST, and also to actually add links to the other libraries mentioned here.

The next part of this series is online here.

I would try to promise that, like Knuth, I will have the decorum to not continue the digestive metaphor beyond the stomach, but I fear I would be promising too much.↩︎
Which is the real procedural codata counterpart to the data cons.↩︎
The old, slow version of the decision tree generator had a pretty awful procedure whose job was to push core:subject patterns down the core pattern tree lazily.↩︎
Sheesh, that one missing expander feature alone would probably let me get rid of about a quarter of the lines of implementation in extensible-match.↩︎
For very sad reasons explained in the SRFI, you can’t use a => pattern to match on the number of values a procedure returns. It’s nonetheless possible to construct valid, non-erroring patterns where the same procedure applied to the same subject would return different numbers of values and match different numbers of patterns. Writing one is an exercise for the reader.↩︎

Tour of a pattern matcher: expression and pattern expansion

2025-11-09T18:18:44Z

I’ve posted a lot on Mastodon (and a little bit here) about the adventures I’ve had implementing an extensible pattern matcher in Scheme. Well, the SRFI isn’t too far off finalization now, and the implementation is unlikely to change dramatically, so I reckoned it might be worthwhile to take readers on a tour of how the implementation works: for one thing, for those who want to be able to dive in and understand the code in future and maybe hack on it; for another, as a further propaganda piece in favour of a style of programming still under-used in Scheme. But also I think there’s a lot the budding Schemer can learn about Scheme, and a lot the budding functional programmer can learn about this central construct of functional languages, from studying the implementation. There’s still a lot of mess in the code that I’m moderately embarassed about – we’ll deal with one such embarassment today, albeit a less immediately avoidable one. But I hope that will get cleaned up over time; like visiting a newly-furnished building where they haven’t yet discovered that putting that cabinet there will make the door handle hit the wall and damage the wallpaper, you can see the idea of the place, and those little things will get sorted eventually.

As mentioned last time I talked about the implementation, extensible-match is a Scheme macro complex enough that it can be seen as a compiler unto itself. There’s a sense in which this is a truism – every Scheme macro is a compiler from a dialect of Scheme containing that macro’s syntactic extension into a dialect that doesn’t use it, at least locally – and attempting to define what makes a macro a compiler beyond that is somewhat impressionistic. But extensible-match has intermediate representations, multiple passes, an optimizer, and a code generator to turn all of that back into Scheme code: it’s definitely on the side of a compiler, taken as a whole. But today, following the path that your patterns actually take when the compiler digests them and turns them into running Scheme code, we’ll just take a look at the front end, which looks more like a traditional Scheme macro definition. Even here, there are a few things worth pointing out along the way!

The match forms

We start our journey in the uncreatively named (extensible-match match) library, which provides all the Scheme expression forms with which pattern match programmers actually use to interact with the pattern matcher. There are a dozen distinct macros for getting access to the pattern matcher – from the very basic match itself (which will probably consistute about 90% of real-world uses) to the somewhat obscure match-letrec*.

Under the hood, though, all of them ultimately expand into a use of match-lambda. Partly this is for party reasons – one thing to name them all, one thing to define them and so on – but also match-lambda is, in fact, the fundamental pattern matching form, in that it offers almost everything that all of the subordinate pattern matching forms need. Specifically, unlike match it handles multiple values without consing them up into a list first; unlike match-let and friends it has multiple clauses which it tries in succession.

Match-lambda in turn is implemented in terms of case-lambda, the Scheme way of making a procedure which takes varying number of arguments by simply dispatching to different code based on the number of them. Case-lambda is built in to Scheme, so the first bit of pattern matching – discerning what to do based on the number of values given – is done for us. All match-lambda does is group the pattern matching clauses (which of course can discern more finely) by the number of values they expect, creates a case-lambda clause for each of those groups, and puts a %match-lambda inside each of those clauses. This %match-lambda handles the next stage: it still handles multiple clauses, but each of them expect the same number of values.

This is a good moment to take a short diversion into macro efficiency, or rather into deliberate inattention to efficiency. An ideal macro expansion will look different depending on the exact forms it receives. In this case, you might think it would somehow optimize the macro if it treated the (common!) case where every match-lambda clause takes the same number of values specially, and expanded directly into a single lambda instead of a case-lambda with only one clause, which itself is going to turn into a call to a lambda expression once this %match-lambda is expanded. Not so!

When writing Scheme macros, it’s always better to just handle the general case and let the compiler itself deal with the redundant code you might generate in special cases. This is the essence of the Guaranteed-Optimization Clause of the Macro-Writer’s Bill of Rights. A case-lambda with only one clause is usually treated by the expander as identical to a lambda even before any optimization begins; resulting structures like (lambda (x y z) ((lambda (x* y* z*) ...) x y z)) are trivial for the compiler itself to remove and turn into a more sane form. Adding it into the macro implementation directly would add more lines of code, but would bring no benefit at all the moment the expansion got out of the expander: in a Scheme compiler, getting rid of pointless make-work code like this is usually the first pass done after macro expansion, a combined inlining/partial evaluation/constant folding pass which Chez Scheme calls ‘cp0’ and Guile calls ‘peval’.¹ Compiler optimizations which, in a language without macros, seem fairly pointless – ‘who’d ever write code like that?’ – become valuable with macros, and become more valuable the more powerful the macro system is – ‘oh, code generated by other code looks like that!’

This is the point of the ‘guaranteed-optimization clause’: it lets us be laudably lazy as macro authors by guaranteeing us that our code will still run fast even if we don’t take the time to deal with redundant expressions; and it lets the users of macros know that they don’t have to worry about the performance impact of using a higher-level syntactic construct. Macros generate all sorts of silly code structures, not only because their authors don’t bother to deal with special cases, but also because there’s a lot that macros don’t know about the context they’re being used in. But that information which is trivially accessible for the Scheme optimizer once it has a fully-expanded abstract syntax tree. An example of not needing to worry about special cases is that, even in performance-sensitive code, if your macro takes in an expression but only wants to evaluate it once, you might be tempted to try to recognize if the expression is a single identifier to avoid redundantly generating a let expression giving it a name local to your macro. But you don’t need to: the Scheme compiler can just take care of it! An example which isn’t just laudable laziness is when the expansion of a macro includes code which treats its input specially if it’s a certain type – but the Scheme compiler with the fully-expanded program at hand can run a type propagation pass and change the code to reflect its knowledge of the possible types a variable might have at the specific point of each individual use of the macro.

We’ll see a few more examples of this philosophy of ‘just let the Scheme compiler deal with it later’ throughout our tour. In general: ask not what you can do for your optimizer, ask what your optimizer can do for you! The expand/optimize procedure in Chez and Loko, or the ,optimize REPL command in Guile, is your friend whenever you’re trying to work out what special cases it’s worth writing extra code to deal with vs. which would be wasted effort on something the compiler itself could do better anyway.

Anyway, %match-lambda turns out not to be the end of the chain of delegating to yet more primitive forms called lambda, although this final one doesn’t generate a redundant expression in the final code. This is an extensible pattern matching library, so patterns can be forms that the matcher itself doesn’t natively know how to interpret. In order to deal with those forms, we have to expand them into forms which it can natively deal with. That’s right, it’s a macro expander within a macro! (Kind of!)

Pattern expansion

So we move on to the (extensible-match expand) library, which contains the expand-pattern form which %match-lambda delegates to. The job of this library is to turn unexpanded patterns, containing all the weird and wonderful pattern syntax that users defined themselves, into ‘core patterns’ that the rest of the pattern match compiler can deal with directly. We’ll deal with the structure and rationale of core patterns in the next installment; for now, if you’re familiar with the structure of SRFI 262 patterns, they should look pretty familiar with an extra core: at the beginning of their name. For disambiguation, variables are now spelled (core:var name-of-the-var) and _ is spelled (core:wildcard), but they’re generally pretty similar in structure.

The interpretation of non-core patterns is actually done by yet another library, (extensible-match pattern-syntax), but this was a fairly late change which I made in order to be able to write an implementation-specific version of only that functionality for Guile, which doesn’t yet support identifier properties. Potentially in future, implementation-specific versions could appear for other Schemes which provide enough access to their internals to allow correct (or nearly correct) pattern syntax semantics to be impemented in some way other than actually using SRFI 213. Anyway, the two libraries still work together pretty much as one.

The forms in these two libraries are, surprisingly, themselves macros and not procedures, and they’re used in a bit of an unusual way. The problem is applying macro hygiene to the pattern syntax expansions. Put (very) roughly, macro hygiene works by the expander creating a new, unique token every time one expansion of one macro use takes place; when the user-written code which writes the expansion has returned, the expander goes in and changes the names of all the identifiers in the returned code by adding that magic token.

Well, extensible-match itself is a macro, and doesn’t have a way to create one of those magic tokens to apply to identifiers. We have to let the macro expander of the Scheme implementation itself do it for us – but there’s no standard Scheme API for that. What do?

The answer is macros in continuation-passing style. Hilsdale and Friedman’s paper, linked, is a little obscure and doesn’t really make a compelling case for using this in practice apart from hinting at the possibility of some really dirty tricks (Petrofsky extraction, Kiselyov defilement, etc.). It’s a well-known trick, though, for implementing Scheme macros which can expand to a language other than Scheme (embedded within Scheme). Patterns are an example of such a language; Alex Shinn’s loop macro clauses are another (Taylor R. Campbell’s version of this looping macro is more popular, but I’ve linked Shinn’s original implementation as it’s shorter and shows the continuation-passing trick more clearly).

The idea is that a continuation-passing macro matches code of the form

(the-keyword input-of-the-macro ... next-keyword . next-arguments)

and always transforms it into the form

(next-keyword output-of-the-macro . next-arguments)

So the macro receives the name of another macro which will further process its own output. Because the whole output after this transformation goes back into the expander, which thinks a complete expansion is finished, it will apply a new magic token to everything in the output-of-the-macro. If output-of-the-macro is itself a macro use in the sublanguage being expanded into, next-keyword can again recursively transform it into a continuation-passing style macro use at the Scheme level, referring back to itself as the continuation.

You can do this entirely in syntax-rules! Shinn’s loop does exactly this – macro-extensible macros don’t have to involve the most advanced expander features available. But offering direct access to an extension API based on continuation-passing macros has some disadvantages:

It exposes implementation details of the sublanguage to writers of extensions to the sublanguage, failing to maintain an appropriately opaque abstraction barrier between usage and implementation. One can imagine that someone might come along and decide to actually destructure the next-arguments, thinking that they can understand what the sublanguage implementation is packing in there and re-use that information for a nifty feature. In so doing they create a Hyrumesque nightmare for the developer of the sublanguage, who will break their nifty feature the next time they decide that this internal continuation structure needs changing.

The problem is only exacerbated by continuation-passing macro extension APIs which provide more than one continuation, or more information, to their writers. Olin Shivers’s loop macro and Taylor R. Campbell’s implementation of foof-loop both do this: the result may be powerful, but from the point of the designer of the sublanguage, it is inflexible, hemming in future extensions of the power of the sublanguage itself.
It fails to establish a distinct namespace for extension keywords to safely live in, outside of which uses of those keywords are either errors (as in (chibi loop)) or have distinct but related semantics (as in the pattern matcher). It also fails to prevent unrelated keywords being accidentally, mistakenly used as keywords within the sublanguage, which can expand into unrelated forms with unexpected behaviour.
It makes error reporting for incorrect uses of extensions to the sublanguage confusing, since upon failure to match any syntax-rules clause, it has no idea which parts of the macro are implementation details of the sublanguage and which are parts written in the sublanguage and directly entered by the programmer.
Macro writers have to be careful to always return to the continuation they were given. (On the other hand, providing quote or quote-syntax as the continuation keyword makes it easy to debug more complex expansions by single-stepping; whereas most Scheme implementations unfortunately do not provide macroexpand-1 for forms in the Scheme language itself.)

The extensible pattern matcher side-steps all of this by capturing the transformer procedures of its extension macros directly (in identifier properties) and calling them only on the user input, then embedding the output of the transformer procedure into a continuation-passing macro. The only information the transformer for pattern syntax has access to is the syntax object representing the pattern use itself; if there’s an error, it’ll be expressed in terms of this only and not in terms of some low-level form the pattern syntax user isn’t interested it. All the pattern syntax transformer has to do is return its expansion, and what the pattern expander does with it after that is none of its business: a clean abstraction barrier, just like real Scheme macros.

You might ask why, having thus eliminated the usual reasons for writing continuation-passing macros, the pattern syntax expander isn’t just a simple procedural recursion like this:

(define (expand-pattern-syntax stx)
  (syntax-case stx (core-pattern)
    ;; Base case, a fully expanded pattern identified by some special head:
    ((core-pattern expansion) #'expansion)
    ;; Recursive case of a pattern with a pattern syntax keyword:
    ((keyword . _)
     (expand-pattern-syntax ((look-up-pattern-transformer #'keyword) stx)))))

Indeed, Sam Tobin-Hochstadt’s paper on extensible pattern matching in Racket presents it as if it did work this way.

As mentioned above, the reason is hygiene: we want to go back into the Scheme expander after each step and apply a unique magic token to any inserted identifiers for each individual use of pattern syntax, so that the names don’t collide with one another. This is, admittedly, somewhat pedantic: there’s no real point to introducing identifiers with the current structure of patterns, but with future extensions there might be, and it’s possible that some unusual pattern macro implementation might have a reason to do it anyway – so it’s worth taking the effort to do the right thing.

All of (extensible-match expand) is thus concerned with implementing a recursive expansion of pattern syntax while maintaining this hygienic property, which it has to do in continuation-passing style. For example, expanding a core:and or core:or means each of their sides have to be expanded recursively, then those expanded forms placed back in a new core:and or core:or which is then deemed fully expanded. If the core:and or core:or is actually a subpattern of some larger pattern – another core:and or core:or even (the core versions of these forms can only be two-armed, as a simplification) – they then have to pass this fully-expanded form back to the ongoing expansion of that larger pattern, so that it in turn can continue and reconstruct its input.

Tobin-Hochstadt’s presentation in his paper is actually something of a pedagogical lie, although it’s not much of one since Racket does have a way to apply its expander’s magic tokens without going all the way back into the expander, and Racket’s pattern matcher uses that to avoid this continuation-passing macro malarkey. This is therefore all a bit embarassing for a Schemer who can look across the aisle and see that Racket has always had better ways of doing this; hopefully, one day, Scheme will too, and all this continuation-passing nonsense will be gone.

As one last point, we don’t expand only one pattern at once – in match-lambda we have a number of clauses, each of which has a number of patterns (for each of the values handled by that clause). So we have not only an expand-pattern in continuation-passing style, but an expand-patterns too, which expands all the patterns in a list of patterns and passes them all on as a list of core patterns; then an expand-patternses which expands all the patterns in a list of list of patterns, passing them on as a list of lists of core patterns, which is what %core-match-lambda actually uses directly. These, of course, work by invoking expand-pattern with a continuation which keeps track of the already-expanded patterns and the ones yet to be expanded. Phew!

All of that gets bundled up and, as %match-lambda requested, the final continuation of the expansion process is %core-match-lambda, the form which co-ordinates the whole pipeline of the actual compilation of patterns into Scheme code.² We’ll start talking about the stages %core-match-lambda puts the patterns through next time – from here on in, it’s all procedures, and no more expanding match macros into yet more macros!

One last thing: that ‘almost’

There’s one wrinkle in using match-lambda for everything: the forms which do recursive binding – match-letrec, match-letrec*, and match-define – practically speaking need access to something that match-lambda doesn’t give them: they need to know the names of all the variables bound by the pattern match. In order to know that, they have to expand the patterns and extract the list of variables before match-lambda does.

Racket actually expands the patterns twice for this functionality, but I think this is unnecessarily nondeterministic. There is no guarantee that any syntax transformation has to be a pure function; while every expansion of a macro should have the same effect, if you throw in generated identifiers and the possibility of seeded hash functions used in a macro implementation and similar such things, there’s no guarantee that every expansion will be detectably identical. So my view is that it’s better to expand the patterns only once. Match-letrec and match-define therefore invoke expand-patterns or expand-pattern themselves, skip around the publically-exposed match-lambda, and go directly to %core-match-lambda. Match-letrec* is implemented in terms of match-define, letting the Scheme implementation take over the implementation of letrec* semantics.

Next time

I’m not sure exactly how much it will make sense to cover in each individual entry in this series, and thus also not sure how many entries there’ll be. But next time we’ll certainly look at the structure of core patterns and the abstract syntax tree, and maybe start to look at the real meat in the decision tree generator.

Greetings, reader from the future! You can read the next part here.

Most compiler courses: ‘Here is 90% on parsing and a tiny bit on register allocation; good luck and have fun!’

Kent Dybvig: calls the pass of his compiler which comes right after parsing and macro expansion ‘Compiler Pass Zero’, because parsing and expansion aren’t really compiling.↩︎
That’s right – just like Chez Scheme, the architecture of extensible-match is such that it doesn’t really consider expansion per se to be compilation.↩︎

Scheme Reports at Fifty: Where do we go from here?

2025-10-18T16:43:53Z

Based on my talk at the Scheme Workshop 2025. You might prefer to have me talk it at you instead.

We are holding an election to the Scheme Steering Committee. Register to vote and nominate candidates!

In December this year, the Scheme reports will turn fifty. As chair of the working group entrusted with the next major revision of the report, I want to start a discussion in the Scheme community about what a good Scheme report looks like in 2025, and how it’s different from what it looked like in 1975, and from other times when the Scheme report was revised.

Who is the Scheme report for?

When making any document, we have to consider who we’re writing for, how they’re going to use it, and what they want and need from it in order to do their thing. For the Scheme report, the obvious two groups are users and implementers. These are very vague groupings, and like any attempt to divide up a group as large and diverse as the Scheme user base, one can’t at all pretend like every member of these groupings has the same views on every issue. Nonetheless, a rule of thumb is that users want their pet features added, creating pressure to grow the report;¹ implementers want to be able to actually create a compiler, creating pressure to keep the report smaller and simpler. But there are also clear subdivisions where even this clear breakdown falls apart, like in the embedded space, where both users and implementers want something that can work on limited-resource environments.

Another tension still rarely recognized is between users in education – students learning the language – and long-time Schemers who know it very well. The How to Design Programs approach seems to have worked for many more beginning students than Structure and Interpretation of Computer Programs ever did, because HtDP acknowledges from the start that assuming brand new students will be able to learn with a tool designed for people who are already experts means that tool can’t be as helpful to those just finding their feet in a new way of thinking.

But in Scheme we often get into philosophical and even political debates when we talk about what the Scheme report should be like; debates which have much more to do with an ideological belief about the nature of the language than with the direct, practical needs of the debaters.²

Big and little languages

A classic such argument is over whether certain versions of Scheme – usually R6RS, and sometimes also the one that my working group is producing – are ‘too big’. On the other hand, Scheme reports that don’t have this quality can be seen as being too ‘little’ for doing real programming work in.

Earlier, littler Scheme reports are often described by the makers of these arguments as a ‘diamond-like jewel’, or similar. The ‘diamond-like jewel’ wording is, I think, from the infamous ‘Worse is Better’ section of Richard P. Gabriel’s paper ‘Lisp: Good News, Bad News, and How to Win Big’.

In 1998 Guy L. Steele, co-inventor of Scheme, read a keynote paper at OOPSLA entitled ‘Growing a Language’ (transcript). In the paper, he talks a lot about the tension between the qualities of a big language vs a little language, and in particular talks about Scheme, saying:

Could not “The Right Thing” be a small language, not hard to port but with no warts?

I guess it could be done, but I, for one, am not that smart (or have not had that much luck). But in fact I think it can not be done. Five-and-twenty years ago, when users did not want that much from a programming language, one could try. Scheme was my best shot at it.

But users will not now with glad cries glom on to a language that gives them no more than what Scheme or Pascal gave them. They need to paint bits, lines, and boxes on the screen in hues bright and wild; they need to talk to printers and servers through the net; they need to load code on the fly; they need their programs to work with other code they don’t trust; they need to run code in many threads and on many machines; they need to deal with text and sayings in all the world’s languages. A small programming language just won’t cut it.

From the things he names as examples of what users want to do with languages, it’s clear that Steele’s mind was very much on Java, and Java in 1998 at that. Some of these things just aren’t in fashion any more: some were thought to be good ideas at the time, but now seen as bad ideas; some of them just never caught on. But some of them are just as important as ever.

Of the things that are just as important as ever, though, many of those no longer seem like things a language should have to take care of; they are, rather, the province of third-party libraries.

That, in fact, is how Steele concludes ‘Growing a Language’: users should be able to grow the language itself by adding libraries.

But, it seems, the base of Scheme as it was in 1975, and even in 1998, was not enough. Most people today would agree that a programming language without even a standard error-handling mechanism is quite deficient, even if it does give one the ability to build one for oneself.

And, in fact, most Schemers agree that the best way for Scheme to go forward is to find better ways for Scheme users to grow the language. Our disagreement, then, is not about ‘bigness’ per se, but where the bigness comes from. Few would swallow a thousand-page Scheme report; setting aside the actual labour of designing and implementing it, most people would be perfectly content with using their own personal library of language extensions whose documentation would run to a thousand pages if it were all documented.³ Our disagreement is over how big the base needs to be in order to provide a foundation to grow from; and over what needs to be in the foundation, to enable users to give themselves this bigness.

Dynamic systems, static languages

So big vs little is one axis of disagreement within the Scheme community. How about another?

In 2012 Richard P. Gabriel wrote an essay paper, ‘The Structure of a Programming Language Revolution’. He talks about how he took a break from programming languages in the early 1990s in order to do a fine arts degree. When he came back in the late 1990s, everything had changed. When he left, people in programming languages talked about dynamic, evolving systems like Common Lisp and Smalltalk. By the time he returned, the conversation had changed to static languages like Haskell.

I think one of the other axes of division in the Scheme community is around this matter, and the reason is that our community lived through this paradigm shift without, collectively, noticing it. Some of the community moved on to the new paradigm; others continued to think in terms of the old.⁴ The two sides do not understand one another’s concerns.

The dynamic systems viewpoint is epitomized by the R5RS, whose only entrypoint for programs is the interactive top level, and which even began to lay the groundwork for a library and namespacing system that would have been based on first-class environments and the eval procedure. Implementations which follow the dynamic systems model include MIT Scheme and Guile.

The static languages viewpoint is epitomized by the R6RS, which doesn’t have an interactive top level at all. Implementations which adhere to this model include Chez, Racket, and Loko, but also niche implementations like Stalin which don’t have their own interactive top level at all.

Put briefly, the dynamic languages people see Scheme as a dynamic language, plain and simple, and want to be able to do all the run-time modification that they should be able to do in any such language. The static languages people see Scheme as a static language in all regards except its type system.

And yet more

Already with these two dimensions of belief about what kind of language Scheme is – big vs little, static vs dynamic – we have a huge range of potential expectations for what the Scheme report might be like. When we compare these viewpoints in terms of the models their advocates might prefer to take as guidance for the future development of the language, we can see just how far apart these ideas are.

Those who want a little, dynamic language will look to earlier versions of the report, and maybe to the cores of languages like Lua and JavaScript. A little static language might look like SML. Common Lisp might be a model to follow for a big dynamic language, or Haskell for a big static language. I’ve seen almost every one of these languages used to suggest what a suitable future direction for Scheme might look like.

These are hugely different rôle models to follow – and these are only two matters on which people have different feelings. I could add more for more philosophical disagreements about the nature of Scheme, ideas about what the right way to make and publish a Scheme report should be, and even very specific individual language features. Clearly, we can’t satisfy all of these expectations at once.

Does a Scheme report still make sense?

Given these deep divisions over the essential nature of the Scheme language, does it even make sense that we still keep making a Scheme report?

‘No’ is an entirely possible answer to this question. Already in the R6RS and R7RS small days, people were arguing that Scheme standardization should stop.

If we went this way then, just like Racket in its default mode no longer claims to be a Scheme report implementation, Schemes would slowly diverge into different languages. Guile Scheme would one day simply be Guile; Chicken Scheme would be Chicken, and so on. Like the many descendants of Algol 60 and 68, and the many dialects of those descendants, each of these languages would have a strongly recognizable common ancestor, but each would still be distinct and, ultimately, likely incompatible.

It shouldn’t surprise you that I don’t think this is the way to go. As a user of Scheme, I can still see ways I want to extend the common core of all of these implementations. It’s much better to be able to do this once and have my extension available on many implementations, than to have to work hard to port my extension to multiple different languages. All implementations benefit by agreeing on a common core that works to be able to grow the language.

Racket is already a cautionary tale. It has some amazing features, but those of us on other implementations can’t take advantage of them because Racket’s implementations of those features aren’t portable. Even Guile has this problem, because it’s still culturally normal among users of that implementation to use its own define-module declaration instead of the standard R6RS or R7RS library or define-library, even if your code only uses standard Scheme features and maybe some common SRFIs. There’s much Guile code I’d like to use on Chez for more speed, or on Chibi for a smaller implementation footprint, that currently requires at least some porting effort for not much good reason.

This is more or less the same rationale for Scheme reports that there has always been, and I think it continues to apply today.

R6RS and R7RS small considered more alike than widely believed

So our thoughts must naturally turn to unification. The co-existence and incompatibility of R6RS and R7RS small have become emblematic of the splits in our community. The R6RS editors and community are, to some extent, justified in feeling that R7RS small unfairly abandoned their work. But in fact the two are more similar than is widely recognized, and in many cases the design of R7RS small in fact vindicates the decisions of the R6RS editors.

This may seem a shocking claim about a division which some predicted would end up killing Scheme as a programming language. But, I think, WG1 stumbled their way towards a realization, which I’ll get to just below, which led them to a develop a library system working on basically the same model as R6RS. R7RS small library declarations support conditional compilation and other features which, in R6RS, is exclusively the province of the ‘main’ level language; but once all the cond-expands in define-library are dealt with, the two are essentially the same. R7RS small’s define-library declarations can be compiled entirely ahead of time to R6RS’s library style – there is already a program which does this, Akku.

Between 2007 and 2013 the landscape of programming tools also shifted so some of the things which were criticized in R6RS’s day – bytevectors for binary data, and Unicode throughout – now seem perfectly normal. It now seems wrong for a programming language not to support these. R7RS small also adopted, for example, the same exception raising and handling mechanism as R6RS.⁵

The changing landscape of programming tools around Scheme means that programmers now have a much different sense of the ‘littleness’ of a language than in 2007. Since Rust there’s increasing recognition that memory safety should be treated as non-optional even in most low-level programming, and is cheap to implement; it now seems strange that R5RS and below, and R7RS small, make bounds violations undefined behaviour in the C sense.⁶

Moreover, what R7RS small’s authors seemed to realize is that the report will always be a minimum, and not a maximum. I think this is why, although it came from a group of users who started out highly critical of the R6RS for its static model of libraries and programs, R7RS small adopted essentially that model. It doesn’t restrict implementations from providing more dynamic library features.

The small/large report split helps even more, because now the report offers two different minimums for implementations to choose from, according to what their needs are. The minimum for ‘the practical needs of mainstream software development’ is, as Guy Steele says, much bigger than it used to be, and that’s fine!

Join, or die

This is my appeal to the Scheme community, then. If we don’t have a Scheme report that the majority of implementations and users accept, Scheme as a coherent single language will simply die. Not all implementations – it’ll be like if, as above, we stopped making Scheme reports and everyone went their own way. But even the implementations which survived would clearly be weaker for not being able to share the libraries of the others.

R7RS is our best shot at joining together to still be able to share code with one another, and to grow the language together – not only in the report itself, but much more in the tools we build with it.

The sacred craft

Joining, however, means compromising.

Nobody will look at any Scheme report and say it is perfect. I can’t look at any version of the report, not even R5RS, and say it was a ‘diamond-like jewel’.⁷ Even if it were perfect for one person, it cannot be all things to all people. Scheme has historically been too shy of compromise; the uncompromising pursuit of perfection, even though perfection is a quality nobody will ever agree on, may be why we have struggled so much to move onwards while other languages have not.

‘Compromise’ is a dirty word for some people, but I would like to offer another perspective. Although R7RS is sometimes perceived as anti-R6RS, two of the small report’s editors actually voted in favour of R6RS ratification.⁸ One of them, John Cowan – also my predecessor as chair of WG2 for the large language – wrote in the rationale to his vote:

a well-crafted compromise is in my opinion the most sacred thing that a secular age knows

We may not be able to have a diamond-like jewel, but we can have the most sacred thing a secular age can make.

As mentioned at the top, I hope this essay will really open a conversation about these matters. You can write a comment on Mastodon, or if that’s too limiting, on the Scheme Reports mailing list.

And don’t forget the election to the Scheme Steering Committee!

26 people who voted against R6RS ratification said, in some form of words, that the language was too big; 13 people said they wanted specific new features and voted against because they weren’t there. Despite the apparent contradiction, the latter group overlaps with the former almost completely.↩︎
Ideology is not bad; I do not use the word to impugn anyone or their arguments. Without ideology, we have no structure with which to build our understanding of the world around us. I have my own ideology of Scheme as much as anyone in the community.↩︎
We may also need to set aside that only the person who wrote such a library would want to actually use it or any software written with it. Today we are also happy to shop in the free market of language extensions, choosing our own personal favourite design of pattern matcher or HTML templater out of many that have already been written by other people.↩︎
As an example of the kind of new work that came out of the Scheme community which didn’t make sense under the old paradigm but seems vital under the new, consider that it was only in 2002 that Matthew Flatt came out with ‘Composable and Compilable Macros: You Want it When?’, introducing phasing as a model of compile-time evaluation compatible with static ahead-of-time compilation. This problem only began to be considered by Schemers in the 1990s, because the problem itself doesn’t really make sense under the dynamic, evolving systems paradigm.↩︎
Although the R7RS small authors didn’t seem to understand what raise-continuable was for and didn’t include an analogous construct to the non-&serious condition types of R6RS.↩︎
My (least) favourite example is that it isn’t safe to decode UTF-8 text from an untrusted source in R7RS small, because invalid UTF-8 passed to utf8->string is also undefined behaviour. Anal monkeys galore.↩︎
What kind of diamond has undefined behaviour when you try and write beyond the end of a buffer? Does diamond-like perfection in a language inherently demand diamond-like perfection of the programs written in it?↩︎
Another WG1 member, Aaron Hsu, voted against R6RS ratification, but if one looks at his WG1 proposals and votes, later consistently pushed to adopt things from R6RS within WG1.↩︎

A story about data representation and performance

2025-09-29T16:51:11Z

If you go poking around the source code for extensible-match, you’ll find places where I define auxiliary syntax keywords and use syntax objects like records belonging to some sum type. Like here or here or here.

Extensible-match belongs to the comparatively rare class of Scheme macros that are complex enough that they count as compilers in themselves. As Olin says when defending the use of records inside of his loop macro implementation, once you recognize that your macro has reached that level of complexity, it’s simply a matter of good programming practice to adjust your data representation and use records instead of trying to make ad hoc transformations on S-expressions. This goes up even to things that are themselves unambiguously compilers: the Nanopass framework helped compiler courses at Indiana and then Chez Scheme itself make the leap from trying to do everything with S-expressions to using proper records to represent and transform the input code. So why does extensible-match use these weird syntax object pseudo-records?

Well, there’s a story about that, and I consider myself to have learned my lesson here: eventually these things will probably mostly disappear.

Since extensible-match is a compiler, I thought when writing that I would benefit from being able to use a pattern matcher – but I also didn’t want to get into the mess that psyntax has where you have to use a pre-expanded version of the pattern matcher in order to be able to work on a new version. So I messily decided to abuse syntax objects as records to be able to consistently use syntax-case as the pattern matcher used to implement another, more general pattern matcher.

This came back to bite me quite badly.

The algorithm extensible-match uses to generate an optimal set of conditionals from patterns is Luc Maranget’s decision tree algorithm. The first problem that started to appear with using syntax objects here is that they don’t, in general, have a well-defined equivalence relation. For example, in order to reduce the number of cases the main decision tree generator has to deal with, there are some ‘source-level transformations’ which simplify some complex cases in the patterns themselves before generating the decision tree. One source-level transformation is rewriting disjunctive (or) subpatterns as separate top-level patterns. Another handles the case that a ‘row’ – essentially an and pattern where the order of checking subpatterns isn’t fixed – contains another row, by flattening the inner row into the outer row. But what if an or contains a row, or vice versa? The easiest thing to do is to iterate these source-level transformations to a fixed point: repeat the process, each time feeding last time’s output back in, until the output is the same as the input, at which point we know there’s nothing more that needs to be done. But you need an equivalence relation to be able to test if the output was the same as the input! So I had a pretty awful procedure that tried to tell if the transformation procedure would find any transformations to do on its input, which in any other circumstance would be the worst kind of premature optimization.

But the real problem came with the performance of Maranget’s algorithm. The algorithm has basically the same run-time complexity as the naïve recursive Fibonacci function: each step of generating the decision tree recurses on a set of patterns which is a little bit smaller than the input set, and on a set that’s a little bit smaller than that again. The hardest torture test I had when implementing Maranget’s algorithm was a port of the bytecode interpreter which is Example 3 in his paper. On Chez Scheme on a pretty recent MacBook, it took 8 seconds to generate the decision tree when I represented patterns this way. This is just 14 cases; Chez is one of the best-optimizing compilers around, and running on most implementations it would be even slower. 8 seconds for a moderately complex match is already unacceptable for incremental development – and once extensible-match starts supporting more Scheme implementations, for many people it would be even worse, since probably no other implementation would be as fast as Chez at this, and many people have slower computers than mine.

Finding the cause of this performance problem was a nightmare. It was a classic ‘peanut butter’ performance problem: because it was smeared out over each recursive step of the main algorithm, there was no obvious individual hot spot that was making it slow. Since I’ve already spoiled half the solution with the premise of this post, I’ll cut to the chase and spoil the rest: in order to distinguish which case of the (conceptual) sum type syntax-case is looking at, it uses the free-identifier=? comparision procedure. This procedure is actually quite slow because it has to check all the local bindings in scope where its inputs come from; since it’s rarely used intensively – usually only a few times per macro expansion, if even that – you don’t notice how slow it is until it starts to add up from being called again and again in the middle of an exponentially-recursive algorithm.

Ending appropriately spoiled, here’s the rest of the sorry story of me trying to work out what was wrong.

I initially thought it was because the hash consing of nodes wasn’t efficient enough (Chez’s time syntax told me that a lot of GCing was happening) – so I tried to optimize that, but it made no real difference. For all its strengths, Chez Scheme’s options for debugging and profiling are pretty weak: you can ask it to annotate code for profiling when loaded, but the resulting profile will only count how many times each bit of code was evaluated, not how much time the evaluation took. It also won’t tell you many times procedures hidden behind macro expansions are called (like free-identifier=? in syntax-case); nor will it go inside any code it hasn’t annotated or can’t annotate, like the procedures within the Chez Scheme standard library (like, again, free-identifier=?). Eventually it occurred to me that, while the whole extensible-match package wouldn’t load on Guile because Guile lacks identifier property support, the decision tree generator didn’t need that support, so I could at least use its far superior profiling options (a real statistical profiler! that also looks in the code you depend on and not only your own code!) to diagnose the problem.

That only confirmed my suspicions that 8 seconds in Chez would translate into something much, much worse on other implementations: the same 14 cases of Maranget’s bytecode interpreter took over 5 minutes to generate a decision tree on Guile! What’s more, although the profiler correctly identified free-identifier=? as the culprit, with 70% of execution time spent there, I couldn’t work out why. You see, the code does also explicitly use free-identifier=? to determine which predicate and accessor calls it can coalesce into a single node in the output decision tree. But I couldn’t believe that was the expensive part of this case!

I can’t remember what made me realize that it might be the free-identifier=? that was internal to the syntax-case form that discriminated between different types of patterns that might be issue, but since free-identifier=? seemed to be the culprit, I started to work on getting it out of the main decision tree generation procedure entirely: not only the explicit uses, but the implicit use in syntax-case as well. I had a very painful refactoring session adding a new layer to the representation of patterns, converting their representation from syntax objects to real Scheme records. Free-identifier=? during decision tree generation was replaced by interning patterns’ embedded expressions into a table so instances of the same identifier, which could be coalesced into a single node in the output decision tree, could be checked by a simple integer comparison instead of free-identifier=?’s recursive scope analysis. (Building this table has quadratic complexity since Scheme has no free-identifier-hash, but it should still save time.)

This refactoring meant I could no longer use the syntax-case pattern matcher to implement the decision tree generator – but there was one nice side-effect, because I could use the opportunity to make explicit one bit of state that was only implicit in the old representation: the name of the variable which has the actual value each subpattern is being tested against.

Whoosh! That alone solved the performance problem: from 8 seconds, the decision tree generation for Maranget’s interpreter was down to 0.2 seconds in Chez. On Guile it’s still 9.3 seconds – although this can’t fairly be compared with the 314 seconds it took there before, as I have a newer and slightly faster computer since I originally ran that benchmark, and (for whatever reason) I didn’t repeat this performance test on Guile on the old computer once I’d solved the performance issue.

Ten seconds for 14 cases still isn’t great, but I’m still much less concerned about this. If you recall my mention of the similarity to a Fibonacci recursion above, it might seem that the solution is memoization. In theory, memoizing the recursion over the smaller of the two cases – the one which excludes the topmost row, which generates the side of the tree used when a test required by that row failed – might help. I haven’t tried this yet, though: based on anecdotal evidence, 14 cases is already larger than the vast majority of pattern matches out there, and 0.2 seconds for this on the main implementation extensible-match supports is pretty okay.

If I do go in and start looking for expand-time performance improvements again, another option for faster compilation might be to just split the input pattern list into chunks, and give up on optimizing the decision tree between the chunks. Since the performance characteristic is exponential and kⁿ + k^m is usually much smaller than k^n + m, this indeed seems to work: splitting these 14 cases up into 10 + 4 cases gets a decision tree in under a second on Guile – but this feels like cheating. Maybe with a compile-time option to set the chunk size or disable the cheat (so production builds get fully-optimized, fast-running decision trees while development builds get fast compilation but slower matching) … I’ll probably wait until match compilation speed on Guile isn’t quite such a shitshow for other, unrelated reasons before starting to really consider these options. (For one thing, how do or patterns count here? As mentioned above, they hide multiple cases, potentially buried deep inside a nested subpattern, and have to be unfolded to the top level to get a true count of the number of cases the decision tree generator will see.)

Anyway, the point here is the main representation of patterns is now in real Scheme records, as God intended, but there are other places in the code that still abuse syntax objects as records. Those don’t hit any performance-sensitive code, though. Probably it would be better to refactor them out as well, some day: you might notice that this isn’t the only place where records which start out as syntax objects get turned into another layer using real records (oops, bad idiom for sum types there – again, I wanted to be able to use case and not a load of type test predicates in a cond, which was really very petty of me :-( ). But that concludes today’s missive. Moral of the story: use real records in your macros when you need to; don’t try and fake them with syntax objects because you think syntax-case will be a nice pattern matcher to implement your compiler with. If you’re implementing anything other than another pattern matcher, you should have no reason to even think of abusing syntax-case for this in the first place – just use a real pattern matcher like extensible-match inside of your macro definition!

Fun fact: Although this is the first post on this new blog, it’s actually a footnote to another blog post I started to write. The footnote got … rather out of hand, and ended up being a post of its own.

Update, 3 October: Zhu Zihao asked why it is that free-identifier=? is so slow; he asked if changing the representation of ribcages in syntax objects to something like a hash table (from an alist) might help. Well, really, an alternative way to look at it isn’t so much that free-identifier=? is slow – it’s more just that normal record type checking is really fast. Since none of the record types that are actually used in representing the AST of patterns are subtyped, a successful type check involves only dereferencing the pointer to the pattern itself then checking pointer equality for the heap tag. Since I didn’t mark any of these record types as sealed, an unsuccessful check is at least another size check (size of the record type’s ancestry) and dereference (the vector of its ancestors) and pointer comparison (the type in the RTD’s place in the ancestry vector), but that’s still minimal, and pretty cache-friendly to boot. (Maybe I could squeeze an extra fraction of a fraction of a second of performance out by declaring these types sealed to avoid all that.)

By comparison, any amount of checking which of multiple bindings the identifier-abused-as-type-tag might have is always going to be slower than this. Since alists are faster than hash tables up to about 25 entries (depending on Scheme implementation, measurement technique, etc.) it’s not even clear there would be any win to be had in that direction.

Besides, another source of speed-up I didn’t mention in this post was field access. Field access with the syntax object representation involved chasing pointers, checking the wrapped list was the right length (even though it always would be, by construction) … and now, like the heap tag test, it’s always a simple, safe pointer dereference. The 70% number from Guile’s profiler gives an impression of the ratio of how much performance improvement came from getting rid of free-identifier=? and how much from not having to chase pointers down a list for field access.

8 seconds to 0.2 seconds is, actually, ‘only’ a speed-up factor of 40. While it sounds like a lot, I find it prima facie utterly believable that memory accesses and branches went down by this factor due to this change in data representation: if I think about the difference in what the computer is doing for each, it makes total sense.

crumbles.blog

HOWTO: Unlock LUKS encrypted disks over SSH on a Raspberry Pi 4 running NixOS

HOWTO: Set up an IPsec VPN on NixOS and connect to it with Mac OS and iOS

Generating the needed certificates, etc.

Configuring Strongswan in configuration.nix

Setting up users

Run it!

Connecting on Mac OS and iOS

Improvements I want to make in a future version of this guide

Make the font bigger

Tour of a pattern matcher: decision trees

Motivating example

Generating a decision tree

Representing a decision tree

Step by step

Picking a pattern

So why was it so hard/why didn’t you do it slightly differently

Optimizing space and time

Visualization

Tour of a pattern matcher: core patterns and the AST

Subject variables vs pattern variables

Core patterns

The Abstract Syntax Tree (AST)

The other half of the clause

Unifying subject variables

Potential future AST transformations

Next time

Tour of a pattern matcher: expression and pattern expansion

The match forms

Pattern expansion

One last thing: that ‘almost’

Next time

Scheme Reports at Fifty: Where do we go from here?

Who is the Scheme report for?

Big and little languages

Dynamic systems, static languages

And yet more

Does a Scheme report still make sense?

R6RS and R7RS small considered more alike than widely believed

Join, or die

The sacred craft

A story about data representation and performance

Configuring Strongswan in `configuration.nix`