I’m tired of seeing “Your connection is not private” when hitting HTTPS services within my internal network. I considered setting up an internal CA, but that seems like a lot of work. I have a real domain name, so Let’s Encrypt might be a very easy solution.

To use Let’s Encrypt on an internal network you need to be able to create a DNS TXT record to prove ownership of the domain. ZoneEdit (my DNS host) recently added an API for creating and deleting TXT records.

There’s an acme.sh plugin for ZoneEdit, but I wanted something I could use with NixOS’s security.acme option. security.acme uses Lego, which supports many DNS providers, but not ZoneEdit.

Thankfully, somebody gave some thought to extensibility and wrote an “External Program” DNS provider for Lego. The documentation is excellent, and it was easy to write a shell script to create and delete the required TXT record.

With the shell script written it can all be hooked up with a NixOS module:

security.acme.acceptTerms = true;
security.acme.certs.${cfg.hostname} = {
  email = cfg.email;
  group = cfg.group;

  # set DNS TXT records by exec-ing acme-zoneedit.sh
  # (configured below)
  dnsProvider = "exec";
  credentialsFile = cfg.credentialsFile;
};

# configure the "exec" DNS provider
systemd.services."acme-${cfg.hostname}".environment = let
  acme-zoneedit-sh = pkgs.writeShellApplication {
    name = "acme-zoneedit.sh";
    runtimeInputs = [ pkgs.curl ];

    text = builtins.readFile ./acme-zoneedit.sh;
  };
in {
  EXEC_PATH = "${acme-zoneedit-sh}/bin/acme-zoneedit.sh";
  EXEC_PROPAGATION_TIMEOUT = "600";
};

There are 2 interesting things here:

  • writeShellApplication takes a normal, Nix-unaware shell script, and wraps it to put curl on the PATH.
  • I’m passing 2 sets of environment variables to the systemd service:
    • systemd.services.<name>.environment contains the configuration for the “External Program” DNS provider.
    • security.acme.certs.<name>.credentialsFile contains the ZoneEdit API key.

    Separating these allows the credentials to be encrypted using e.g. agenix.

gai.conf

There was one unexpected complication along the way: Lego failed every time with an error like:

2023/11/27 05:14:00 [INFO] [redacted.diffeq.com] acme: Waiting for DNS record propagation.
2023/11/27 05:14:02 [INFO] [redacted.diffeq.com] acme: Cleaning DNS-01 challenge
2023/11/27 05:14:04 [INFO] Unable to get the authorization for: https://acme-v02.api.letsencrypt.org/acme/authz-v3/redacted
2023/11/27 05:14:04 Could not obtain certificates:
	error: one or more domains had a problem:
[redacted.diffeq.com] time limit exceeded: last error: read udp [1faa:8a59:8df7:53ca:c50d:18a8:5be2:1d65]:34358->[2600:3c01::f03c:91ff:fecc:142b]:53: i/o timeout

What is this DNS server that Lego is trying to talk to, and why won’t it respond?

tcpdump gave me the answer: it’s ns19.zoneedit.com. Presumably Lego is trying to contact an authoritative DNS server for the domain, and it’s getting this address that doesn’t respond.

ns19 seems to have some kind of misconfiguration - ns1 through ns18 work fine via IPv6, and ns19 works fine via IPv4.

My ugly (and hopefully temporary) solution is to configure glibc to prefer using an IPv4 address when a host has both IPv4 and IPv6 available:

# /etc/gai.conf

label  ::1/128       0
label  ::/0          1
label  2002::/16     2
label ::/96          3
label ::ffff:0:0/96  4

precedence  ::1/128       50
precedence  ::/0          40
precedence  2002::/16     30
precedence ::/96          20

# prefer IPv4 addresses when both are available
# (because zoneedit's DNS servers don't work properly over IPv6, which
# breaks ACME)
precedence ::ffff:0:0/96  100

With this file in place, Lego gets ns19’s IPv4 address, and everything works beautifully.