NixOS integration

[musjj]

Any chance that there could be an official NixOS module integration, just like https://github.com/Mic92/sops-nix?

[domenkozar]

Would be great if someone makes this! I'm happy to review PR to nixpkgs.

[blaggacao]

_{Here are my notes from exploring to use it.}

(as shared on devenv matrix chat):

TL;DR;

I explored using secretspec with devenv for NixOS deployments but found a gap in end-to-end secret provisioning (especially vs. nixos-rebuild). Due to issues with WSL keyring/dbus and the lack of native service wrapping, I settled on using systemd-creds on the target host. I now use secretspec locally only to fetch credentials, which are then piped over SSH and encrypted at rest on the RPi (leveraging a TPM).

End Result (script)

    deploy-creds-rpi5.exec =
      let
        emplaceSecret =
          file: name:
          "mkdir -p /etc/credstore && systemd-creds encrypt - /etc/credstore/${file} --name ${name} && chmod 0400 /etc/credstore/${file}";
      in
      ''
        secretspec run -- bash -lc '
          set -e

          cat "$CLOUDFLARE_CERT_PEM" | ssh ${rpi5} "${emplaceSecret "cloudflared-cert.pem" "cert.pem"}"

          cat "$CLOUDFLARE_CREDJSON" | ssh ${rpi5} "${emplaceSecret "cloudflared-rpi-foo.bar" "credentials.json"}"

          printf "ADMIN_TOKEN=%s\n" "$VAULTWARDEN_ADMIN_TOKEN" | ssh ${rpi5} "${emplaceSecret "vaultwarden-admin-token" "vaultwarden.env"}"
        '
      '';

Storyline

• https://devenv.sh/integrations/secretspec/ (the page) seems to induce the agent to always assume that I want secrets inside the devenv environment, even if I want them only to "deploy" to the remote nixos machine. Probably these docs could be touched over to clarify this distinctions to humans and machines.
• ... actually, there seems to be a gap in the end-to-end process. nixos-rebuild does not support populating files/secrets, nixos-anywhere otoh does. It's not a polished experience at all. I'm now hinted again at sops-nix et al, but I can't allow myself to go down another tooling integration rabithole.
• I'm not entirely sure how secretspec actually fits the picture here... Maybe by fetching secrets on-demand from a remote service? Sounds doable, except for bootstrapping that very service (e.g. vaultwarden).
• I struggle to understand how to wrap a systemd service with secretspec, e.g.:

  services.cloudflared = {
    enable = true;
    certificateFile = "/etc/cloudflared/cert.pem";
    tunnels = {
      enclosed = {
        credentialsFile = "/etc/cloudflared/credentials/enclosed.json";
        ingress = {
          "${tunnelHostname}" = {
            service = "http://${enclosedHost}:${toString enclosedPort}";
          };
        };
      };
    };
  };

• certificateFile needs to be a file, so somehow, I need secretspec to emplace it in a services pre-hook ? any good ideas how to do that reliably - has been a while.

• NixOS-WSL might not support keyring? Seems it doesn't -even after enabling services.gnome.gnome-keyring.enable = true;:

• dbus on NixOS-WSL also seems a bit unreliable - no clear picture - I could find this issue, but my investigation and observation remaind inconclusive, unfortunately: systemd doesn't create user (d)bus microsoft/WSL#8842

• Just wanted to share the outcome of the secrest journey with devenv - There may be a conclusing sitting somewher in this journey, that's why I share it.

# denvenv.nix

{ inputs, ... }:
let
  rpi5ip = "192.168.88.192";
  rpi5 = "root@${rpi5ip}";
in

{
  scripts = {
    nixos-rebuild-rpi5.exec = "nixos-rebuild switch --flake ./rpi#rpi5 --target-host ${rpi5}";
    nixos-anywhere-rpi5.exec = "nixos-anywhere --flake ./rpi#rpi5 --target-host ${rpi5}";
    deploy-creds-rpi5.exec = ''
      cat /home/nixos/.cloudflared/cert.pem | ssh ${rpi5} \
        'mkdir -p /etc/credstore && systemd-creds encrypt - /etc/credstore/cloudflared-cert.pem --name cert.pem && chmod 0400 /etc/credstore/cloudflared-cert.pem'
      cat /home/nixos/.cloudflared/f20deb47-b7df-4505-b644-2b282bae1a75.json | ssh ${rpi5} \
        'mkdir -p /etc/credstore && systemd-creds encrypt - /etc/credstore/cloudflared-rpi-foo.bar --name credentials.json && chmod 0400 /etc/credstore/cloudflared-rpi-foo.bar'
    '';
  };
}

  services.cloudflared = {
    enable = true;
    certificateFile = "/not/null"; # cloudflared requires this option, but we'll load the cert from systemd-creds instead
    tunnels = {
      "foo.bar" = {
        credentialsFile = "/not/null"; # cloudflared requires this option, but we'll load the credentials from systemd-creds instead
        default = "http_status:404";
        ingress = {
          "enclosed.foo.bar" = {
            service = "http://${enclosedHost}:${toString enclosedPort}";
          };
        };
      };
    };
  };

  systemd.services."cloudflared-tunnel-rpi-foo.bar" = {
    serviceConfig = {
      LoadCredential = lib.mkForce [ ];
      LoadCredentialEncrypted = [
        "credentials.json:/etc/credstore/cloudflared-rpi-foo.bar"
        "cert.pem:/etc/credstore/cloudflared-cert.pem"
      ];
    };
  };

• So I chose to utilize systemd-creds and override the service definition for that. This seems just the way it systemd services should acquire secrets. I'm considering slapping a 25USD TPM onto the PI to protect the decryption keys.

• My conlusion is that secretspec may not be in a really good position to cover these types of secrets as it stands. Maybe the story would be different with a systemd-creds provider in secretspec so that you could have a system wide secretspec.toml - but then, it also would also flood the zone with unrelated secrets.

• Maybe, for NixOS, sticking with systemd-creds and ssh pipes is simply the best and most transparent option.

• One could imagine integrating secretspec on the devenv script's side to aquire secrets from there instead of a file.

• This is what I now ultimately settled on:

    deploy-creds-rpi5.exec =
      let
        emplaceSecret =
          file: name:
          "mkdir -p /etc/credstore && systemd-creds encrypt - /etc/credstore/${file} --name ${name} && chmod 0400 /etc/credstore/${file}";
      in
      ''
        secretspec run -- bash -lc '
          set -e

          cat "$CLOUDFLARE_CERT_PEM" | ssh ${rpi5} "${emplaceSecret "cloudflared-cert.pem" "cert.pem"}"

          cat "$CLOUDFLARE_CREDJSON" | ssh ${rpi5} "${emplaceSecret "cloudflared-rpi-foo.bar" "credentials.json"}"

          printf "ADMIN_TOKEN=%s\n" "$VAULTWARDEN_ADMIN_TOKEN" | ssh ${rpi5} "${emplaceSecret "vaultwarden-admin-token" "vaultwarden.env"}"
        '
      '';

and they are acquired by secretspec nicely before on deployment...

• I ordered a rpi TPM from Reichelt Electronics for 25 Eur to securely encrypt at rest on the target host.

• This approach forces me to harden each nixos service that I deploy with something like the above LoadCredentialEncrypted= approach.

[blaggacao]

x-ref: #41

[blaggacao]

In order to account for optional secrets, I updated to:

    deploy-creds-rpi5.exec =
      let
        emplaceSecret =
          file: name:
          "mkdir -p /etc/credstore && systemd-creds encrypt - /etc/credstore/${file} --name ${name} && chmod 0400 /etc/credstore/${file}";
      in
      ''
        secretspec run -- bash -lc '
          set -e

          upload_file_secret() {
            local src="$1"; local file="$2"; local name="$3"
            if [ -n "''${src:-}" ]; then
              cat "$src" | ssh ${rpi5} "${emplaceSecret "$file" "$name"}"
            fi
          }

          upload_value_secret() {
            local value="$1"; local file="$2"; local name="$3"; local fmt="${4:-%s}"
            if [ -n "''${value:-}" ]; then
              printf "$fmt" "$value" | ssh ${rpi5} "${emplaceSecret "$file" "$name"}"
            fi
          }

          upload_file_secret "$CLOUDFLARE_CERT_PEM" "cloudflared-cert.pem" "cert.pem"
          upload_file_secret "$CLOUDFLARE_CREDJSON" "cloudflared-rpi-fee.bar" "credentials.json"
          upload_file_secret "$MAXMIND_LICENSE_KEY_FILE" "maxmind-license-key" "maxmind-license-key"
          upload_value_secret "$POCKET_ID_ENCRYPTION_KEY" "pocket-id-encryption-key" "pocket-id-encryption-key"
          upload_value_secret "$VAULTWARDEN_ADMIN_TOKEN" "vaultwarden.env" "vaultwarden-admin-token" "ADMIN_TOKEN=%s\n"
        '
      '';
  };