CrashLoopBackOff on Kubernetes: DuckDB WAL replay fails on fresh database initialization

[marcochiodo]

Environment

• Liwan version: 1.4 (image ghcr.io/explodingcamera/liwan:1.4, SHA sha256:80c696af40b84abb1a008ee307e297a7722c178e654ab88dee5e953e2f93f661)
• DuckDB version: 1.5.0 (bundled in liwan 1.4)
• Kubernetes: k3s v1.33.6+k3s1
• Container runtime: containerd 2.1.5-k3s1.33
• Host OS: Debian GNU/Linux 13 (trixie)
• Kernel: 6.12.38+deb13-cloud-amd64
• CPU: AMD EPYC with AVX2 support
• Filesystem: ext4
• Storage: local-path provisioner (k3s default), bind-mount into pod

Problem

Liwan crashes immediately (<1 second) on every startup inside a Kubernetes pod,
even on a completely clean /data directory. The error is:

WARN liwan::app::db: Failed to create DuckDB connection. If you've just upgraded
to Liwan 1.2, please downgrade to version 1.1.1 first, start and stop the server,
and then upgrade to 1.2 again.
Error: Failed to create DuckDB connection: INTERNAL Error: Failure while replaying
WAL file "/data/liwan-events.duckdb.wal": Calling DatabaseManager::GetDefaultDatabase
with no default database set
This error signals an assertion failure within DuckDB.

DuckDB creates the WAL file on fresh init, then immediately fails to replay it as
part of the initial checkpoint. The result is a CrashLoopBackOff with a deterministic
4494-byte WAL file created every time.

What We Tested

Scenario	Result
ctr run with overlay filesystem (no volume)	✅ Works
ctr run with bind-mount to the same PVC path (clean dir)	✅ Works
Kubernetes pod, clean PVC, default security context	❌ Crashes
Kubernetes pod + seccompProfile: Unconfined	❌ Crashes
Kubernetes pod + capabilities: add: ["ALL"]	❌ Crashes
Kubernetes pod + runAsUser: 0	❌ Crashes
Kubernetes pod + memory limit 512Mi	❌ Crashes
Kubernetes pod + LIWAN_DUCKDB_THREADS=1	❌ Crashes

The crash is deterministic and always produces the same 4494-byte WAL file before failing.

Additional Findings

• LIWAN_LISTEN cannot be set via environment variable — causes Error: duplicate field 'listen', suggesting the distroless image includes a TOML config with listen already set.
• LIWAN_BASE_URL must point to a resolvable domain before the pod starts, otherwise liwan
  panics with failed to lookup address information: Name does not resolve (src/web/mod.rs:143).
• DuckDB 1.5.1 release notes mention a fix for "WAL corruption related to
  MarkBlockAsCheckpointed on fresh database initialization", which matches this exact failure.
  Liwan 1.4 bundles DuckDB 1.5.0.

Suspected Root Cause

DuckDB 1.5.0 bug in fresh database initialization, fixed in DuckDB 1.5.1. The bug surfaces
specifically in the Kubernetes pod execution context (possibly related to cgroup constraints or
subtle runtime differences vs. bare ctr run). Upgrading the bundled DuckDB to ≥1.5.1 would
likely fix the issue.

Workaround

None found. The application is currently not usable on Kubernetes with liwan 1.4.

[explodingcamera]

Hi, thanks for report this! I sadly can't reproduce your issues with k3s on my local machine, but I'll probably push an update with duckdb 1.5.1 tomorrow.

The duplicate field is most likely due to both port and listen being set somewhere in a config or env, I'll also include a fix to allow this in the next update by prioritizing listen.

As for the BASE_URL error, do you maybe have listen set to a domain + port? The failed to lookup address information: Name does not resolve should be coming from a listen address that can't be resolved to an ip address.

[marcochiodo]

Thanks for the quick response!

To clarify the BASE_URL panic: we did not have LIWAN_LISTEN set at all when that error occurred. Our only environment variable was LIWAN_BASE_URL=mydomain.com. The listen variable was added later in an attempt to explicitly bind to 0.0.0.0:9042 — which is when we hit the duplicate field error.

So it seems like liwan may be using the hostname from LIWAN_BASE_URL as the listen address, or the config embedded in the distroless image has a non-standard default for listen that references the base URL. Either way, the panic at src/web/mod.rs:143 was triggered purely by LIWAN_BASE_URL being set, with no explicit listen configuration on our side.

Looking forward to the DuckDB 1.5.1 update — we'll test it as soon as it's out!

[marcochiodo]

Would you be able to push a new image tag to GHCR with the DuckDB 1.5.1 fix before the official release? Even a 1.4.1 or edge tag would be enough for us to test it. Thanks!

[explodingcamera]

@marcochiodo I just pushed a new image if you want to try it out (edge/1.4.1-rc.0)

[marcochiodo]

Just tested edge/1.4.1-rc.0 — great progress!

• The DuckDB WAL issue on fresh initialization is fixed ✅
• The LIWAN_LISTEN duplicate field error is fixed ✅ (now correctly logs a warning and takes precedence over port)

One remaining issue: the pod still crashes on startup without explicitly setting LIWAN_LISTEN=0.0.0.0:9042. The default listen address from the embedded config resolves to the node's external IP (193.203.15.59), which can't be bound from inside a container. Setting LIWAN_LISTEN=0.0.0.0:9042 via environment variable fixes it.

Overall liwan is now working on Kubernetes. Thanks for the quick turnaround!