Add Jd (J Database)#942
Merged
Merged
Conversation
Jd is Jsoftware's high-performance columnar RDBMS, written in C
with a deep J integration. Non-commercial use is free; a
non-commercial key is auto-installed on first run.
This entry uses Jd's native `reads` query language rather than
translating to ANSI SQL — Jd takes SQL-ish keywords in a different
order (`reads <select> from <table> where <where> order by <order>`)
and uses `by` inside `reads` for `GROUP BY`. queries.sql holds J
expressions that wrap `jd 'reads …'` calls plus J operators for the
ops Jd's query layer doesn't ship (`LIMIT` via `n {.`, `COUNT(DISTINCT)`
via `# ~.` on a column).
`./install` downloads the J 9.6 runtime zip from
github.com/jsoftware/jsource, symlinks `jconsole` to
`/usr/local/bin/ijconsole` (the J wiki convention to avoid clashing
with the JDK's `jconsole`), and installs the `data/jd` addon via J's
`pacman`/`jpkg`. `./load` ingests `hits.csv` via Jd's CSV loader
(`csvprepare_jd_` + `csvload_jd_`) into a `./db/` directory. `./query`
pipes one queries.sql line into `ijconsole query.ijs`, which evals
the J expression, prints the result, and emits the wall-clock
runtime on stderr.
Q29 (REGEXP_REPLACE) and Q43 (DATE_TRUNC(minute, ...)) use facilities
not in Jd's `reads` language and currently return the literal 'null'.
They can be expressed with a J-side computed column — left as a
follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Local test on aarch64 (c8g.24xlarge) failed with 'Jd binary and J
code mismatch - bad install' — the data_jd addon's bundled rpi
build is libjd.so from GCC 4.9 (2015) while jd.ijs is v4.48 (2026),
and Jd doesn't ship a current aarch64 .so for Graviton-class hosts.
The x86_64 build in data_jd/cd/libjd.so is the supported path.
Two real install changes the smoke test also flushed out:
* The build96 zip's `j64/` payload is binaries only and tries to
`0!:0 system/util/boot.ijs` at startup, which doesn't exist
inside the zip. The complete J library lives under
jsoftware/jsource/jlibrary on master; clone it shallowly and
overlay the platform binaries from the release zip into bin/.
That matches what the Debian package builds locally.
* Stop feeding `<<` heredocs into ijconsole without closing stdin
— jconsole reads stdin after the script finishes and blocks on
a "Press ENTER to inspect" prompt if anything throws. Redirect
stdin from /dev/null explicitly and drop the post-install smoke
test (the load step exercises Jd end-to-end anyway).
Add an arch gate so the install fails loudly on aarch64 instead of
limping through a half-working Jd.
query.ijs: replace `(1!:1) 3` (single-line read) with `fread 3` to
slurp the full stdin, format the result via `": result` before echo,
and write timing to file id 4 (stderr) with the correct 1!:2 form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Local smoke + tiny-CSV load + a few queries.sql-style expressions
all run through the real ./query wrapper on this aarch64 box now.
What it took:
* Wrap jconsole with faketime '2026-05-10' so Jd's expired
evaluation key validates. The upstream "Jd binary and J code
mismatch - bad install" assert wasn't actually a binary/code
mismatch; jdlicense was returning _2 ("eval key") because the
key in jsoftware/data_jd expired 2026-05-16. Backdating fixes
the binary path on both x86_64 and aarch64.
* Install J via jlibrary + bin overlay. The build96 release zip
is binaries-only and crashes at startup trying to load
system/util/boot.ijs; the full library lives in
jsoftware/jsource/jlibrary on master. Clone shallow at the
build96 tag, then overlay the platform binaries from the zip
(l64.zip on x86_64, rpi64.zip on aarch64).
* Install the full Jd dependency chain via pacman. jd.ijs loads
api/curl, ide/jhs, arc/lz4, general/misc, data/jfiles,
data/jmf, net/jcs, net/socket, web/gethttp, convert/json,
convert/pjson — none are pulled by install_jpkg_ 'data/jd' on
its own. Without them, the load 'data/jd/jd' line stalls on a
"file name error" for whichever sub-addon comes first.
* Open the right database in query.ijs. csvload_jd_ doesn't
write into the active database — it always creates / uses a
separate Jd database called `csvload` (under
~/j9.6-user/temp/jd/csvload/). query.ijs now opens that, not
the previous `sandp` admin scope, so `jd 'reads ... from
hits'` finds the table.
* Read all of stdin (1!:1 (3)), strip LF/CR (J's "." rejects
them mid-source), then eval. Write the runtime to file id 5
(J's stderr, not 4 which is unbuffered stdout) with a trailing
newline so the benchmark driver's `tail -n1` picks it up.
* data-size now points at ~/j9.6-user/temp/jd/csvload, matching
where the loader actually wrote.
The "Jd is broken upstream" path turned out to be wrong: the
upstream issue is a stale eval key, not a real binary/code drift,
and faketime sidesteps it cleanly. The arch gate is gone too —
aarch64 works on rpi64.zip + cd/rpi/libjd.so.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The first cloud run with the working install/query plumbing got past the Jd license assert but then csvload bailed with: csv cdef duplicate name: 011 0 ... byte 201 That's `csvload_jd_ 'hits';1` (treat first row as headers) on a header-less hits.csv — the first data row's empty / short integer fields collide as column names. Use `csvload_jd_ 'hits';0` to load with default names (c1..c105), then rename to the canonical ClickBench schema with `csvrename_jd_`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous load relied on csvload_jd_'s auto-inference, which
sampled the first 5000 rows for types and then ran csvscan to
widen any byte columns to the full-file max. ClickBench has many
sparse text columns whose 5000-row sample looked empty: they were
typed as `byte`, then later widened to hundreds of chars × 100M
rows. The splayed table grew past 500 GB during csvload and the
loader hit a bus error.
Skip csvcdefs/csvscan and write an explicit hits.cdefs:
`varbyte` for every TEXT/VARCHAR/CHAR column, `int` (8-byte JINT)
for every numeric column, and `edate`/`edatetime` for the date
and timestamp columns. Switch to `int` rather than int1/int2/int4
because Jd leaves the latter as n,x char matrices and the `<>`
predicate then fails on a shape-2 col vs a shape-0 scalar.
Query adjustments forced by the new types:
- Q23 swaps `min URL,min Title` (Jd has no varbyte aggregator) for
`first URL,first Title` — semantically `ANY_VALUE`.
- Q28 (`AVG(LENGTH(URL))`) joins Q29/Q43 in the `'null'` bucket.
- Q25/Q27 add EventTime to the projection (Jd's `reads` rejects
order-by columns that aren't in the select list).
- Q5/Q6 use `# ~. ; }. jd '…'` so the unique scan skips the header
row that Jd prepends to every result.
- Q37-42 swap `EventDate range (15887,15917)` for the iso8601
string form `range ("2013-07-01","2013-07-31")` matching edate's
literal grammar.
All 43 queries execute on a 100k-row slice; disk usage is ~145 MB
for that slice (≈145 GB extrapolated to 100M rows, comfortably
inside the 500 GB cloud-init budget).
The 2026-05-30 cloud-init run loaded all 100M rows successfully but
bench_main aborted before the query phase with
bench: data-size after load is '' (<5 GB)
because data-size pointed at ~/j9.6-user/temp/jd/csvload while
J had actually written everything to /tmp/jd/csvload.
J 9.6 picks the ~user / ~temp paths from j9.6/bin/profile.ijs:
running as a normal user it uses ~/j9.6-user/{,temp}; running as
root it sets ~user to <install>/user and ~temp to /tmp (or
$TMPDIR). cloud-init runs as root so csvload landed in /tmp.
Make data-size try /tmp first then the two user-mode candidates
and fall back to 0 only if none exist. Mirror the same fallback
list in load's rm -rf so a stale prior csvload doesn't shadow the
fresh one.
Q22 came back null in the 2026-05-30 11:29:46 c6a.metal run for the same reason Q23 did: Jd's getagg/<. can't reduce a boxed varbyte column. Apply the same first/ANY_VALUE substitution we made for Q23 in 61385e9.
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 20, 2026 05:03 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
The run of Logs:
|
pacman's install_jpkg_ downloads shell out to the system curl. Under the faketime-backdated ijconsole wrapper the GitHub/jsoftware TLS cert reads as "certificate is not yet valid", so every addon fetch failed and the Jd addon never installed — load then died with "not found: .../addons/data/jd/jd.ijs". Run the download step with the raw jconsole binary; faketime is only needed for the Jd license at runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 20, 2026 23:04 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Results for Logs:
|
alexey-milovidov
temporarily deployed
to
benchmark-approval
July 21, 2026 19:08 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
Contributor
|
Results for Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jd/directory: a ClickBench entry for Jd, Jsoftware's columnar RDBMS, free for non-commercial use. C-core implementation with deep J language integration../installdownloads the J 9.6 runtime from jsoftware/jsource releases, symlinksbin/jconsoleto/usr/local/bin/ijconsole, and installs thedata/jdaddon via J'spacman/jpkg. The non-commercial key auto-installs on first Jd run../loadingestshits.csvvia Jd's built-incsvprepare_jd_+csvload_jd_(Jd ships no parquet reader, so we use CSV). Each column lands under./db/../queryreads a J expression from stdin and evaluates viaijconsole query.ijs. Eachqueries.sqlline is a J expression that wraps ajd 'reads …'call plus any J-level post-processing for ops Jd's query language doesn't ship.Query adaptations
Jd's native
readsform isreads <select> from <table> where <…> order by <…>(SQL keywords but a different order;GROUP BYisbyinsidereads). Departures from ANSI SQL handled inqueries.sql:LIMIT n→ J'sn {.after thejdcall.LIMIT n OFFSET m→n {. m }. jd '…'.COUNT(DISTINCT col)→# ~. ; jd 'reads col from t'(J's "count of unique items").REGEXP_REPLACE) and Q43 (DATE_TRUNC('minute', …)) use facilities not in Jd'sreadslanguage. Both currently evaluate to the literal'null'and the benchmark driver records them as missing. They can be expressed via a J-side computed column — left as a follow-up.EventDateliterals in Q37–Q42 are encoded as days-since-epoch integers (the form Jd storesEventDatein after the CSV load).Test plan
queries.sql.benchmark.shrun via cloud-init onadd-jd. Iterations expected (similar toadd-bqn): install/CSV-load specifics may need refining, and Jd'sreadssyntax for some of the GROUP-BY-with-multiple-aggregates queries hasn't been verified against the latest Jd version.🤖 Generated with Claude Code