CrateDB integration tests by amotl · Pull Request #192 · panodata/omniload

amotl · 2026-07-10T01:24:23Z

About

CrateDB is an emerging open-source data warehouse that can store data in the terabyte ranges and is largely compatible with PostgreSQL.

Details

An earlier attempt to validate CrateDB caused too much noise, because the database didn't support the DATE type. Now that this has changed recently, warehouse operations become actually viable.

Tests: Adjust software tests for CrateDB bruin-data/ingestr#284
PostgreSQL compatibility: AAUIW support for storing data using the Postgres DATE data type crate/crate#19414

@DHRUV6029: Thank you very much for submitting the relevant patch to CrateDB. 👍

Support PostgreSQL DATE as a storable column type crate/crate#19588

Status

The adjusted test suite currently needs to skip a few test cases that raise the venerable DestinationSchemaTampered exception of dlt. We think dlt-cratedb might need further adjustments beyond the patch recently submitted by @florinutz.

pytest --no-cov -k cratedb
87 passed, 71 skipped, 54 warnings in 124.39s (0:02:04)

References

Database result transformations for better PostgreSQL compatibility crate/dlt-cratedb#54

coderabbitai · 2026-07-10T01:24:30Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6e01ebfa-f033-4cc6-9f2e-24ec55a032ec

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cratedb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-11T22:05:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.36%. Comparing base (dc5aa50) to head (116d76d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #192      +/-   ##
==========================================
+ Coverage   54.33%   54.36%   +0.03%     
==========================================
  Files         203      203              
  Lines        9574     9574              
==========================================
+ Hits         5202     5205       +3     
+ Misses       4372     4369       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

amotl · 2026-07-11T22:09:25Z

  "pytest-cov<8",
  "pytest-xdist[psutil]<4",
-  "testcontainers[mysql,postgres]>=4.9.1,<4.15",
+  "testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",


Thank you very much for adding a native CrateDB adapter to Testcontainers for Python. With testcontainers 4.15, we will be able to use it in downstream packages swiftly.

Suggested change

"testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",

"testcontainers[cratedb,mysql,postgres]>=4.15,<4.16",

/cc @florinutz, @alexanderankin

amotl · 2026-07-11T22:10:45Z

  "dataclasses-json<0.7",
  "dlt<1.29",
-  "dlt-cratedb<0.2",
+  "dlt-cratedb @ git+https://github.com/zerotired/dlt-cratedb.git@fix-dynamic-install",


@florinutz: It looks like even when using the newest updates in the dlt-cratedb repository, including yours, the venerable DestinationSchemaTampered error is still present across the board.

amotl · 2026-07-11T22:15:15Z

+    if dest_uri.startswith("cratedb://"):
+        pytest.skip(
+            "Fails on CrateDB with `DestinationSchemaTampered`, see "
+            "https://github.com/crate/dlt-cratedb/issues/14"
+        )
+


@florinutz: It looks like even when using the newest updates in the dlt-cratedb repository, including yours, the venerable DestinationSchemaTampered error is still present across the board.

Like demonstrated here, we had to skip all test cases that are still failing on CrateDB. It would be so sweet to resolve them in one way or another, so CrateDB can become a premium-grade data warehouse within omniload and other applications using dlt. 🙏

read-the-docs-community · 2026-07-11T23:03:40Z

Documentation build overview

📚 omniload | 🛠️ Build #33547231 | 📁 Comparing 116d76d against latest (1beee34)

🔍 Preview build

4 files changed

+ getting-started/file-format-routing.html
+ supported-sources/cbor.html
+ supported-sources/msgpack.html
± supported-sources/index.html

amotl

Let me flag two more errors.

amotl · 2026-07-11T23:08:44Z

+    if dest_uri.startswith("cratedb://"):
+        pytest.skip(
+            'Fails on CrateDB with `"_id" conflicts with system column`, '
+            "see https://github.com/crate/dlt-cratedb/issues/19"
+        )
+


Currently we need to skip the MongoDB -> CrateDB integration tests, because of the venerable “_id” conflicts with system column pattern error that apparently still happens.

Problem importing from MongoDB crate/dlt-cratedb#19

/cc @florinutz

amotl · 2026-07-11T23:10:31Z

    def append_test(dest_uri, dynamodb, schema: str):
+
+        if dest_uri.startswith("cratedb://"):
+            pytest.skip(
+                "Fails on CrateDB with twice the amount of expected results: AssertionError: assert 6 == 3"
+            )
+


That is another type of error we haven't seen before with dlt and CrateDB. It needs to be investigated.

amotl · 2026-07-11T23:15:55Z

-    assert res[0] == ("message1",)
-    assert res[1] == ("message2",)
-    assert res[2] == ("message3",)
+    if dest_uri.startswith("cratedb://"):
+        messages_db = [res[0][0], res[1][0], res[2][0]]
+        assert "message1" in messages_db
+        assert "message2" in messages_db
+        assert "message3" in messages_db
+    else:
+        assert res[0] == ("message1",)
+        assert res[1] == ("message2",)
+        assert res[2] == ("message3",)


Note to self: Maybe adjust the database query instead to emit ordered results?

amotl force-pushed the cratedb branch 4 times, most recently from 49dd807 to 190f1af Compare July 11, 2026 21:56

amotl commented Jul 11, 2026

View reviewed changes

amotl added 2 commits July 12, 2026 01:01

Tests: Update to testcontainers 4.15

0ea82af

Tests: Add integration tests for CrateDB destination

116d76d

amotl force-pushed the cratedb branch from 190f1af to 116d76d Compare July 11, 2026 23:02

amotl commented Jul 11, 2026

View reviewed changes

This was referenced Jul 12, 2026

Tests: Probe pgrust instead of PostgreSQL #201

Draft

PostgreSQL compatibility: AAUIW support for storing data using the Postgres DATE data type crate/crate#19414

Closed

Chore: Fix packaging for dynamic installs from Git repository crate/dlt-cratedb#56

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CrateDB integration tests#192

CrateDB integration tests#192
amotl wants to merge 2 commits into
mainfrom
cratedb

amotl commented Jul 10, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

Review skipped

Uh oh!

codecov Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

amotl Jul 11, 2026 •

edited

Loading

Uh oh!

amotl Jul 11, 2026 •

edited

Loading

Uh oh!

amotl Jul 11, 2026 •

edited

Loading

Uh oh!

read-the-docs-community Bot commented Jul 11, 2026

Uh oh!

amotl left a comment

Uh oh!

amotl Jul 11, 2026

Uh oh!

amotl Jul 11, 2026 •

edited

Loading

Uh oh!

amotl Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	"testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",
	"testcontainers[cratedb,mysql,postgres]>=4.15,<4.16",

Uh oh!

Conversation

amotl commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Details

Status

References

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

codecov Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

read-the-docs-community Bot commented Jul 11, 2026

Documentation build overview

Uh oh!

amotl left a comment

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amotl commented Jul 10, 2026 •

edited

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading