Add read-only support for PostgreSQL array columns by bilby91 · Pull Request #3402 · Azure/data-api-builder

bilby91 · 2026-03-30T04:01:16Z

Summary

Add read-only support for PostgreSQL array columns (int[], text[], boolean[], bigint[], etc.) which previously caused initialization failures
Array columns are exposed as GraphQL list types ([Int], [String], etc.) and returned as JSON arrays in REST/MCP responses
Array columns are marked read-only and excluded from create/update mutation inputs until write support is implemented

Changes

Core plumbing (database-agnostic):

ColumnDefinition: new IsArrayType and ElementSystemType properties
SqlMetadataProvider: detect array types (System.Array) during schema introspection
SchemaConverter: unwrap array element types and generate ListTypeNode for GraphQL fields
EdmModelBuilder: represent array columns as EdmCollectionTypeReference in OData model
TypeHelper: fallback handling for unresolved System.Array type

PostgreSQL-specific:

PostgreSqlMetadataProvider: override PopulateColumnDefinitionWithHasDefaultAndDbType to resolve element types from information_schema udt_name (_int4→int, _text→string, etc.)

Tests:

14 new unit tests (array→ListTypeNode generation, nullability, AutoGenerated directive, byte[] exclusion)
3 new PostgreSQL e2e tests (query by PK, null arrays, multi-row queries)
Test schema and config for array_type_table entity

Limitations

Read-only: mutations on array columns are blocked (marked as auto-generated/read-only)
No OData $filter: filtering on array columns is not supported (no OData array operators)
PostgreSQL only: other databases don't have native array types; generic plumbing is in place for future use
1D arrays only: multi-dimensional arrays are not tested/supported

Test plan

Unit tests pass (1325/1325, 0 failures)
PostgreSQL e2e tests pass (3/3 array tests, 711/711 non-preexisting PG tests)
CI pipeline validation (PostgreSQL Docker)

Copilot

Pull request overview

Adds read-only support for PostgreSQL array columns in DAB so schema introspection and runtime initialization no longer fail when encountering int[], text[], boolean[], bigint[], etc. The change threads array-awareness through metadata, GraphQL schema generation, and the EDM model while keeping arrays excluded from mutation inputs.

Changes:

Detect PostgreSQL array columns during metadata introspection, resolve array element CLR types, and mark array columns read-only.
Expose array columns as GraphQL list types and represent them as EDM collection properties.
Add/extend unit + PostgreSQL e2e test coverage plus schema/config updates for a new array_type_table test entity.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Config/DatabasePrimitives/DatabaseObject.cs	Extends `ColumnDefinition` with `IsArrayType`/`ElementSystemType` to carry array metadata.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs	Detects array-typed columns from provider schema and marks them read-only.
src/Core/Services/MetadataProviders/PostgreSqlMetadataProvider.cs	Resolves PostgreSQL array element types via `information_schema` (`udt_name`) and updates `ColumnDefinition`.
src/Service.GraphQLBuilder/Sql/SchemaConverter.cs	Generates GraphQL fields as `ListTypeNode` for array columns and adds array-aware system type mapping.
src/Core/Parsers/EdmModelBuilder.cs	Emits EDM collection properties for array columns to support OData model building.
src/Core/Services/TypeHelper.cs	Adds fallback handling for abstract `System.Array` in EDM primitive type mapping.
src/Service.Tests/GraphQLBuilder/Sql/SchemaConverterTests.cs	Adds unit tests for array→GraphQL list type generation, nullability, directives, and `byte[]` exclusion.
src/Service.Tests/SqlTests/GraphQLSupportedTypesTests/PostgreSqlGQLArrayTypesTests.cs	Adds PostgreSQL e2e GraphQL tests validating array values and null handling.
src/Service.Tests/DatabaseSchema-PostgreSql.sql	Adds `array_type_table` DDL + seed data for PostgreSQL integration tests.
src/Service.Tests/dab-config.PostgreSql.json	Adds `ArrayType` entity configuration (read-only permissions) for PostgreSQL test runs.
src/Service.Tests/UnitTests/SerializationDeserializationTests.cs	Updates serialization test expectations to account for added `ColumnDefinition` fields.

src/Service.Tests/SqlTests/GraphQLSupportedTypesTests/PostgreSqlGQLArrayTypesTests.cs

aaronburtle · 2026-03-31T15:29:34Z

/azp run

azure-pipelines · 2026-03-31T15:30:04Z

Azure Pipelines successfully started running 6 pipeline(s).

bilby91 · 2026-03-31T15:39:03Z

@aaronburtle Thanks for approving CI. I'll take a look at the failing tests and address them.

bilby91 · 2026-03-31T17:35:53Z

@aaronburtle I think fixes should be in place.

aaronburtle · 2026-03-31T18:46:01Z

/azp run

azure-pipelines · 2026-03-31T18:46:29Z

Azure Pipelines successfully started running 6 pipeline(s).

src/Core/Services/MetadataProviders/PostgreSqlMetadataProvider.cs

aaronburtle · 2026-03-31T20:13:31Z

Looks like we are missing tests on the REST side of things. Should add some simple tests to validate REST works with array types, eg. GET /api/ArrayType and GET /api/ArrayType/id/1

src/Service.GraphQLBuilder/Queries/InputTypeBuilder.cs

aaronburtle · 2026-03-31T20:25:20Z

The PG_SQL Integration test failures are from mismatched configs, need to add the new section to ConfigurationTests.TestReadingRuntimeConfigForPostgreSql.verified.txt

PostgreSQL array columns (int[], text[], boolean[], bigint[], etc.) were previously unsupported and caused initialization failures. This change adds read-only support so array columns are exposed as list types in GraphQL ([Int], [String], etc.) and as JSON arrays in REST responses. Array columns are marked read-only and excluded from mutation input types until write support is implemented. The implementation adds generic array plumbing in the shared SQL layers and PostgreSQL-specific element type resolution via information_schema udt_name mapping. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add ArrayType entity to postgresql-commands.txt so CI's config generator includes it, and skip list-type fields in InputTypeBuilder filter/orderby generation to prevent array columns from being incorrectly processed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add _json, _jsonb, _money to PostgreSQL array type mapping dictionary - Add json[], jsonb[], money[] columns to array_type_table test schema - Add REST integration tests for array type endpoints (list, by PK, nulls) - Update GraphQL array tests to cover new array column types - Update PostgreSQL snapshot to include ArrayType entity Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

bilby91 · 2026-03-31T22:14:27Z

I handle most of the stuff already. Need to replicate the Cosmos issue locally and get it fixed. Will update when I have that aspect ready.

…orderby The previous change to skip list-type fields was too broad — it excluded all list types, which broke Cosmos DB nested object array filtering (e.g., additionalAttributes, moons). Narrowed the condition to only skip scalar list types (IsListType && IsBuiltInType) so PostgreSQL array columns (int[], text[]) are still excluded while Cosmos nested arrays remain filterable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

bilby91 · 2026-04-01T12:40:01Z

@aaronburtle I think CI should pass now :)

aaronburtle · 2026-04-02T09:40:25Z

/azp run

azure-pipelines · 2026-04-02T09:40:32Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

aaronburtle · 2026-04-02T09:41:13Z

/azp run

azure-pipelines · 2026-04-02T09:41:38Z

Azure Pipelines successfully started running 6 pipeline(s).

aaronburtle · 2026-04-03T09:43:52Z

Great progress on the fix @bilby91! The IsListType() && IsBuiltInType() approach fixed the Cosmos nested object array tests, but 2 Cosmos tests are still failing:

TestQueryFilterContains_WithStringArray
TestQueryFilterNotContains_WithStringArray

The issue is that IsBuiltInType() unwraps the list to check the inner type, so [String] (Cosmos tags field) returns true and gets skipped but Cosmos legitimately filters on [String] via contains or notContains.

I think the fix here is to use the @autoGenerated directive instead of IsBuiltInType as the discriminator. PostgreSQL array columns always carry this directive (because IsReadOnly = true), while Cosmos schema fields never do.

So in InputTypeBuilder.cs, if we replace both occurrences of:

if (field.Type.IsListType() && IsBuiltInType(field.Type))
{
    continue;
}

with

if (field.Type.IsListType() &&
    field.Directives.Any(d => d.Name.Value == AutoGeneratedDirectiveType.DirectiveName))
{
    continue;
}

I think that will solve the problem. Thanks again for working on this!

@autogenerated

The previous fix used IsListType() && IsBuiltInType() to skip PostgreSQL array columns from filter generation, but this also excluded Cosmos scalar arrays (e.g., tags: [String]) which support ARRAY_CONTAINS filtering. Added @autogenerated directive check so only SQL read-only array columns are skipped — Cosmos fields lack this directive. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

bilby91 · 2026-04-03T12:31:50Z

@aaronburtle Thanks a lot for you help and guidance during all the PR :)

I had identified the issue last night and forgot to push it. I had some issues with CosmoDb simulator crushing in my environment so I was not able to run the entire suite. I was able to confirm the relevant tests are passing.

Thanks again!

aaronburtle · 2026-04-03T20:45:45Z

/azp run

azure-pipelines · 2026-04-03T20:46:11Z

Azure Pipelines successfully started running 6 pipeline(s).

bilby91 · 2026-04-03T21:27:13Z

@aaronburtle All green :)

Thanks again.

aaronburtle · 2026-04-06T15:19:45Z

@aaronburtle All green :)

Thanks again.

Thanks for the quick iterations, taking another pass and will need another reviewer, hopefully will get this merged soon!

RubenCerna2079

So I am half-way through the PR, will try to finish it by tomorrow

RubenCerna2079 · 2026-04-07T00:24:14Z

src/Core/Parsers/EdmModelBuilder.cs

+                        {
+                            // Array columns are represented as EDM collection types (e.g., Collection(Edm.Int32) for int[]).
+                            sqlMetadataProvider.TryGetExposedColumnName(entityAndDbObject.Key, column, out exposedColumnName!);
+                            EdmPrimitiveTypeReference elementTypeRef = new(EdmCoreModel.Instance.GetPrimitiveType(type), isNullable: true);


Is there a reason why we want isNullable to be true?

isNullable: true is used because PostgreSQL arrays can contain NULL elements ('{1,NULL,3}'::int[] is valid). The element type reference should reflect this.

It's also consistent with how non-PK columns are handled in the same method — line 140 uses isNullable: true for regular structural properties. Only primary keys use isNullable: false.

I verified that changing to isNullable: false doesn't cause a runtime failure — DAB's OData pipeline doesn't enforce element-level nullability in collections at serialization time. So this is about correctness of the EDM model representation rather than a breaking behavior difference.

I added a test row (id=4) with NULL elements inside arrays ('{1,NULL,3}', '{hello,NULL,world}', etc.) and two new tests that query it:

QueryArrayColumnsWithNullElements (GraphQL)

GetArrayTypeWithNullElementsInsideArrays (REST)

Both pass with isNullable: true and also with false, confirming the pipeline handles both.

src/Core/Parsers/EdmModelBuilder.cs

…mment Add test row (id=4) with NULL elements inside arrays (e.g., '{1,NULL,3}') and two new tests (GraphQL and REST) that verify null elements within arrays are handled correctly. Update the EdmModelBuilder block comment to document the array column branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

aaronburtle

LGTM! Thanks again for so diligently working on this one @bilby91 <3

bilby91 marked this pull request as ready for review March 30, 2026 04:03

Copilot AI review requested due to automatic review settings March 30, 2026 04:03

bilby91 requested review from Alekhya-Polavarapu, Aniruddh25, JerryNixon, RubenCerna2079, aaronburtle, anushakolan, rusamant, sourabh1007, souvikghosh04, stuartpa and vadeveka as code owners March 30, 2026 04:03

Copilot started reviewing on behalf of bilby91 March 30, 2026 04:04 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

bilby91 force-pushed the feature/postgresql-array-columns-readonly branch from 5bb9456 to 29833c7 Compare March 30, 2026 04:19

aaronburtle self-assigned this Mar 31, 2026

aaronburtle reviewed Mar 31, 2026

View reviewed changes

src/Core/Services/MetadataProviders/PostgreSqlMetadataProvider.cs Show resolved Hide resolved

aaronburtle reviewed Mar 31, 2026

View reviewed changes

src/Service.GraphQLBuilder/Queries/InputTypeBuilder.cs Outdated Show resolved Hide resolved

bilby91 and others added 3 commits March 31, 2026 17:49

bilby91 force-pushed the feature/postgresql-array-columns-readonly branch from b9e3435 to 684ced5 Compare March 31, 2026 22:14

Merge branch 'main' into feature/postgresql-array-columns-readonly

3d84621

RubenCerna2079 self-assigned this Apr 2, 2026

Merge branch 'main' into feature/postgresql-array-columns-readonly

8928278

RubenCerna2079 reviewed Apr 7, 2026

View reviewed changes

aaronburtle approved these changes Apr 7, 2026

View reviewed changes

aaronburtle added the pgsql an issue thats specific to pgsql label Apr 7, 2026

aaronburtle added this to Data API builder Apr 7, 2026

github-project-automation bot moved this to Todo in Data API builder Apr 7, 2026

aaronburtle moved this from Todo to Review In Progress in Data API builder Apr 7, 2026

aaronburtle added this to the April 2026 milestone Apr 7, 2026

Conversation

bilby91 commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Limitations

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronburtle commented Mar 31, 2026

Uh oh!

azure-pipelines bot commented Mar 31, 2026

Uh oh!

bilby91 commented Mar 31, 2026

Uh oh!

bilby91 commented Mar 31, 2026

Uh oh!

aaronburtle commented Mar 31, 2026

Uh oh!

azure-pipelines bot commented Mar 31, 2026

Uh oh!

Uh oh!

aaronburtle commented Mar 31, 2026

Uh oh!

Uh oh!

aaronburtle commented Mar 31, 2026

Uh oh!

bilby91 commented Mar 31, 2026

Uh oh!

bilby91 commented Apr 1, 2026

Uh oh!

aaronburtle commented Apr 2, 2026

Uh oh!

azure-pipelines bot commented Apr 2, 2026

Uh oh!

aaronburtle commented Apr 2, 2026

Uh oh!

azure-pipelines bot commented Apr 2, 2026

Uh oh!

aaronburtle commented Apr 3, 2026

Uh oh!

bilby91 commented Apr 3, 2026

Uh oh!

aaronburtle commented Apr 3, 2026

Uh oh!

azure-pipelines bot commented Apr 3, 2026

Uh oh!

bilby91 commented Apr 3, 2026

Uh oh!

aaronburtle commented Apr 6, 2026

Uh oh!

RubenCerna2079 left a comment

Choose a reason for hiding this comment

Uh oh!

RubenCerna2079 Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

bilby91 Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aaronburtle left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bilby91 commented Mar 30, 2026 •

edited

Loading