Skip to content

CHANGELOG.md

Latest commit

 

History

History
302 lines (238 loc) · 22.1 KB

CHANGELOG.md

File metadata and controls

302 lines (238 loc) · 22.1 KB

Apache Polaris Changelog

This changelog is used to give users and contributors condensed information about the contents of Polaris releases. Entries are grouped in sections like Highlights or Upgrade notes, the provided sections can be adjusted as necessary. Empty sections will not end up in the release notes. Contributors are encouraged to incorporate CHANGELOG updates into their PRs when appropriate. Reviewers should be mindful of the impact of PRs and request adding CHANGELOG notes for breaking (!) changes and possibly other sections as appropriate.

Unreleased

Highlights

Upgrade notes

Breaking changes

  • The ExternalCatalogFactory interface has been renamed to FederatedCatalogFactory. Its createCatalog() and createGenericCatalog() method signatures have been extended to include a catalogProperties parameter of type Map<String, String> for passing through proxy and timeout settings to federated catalog HTTP clients.
  • The ConnectionCredentials.of() method now throws an exception when more than one expiration timestamp property is present in the credentials map. Only a single expiration timestamp is allowed per credentials bundle.

New Features

  • Added envFrom support in Helm chart.
  • Added summarize subcommand to Polaris CLI.
  • Added find and tables options to Polaris CLI.
  • Added support for multiple event listeners. Set polaris.event-listener.types to a comma-separated list of event listener types to enable multiple event listeners.
  • Added support for enabling only a subset of event types and event categories per event listener. Set polaris.event-listener.<name>.enabled-event-types or polaris.event-listener.<name>.enabled-event-categories to the list of event types or categories for the specified event listener to only consume the selected subset of events.
  • Added support for Apache Ranger as an external authorizer (Beta).

Changes

  • Removed unused PolarisAuthorizableOperation values: REVOKE_PRINCIPAL_GRANT_FROM_PRINCIPAL_ROLE, REVOKE_PRINCIPAL_ROLE_GRANT_FROM_PRINCIPAL_ROLE, LIST_GRANTS_ON_ROOT, ADD_PRINCIPAL_GRANT_TO_PRINCIPAL_ROLE, LIST_GRANTS_ON_PRINCIPAL, ADD_PRINCIPAL_ROLE_GRANT_TO_PRINCIPAL_ROLE, LIST_GRANTS_ON_PRINCIPAL_ROLE, ADD_CATALOG_ROLE_GRANT_TO_CATALOG_ROLE, REVOKE_CATALOG_ROLE_GRANT_FROM_CATALOG_ROLE, LIST_GRANTS_ON_CATALOG_ROLE, LIST_GRANTS_ON_CATALOG, LIST_GRANTS_ON_NAMESPACE, LIST_GRANTS_ON_TABLE, LIST_GRANTS_ON_VIEW.
  • Changed deprecated APIs in JUnit 5. This change will force downstream projects that pull in the Polaris test packages to adopt JUnit 6.

Deprecations

  • The configuration option polaris.event-listener.type is deprecated and will be removed later. Please use polaris.event-listener.types instead.

Fixes

Commits

[1.4.0]

Upgrade notes

  • The custom token-bucket based rate limiter has been replaced with Guava's rate limiter implementation.
  • The Helm chart now includes a JSON schema file for easy validation of values files. Because types are now validated, existing values files may need to be updated to match the new schema.

Breaking changes

  • The (Before/After)CommitViewEvent has been removed.
  • The (Before/After)CommitTableEvent has been removed.
  • The PolarisMetricsReporter.reportMetric() method signature has been extended to include a receivedTimestamp parameter of type java.time.Instant.
  • The ExternalCatalogFactory.createCatalog() and createGenericCatalog() method signatures have been extended to include a catalogProperties parameter of type Map<String, String> for passing through proxy and timeout settings to federated catalog HTTP clients.
  • Metrics reporting now requires the TABLE_READ_DATA privilege on the target table for read (scan) metrics and TABLE_WRITE_DATA for write (commit) metrics.
  • The REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE operation no longer requires the PRINCIPAL_ROLE_MANAGE_GRANTS_FOR_GRANTEE privilege. Only CATALOG_ROLE_MANAGE_GRANTS_ON_SECURABLE on the catalog role is now required, making revoke symmetric with assign. This allows catalog administrators to fully manage catalog role assignments without requiring elevated privileges on principal roles.

New Features

  • Added deploymentAnnotations support in Helm chart.
  • Added KMS properties (optional) to catalog storage config to enable S3 data encryption.
  • Added topologySpreadConstraints support in Helm chart.
  • Added priorityClassName support in Helm chart.
  • Added support for including principal name in subscoped credentials. INCLUDE_PRINCIPAL_NAME_IN_SUBSCOPED_CREDENTIAL (default: false) can be used to toggle this feature. If enabled, cached credentials issued to one principal will no longer be available for others.
  • Added per-field selection for AWS STS session tags in credential vending. The new SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL configuration accepts a comma-separated list of fields to include as session tags (supported: realm, catalog, namespace, table, principal, roles, trace_id). This replaces the previous INCLUDE_SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL and INCLUDE_TRACE_ID_IN_SESSION_TAGS boolean flags. Selecting only the fields needed helps avoid AWS STS packed policy size limit errors (2048 characters) caused by long namespace paths. Note: including trace_id disables credential caching, which may increase STS calls and latency.
  • Added support for Kubernetes Gateway API to the Helm Chart.
  • Added hierarchical flag to AzureStorageConfigInfo to allow more precise SAS token down-scoping in ADLS when the hierarchical namespace feature is enabled in Azure.
  • Relaxed client_id, client_secret regex/pattern validation on reset endpoint call
  • Added support for S3-compatible storage that does not have KMS (use kmsUavailable: true in catalog storage configuration)
  • Added support for storage-scoped AWS credentials, allowing different AWS credentials to be configured per named storage. Enable with the RESOLVE_CREDENTIALS_BY_STORAGE_NAME feature flag (default: false). Storage names can be set explicitly via the storageName field on storage configuration, or inferred from the first allowed location's host.
  • Added support for persisting Iceberg metrics (ScanReport, CommitReport) to the database. Enable by setting polaris.iceberg-metrics.reporting.type=persisting in configuration. Metrics tables are included in the main JDBC schema.
  • Added setup options to Polaris CLI.
  • Added CockroachDB as a supported database for the relational JDBC persistence backend. Set polaris.persistence.relational.jdbc.database-type to cockroachdb to get started.

Changes

  • The gcpServiceAccount configuration value now affects Polaris behavior (enables service account impersonation). This value was previously defined but unused. This change may affect existing deployments that have populated this property.
  • (Before/After)UpdateTableEvent is emitted for all table updates within a transaction.
  • Added KMS options to Polaris CLI.
  • Changed from Poetry to UV for Python package management.
  • Exclude KMS policies when KMS is not being used for S3.
  • Improved default KMS permission handling to better distinguish read-only and read-write access.

Deprecations

  • The configuration option polaris.rate-limiter.token-bucket.window is no longer supported and should be removed.
  • PolarisConfigurationStore has been deprecated for removal.

Fixes

  • Fixed error propagation in drop operations (dropTable, dropView, dropNamespace). Server errors now return appropriate HTTP status codes based on persistence result instead of always returning NotFound
  • Enable non-AWS STS role ARNs
  • Helm chart: fixed a bug that prevented CORS settings to be properly applied. A new setting cors.enabled has been introduced in the chart as part of the fix.

1.3.0-incubating

Highlights

  • Support for Iceberg Metrics Reporting has been introduced in Polaris. Out of the box, metrics can be printed to the logs by setting the org.apache.polaris.service.reporting logger level to INFO (it's set to OFF by default). Custom reporters can be implemented and configured to send metrics to external systems for further analysis and monitoring.
  • Support for Open Policy Agent (OPA) integration has been added to Polaris. This enables delegating authorization decisions to external policy decision points, allowing organizations to centralize policy management and implement complex authorization rules. OPA integration can be enabled by setting polaris.authorization.type=opa in the Polaris configuration.

Upgrade notes

  • The legacy management endpoints at /metrics and /healthcheck have been removed. Please use the standard management endpoints at /q/metrics and /q/health instead.

Breaking changes

  • The EclipseLink Persistence implementation has been completely removed.
  • The default request ID header name has changed from Polaris-Request-Id to X-Request-ID.

New Features

  • Added --no-sts flag to CLI to support S3-compatible storage systems that do not have Security Token Service available.
  • Support credential vending for federated catalogs. ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING (default: true) was added to toggle this feature.
  • Enhanced catalog federation with SigV4 authentication support, additional authentication types for credential vending, and location-based access restrictions to block credential vending for remote tables outside allowed location lists.

Changes

  • client.region is no longer considered a "credential" property (related to Iceberg REST Catalog API).
  • Relaxed the requirements for S3 storage's ARN to allow Polaris to connect to more non-AWS S3 storage appliances.
  • Added checksum to helm deployment so that it will restart when the configmap has changed.
  • Generic Table is no longer in beta and is generally-available.
  • Added Windows support for Python client.

1.2.0-incubating

Upgrade Notes

  • Amazon RDS plugin enabled, this allows polaris to connect to AWS Aurora PostgreSQL using IAM authentication.

Breaking changes

  • Python3.9 support will be dropped due to EOL

New Features

  • Added a finer grained authorization model for UpdateTable requests. Existing privileges continue to work for granting UpdateTable, such as TABLE_WRITE_PROPERTIES. However, you can now instead grant privileges just for specific operations, such as TABLE_ADD_SNAPSHOT
  • Added a Management API endpoint to reset principal credentials, controlled by the ENABLE_CREDENTIAL_RESET (default: true) feature flag.
  • The ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS was added to support sub-catalog (initially namespace and table) RBAC for federated catalogs. The setting can be configured on a per-catalog basis by setting the catalog property: polaris.config.enable-sub-catalog-rbac-for-federated-catalogs. The realm-level feature flag ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS (default: true) controls whether this functionality can be enabled or modified at the catalog level.
  • Added support for S3-compatible storage that does not have STS (use stsUnavailable: true in catalog storage configuration)
  • Python client: added support for custom realm and header
  • Python client: added support for policy management

Changes

  • The following APIs will now return the newly-created objects as part of the successful 201 response: createCatalog, createPrincipalRole, createCatalogRole.

Deprecations

  • The property polaris.active-roles-provider.type is deprecated and has no effect anymore.
  • The EclipseLink Persistence implementation has been deprecated since 1.0.0 and will be completely removed in 1.3.0 or in 2.0.0 (whichever happens earlier).
  • The legacy management endpoints at /metrics and /healthcheck have been deprecated in 1.2.0 and will be completely removed in 1.3.0 or in 2.0.0 (whichever happens earlier). Please use the standard management endpoints at /q/metrics and /q/health instead.

Fixes

  • Fixed incorrect Azure expires at field for the credentials refresh response, leading to client failure via #2633

1.1.0-incubating

Apache Polaris 1.1.0-incubating was released on September 19th, 2025.

  • Highlights
    • HMS Federation Support: Added support for Hive Metastore (HMS) federation, enabling integration with existing Hive metastores.
    • Modularized Federation: Introduced modularized federation architecture to support multiple catalog types and improve extensibility.
    • External Authentication: Added comprehensive support for external identity providers including Keycloak integration and Helm chart configuration options.
    • Python Client Distribution: The Python client is now packaged and distributed as a proper Python package for easier installation and usage.
    • Catalog Federation CLI: Extended the CLI with support for managing federated catalogs, making it easier to configure and operate catalog federation.
    • MinIO: Added MinIO integration support with comprehensive getting started documentation.
  • New features
    • Added Catalog configuration for S3 and STS endpoints. This also allows using non-AWS S3 implementations.
    • The realm-level feature flag ALLOW_SETTING_S3_ENDPOINTS (default: true) may be used to disable this functionality.
    • The IMPLICIT authentication type enables users to create federated catalogs without explicitly providing authentication parameters to Polaris. When the authentication type is set to IMPLICIT, the authentication parameters are picked from the environment or configuration files.
    • The DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED feature was added to support placing tables at locations that better optimize for object storage.
    • The LIST_PAGINATION_ENABLED (default: false) feature flag can be used to enable pagination in the Iceberg REST Catalog API.
    • The Helm chart now supports Pod Disruption Budgets (PDBs) for Polaris components. This allows users to define the minimum number of pods that must be available during voluntary disruptions, such as node maintenance.
    • Feature configuration PURGE_VIEW_METADATA_ON_DROP was added to allow dropping views without purging their metadata files.
    • Introduced S3 path-style access support for improved compatibility with S3-compatible storage systems.
    • Enhanced Python client with integration tests and improved error handling.
    • Introduced extensible pagination token implementation for better API performance.
    • Added support for s3a scheme in addition to existing S3 schemes.
    • Enhanced Helm chart with support for external authentication configuration and relational JDBC backend options.
    • Added comprehensive diagnostics and monitoring capabilities throughout the system.
    • Introduced bootstrap command options to specify custom schema files for database initialization.
    • Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required. The endpoint path is always returned when using vended credentials, but clients must enable the refresh-credentials flag for the desired storage provider.
    • Added a Management API endpoint to reset principal credentials, controlled by the ENABLE_CREDENTIAL_RESET (default: true) feature flag.
  • Changes
    • Polaris Management API clients must be prepared to deal with new attributes in AwsStorageConfigInfo objects.
    • S3 configuration property role-ARN is no longer mandatory.
  • Breaking changes
    • Helm chart: the default value of the authentication.tokenBroker.secret.symmetricKey.secretKey property has changed from symmetric.pem to symmetric.key.
    • For migrations from 1.0.x to 1.1.x, users using JDBC persistence and wanting to continue using v1 schema, must ensure that they, run following SQL statement under POLARIS_SCHEMA to make sure version table exists: 
      CREATE TABLE IF NOT EXISTS version (
   version_key TEXT PRIMARY KEY,
   version_value INTEGER NOT NULL
);
INSERT INTO version (version_key, version_value)
  VALUES ('version', 1)
ON CONFLICT (version_key) DO UPDATE
                        SET version_value = EXCLUDED.version_value;
COMMENT ON TABLE version IS 'the version of the JDBC schema in use';

ALTER TABLE polaris_schema.entities ADD COLUMN IF NOT EXISTS location_without_scheme TEXT;
      • Please don't enable OPTIMIZED_SIBLING_CHECK feature configuration, once the above SQL statements are run. As it may lead to incorrect behavior, due to missing data for location_without_scheme column.
  • Deprecations
    • The property polaris.active-roles-provider.type is deprecated for removal.
    • The ActiveRolesProvider interface is deprecated for removal.

1.0.1-incubating

Apache Polaris 1.0.1-incubating was released on August 16th, 2025. It’s a maintenance release on the 1.0.0 release fixing a couple of issues on the Helm Chart:

  • remove db-kind in Helm Chart
  • add relational-jdbc to helm

1.0.0-incubating

Apache Polaris 1.0.0-incubating was released on July 9th, 2025.

  • Highlights
    • First release ready for real-world workloads after the public beta 0.9.0
    • Binary distribution – first release with single downloadable .tgz or .zip artifact.
    • Helm Chart – debut of an official Helm chart for seamless Kubernetes deployment
  • New features & enhancements
    • Policy Store — persistence with schema evolution, built‑in TMS policies (Data compaction, Snapshot expiry, etc) and REST CRUD endpoints
    • Postgres JDBC persistence — native JDBC backend with robust support for concurrent changes.
    • Rollback Compaction on Conflicts - makes Polaris smarter, to revert the compaction commits in case of crunch to let the writers who are actually adding or removing the data to the table succeed. In a sense treating compaction as always a lower priority process.
    • Enhanced runtime — new runtime powered by Quarkus delivers out‑of‑the‑box Kubernetes readiness, quick startup, OIDC integration, and many other benefits. Polaris server and admin tool are now using Quarkus as a runtime framework.
    • HTTP caching via ETag — the loadTable endpoint supports ETag, reducing bandwidth and improving perceived latency
    • Support for external identity providers (IdP) — Polaris can now be its own IdP, delegate to an external IdP, or both
    • Snapshot filtering – clients can choose to load only referenced snapshots
    • Catalog Federation (experimental) – federate requests to an external Iceberg REST or Hadoop Catalog
    • Generic Tables (experimental) — serve multiple table formats besides Iceberg tables; initial Spark 3.5 plugin supports Delta Lake
    • Event Listener framework (experimental) — subscribe to catalog events (AfterTableCommitedEvent, BeforeViewCommitedEvent, etc)
  • Notable bug fixes
    • CLI and Python Client improvements – Support for new features, CLI repair, changes to the update subcommand, and various fixes
    • Safe configurations – Catalog-level Polaris configurations follow a strict naming convention to avoid name clashes with user-provided configuration entries. Legacy Polaris configuration names are still supported in 1.0 to allow existing deployments to migrate without rush.
    • TableOperations optimizations – Changes to BasePolarisTableOperations result in less traffic to object storage during commits
    • Bounded entity cache – The entity cache is now more memory-aware and less likely to lead to OOMs
    • Bootstrapping fixes – Users can more easily bootstrap a new realm. Root credentials can be provided by the user or generated by Polaris (and returned to the user).
  • Breaking changes
    • Server Configuration – The format used to configure the Polaris service in 0.9 has changed with the migration to Quarkus and changes to configurations
    • Bootstrap Flow – The bootstrap flow used in 0.9 has changed with the migration to Quarkus and the new admin tool

0.9.0-incubating

Apache Polaris 0.9.0 was released on March 11, 2025 as the first Polaris release. Only the source distribution is available for this release.