[Backport 3.x] gRPC support for opensearch-java - #2065
Merged
reta merged 2 commits intoJul 22, 2026
Merged
Conversation
* feat: add transparent gRPC transport as separate java-client-grpc module Adds a transparent gRPC transport layer that routes bulk operations over gRPC for improved performance while falling back to REST for all other operations. Isolated in a separate java-client-grpc module to prevent classpath conflicts. Includes: - GrpcTransport + HybridTransport (automatic routing and fallback) - Translation layer (BulkRequest/Response <-> protobuf conversion) - TLS support (trust cert, trust store, mTLS, insecure, hostname override) - Basic auth, AWS SigV4, and JWT authentication interceptors - Channel health monitoring via gRPC connectivity state machine - Integration tests (framework-compliant, version-gated to 3.5.0+) - Sample code and CI configuration Signed-off-by: Dayana Jean <jeadayao@amazon.com> * ci: retrigger CI Signed-off-by: Dayana Jean <jeadayao@amazon.com> * Making the FieldMappingUtil package private Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com> * Making this package private Apply suggestion from @reta Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com> * Making BasicAuthInterceptor package private Apply suggestion from @reta Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com> * Remove unused toHttpStatus method and fix test package for package-private classes - Removed GrpcStatusConverter.toHttpStatus() which was unused anywhere in the codebase (per maintainer feedback) - Moved TranslationTest to the translation package so it can access package-private FieldMappingUtil and GrpcStatusConverter.convert() Signed-off-by: Dayana Jean <jeadayao@amazon.com> * refactor: split AWS SigV4 into AwsGrpcTransport per maintainer feedback Follows the same pattern as ApacheHttpClient5Transport (general) vs AwsSdk2Transport (AWS-specific) in the existing codebase. Changes: - Created AwsGrpcTransport: extends GrpcTransport, adds SigV4 signing - AwsGrpcTransport.awsBuilder(host, port).sigV4(...).tls(...).build() - Requires sigV4Config and TLS - Overrides preProcessBulk() for payload hash computation - Removed SigV4 from GrpcTransport: - Removed .sigV4() builder method - Removed sigV4Interceptor field - Added protected preProcessBulk() hook for subclasses - GrpcTransport is now purely general-purpose (basic auth, JWT, TLS) - Updated tests to use AwsGrpcTransport for SigV4 tests Signed-off-by: Dayana Jean <jeadayao@amazon.com> * refactor: remove silent REST fallback from HybridTransport Per maintainer feedback: if the intent is to use gRPC for a supported endpoint, errors should propagate to the user rather than silently falling back to REST. Changes: - HybridTransport no longer catches gRPC errors and retries via REST - Removed fallbackOnError constructor parameter and field - Routing behavior preserved: unsupported endpoints → REST directly - gRPC-supported endpoints: errors propagate to caller - Simplified performRequest/performRequestAsync (no try/catch) - Updated tests: testGrpcErrorPropagatesForSupportedEndpoint Behavior: client.bulk(req) → gRPC (errors propagate if gRPC fails) client.search(req) → REST (not supported by gRPC, routed directly) Signed-off-by: Dayana Jean <jeadayao@amazon.com> * docs: update comments to reflect routing instead of fallback Stale references to 'automatic REST fallback' replaced with accurate 'REST routing for unsupported endpoints' language throughout. - GrpcTransport: updated javadoc and error message - GrpcDemo: rewrote Demo 3 to show REST routing (not fallback on error) - GrpcAwsSigV4: updated to use AwsGrpcTransport.awsBuilder() - GrpcBulkIT: renamed testRestFallback → testRestRouting Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: make integration test base classes self-contained Per maintainer feedback: OpenSearchJavaClientTestCase and TestcontainersThreadFilter are internal test classes not exposed for external module use. Changes: - AbstractGrpcIT: now extends nothing, uses standard JUnit 4 + Assume - Removed dependency on OpenSearchJavaClientTestCase - Removed @ThreadLeakFilters(TestcontainersThreadFilter) - Uses Assume.assumeTrue() with manual version parsing - Reads cluster config from system properties directly - GrpcTransportSupport: converted from interface (extending OpenSearchTransportSupport) to utility class - GrpcBulkIT: no longer implements GrpcTransportSupport interface Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: lower java-client-grpc baseline from JDK 11 to JDK 8 Per maintainer feedback: opensearch-java baseline is JDK 8 and gRPC-Java supports Java 8. No technical reason to require JDK 11. Changes: - build.gradle.kts: targetCompatibility/sourceCompatibility → 1.8 - GrpcSigV4Test: replaced 'var' (Java 10+) with explicit types Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: remove explicit jackson test deps (transitive via java-client) Per maintainer feedback: opensearch-java has a hard dependency on Jackson 3.x, so it comes transitively. No need to declare it again. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: wire up integration tests with java21 source set Per maintainer feedback: integration tests were not being compiled or run. Added the standard java21 source set configuration used by java-client to java-client-grpc. Changes: - Added unitTest/integrationTest task definitions - Added java21 source set (src/test/java11) gated on JDK 21+ - Added test framework, testcontainers, opensearch-testcontainers deps - Added static import for JUnit Assert in GrpcBulkIT - Integration tests now compile and will run with: ./gradlew :java-client-grpc:integrationTest -Dtests.opensearch.version=3.5.0 Signed-off-by: Dayana Jean <jeadayao@amazon.com> * ci: add OpenSearch 3.5.0 to integration test matrix Adds the first gRPC-capable version to the test matrix so the gRPC integration tests run in CI. Includes both Java 21 and Java 25. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: replace wildcard import with explicit imports (spotless) Spotless rejects wildcard imports. Replaced 'import static org.junit.Assert.*' with explicit imports for assertEquals, assertFalse, assertNotNull, assertTrue. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: remove java21 classes from unitTest task The java21 source set only contains integration tests (integTest package). Adding it to unitTest causes 'No tests found' failure since the filter excludes integTest classes. Only integrationTest needs the java21 classes. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: spotless formatting for samples (license header, import order) - GrpcDemo.java: replaced custom header with standard Apache-2.0 license - GrpcAwsSigV4.java: fixed import ordering (AwsGrpcTransport alphabetical) - Removed unused imports Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: skip gRPC integration tests when port is unreachable assumeGrpcSupported() now verifies both: 1. Server version is 3.5.0+ (via REST info endpoint) 2. gRPC port is actually reachable (TCP socket check) This prevents test failures when running integrationTest against OpenSearch versions that don't have gRPC enabled or when the gRPC port isn't exposed by testcontainers. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: only enable gRPC in testcontainer for OpenSearch 3.5.0+ GrpcTestContainerRule now checks the tests.opensearch.version property before adding gRPC configuration. Older versions (1.x, 2.x) don't support aux.transport.types and would fail to start. On pre-3.5.0 versions: - Container starts without gRPC config (REST only) - gRPC port not exposed - Tests skip via assumeGrpcSupported() (version check + port check) Signed-off-by: Dayana Jean <jeadayao@amazon.com> * ci: exclude grpc integration tests until OpenSearch 3.5.0 is released OpenSearch 3.5.0 Docker image is not yet published, so the gRPC integration tests cannot run in CI. Changes: - Exclude :java-client-grpc:integrationTest from the main CI run (all matrix versions are pre-3.5.0) - Comment out 3.5.0 entries in test matrix (uncomment when released) The gRPC integration tests can still be run locally against a container started manually or once 3.5.0 is published. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: graceful skip on container failure + re-enable 3.5.0 in CI OpenSearch 3.5.0 Docker image is published. Re-enabled in CI matrix. Changes: - GrpcTestContainerRule.before(): catches ContainerLaunchException and calls Assume.assumeTrue(false) to skip tests gracefully instead of failing the build - Added 5-minute startup timeout for CI environments - Removed duplicate disk watermark env var - Re-enabled 3.5.0 in test-integration.yml matrix - Removed -x :java-client-grpc:integrationTest exclusion On pre-3.5.0 versions: container starts without gRPC, tests skip via assumeGrpcSupported(). On 3.5.0+: if container fails for any reason, tests skip instead of failing the build. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: spotless formatting on GrpcTestContainerRule Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: remove grpcStatusToHttpStatus from FieldMappingUtil Per maintainer feedback: method was only used in tests, not in production code. Removed the method and its 7 associated tests. Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: skip gRPC tests early with assumeTrue for unsupported versions Per maintainer feedback: use assumeTrue in the class rule to skip tests immediately for older versions, rather than conditionally configuring the container. Changes: - GrpcTestContainerRule.before(): assumeTrue("OpenSearch should support gRPC", supportsGrpc(version)) skips for pre-3.5.0 - createContainer(): always configures gRPC (only reached for 3.5.0+) - Removed conditional gRPC config logic Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: make GrpcChannelFactory methods package-private Signed-off-by: Dayana Jean <jeadayao@amazon.com> * fix: apply maintainer suggestions on visibility and naming 1. BasicAuthInterceptor: constructor now package-private 2. GrpcChannelFactory: class now package-private (final class, no public) 3. GrpcTestContainerRule renamed to OpenSearchGrpcTestContainerRule 4. Updated references in AbstractGrpcIT Signed-off-by: Dayana Jean <jeadayao@amazon.com> --------- Signed-off-by: Dayana Jean <jeadayao@amazon.com> Signed-off-by: Dayana <jeandayana28@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 44bbc73) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com>
opensearch-ci-bot
requested review from
Bukhtawar,
VachaShah,
Xtansia,
madhusudhankonda,
reta,
saratvemulapalli and
szczepanczykd
as code owners
July 22, 2026 22:02
reta
force-pushed
the
backport/backport-2062-to-3.x
branch
2 times, most recently
from
July 22, 2026 22:49
e34ae90 to
bb68088
Compare
Signed-off-by: Andriy Redko <drreta@gmail.com>
reta
force-pushed
the
backport/backport-2062-to-3.x
branch
from
July 22, 2026 22:58
bb68088 to
94e4ca4
Compare
reta
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 44bbc73 from #2062.