Add gRPC supports for Search foundation; match-all query - #2071
Open
dayana-j wants to merge 29 commits into
Open
Add gRPC supports for Search foundation; match-all query#2071dayana-j wants to merge 29 commits into
dayana-j wants to merge 29 commits into
Conversation
Adds a transparent gRPC transport layer that routes bulk operations over gRPC for improved performance while falling back to REST for all other operations. Isolated in a separate java-client-grpc module to prevent classpath conflicts. Includes: - GrpcTransport + HybridTransport (automatic routing and fallback) - Translation layer (BulkRequest/Response <-> protobuf conversion) - TLS support (trust cert, trust store, mTLS, insecure, hostname override) - Basic auth, AWS SigV4, and JWT authentication interceptors - Channel health monitoring via gRPC connectivity state machine - Integration tests (framework-compliant, version-gated to 3.5.0+) - Sample code and CI configuration Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com>
Apply suggestion from @reta Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com>
Apply suggestion from @reta Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Dayana <jeandayana28@gmail.com>
…ivate classes - Removed GrpcStatusConverter.toHttpStatus() which was unused anywhere in the codebase (per maintainer feedback) - Moved TranslationTest to the translation package so it can access package-private FieldMappingUtil and GrpcStatusConverter.convert() Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Follows the same pattern as ApacheHttpClient5Transport (general) vs AwsSdk2Transport (AWS-specific) in the existing codebase. Changes: - Created AwsGrpcTransport: extends GrpcTransport, adds SigV4 signing - AwsGrpcTransport.awsBuilder(host, port).sigV4(...).tls(...).build() - Requires sigV4Config and TLS - Overrides preProcessBulk() for payload hash computation - Removed SigV4 from GrpcTransport: - Removed .sigV4() builder method - Removed sigV4Interceptor field - Added protected preProcessBulk() hook for subclasses - GrpcTransport is now purely general-purpose (basic auth, JWT, TLS) - Updated tests to use AwsGrpcTransport for SigV4 tests Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: if the intent is to use gRPC for a supported endpoint, errors should propagate to the user rather than silently falling back to REST. Changes: - HybridTransport no longer catches gRPC errors and retries via REST - Removed fallbackOnError constructor parameter and field - Routing behavior preserved: unsupported endpoints → REST directly - gRPC-supported endpoints: errors propagate to caller - Simplified performRequest/performRequestAsync (no try/catch) - Updated tests: testGrpcErrorPropagatesForSupportedEndpoint Behavior: client.bulk(req) → gRPC (errors propagate if gRPC fails) client.search(req) → REST (not supported by gRPC, routed directly) Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Stale references to 'automatic REST fallback' replaced with accurate 'REST routing for unsupported endpoints' language throughout. - GrpcTransport: updated javadoc and error message - GrpcDemo: rewrote Demo 3 to show REST routing (not fallback on error) - GrpcAwsSigV4: updated to use AwsGrpcTransport.awsBuilder() - GrpcBulkIT: renamed testRestFallback → testRestRouting Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: OpenSearchJavaClientTestCase and TestcontainersThreadFilter are internal test classes not exposed for external module use. Changes: - AbstractGrpcIT: now extends nothing, uses standard JUnit 4 + Assume - Removed dependency on OpenSearchJavaClientTestCase - Removed @ThreadLeakFilters(TestcontainersThreadFilter) - Uses Assume.assumeTrue() with manual version parsing - Reads cluster config from system properties directly - GrpcTransportSupport: converted from interface (extending OpenSearchTransportSupport) to utility class - GrpcBulkIT: no longer implements GrpcTransportSupport interface Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: opensearch-java baseline is JDK 8 and gRPC-Java supports Java 8. No technical reason to require JDK 11. Changes: - build.gradle.kts: targetCompatibility/sourceCompatibility → 1.8 - GrpcSigV4Test: replaced 'var' (Java 10+) with explicit types Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: opensearch-java has a hard dependency on Jackson 3.x, so it comes transitively. No need to declare it again. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: integration tests were not being compiled or run. Added the standard java21 source set configuration used by java-client to java-client-grpc. Changes: - Added unitTest/integrationTest task definitions - Added java21 source set (src/test/java11) gated on JDK 21+ - Added test framework, testcontainers, opensearch-testcontainers deps - Added static import for JUnit Assert in GrpcBulkIT - Integration tests now compile and will run with: ./gradlew :java-client-grpc:integrationTest -Dtests.opensearch.version=3.5.0 Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Adds the first gRPC-capable version to the test matrix so the gRPC integration tests run in CI. Includes both Java 21 and Java 25. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Spotless rejects wildcard imports. Replaced 'import static org.junit.Assert.*' with explicit imports for assertEquals, assertFalse, assertNotNull, assertTrue. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
The java21 source set only contains integration tests (integTest package). Adding it to unitTest causes 'No tests found' failure since the filter excludes integTest classes. Only integrationTest needs the java21 classes. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
- GrpcDemo.java: replaced custom header with standard Apache-2.0 license - GrpcAwsSigV4.java: fixed import ordering (AwsGrpcTransport alphabetical) - Removed unused imports Signed-off-by: Dayana Jean <jeadayao@amazon.com>
assumeGrpcSupported() now verifies both: 1. Server version is 3.5.0+ (via REST info endpoint) 2. gRPC port is actually reachable (TCP socket check) This prevents test failures when running integrationTest against OpenSearch versions that don't have gRPC enabled or when the gRPC port isn't exposed by testcontainers. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
GrpcTestContainerRule now checks the tests.opensearch.version property before adding gRPC configuration. Older versions (1.x, 2.x) don't support aux.transport.types and would fail to start. On pre-3.5.0 versions: - Container starts without gRPC config (REST only) - gRPC port not exposed - Tests skip via assumeGrpcSupported() (version check + port check) Signed-off-by: Dayana Jean <jeadayao@amazon.com>
OpenSearch 3.5.0 Docker image is not yet published, so the gRPC integration tests cannot run in CI. Changes: - Exclude :java-client-grpc:integrationTest from the main CI run (all matrix versions are pre-3.5.0) - Comment out 3.5.0 entries in test matrix (uncomment when released) The gRPC integration tests can still be run locally against a container started manually or once 3.5.0 is published. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
OpenSearch 3.5.0 Docker image is published. Re-enabled in CI matrix. Changes: - GrpcTestContainerRule.before(): catches ContainerLaunchException and calls Assume.assumeTrue(false) to skip tests gracefully instead of failing the build - Added 5-minute startup timeout for CI environments - Removed duplicate disk watermark env var - Re-enabled 3.5.0 in test-integration.yml matrix - Removed -x :java-client-grpc:integrationTest exclusion On pre-3.5.0 versions: container starts without gRPC, tests skip via assumeGrpcSupported(). On 3.5.0+: if container fails for any reason, tests skip instead of failing the build. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: method was only used in tests, not in production code. Removed the method and its 7 associated tests. Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: use assumeTrue in the class rule to skip
tests immediately for older versions, rather than conditionally
configuring the container.
Changes:
- GrpcTestContainerRule.before(): assumeTrue("OpenSearch should
support gRPC", supportsGrpc(version)) skips for pre-3.5.0
- createContainer(): always configures gRPC (only reached for 3.5.0+)
- Removed conditional gRPC config logic
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
1. BasicAuthInterceptor: constructor now package-private 2. GrpcChannelFactory: class now package-private (final class, no public) 3. GrpcTestContainerRule renamed to OpenSearchGrpcTestContainerRule 4. Updated references in AbstractGrpcIT Signed-off-by: Dayana Jean <jeadayao@amazon.com>
dayana-j
force-pushed
the
grpc-search
branch
2 times, most recently
from
July 29, 2026 19:13
6e09875 to
eaae53d
Compare
Adds search-over-gRPC support to the Java client transport, starting
with match_all query. Follows the same pattern as Bulk.
New files:
- SearchRequestConverter: converts SearchRequest → protobuf
- match_all query fully implemented
- Unsupported query types throw UnsupportedOperationException
with message directing to REST or contributing the converter
- SearchResponseConverter: converts protobuf SearchResponse → java
- Full response: took, timed_out, _shards, hits (total, max_score)
- Per-hit: _index, _id, _score, _version, _seq_no, _primary_term
- _source deserialization: bytes → JSON → TDocument via JsonpMapper
- SearchRequestConverterTest: 7 unit tests
- GrpcSearchIT: 3 integration tests (match_all, pagination, empty)
Modified files:
- GrpcTransport: added SearchServiceBlockingStub, registered
SearchRequest._ENDPOINT, added performSearch() handler
Query types not yet implemented (server supports them, client
converter needs extending):
- term, terms, match, match_phrase, bool, range, prefix, wildcard,
regexp, fuzzy, exists, ids, nested, geo_distance, knn, hybrid
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
dayana-j
marked this pull request as ready for review
July 29, 2026 19:38
dayana-j
requested review from
Bukhtawar,
VachaShah,
Xtansia,
madhusudhankonda,
reta,
saratvemulapalli and
szczepanczykd
as code owners
July 29, 2026 19:38
reta
reviewed
Jul 29, 2026
| java.util.Set<Endpoint<?, ?, ?>> endpoints = new java.util.HashSet<>(); | ||
| endpoints.add(BulkRequest._ENDPOINT); | ||
| // Future: SearchRequest._ENDPOINT, KnnSearchRequest._ENDPOINT | ||
| endpoints.add(SearchRequest._ENDPOINT); |
Collaborator
There was a problem hiding this comment.
Thanks @dayana-j , I think this change needs a bit more thoughts:
- the GRPC transport support is strictly endpoint base (for now), however the
_searchis limited to onlymatch_allquery (it is not enough to filter endpoint, we need params or/and payload) - the hybrid transport does internal routing based on endpoints as well, so we basically end up with the situation that user could only invoke
match_alllike queries even if REST based transport supports everything else
That would significantly limit its applicability I believe. The suggestion I have is to enhance endpoint support with request argument GrpcTransport.isEndpointSupported(endpoint, request) so the GRPC transport could check endpoint and perform the request conversion (yes, could be a bit wasteful work but at least we will properly route to REST transport till all queries are covered).
Curious WDYT? Thank you.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds gRPC support for Search for the opensearch-java client, starting with the match_all query type. It follows the same architecture pattern used for the Bulk gRPC transport.
The SearchRequestConverter converts the opensearch-java SearchRequest to the protobuf SearchRequest used by the gRPC SearchService. Currently only match_all is implemented:
Other query types supported by the server (term, match, bool, range, k-NN, etc.) are not yet supported in this version and will throw an UnsupportedOperationException directing users to REST:
The SearchResponseConverter converts the protobuf SearchResponse back to the opensearch-java SearchResponse. This includes full response metadata (took, timed_out, _shards), hits metadata (total, max_score), and per-hit fields. Document _source is deserialized from raw bytes to the user's target class via the JsonpMapper:
Unit tests cover the request conversion for match_all with various parameters and verify that unsupported query types throw appropriately. Integration tests index documents via gRPC bulk, then search with match_all over gRPC and verify the full response.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.