Skip to content

Add gRPC supports for Search foundation; match-all query - #2071

Open
dayana-j wants to merge 29 commits into
opensearch-project:mainfrom
dayana-j:grpc-search
Open

Add gRPC supports for Search foundation; match-all query#2071
dayana-j wants to merge 29 commits into
opensearch-project:mainfrom
dayana-j:grpc-search

Conversation

@dayana-j

@dayana-j dayana-j commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds gRPC support for Search for the opensearch-java client, starting with the match_all query type. It follows the same architecture pattern used for the Bulk gRPC transport.

The SearchRequestConverter converts the opensearch-java SearchRequest to the protobuf SearchRequest used by the gRPC SearchService. Currently only match_all is implemented:

private static org.opensearch.protobufs.MatchAllQuery convertMatchAll(MatchAllQuery matchAll) {
    org.opensearch.protobufs.MatchAllQuery.Builder builder =
        org.opensearch.protobufs.MatchAllQuery.newBuilder();
    if (matchAll.boost() != null) {
        builder.setBoost(matchAll.boost().floatValue());
    }
    return builder.build();
}

Other query types supported by the server (term, match, bool, range, k-NN, etc.) are not yet supported in this version and will throw an UnsupportedOperationException directing users to REST:

throw new UnsupportedOperationException(
     "Query type '" + query._kind() + "' is not yet supported for gRPC transport in this version. "
         + "Use REST transport for this query type."
 );

The SearchResponseConverter converts the protobuf SearchResponse back to the opensearch-java SearchResponse. This includes full response metadata (took, timed_out, _shards), hits metadata (total, max_score), and per-hit fields. Document _source is deserialized from raw bytes to the user's target class via the JsonpMapper:

static <TDocument> TDocument deserializeSource(
    byte[] sourceBytes, JsonpMapper jsonpMapper, Class<TDocument> tDocumentClass
) {
    InputStream stream = new ByteArrayInputStream(sourceBytes);
    jakarta.json.stream.JsonParser parser = jsonpMapper.jsonProvider().createParser(stream);
    parser.next();
    return jsonpMapper.deserialize(parser, tDocumentClass);
}

Unit tests cover the request conversion for match_all with various parameters and verify that unsupported query types throw appropriately. Integration tests index documents via gRPC bulk, then search with match_all over gRPC and verify the full response.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dayana-j and others added 28 commits July 15, 2026 14:18
Adds a transparent gRPC transport layer that routes bulk operations
over gRPC for improved performance while falling back to REST for
all other operations. Isolated in a separate java-client-grpc module
to prevent classpath conflicts.

Includes:
- GrpcTransport + HybridTransport (automatic routing and fallback)
- Translation layer (BulkRequest/Response <-> protobuf conversion)
- TLS support (trust cert, trust store, mTLS, insecure, hostname override)
- Basic auth, AWS SigV4, and JWT authentication interceptors
- Channel health monitoring via gRPC connectivity state machine
- Integration tests (framework-compliant, version-gated to 3.5.0+)
- Sample code and CI configuration

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Dayana <jeandayana28@gmail.com>
Apply suggestion from @reta

Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Dayana <jeandayana28@gmail.com>
Apply suggestion from @reta

Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Dayana <jeandayana28@gmail.com>
…ivate classes

- Removed GrpcStatusConverter.toHttpStatus() which was unused anywhere
  in the codebase (per maintainer feedback)
- Moved TranslationTest to the translation package so it can access
  package-private FieldMappingUtil and GrpcStatusConverter.convert()

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Follows the same pattern as ApacheHttpClient5Transport (general) vs
AwsSdk2Transport (AWS-specific) in the existing codebase.

Changes:
- Created AwsGrpcTransport: extends GrpcTransport, adds SigV4 signing
  - AwsGrpcTransport.awsBuilder(host, port).sigV4(...).tls(...).build()
  - Requires sigV4Config and TLS
  - Overrides preProcessBulk() for payload hash computation
- Removed SigV4 from GrpcTransport:
  - Removed .sigV4() builder method
  - Removed sigV4Interceptor field
  - Added protected preProcessBulk() hook for subclasses
- GrpcTransport is now purely general-purpose (basic auth, JWT, TLS)
- Updated tests to use AwsGrpcTransport for SigV4 tests

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: if the intent is to use gRPC for a supported
endpoint, errors should propagate to the user rather than silently
falling back to REST.

Changes:
- HybridTransport no longer catches gRPC errors and retries via REST
- Removed fallbackOnError constructor parameter and field
- Routing behavior preserved: unsupported endpoints → REST directly
- gRPC-supported endpoints: errors propagate to caller
- Simplified performRequest/performRequestAsync (no try/catch)
- Updated tests: testGrpcErrorPropagatesForSupportedEndpoint

Behavior:
  client.bulk(req)   → gRPC (errors propagate if gRPC fails)
  client.search(req) → REST (not supported by gRPC, routed directly)

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Stale references to 'automatic REST fallback' replaced with accurate
'REST routing for unsupported endpoints' language throughout.

- GrpcTransport: updated javadoc and error message
- GrpcDemo: rewrote Demo 3 to show REST routing (not fallback on error)
- GrpcAwsSigV4: updated to use AwsGrpcTransport.awsBuilder()
- GrpcBulkIT: renamed testRestFallback → testRestRouting

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: OpenSearchJavaClientTestCase and
TestcontainersThreadFilter are internal test classes not exposed for
external module use.

Changes:
- AbstractGrpcIT: now extends nothing, uses standard JUnit 4 + Assume
  - Removed dependency on OpenSearchJavaClientTestCase
  - Removed @ThreadLeakFilters(TestcontainersThreadFilter)
  - Uses Assume.assumeTrue() with manual version parsing
  - Reads cluster config from system properties directly
- GrpcTransportSupport: converted from interface (extending
  OpenSearchTransportSupport) to utility class
- GrpcBulkIT: no longer implements GrpcTransportSupport interface

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: opensearch-java baseline is JDK 8 and gRPC-Java
supports Java 8. No technical reason to require JDK 11.

Changes:
- build.gradle.kts: targetCompatibility/sourceCompatibility → 1.8
- GrpcSigV4Test: replaced 'var' (Java 10+) with explicit types

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: opensearch-java has a hard dependency on
Jackson 3.x, so it comes transitively. No need to declare it again.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: integration tests were not being compiled or
run. Added the standard java21 source set configuration used by
java-client to java-client-grpc.

Changes:
- Added unitTest/integrationTest task definitions
- Added java21 source set (src/test/java11) gated on JDK 21+
- Added test framework, testcontainers, opensearch-testcontainers deps
- Added static import for JUnit Assert in GrpcBulkIT
- Integration tests now compile and will run with:
  ./gradlew :java-client-grpc:integrationTest -Dtests.opensearch.version=3.5.0

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Adds the first gRPC-capable version to the test matrix so the gRPC
integration tests run in CI. Includes both Java 21 and Java 25.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Spotless rejects wildcard imports. Replaced 'import static org.junit.Assert.*'
with explicit imports for assertEquals, assertFalse, assertNotNull, assertTrue.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
The java21 source set only contains integration tests (integTest package).
Adding it to unitTest causes 'No tests found' failure since the filter
excludes integTest classes. Only integrationTest needs the java21 classes.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
- GrpcDemo.java: replaced custom header with standard Apache-2.0 license
- GrpcAwsSigV4.java: fixed import ordering (AwsGrpcTransport alphabetical)
- Removed unused imports

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
assumeGrpcSupported() now verifies both:
1. Server version is 3.5.0+ (via REST info endpoint)
2. gRPC port is actually reachable (TCP socket check)

This prevents test failures when running integrationTest against
OpenSearch versions that don't have gRPC enabled or when the gRPC
port isn't exposed by testcontainers.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
GrpcTestContainerRule now checks the tests.opensearch.version property
before adding gRPC configuration. Older versions (1.x, 2.x) don't
support aux.transport.types and would fail to start.

On pre-3.5.0 versions:
- Container starts without gRPC config (REST only)
- gRPC port not exposed
- Tests skip via assumeGrpcSupported() (version check + port check)

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
OpenSearch 3.5.0 Docker image is not yet published, so the gRPC
integration tests cannot run in CI. Changes:

- Exclude :java-client-grpc:integrationTest from the main CI run
  (all matrix versions are pre-3.5.0)
- Comment out 3.5.0 entries in test matrix (uncomment when released)

The gRPC integration tests can still be run locally against a
container started manually or once 3.5.0 is published.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
OpenSearch 3.5.0 Docker image is published. Re-enabled in CI matrix.

Changes:
- GrpcTestContainerRule.before(): catches ContainerLaunchException and
  calls Assume.assumeTrue(false) to skip tests gracefully instead of
  failing the build
- Added 5-minute startup timeout for CI environments
- Removed duplicate disk watermark env var
- Re-enabled 3.5.0 in test-integration.yml matrix
- Removed -x :java-client-grpc:integrationTest exclusion

On pre-3.5.0 versions: container starts without gRPC, tests skip via
assumeGrpcSupported(). On 3.5.0+: if container fails for any reason,
tests skip instead of failing the build.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: method was only used in tests, not in
production code. Removed the method and its 7 associated tests.

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Per maintainer feedback: use assumeTrue in the class rule to skip
tests immediately for older versions, rather than conditionally
configuring the container.

Changes:
- GrpcTestContainerRule.before(): assumeTrue("OpenSearch should
  support gRPC", supportsGrpc(version)) skips for pre-3.5.0
- createContainer(): always configures gRPC (only reached for 3.5.0+)
- Removed conditional gRPC config logic

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
Signed-off-by: Dayana Jean <jeadayao@amazon.com>
1. BasicAuthInterceptor: constructor now package-private
2. GrpcChannelFactory: class now package-private (final class, no public)
3. GrpcTestContainerRule renamed to OpenSearchGrpcTestContainerRule
4. Updated references in AbstractGrpcIT

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
@dayana-j
dayana-j force-pushed the grpc-search branch 2 times, most recently from 6e09875 to eaae53d Compare July 29, 2026 19:13
Adds search-over-gRPC support to the Java client transport, starting
with match_all query. Follows the same pattern as Bulk.

New files:
- SearchRequestConverter: converts SearchRequest → protobuf
  - match_all query fully implemented
  - Unsupported query types throw UnsupportedOperationException
    with message directing to REST or contributing the converter
- SearchResponseConverter: converts protobuf SearchResponse → java
  - Full response: took, timed_out, _shards, hits (total, max_score)
  - Per-hit: _index, _id, _score, _version, _seq_no, _primary_term
  - _source deserialization: bytes → JSON → TDocument via JsonpMapper
- SearchRequestConverterTest: 7 unit tests
- GrpcSearchIT: 3 integration tests (match_all, pagination, empty)

Modified files:
- GrpcTransport: added SearchServiceBlockingStub, registered
  SearchRequest._ENDPOINT, added performSearch() handler

Query types not yet implemented (server supports them, client
converter needs extending):
- term, terms, match, match_phrase, bool, range, prefix, wildcard,
  regexp, fuzzy, exists, ids, nested, geo_distance, knn, hybrid

Signed-off-by: Dayana Jean <jeadayao@amazon.com>
java.util.Set<Endpoint<?, ?, ?>> endpoints = new java.util.HashSet<>();
endpoints.add(BulkRequest._ENDPOINT);
// Future: SearchRequest._ENDPOINT, KnnSearchRequest._ENDPOINT
endpoints.add(SearchRequest._ENDPOINT);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dayana-j , I think this change needs a bit more thoughts:

  • the GRPC transport support is strictly endpoint base (for now), however the _search is limited to only match_all query (it is not enough to filter endpoint, we need params or/and payload)
  • the hybrid transport does internal routing based on endpoints as well, so we basically end up with the situation that user could only invoke match_all like queries even if REST based transport supports everything else

That would significantly limit its applicability I believe. The suggestion I have is to enhance endpoint support with request argument GrpcTransport.isEndpointSupported(endpoint, request) so the GRPC transport could check endpoint and perform the request conversion (yes, could be a bit wasteful work but at least we will properly route to REST transport till all queries are covered).

Curious WDYT? Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants