Add configurable indexing event decoder#1220
Open
jwils wants to merge 2 commits into
Open
Conversation
f31c540 to
dbc7b07
Compare
dbc7b07 to
6b885e2
Compare
This was referenced Jun 1, 2026
c1771af to
d6dba0f
Compare
1a7aee7 to
d18af46
Compare
ff22ea0 to
ef120ae
Compare
d18af46 to
0720530
Compare
ef120ae to
f3a3547
Compare
0720530 to
bc078de
Compare
f3a3547 to
12be6cd
Compare
bc078de to
a07b547
Compare
12be6cd to
2e2996c
Compare
a07b547 to
1228bcd
Compare
2e2996c to
50e8b8c
Compare
1228bcd to
a0d1136
Compare
50e8b8c to
e243aa7
Compare
e2feffe to
316f4cd
Compare
3285c93 to
dcd2d33
Compare
a0d30f5 to
460d40d
Compare
3efeb9a to
69c2676
Compare
460d40d to
56e06cf
Compare
69c2676 to
36c7a80
Compare
56e06cf to
9149b33
Compare
36c7a80 to
16bd2d6
Compare
16ae034 to
253bda5
Compare
16bd2d6 to
d29af39
Compare
253bda5 to
898d197
Compare
d29af39 to
a770dcb
Compare
898d197 to
9c881ac
Compare
a770dcb to
cd3aac9
Compare
9c881ac to
81f85dc
Compare
cd3aac9 to
e8e1a67
Compare
This was referenced Jun 11, 2026
1631628 to
4a79422
Compare
- Make `indexing_event_decoder` a required `SqsProcessor` kwarg instead of a nilable one with a second, duplicate default-construction path. The warehouse lambda now passes its indexer's configured decoder, so there is a single source of truth for the default and decoders can rely on receiving non-nil `schema_artifacts`. - Make `JSONLines` inherit from `Interface` so its RBS superclass declaration matches the runtime class. - Validate the decoder's `name` config with the same class-name pattern used for query interceptors, and regenerate the config schema artifact. - Match the established extension-interface style (explanatory comments, `:nocov:` only around the body that must return a value) and document the interface publicly since it is the contract decoder authors implement. - Replace the `_ =` cast with an inline type annotation, alphabetize requires, and simplify the SQS processor spec helper.
4a79422 to
b1abdb0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The indexer lambda currently assumes SQS payload bodies are JSON Lines. To support ingestion sources that serialize events differently, the payload decoding step needs to be configurable before the normal ElasticGraph event validation and indexing pipeline runs.
What
indexer.indexing_event_decoderextension setting using the existing runtime extension loader pattern.ElasticGraph::Indexer::IndexingEventDecoder::JSONLinesdecoder to preserve current behavior.elasticgraph-indexer_lambdaSQS processing through the configured decoder while keeping warehouse lambda compatibility.Verification
script/type_checkbundle exec rspec elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/lambda_function_spec.rb elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/lambda_function_spec.rbbundle exec standardrb elasticgraph-indexer/lib/elastic_graph/indexer.rb elasticgraph-indexer/lib/elastic_graph/indexer/config.rb elasticgraph-indexer/lib/elastic_graph/indexer/event_id.rb elasticgraph-indexer/lib/elastic_graph/indexer/indexing_event_decoder.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer/spec/support/example_extensions/indexing_event_decoder.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/lambda_function.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/sqs_processor.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rbscript/update_config_artifacts --verifyDesign notes
SqsProcessortakes the decoder as a required keyword argument; both the indexer lambda and the warehouse lambda pass their indexer's configured decoder, so the JSON Lines default is defined in exactly one place (theindexer.indexing_event_decoderconfig default) and decoders can rely on receiving non-nilschema_artifacts.JSONLinesdecoder lives inelasticgraph-indexer(rather thanelasticgraph-json_ingestion) because the indexer needs a default it canrequirewithout depending on an optional extension gem. If/when JSON-specific indexer logic moves intoelasticgraph-json_ingestion(per the long-term direction discussed on Add JSON ingestion gem shell #1199), the decoder can move with it.Stack
Current PR is marked with
->.