Skip to content

Add configurable indexing event decoder#1220

Open
jwils wants to merge 2 commits into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters
Open

Add configurable indexing event decoder#1220
jwils wants to merge 2 commits into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters

Conversation

@jwils

@jwils jwils commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Why

The indexer lambda currently assumes SQS payload bodies are JSON Lines. To support ingestion sources that serialize events differently, the payload decoding step needs to be configurable before the normal ElasticGraph event validation and indexing pipeline runs.

What

  • Add an indexer.indexing_event_decoder extension setting using the existing runtime extension loader pattern.
  • Add a default ElasticGraph::Indexer::IndexingEventDecoder::JSONLines decoder to preserve current behavior.
  • Wire elasticgraph-indexer_lambda SQS processing through the configured decoder while keeping warehouse lambda compatibility.
  • Add RBS, focused specs, README docs, and regenerated config schema.

Verification

  • script/type_check
  • bundle exec rspec elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/lambda_function_spec.rb elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/lambda_function_spec.rb
  • bundle exec standardrb elasticgraph-indexer/lib/elastic_graph/indexer.rb elasticgraph-indexer/lib/elastic_graph/indexer/config.rb elasticgraph-indexer/lib/elastic_graph/indexer/event_id.rb elasticgraph-indexer/lib/elastic_graph/indexer/indexing_event_decoder.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer/spec/support/example_extensions/indexing_event_decoder.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/lambda_function.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/sqs_processor.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb
  • script/update_config_artifacts --verify

Design notes

  • SqsProcessor takes the decoder as a required keyword argument; both the indexer lambda and the warehouse lambda pass their indexer's configured decoder, so the JSON Lines default is defined in exactly one place (the indexer.indexing_event_decoder config default) and decoders can rely on receiving non-nil schema_artifacts.
  • The default JSONLines decoder lives in elasticgraph-indexer (rather than elasticgraph-json_ingestion) because the indexer needs a default it can require without depending on an optional extension gem. If/when JSON-specific indexer logic moves into elasticgraph-json_ingestion (per the long-term direction discussed on Add JSON ingestion gem shell #1199), the decoder can move with it.

Stack

Current PR is marked with ->.

@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from f31c540 to dbc7b07 Compare May 31, 2026 05:24
@CLAassistant

CLAassistant commented May 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from dbc7b07 to 6b885e2 Compare May 31, 2026 23:52
@jwils jwils changed the base branch from main to joshuaw/json-ingestion-test-migration May 31, 2026 23:53
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from c1771af to d6dba0f Compare June 1, 2026 18:28
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 1a7aee7 to d18af46 Compare June 1, 2026 18:42
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch 2 times, most recently from ff22ea0 to ef120ae Compare June 1, 2026 18:58
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from d18af46 to 0720530 Compare June 1, 2026 18:59
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from ef120ae to f3a3547 Compare June 1, 2026 19:01
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 0720530 to bc078de Compare June 1, 2026 19:01
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from f3a3547 to 12be6cd Compare June 4, 2026 13:59
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from bc078de to a07b547 Compare June 4, 2026 14:00
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 12be6cd to 2e2996c Compare June 4, 2026 14:19
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from a07b547 to 1228bcd Compare June 4, 2026 14:19
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 2e2996c to 50e8b8c Compare June 5, 2026 18:36
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 1228bcd to a0d1136 Compare June 5, 2026 18:36
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 50e8b8c to e243aa7 Compare June 5, 2026 18:45
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from e2feffe to 316f4cd Compare June 5, 2026 22:35
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 3285c93 to dcd2d33 Compare June 5, 2026 22:43
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from a0d30f5 to 460d40d Compare June 5, 2026 23:37
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 3efeb9a to 69c2676 Compare June 6, 2026 00:25
@jwils jwils marked this pull request as ready for review June 8, 2026 15:47
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 460d40d to 56e06cf Compare June 9, 2026 15:10
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 69c2676 to 36c7a80 Compare June 9, 2026 15:10
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 56e06cf to 9149b33 Compare June 9, 2026 15:31
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 36c7a80 to 16bd2d6 Compare June 9, 2026 15:31
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch 2 times, most recently from 16ae034 to 253bda5 Compare June 9, 2026 19:07
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 16bd2d6 to d29af39 Compare June 9, 2026 19:11
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 253bda5 to 898d197 Compare June 9, 2026 19:52
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from d29af39 to a770dcb Compare June 9, 2026 19:52
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 898d197 to 9c881ac Compare June 9, 2026 20:09
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from a770dcb to cd3aac9 Compare June 9, 2026 20:09
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 9c881ac to 81f85dc Compare June 10, 2026 12:48
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from cd3aac9 to e8e1a67 Compare June 10, 2026 12:48
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 1631628 to 4a79422 Compare June 11, 2026 17:14
jwils added 2 commits June 11, 2026 21:33
- Make `indexing_event_decoder` a required `SqsProcessor` kwarg instead of a
  nilable one with a second, duplicate default-construction path. The warehouse
  lambda now passes its indexer's configured decoder, so there is a single
  source of truth for the default and decoders can rely on receiving non-nil
  `schema_artifacts`.
- Make `JSONLines` inherit from `Interface` so its RBS superclass declaration
  matches the runtime class.
- Validate the decoder's `name` config with the same class-name pattern used
  for query interceptors, and regenerate the config schema artifact.
- Match the established extension-interface style (explanatory comments,
  `:nocov:` only around the body that must return a value) and document the
  interface publicly since it is the contract decoder authors implement.
- Replace the `_ =` cast with an inline type annotation, alphabetize requires,
  and simplify the SQS processor spec helper.
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 4a79422 to b1abdb0 Compare June 12, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants