Skip to content

Choose a tag to compare

@bpolaszek bpolaszek released this 11 Feb 13:21
9386ed5

What's Changed

  • Feat: Generator fan-out in transform pipeline by @bpolaszek in #59
  • Feat: Batch transform support (BatchTransformerInterface) by @bpolaszek in #60

Full Changelog: 4.1...4.2.0

New Features

Generator fan-out in transform pipeline (#59)

Chained transformers now support flatMap semantics: when a transformer yields multiple items via a Generator, each item is passed individually to the next transformer in the chain instead of passing the whole generator (see BC breaks below).

$etl = (new EtlExecutor())
    ->transformWith(
        fn (string $item) => strrev($item),
        function (string $item): Generator {
            yield $item;
            yield strtoupper($item);
        },
        fn (string $item) => "({$item})",
    );

$report = $etl->process(['foo', 'bar']);
// Output: ['(oof)', '(OOF)', '(rab)', '(RAB)']

Batch transform support (#60)

New BatchTransformerInterface allows transforming multiple items at once — ideal for concurrent HTTP requests or any batchable operation.

$executor = (new EtlExecutor())
    ->transformWith(new CallableBatchTransformer(
        function (array $items, EtlState $state): array {
            return $httpClient->sendConcurrent(
                array_map(fn ($item) => new Request('GET', $item['url']), $items)
            );
        }
    ))
    ->withOptions(new EtlConfiguration(batchSize: 10))
    ->process($source);
  • EtlConfiguration gains a batchSize option (default: 1, no batching)
  • Items are chunked automatically and passed as an array to the batch transformer
  • bentools/iterable-functions is now a production dependency

Breaking Changes

Impact Description
⚠️ ChainTransformer Chained transformers that type-hint Generator as input will break — they now receive individual items instead
⚠️ TransformEvent listeners $event->transformResult now contains a single item instead of the full generator output
⚠️ Skip in TransformEvent skip() now only skips the individual transformed item, not all items from the generator
⚠️ EtlExecutor::$transformer Property type changed from TransformerInterface to TransformerInterface|BatchTransformerInterface