Use compact oneof memory structure by tigrannajaryan · Pull Request #418 · splunk/stef

tigrannajaryan · 2026-05-20T21:56:25Z

Borrows the storage format idea from https://github.com/tigrannajaryan/govariant This reduces memory usage when multiple records are kept in memory and also improves speed of some operations.

Benchmarking results:

goos: darwin
goarch: arm64
pkg: github.com/splunk/stef/benchmarks
cpu: Apple M2 Pro
                                                  │ bench_base.txt │          bench_current.txt          │
                                                  │     sec/op     │   sec/op     vs base                │
SerializeNative/STEF/serialize-10                      5.000m ± 4%   3.193m ± 3%  -36.14% (p=0.000 n=18)
DeserializeNative/STEF/deser-10                        1.479m ± 7%   1.489m ± 5%        ~ (p=0.839 n=18)
SerializeFromPdata/STEF/serialize-10                   70.37m ± 3%   58.67m ± 4%  -16.63% (p=0.000 n=18)
DeserializeToPdata/STEF/deserialize-10                 24.92m ± 2%   24.89m ± 2%        ~ (p=0.815 n=18)
STEFReaderRead-10                                      1.523m ± 3%   1.495m ± 6%        ~ (p=0.323 n=18)
STEFSerializeMultipart/astronomy-otelmetrics-10         1.958 ± 3%    1.808 ± 3%   -7.68% (p=0.000 n=18)
STEFDeserializeMultipart/astronomy-otelmetrics-10      45.87m ± 3%   46.92m ± 6%        ~ (p=0.143 n=18)
ReadSTEF-10                                            1.695m ± 2%   1.655m ± 3%   -2.35% (p=0.008 n=18)
ReadSTEFZ-10                                           2.218m ± 2%   2.227m ± 3%        ~ (p=0.988 n=18)
ReadSTEFZWriteSTEF-10                                  4.682m ± 4%   4.735m ± 5%        ~ (p=0.134 n=18)
geomean                                                11.21m        10.44m        -6.85%

                                                  │ bench_base.txt │          bench_current.txt          │
                                                  │   sec/point    │  sec/point   vs base                │
SerializeNative/STEF/serialize-10                      74.79n ± 4%   47.76n ± 3%  -36.14% (p=0.000 n=18)
DeserializeNative/STEF/deser-10                        22.12n ± 7%   22.27n ± 5%        ~ (p=0.833 n=18)
SerializeFromPdata/STEF/serialize-10                  1053.5n ± 3%   878.0n ± 4%  -16.66% (p=0.000 n=18)
DeserializeToPdata/STEF/deserialize-10                 373.0n ± 2%   372.5n ± 2%        ~ (p=0.784 n=18)
STEFReaderRead-10                                      22.79n ± 3%   22.36n ± 6%        ~ (p=0.319 n=18)
STEFSerializeMultipart/astronomy-otelmetrics-10        2.489µ ± 3%   2.298µ ± 3%   -7.68% (p=0.000 n=18)
STEFDeserializeMultipart/astronomy-otelmetrics-10      58.30n ± 3%   59.63n ± 6%        ~ (p=0.145 n=18)
ReadSTEF-10                                            25.37n ± 2%   24.77n ± 3%   -2.35% (p=0.008 n=18)
ReadSTEFZ-10                                           33.19n ± 2%   33.33n ± 3%        ~ (p=0.981 n=18)
ReadSTEFZWriteSTEF-10                                  70.07n ± 4%   70.86n ± 5%        ~ (p=0.132 n=18)
geomean                                                102.4n        95.39n        -6.85%

                                                  │ bench_base.txt │          bench_current.txt           │
                                                  │      B/op      │     B/op      vs base                │
SerializeNative/STEF/serialize-10                     3.373Mi ± 0%   3.395Mi ± 0%   +0.63% (p=0.000 n=18)
DeserializeNative/STEF/deser-10                       951.4Ki ± 0%   725.8Ki ± 0%  -23.71% (p=0.000 n=18)
SerializeFromPdata/STEF/serialize-10                  79.76Mi ± 0%   36.49Mi ± 0%  -54.25% (p=0.000 n=18)
DeserializeToPdata/STEF/deserialize-10                34.83Mi ± 0%   34.61Mi ± 0%   -0.63% (p=0.000 n=18)
STEFReaderRead-10                                     953.1Ki ± 0%   727.5Ki ± 0%  -23.67% (p=0.000 n=18)
STEFSerializeMultipart/astronomy-otelmetrics-10       4.868Gi ± 0%   4.168Gi ± 0%  -14.37% (p=0.000 n=18)
STEFDeserializeMultipart/astronomy-otelmetrics-10     20.30Mi ± 0%   20.19Mi ± 0%   -0.56% (p=0.000 n=18)
ReadSTEF-10                                           953.1Ki ± 0%   727.5Ki ± 0%  -23.67% (p=0.000 n=18)
ReadSTEFZ-10                                          10.29Mi ± 0%   10.07Mi ± 0%   -2.14% (p=0.000 n=18)
ReadSTEFZWriteSTEF-10                                 13.45Mi ± 0%   13.22Mi ± 0%   -1.67% (p=0.000 n=18)
geomean                                               12.66Mi        10.58Mi       -16.41%

                                                  │ bench_base.txt │          bench_current.txt           │
                                                  │   allocs/op    │  allocs/op   vs base                 │
SerializeNative/STEF/serialize-10                      2.642k ± 0%   2.662k ± 0%    +0.76% (p=0.000 n=18)
DeserializeNative/STEF/deser-10                         463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.000 n=18)
SerializeFromPdata/STEF/serialize-10                   134.7k ± 0%   134.7k ± 0%    +0.03% (p=0.021 n=18)
DeserializeToPdata/STEF/deserialize-10                 756.2k ± 0%   756.2k ± 0%    +0.00% (p=0.000 n=18)
STEFReaderRead-10                                       463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.000 n=18)
STEFSerializeMultipart/astronomy-otelmetrics-10        13.15M ± 0%   13.22M ± 0%    +0.51% (p=0.000 n=18)
STEFDeserializeMultipart/astronomy-otelmetrics-10      1.958k ± 0%   7.923k ± 0%  +304.65% (p=0.000 n=18)
ReadSTEF-10                                             463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.000 n=18)
ReadSTEFZ-10                                            503.0 ± 0%    504.0 ± 0%    +0.20% (p=0.000 n=18)
ReadSTEFZWriteSTEF-10                                  1.230k ± 0%   1.232k ± 0%    +0.16% (p=0.000 n=18)
geomean                                                7.293k        8.406k        +15.27%

Also added a benchmarks that reads many records at once to show how much memory is reduced:

goos: darwin
goarch: arm64
pkg: github.com/splunk/stef/benchmarks
cpu: Apple M2 Pro
                  │   main.txt   │             after.txt              │
                  │    sec/op    │   sec/op     vs base               │
ReaderReadMany-10   12.087m ± 5%   9.782m ± 2%  -19.07% (p=0.002 n=6)

                  │  main.txt   │             after.txt              │
                  │  sec/point  │  sec/point   vs base               │
ReaderReadMany-10   181.0n ± 5%   146.5n ± 2%  -19.06% (p=0.002 n=6)

                  │   main.txt    │              after.txt              │
                  │     B/op      │     B/op      vs base               │
ReaderReadMany-10   2111.4Ki ± 6%   921.2Ki ± 1%  -56.37% (p=0.002 n=6)

                  │  main.txt  │             after.txt             │
                  │ allocs/op  │ allocs/op   vs base               │
ReaderReadMany-10   970.0 ± 5%   845.5 ± 2%  -12.84% (p=0.002 n=6)

Copilot

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

github-actions · 2026-05-20T22:04:25Z

Benchmark Result

Benchmark diff with base branch

goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 9V74 80-Core Processor                
                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │     sec/op     │   sec/op     vs base               │
SerializeNative/STEF/serialize-4                     8.788m ±  7%   6.759m ± 1%  -23.09% (p=0.002 n=6)
SerializeNative/STEFU/serialize-4                    33.45m ±  1%   33.20m ± 1%        ~ (p=0.240 n=6)
DeserializeNative/STEF/deser-4                       2.460m ±  0%   2.457m ± 2%        ~ (p=0.589 n=6)
DeserializeNative/STEFU/deser-4                      7.053m ±  1%   7.215m ± 2%   +2.29% (p=0.002 n=6)
SerializeFromPdata/STEF/serialize-4                  136.2m ±  6%   112.7m ± 1%  -17.29% (p=0.002 n=6)
SerializeFromPdata/STEFU/serialize-4                 33.52m ±  1%   33.99m ± 1%   +1.40% (p=0.009 n=6)
DeserializeToPdata/STEF/deserialize-4                46.47m ±  2%   46.73m ± 4%        ~ (p=0.394 n=6)
DeserializeToPdata/STEFU/deserialize-4               64.08m ±  3%   64.69m ± 2%        ~ (p=0.093 n=6)
STEFReaderRead-4                                     2.572m ±  1%   2.562m ± 1%        ~ (p=0.180 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4        3.135 ±  7%    2.807 ± 4%  -10.48% (p=0.002 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     79.91m ± 10%   67.72m ± 3%  -15.25% (p=0.002 n=6)
ReadSTEF-4                                           2.669m ±  1%   2.524m ± 1%   -5.44% (p=0.002 n=6)
ReadSTEFZ-4                                          3.465m ±  1%   3.176m ± 0%   -8.34% (p=0.002 n=6)
ReadSTEFZWriteSTEF-4                                 7.671m ±  2%   7.068m ± 0%   -7.87% (p=0.002 n=6)
geomean                                              20.97m         19.64m        -6.32%

                                                 │ bench-main.txt │           bench-new.txt            │
                                                 │   sec/point    │  sec/point   vs base               │
SerializeNative/STEF/serialize-4                     131.5n ±  7%   101.1n ± 1%  -23.09% (p=0.002 n=6)
SerializeNative/STEFU/serialize-4                    500.3n ±  1%   496.6n ± 1%        ~ (p=0.240 n=6)
DeserializeNative/STEF/deser-4                       36.79n ±  0%   36.74n ± 2%        ~ (p=0.561 n=6)
DeserializeNative/STEFU/deser-4                      105.5n ±  1%   108.0n ± 2%   +2.32% (p=0.002 n=6)
SerializeFromPdata/STEF/serialize-4                  2.039µ ±  6%   1.687µ ± 2%  -17.29% (p=0.002 n=6)
SerializeFromPdata/STEFU/serialize-4                 502.0n ±  1%   508.5n ± 1%   +1.29% (p=0.011 n=6)
DeserializeToPdata/STEF/deserialize-4                695.5n ±  2%   699.5n ± 4%        ~ (p=0.461 n=6)
DeserializeToPdata/STEFU/deserialize-4               959.0n ±  3%   968.5n ± 2%        ~ (p=0.071 n=6)
STEFReaderRead-4                                     38.47n ±  1%   38.32n ± 1%        ~ (p=0.180 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       3.985µ ±  7%   3.568µ ± 4%  -10.48% (p=0.002 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4    101.55n ± 10%   86.08n ± 3%  -15.24% (p=0.002 n=6)
ReadSTEF-4                                           39.95n ±  1%   37.78n ± 1%   -5.43% (p=0.002 n=6)
ReadSTEFZ-4                                          51.86n ±  1%   47.53n ± 0%   -8.34% (p=0.002 n=6)
ReadSTEFZWriteSTEF-4                                 114.8n ±  2%   105.8n ± 0%   -7.88% (p=0.002 n=6)
geomean                                              220.6n         206.6n        -6.32%

                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │      B/op      │     B/op      vs base               │
SerializeNative/STEF/serialize-4                     3.341Mi ± 0%   3.340Mi ± 0%        ~ (p=0.937 n=6)
SerializeNative/STEFU/serialize-4                    7.557Mi ± 0%   7.327Mi ± 0%   -3.04% (p=0.002 n=6)
DeserializeNative/STEF/deser-4                       951.4Ki ± 0%   725.8Ki ± 0%  -23.71% (p=0.002 n=6)
DeserializeNative/STEFU/deser-4                      1.715Mi ± 0%   1.303Mi ± 0%  -24.03% (p=0.002 n=6)
SerializeFromPdata/STEF/serialize-4                  76.55Mi ± 0%   33.24Mi ± 0%  -56.58% (p=0.002 n=6)
SerializeFromPdata/STEFU/serialize-4                 7.557Mi ± 0%   7.327Mi ± 0%   -3.04% (p=0.002 n=6)
DeserializeToPdata/STEF/deserialize-4                34.83Mi ± 0%   34.61Mi ± 0%   -0.63% (p=0.002 n=6)
DeserializeToPdata/STEFU/deserialize-4               41.89Mi ± 0%   41.48Mi ± 0%   -0.98% (p=0.002 n=6)
STEFReaderRead-4                                     953.1Ki ± 0%   727.5Ki ± 0%  -23.67% (p=0.002 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       3.385Gi ± 0%   2.681Gi ± 0%  -20.79% (p=0.002 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     20.30Mi ± 0%   20.19Mi ± 0%   -0.56% (p=0.002 n=6)
ReadSTEF-4                                           953.1Ki ± 0%   727.5Ki ± 0%  -23.67% (p=0.002 n=6)
ReadSTEFZ-4                                          10.29Mi ± 0%   10.07Mi ± 0%   -2.14% (p=0.002 n=6)
ReadSTEFZWriteSTEF-4                                 13.44Mi ± 0%   13.21Mi ± 0%   -1.67% (p=0.002 n=6)
geomean                                              10.78Mi        9.163Mi       -15.00%

                                                 │ bench-main.txt │             bench-new.txt             │
                                                 │   allocs/op    │  allocs/op   vs base                  │
SerializeNative/STEF/serialize-4                      2.656k ± 0%   2.667k ± 0%    +0.38% (p=0.002 n=6)
SerializeNative/STEFU/serialize-4                      883.0 ± 0%    883.0 ± 0%         ~ (p=1.000 n=6) ¹
DeserializeNative/STEF/deser-4                         463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.002 n=6)
DeserializeNative/STEFU/deser-4                        496.0 ± 0%    496.0 ± 0%         ~ (p=1.000 n=6) ¹
SerializeFromPdata/STEF/serialize-4                   134.7k ± 0%   134.7k ± 0%         ~ (p=0.374 n=6)
SerializeFromPdata/STEFU/serialize-4                   884.0 ± 0%    884.0 ± 0%         ~ (p=1.000 n=6)
DeserializeToPdata/STEF/deserialize-4                 756.2k ± 0%   756.2k ± 0%    +0.00% (p=0.002 n=6)
DeserializeToPdata/STEFU/deserialize-4                944.9k ± 0%   944.9k ± 0%         ~ (p=1.000 n=6) ¹
STEFReaderRead-4                                       463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.002 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4        13.15M ± 0%   13.22M ± 0%    +0.51% (p=0.002 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4      1.956k ± 0%   7.921k ± 0%  +304.96% (p=0.002 n=6)
ReadSTEF-4                                             463.0 ± 0%    464.0 ± 0%    +0.22% (p=0.002 n=6)
ReadSTEFZ-4                                            500.0 ± 0%    500.0 ± 0%         ~ (p=1.000 n=6)
ReadSTEFZWriteSTEF-4                                  1.228k ± 0%   1.230k ± 0%    +0.16% (p=0.002 n=6)
geomean                                               6.300k        6.970k        +10.64%
¹ all samples are equal

Benchmark result

benchstat bench-new.txt
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 9V74 80-Core Processor                
                                                 │ bench-new.txt │
                                                 │    sec/op     │
SerializeNative/STEF/serialize-4                     6.759m ± 1%
SerializeNative/STEFU/serialize-4                    33.20m ± 1%
DeserializeNative/STEF/deser-4                       2.457m ± 2%
DeserializeNative/STEFU/deser-4                      7.215m ± 2%
SerializeFromPdata/STEF/serialize-4                  112.7m ± 1%
SerializeFromPdata/STEFU/serialize-4                 33.99m ± 1%
DeserializeToPdata/STEF/deserialize-4                46.73m ± 4%
DeserializeToPdata/STEFU/deserialize-4               64.69m ± 2%
STEFReaderRead-4                                     2.562m ± 1%
STEFSerializeMultipart/astronomy-otelmetrics-4        2.807 ± 4%
STEFDeserializeMultipart/astronomy-otelmetrics-4     67.72m ± 3%
ReadSTEF-4                                           2.524m ± 1%
ReadSTEFZ-4                                          3.176m ± 0%
ReadSTEFZWriteSTEF-4                                 7.068m ± 0%
geomean                                              19.64m

                                                 │ bench-new.txt │
                                                 │   sec/point   │
SerializeNative/STEF/serialize-4                     101.1n ± 1%
SerializeNative/STEFU/serialize-4                    496.6n ± 1%
DeserializeNative/STEF/deser-4                       36.74n ± 2%
DeserializeNative/STEFU/deser-4                      108.0n ± 2%
SerializeFromPdata/STEF/serialize-4                  1.687µ ± 2%
SerializeFromPdata/STEFU/serialize-4                 508.5n ± 1%
DeserializeToPdata/STEF/deserialize-4                699.5n ± 4%
DeserializeToPdata/STEFU/deserialize-4               968.5n ± 2%
STEFReaderRead-4                                     38.32n ± 1%
STEFSerializeMultipart/astronomy-otelmetrics-4       3.568µ ± 4%
STEFDeserializeMultipart/astronomy-otelmetrics-4     86.08n ± 3%
ReadSTEF-4                                           37.78n ± 1%
ReadSTEFZ-4                                          47.53n ± 0%
ReadSTEFZWriteSTEF-4                                 105.8n ± 0%
geomean                                              206.6n

                                                 │ bench-new.txt │
                                                 │     B/op      │
SerializeNative/STEF/serialize-4                    3.340Mi ± 0%
SerializeNative/STEFU/serialize-4                   7.327Mi ± 0%
DeserializeNative/STEF/deser-4                      725.8Ki ± 0%
DeserializeNative/STEFU/deser-4                     1.303Mi ± 0%
SerializeFromPdata/STEF/serialize-4                 33.24Mi ± 0%
SerializeFromPdata/STEFU/serialize-4                7.327Mi ± 0%
DeserializeToPdata/STEF/deserialize-4               34.61Mi ± 0%
DeserializeToPdata/STEFU/deserialize-4              41.48Mi ± 0%
STEFReaderRead-4                                    727.5Ki ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4      2.681Gi ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4    20.19Mi ± 0%
ReadSTEF-4                                          727.5Ki ± 0%
ReadSTEFZ-4                                         10.07Mi ± 0%
ReadSTEFZWriteSTEF-4                                13.21Mi ± 0%
geomean                                             9.163Mi

                                                 │ bench-new.txt │
                                                 │   allocs/op   │
SerializeNative/STEF/serialize-4                     2.667k ± 0%
SerializeNative/STEFU/serialize-4                     883.0 ± 0%
DeserializeNative/STEF/deser-4                        464.0 ± 0%
DeserializeNative/STEFU/deser-4                       496.0 ± 0%
SerializeFromPdata/STEF/serialize-4                  134.7k ± 0%
SerializeFromPdata/STEFU/serialize-4                  884.0 ± 0%
DeserializeToPdata/STEF/deserialize-4                756.2k ± 0%
DeserializeToPdata/STEFU/deserialize-4               944.9k ± 0%
STEFReaderRead-4                                      464.0 ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4       13.22M ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4     7.921k ± 0%
ReadSTEF-4                                            464.0 ± 0%
ReadSTEFZ-4                                           500.0 ± 0%
ReadSTEFZWriteSTEF-4                                 1.230k ± 0%
geomean                                              6.970k

Copilot

Pull request overview

Copilot reviewed 7 out of 36 changed files in this pull request and generated 2 comments.

Files not reviewed (29)

examples/ints/internal/ints/record.go: Language not supported
examples/jsonl/internal/jsonstef/jsonvalue.go: Language not supported
examples/jsonl/internal/jsonstef/record.go: Language not supported
examples/profile/internal/profile/function.go: Language not supported
examples/profile/internal/profile/labelvalue.go: Language not supported
examples/profile/internal/profile/line.go: Language not supported
examples/profile/internal/profile/location.go: Language not supported
examples/profile/internal/profile/mapping.go: Language not supported
examples/profile/internal/profile/numvalue.go: Language not supported
examples/profile/internal/profile/profilemetadata.go: Language not supported
examples/profile/internal/profile/sample.go: Language not supported
examples/profile/internal/profile/samplevalue.go: Language not supported
examples/profile/internal/profile/samplevaluetype.go: Language not supported
go/otel/otelstef/anyvalue.go: Language not supported
go/otel/otelstef/envelope.go: Language not supported
go/otel/otelstef/event.go: Language not supported
go/otel/otelstef/exemplar.go: Language not supported
go/otel/otelstef/exemplarvalue.go: Language not supported
go/otel/otelstef/exphistogrambuckets.go: Language not supported
go/otel/otelstef/exphistogramvalue.go: Language not supported
go/otel/otelstef/histogramvalue.go: Language not supported
go/otel/otelstef/link.go: Language not supported
go/otel/otelstef/metric.go: Language not supported
go/otel/otelstef/metrics.go: Language not supported
go/otel/otelstef/point.go: Language not supported
go/otel/otelstef/pointvalue.go: Language not supported
go/otel/otelstef/quantilevalue.go: Language not supported
go/otel/otelstef/resource.go: Language not supported
go/otel/otelstef/scope.go: Language not supported

Comments suppressed due to low confidence (1)

stefc/templates/go/struct.go.tmpl:125

The new exported Reset() forwards to reset(), but reset() does not clear modifiedFields.mask (or nested modified state). After a struct has been used once, calling Reset() leaves prior modified bits set, so subsequent encoding/diff operations can behave as if fields are still modified even though values were cleared. Either have Reset() also clear modified flags (e.g., by calling setUnmodifiedRecursively()/setting modifiedFields.mask=0 after resetting children) or adjust the Reset doc to avoid claiming an "initial state" when modified flags are preserved.

Copilot

Pull request overview

Copilot reviewed 7 out of 36 changed files in this pull request and generated 1 comment.

Files not reviewed (29)

examples/ints/internal/ints/record.go: Language not supported
examples/jsonl/internal/jsonstef/jsonvalue.go: Language not supported
examples/jsonl/internal/jsonstef/record.go: Language not supported
examples/profile/internal/profile/function.go: Language not supported
examples/profile/internal/profile/labelvalue.go: Language not supported
examples/profile/internal/profile/line.go: Language not supported
examples/profile/internal/profile/location.go: Language not supported
examples/profile/internal/profile/mapping.go: Language not supported
examples/profile/internal/profile/numvalue.go: Language not supported
examples/profile/internal/profile/profilemetadata.go: Language not supported
examples/profile/internal/profile/sample.go: Language not supported
examples/profile/internal/profile/samplevalue.go: Language not supported
examples/profile/internal/profile/samplevaluetype.go: Language not supported
go/otel/otelstef/anyvalue.go: Language not supported
go/otel/otelstef/envelope.go: Language not supported
go/otel/otelstef/event.go: Language not supported
go/otel/otelstef/exemplar.go: Language not supported
go/otel/otelstef/exemplarvalue.go: Language not supported
go/otel/otelstef/exphistogrambuckets.go: Language not supported
go/otel/otelstef/exphistogramvalue.go: Language not supported
go/otel/otelstef/histogramvalue.go: Language not supported
go/otel/otelstef/link.go: Language not supported
go/otel/otelstef/metric.go: Language not supported
go/otel/otelstef/metrics.go: Language not supported
go/otel/otelstef/point.go: Language not supported
go/otel/otelstef/pointvalue.go: Language not supported
go/otel/otelstef/quantilevalue.go: Language not supported
go/otel/otelstef/resource.go: Language not supported
go/otel/otelstef/scope.go: Language not supported

Borrows the storage format idea from https://github.com/tigrannajaryan/govariant This reduces memory usage when multiple records are kept in memory and also improves speed of some operations. Benchmarking results: ``` goos: darwin goarch: arm64 pkg: github.com/splunk/stef/benchmarks cpu: Apple M2 Pro │ bench_base.txt │ bench_current.txt │ │ sec/op │ sec/op vs base │ SerializeNative/STEF/serialize-10 5.000m ± 4% 3.193m ± 3% -36.14% (p=0.000 n=18) DeserializeNative/STEF/deser-10 1.479m ± 7% 1.489m ± 5% ~ (p=0.839 n=18) SerializeFromPdata/STEF/serialize-10 70.37m ± 3% 58.67m ± 4% -16.63% (p=0.000 n=18) DeserializeToPdata/STEF/deserialize-10 24.92m ± 2% 24.89m ± 2% ~ (p=0.815 n=18) STEFReaderRead-10 1.523m ± 3% 1.495m ± 6% ~ (p=0.323 n=18) STEFSerializeMultipart/astronomy-otelmetrics-10 1.958 ± 3% 1.808 ± 3% -7.68% (p=0.000 n=18) STEFDeserializeMultipart/astronomy-otelmetrics-10 45.87m ± 3% 46.92m ± 6% ~ (p=0.143 n=18) ReadSTEF-10 1.695m ± 2% 1.655m ± 3% -2.35% (p=0.008 n=18) ReadSTEFZ-10 2.218m ± 2% 2.227m ± 3% ~ (p=0.988 n=18) ReadSTEFZWriteSTEF-10 4.682m ± 4% 4.735m ± 5% ~ (p=0.134 n=18) geomean 11.21m 10.44m -6.85% │ bench_base.txt │ bench_current.txt │ │ sec/point │ sec/point vs base │ SerializeNative/STEF/serialize-10 74.79n ± 4% 47.76n ± 3% -36.14% (p=0.000 n=18) DeserializeNative/STEF/deser-10 22.12n ± 7% 22.27n ± 5% ~ (p=0.833 n=18) SerializeFromPdata/STEF/serialize-10 1053.5n ± 3% 878.0n ± 4% -16.66% (p=0.000 n=18) DeserializeToPdata/STEF/deserialize-10 373.0n ± 2% 372.5n ± 2% ~ (p=0.784 n=18) STEFReaderRead-10 22.79n ± 3% 22.36n ± 6% ~ (p=0.319 n=18) STEFSerializeMultipart/astronomy-otelmetrics-10 2.489µ ± 3% 2.298µ ± 3% -7.68% (p=0.000 n=18) STEFDeserializeMultipart/astronomy-otelmetrics-10 58.30n ± 3% 59.63n ± 6% ~ (p=0.145 n=18) ReadSTEF-10 25.37n ± 2% 24.77n ± 3% -2.35% (p=0.008 n=18) ReadSTEFZ-10 33.19n ± 2% 33.33n ± 3% ~ (p=0.981 n=18) ReadSTEFZWriteSTEF-10 70.07n ± 4% 70.86n ± 5% ~ (p=0.132 n=18) geomean 102.4n 95.39n -6.85% │ bench_base.txt │ bench_current.txt │ │ B/op │ B/op vs base │ SerializeNative/STEF/serialize-10 3.373Mi ± 0% 3.395Mi ± 0% +0.63% (p=0.000 n=18) DeserializeNative/STEF/deser-10 951.4Ki ± 0% 725.8Ki ± 0% -23.71% (p=0.000 n=18) SerializeFromPdata/STEF/serialize-10 79.76Mi ± 0% 36.49Mi ± 0% -54.25% (p=0.000 n=18) DeserializeToPdata/STEF/deserialize-10 34.83Mi ± 0% 34.61Mi ± 0% -0.63% (p=0.000 n=18) STEFReaderRead-10 953.1Ki ± 0% 727.5Ki ± 0% -23.67% (p=0.000 n=18) STEFSerializeMultipart/astronomy-otelmetrics-10 4.868Gi ± 0% 4.168Gi ± 0% -14.37% (p=0.000 n=18) STEFDeserializeMultipart/astronomy-otelmetrics-10 20.30Mi ± 0% 20.19Mi ± 0% -0.56% (p=0.000 n=18) ReadSTEF-10 953.1Ki ± 0% 727.5Ki ± 0% -23.67% (p=0.000 n=18) ReadSTEFZ-10 10.29Mi ± 0% 10.07Mi ± 0% -2.14% (p=0.000 n=18) ReadSTEFZWriteSTEF-10 13.45Mi ± 0% 13.22Mi ± 0% -1.67% (p=0.000 n=18) geomean 12.66Mi 10.58Mi -16.41% │ bench_base.txt │ bench_current.txt │ │ allocs/op │ allocs/op vs base │ SerializeNative/STEF/serialize-10 2.642k ± 0% 2.662k ± 0% +0.76% (p=0.000 n=18) DeserializeNative/STEF/deser-10 463.0 ± 0% 464.0 ± 0% +0.22% (p=0.000 n=18) SerializeFromPdata/STEF/serialize-10 134.7k ± 0% 134.7k ± 0% +0.03% (p=0.021 n=18) DeserializeToPdata/STEF/deserialize-10 756.2k ± 0% 756.2k ± 0% +0.00% (p=0.000 n=18) STEFReaderRead-10 463.0 ± 0% 464.0 ± 0% +0.22% (p=0.000 n=18) STEFSerializeMultipart/astronomy-otelmetrics-10 13.15M ± 0% 13.22M ± 0% +0.51% (p=0.000 n=18) STEFDeserializeMultipart/astronomy-otelmetrics-10 1.958k ± 0% 7.923k ± 0% +304.65% (p=0.000 n=18) ReadSTEF-10 463.0 ± 0% 464.0 ± 0% +0.22% (p=0.000 n=18) ReadSTEFZ-10 503.0 ± 0% 504.0 ± 0% +0.20% (p=0.000 n=18) ReadSTEFZWriteSTEF-10 1.230k ± 0% 1.232k ± 0% +0.16% (p=0.000 n=18) geomean 7.293k 8.406k +15.27% ``` Also added a benchmarks that reads many records at once to show how much memory is reduced: ``` goos: darwin goarch: arm64 pkg: github.com/splunk/stef/benchmarks cpu: Apple M2 Pro │ main.txt │ after.txt │ │ sec/op │ sec/op vs base │ ReaderReadMany-10 12.087m ± 5% 9.782m ± 2% -19.07% (p=0.002 n=6) │ main.txt │ after.txt │ │ sec/point │ sec/point vs base │ ReaderReadMany-10 181.0n ± 5% 146.5n ± 2% -19.06% (p=0.002 n=6) │ main.txt │ after.txt │ │ B/op │ B/op vs base │ ReaderReadMany-10 2111.4Ki ± 6% 921.2Ki ± 1% -56.37% (p=0.002 n=6) │ main.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ ReaderReadMany-10 970.0 ± 5% 845.5 ± 2% -12.84% (p=0.002 n=6) ```

Copilot

Pull request overview

Copilot reviewed 7 out of 36 changed files in this pull request and generated no new comments.

Files not reviewed (29)

examples/ints/internal/ints/record.go: Language not supported
examples/jsonl/internal/jsonstef/jsonvalue.go: Language not supported
examples/jsonl/internal/jsonstef/record.go: Language not supported
examples/profile/internal/profile/function.go: Language not supported
examples/profile/internal/profile/labelvalue.go: Language not supported
examples/profile/internal/profile/line.go: Language not supported
examples/profile/internal/profile/location.go: Language not supported
examples/profile/internal/profile/mapping.go: Language not supported
examples/profile/internal/profile/numvalue.go: Language not supported
examples/profile/internal/profile/profilemetadata.go: Language not supported
examples/profile/internal/profile/sample.go: Language not supported
examples/profile/internal/profile/samplevalue.go: Language not supported
examples/profile/internal/profile/samplevaluetype.go: Language not supported
go/otel/otelstef/anyvalue.go: Language not supported
go/otel/otelstef/envelope.go: Language not supported
go/otel/otelstef/event.go: Language not supported
go/otel/otelstef/exemplar.go: Language not supported
go/otel/otelstef/exemplarvalue.go: Language not supported
go/otel/otelstef/exphistogrambuckets.go: Language not supported
go/otel/otelstef/exphistogramvalue.go: Language not supported
go/otel/otelstef/histogramvalue.go: Language not supported
go/otel/otelstef/link.go: Language not supported
go/otel/otelstef/metric.go: Language not supported
go/otel/otelstef/metrics.go: Language not supported
go/otel/otelstef/point.go: Language not supported
go/otel/otelstef/pointvalue.go: Language not supported
go/otel/otelstef/quantilevalue.go: Language not supported
go/otel/otelstef/resource.go: Language not supported
go/otel/otelstef/scope.go: Language not supported

tigrannajaryan requested a review from Copilot May 20, 2026 21:58

Copilot AI reviewed May 20, 2026

View reviewed changes

tigrannajaryan requested a review from Copilot May 20, 2026 22:04

Copilot started reviewing on behalf of tigrannajaryan May 20, 2026 22:05 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread stefc/templates/go/oneof.go.tmpl

Comment thread go/otel/otelstef/pointvalue.go

tigrannajaryan force-pushed the tigran/oneof-variant branch from e46b667 to d510019 Compare May 20, 2026 22:40

tigrannajaryan requested a review from Copilot May 20, 2026 22:41

Copilot started reviewing on behalf of tigrannajaryan May 20, 2026 22:41 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread stefc/templates/go/oneof.go.tmpl

tigrannajaryan force-pushed the tigran/oneof-variant branch from d510019 to af8261b Compare May 20, 2026 23:10

tigrannajaryan requested a review from Copilot May 20, 2026 23:10

Copilot started reviewing on behalf of tigrannajaryan May 20, 2026 23:10 View session

tigrannajaryan marked this pull request as ready for review May 20, 2026 23:16

Copilot AI reviewed May 20, 2026

View reviewed changes

tigrannajaryan merged commit 6e20c15 into main May 20, 2026
10 checks passed

tigrannajaryan deleted the tigran/oneof-variant branch May 20, 2026 23:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use compact oneof memory structure#418

Use compact oneof memory structure#418
tigrannajaryan merged 1 commit into
mainfrom
tigran/oneof-variant

tigrannajaryan commented May 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented May 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tigrannajaryan commented May 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Result

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 20, 2026 •

edited

Loading