Azure Blob storage: pooled reads, streaming serializer, buffered writes by egil · Pull Request #9879 · dotnet/orleans

egil · 2026-01-13T23:15:35Z

Summary

Add IGrainStorageStreamingSerializer plus stream overloads for OrleansJsonSerializer; Orleans and Newtonsoft.Json grain storage serializers implement it.
Azure Blob storage can use pooled read buffers via DownloadStreamingAsync, and supports a buffered stream write mode using pooled segments; default write path remains BinaryData.
Add logging when pooled reads fall back for payloads > int.MaxValue.

Tests/Benchmarks

New Azure Blob storage tests covering pooled reads and streaming serializer behavior (default binary writes vs buffered stream writes).
Added focused grain storage benchmarks (read/write, binary vs streaming) across Orleans, Newtonsoft.Json, and STJ, with LOH-sized payloads. Note: results are from Azurite on a local machine, so CPU timing is noisy but allocation/GC deltas are consistent.

Experiments/Decisions

Tried OpenWriteAsync for fully streaming uploads; rejected due to an ETag/concurrency race when GetPropertiesAsync is needed post-upload.
Explored IBufferWriter/ReadOnlySequence serializer paths; dropped to keep the API surface smaller and because Blob SDK paths are stream-first.
Considered size hints and rented byte[] for writes; not reliable with unknown payload sizes in Orleans. Buffered stream uploads were kept as the best allocation/compatibility tradeoff while keeping BinaryData as the default for throughput.

Ill post measurements from my laptop in comments below.

Copilot

Pull request overview

This PR introduces an opt-in performance optimization for Azure Blob Storage reads by using pooled buffers to reduce memory pressure on the Large Object Heap (LOH) and minimize Gen2 garbage collection. The change adds a new UsePooledBufferForReads option to AzureBlobStorageOptions that, when enabled, switches from DownloadContentAsync to DownloadStreamingAsync with ArrayPool-rented buffers.

Changes:

Added UsePooledBufferForReads option to AzureBlobStorageOptions for opt-in buffer pooling
Modified ReadStateAsync to use streaming download with pooled buffers when the option is enabled
Added comprehensive test coverage with PersistenceGrainTests_AzureBlobStore_PooledReads
Added performance benchmark AzureBlobReadStateBenchmark demonstrating 56-74% memory allocation reduction for large payloads

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorageOptions.cs	Adds new `UsePooledBufferForReads` boolean option with documentation warning about buffer retention
src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs	Implements pooled buffer logic using `DownloadStreamingAsync` and `ArrayPool<byte>.Shared` when option is enabled
test/Extensions/TesterAzureUtils/Persistence/PersistenceGrainTests_AzureBlobStore_PooledReads.cs	New test class verifying functionality with pooled reads enabled
test/Benchmarks/GrainStorage/AzureBlobReadStateBenchmark.cs	New benchmark comparing pooled vs non-pooled read performance
test/Benchmarks/Program.cs	Adds benchmark runner entry for the new Azure Blob read state benchmark
test/Benchmarks/run_test.cmd	Updates script to run the new benchmark
test/Benchmarks/Properties/launchSettings.json	Removes hardcoded launch profile

ReubenBond · 2026-01-14T17:20:44Z

I assume most of the remaining allocations are from the actual state object being loaded. Does that sound right? I also see an eTag string + the BinaryData object + the async state machine, but those are not worth going after at this point.

egil · 2026-01-14T17:24:47Z

I assume most of the remaining allocations are from the actual state object being loaded. Does that sound right?

That is my assumption too.

I also see an eTag string + the BinaryData object + the async state machine, but those are not worth going after at this point.

Agreed. First stab at this is to reduce GC churn. Similar pattern may be applicable to other storage providers though, but I do feel like blob storage is more likely to have larger payloads than e.g. a SQL server.

Long term it would be great to do away with byte[] entirely and pass a stream or similar abstraction directly to the grain storage serializer, but that requires changing API surface.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

egil · 2026-01-15T23:37:42Z


BenchmarkDotNet v0.15.6, Windows 11 (10.0.26200.7623)
13th Gen Intel Core i7-13800H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.102
  [Host]     : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3
  Job-WRIKEU : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3

IterationCount=5  LaunchCount=1  WarmupCount=2

Method	PayloadSize	SerializerKind	Mean	Error	StdDev	Ratio	RatioSD	Gen0	Gen1	Gen2	Allocated	Alloc Ratio
ReadStateNonPooledAsync	4096	Orleans	1.255 ms	0.1456 ms	0.0378 ms	1.00	0.04	-	-	-	20.3 KB	1.00
ReadStatePooledAsync	4096	Orleans	1.204 ms	0.1723 ms	0.0447 ms	0.96	0.04	-	-	-	19.17 KB	0.94

ReadStateNonPooledAsync	4096	NewtonsoftJson	1.675 ms	1.0016 ms	0.1550 ms	1.01	0.12	-	-	-	44.66 KB	1.00
ReadStatePooledAsync	4096	NewtonsoftJson	1.729 ms	0.3535 ms	0.0547 ms	1.04	0.10	-	-	-	41.96 KB	0.94

ReadStateNonPooledAsync	4096	SystemTextJson	1.293 ms	0.5266 ms	0.0815 ms	1.00	0.08	-	-	-	24.88 KB	1.00
ReadStatePooledAsync	4096	SystemTextJson	1.308 ms	0.5689 ms	0.1478 ms	1.01	0.12	-	-	-	22.25 KB	0.89

ReadStateNonPooledAsync	65536	Orleans	2.637 ms	1.1559 ms	0.3002 ms	1.01	0.15	-	-	-	46.33 KB	1.00
ReadStatePooledAsync	65536	Orleans	1.378 ms	0.5652 ms	0.1468 ms	0.53	0.08	-	-	-	34.39 KB	0.74

ReadStateNonPooledAsync	65536	NewtonsoftJson	1.736 ms	0.2337 ms	0.0607 ms	1.00	0.04	10.0000	3.3333	3.3333	440.02 KB	1.00
ReadStatePooledAsync	65536	NewtonsoftJson	1.606 ms	0.3397 ms	0.0526 ms	0.93	0.04	6.6667	3.3333	3.3333	402.3 KB	0.91

ReadStateNonPooledAsync	65536	SystemTextJson	1.539 ms	0.1674 ms	0.0435 ms	1.00	0.04	-	-	-	119.34 KB	1.00
ReadStatePooledAsync	65536	SystemTextJson	1.992 ms	1.2706 ms	0.1966 ms	1.30	0.12	-	-	-	82.46 KB	0.69

ReadStateNonPooledAsync	131072	Orleans	1.438 ms	0.3246 ms	0.0843 ms	1.00	0.08	-	-	-	70.33 KB	1.00
ReadStatePooledAsync	131072	Orleans	1.460 ms	0.6371 ms	0.1655 ms	1.02	0.12	-	-	-	50.39 KB	0.72

ReadStateNonPooledAsync	131072	NewtonsoftJson	2.191 ms	0.6722 ms	0.1746 ms	1.01	0.10	16.0000	12.0000	12.0000	902.13 KB	1.00
ReadStatePooledAsync	131072	NewtonsoftJson	4.126 ms	5.3489 ms	1.3891 ms	1.89	0.60	15.0000	10.0000	10.0000	786.67 KB	0.87

ReadStateNonPooledAsync	131072	SystemTextJson	1.866 ms	0.8013 ms	0.2081 ms	1.01	0.15	3.3333	-	-	271.33 KB	1.00
ReadStatePooledAsync	131072	SystemTextJson	2.052 ms	0.7261 ms	0.1886 ms	1.11	0.15	-	-	-	146.73 KB	0.54

egil · 2026-01-15T23:38:31Z


BenchmarkDotNet v0.15.6, Windows 11 (10.0.26200.7623)
13th Gen Intel Core i7-13800H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.102
  [Host]     : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3
  Job-WRIKEU : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3

IterationCount=5  LaunchCount=1  WarmupCount=2

Method	PayloadSize	SerializerKind	Mean	Error	StdDev	Median	Ratio	RatioSD	Gen0	Gen1	Gen2	Allocated	Alloc Ratio
ReadStateBinaryAsync	65536	Orleans	1.434 ms	0.8179 ms	0.2124 ms	1.402 ms	1.02	0.19	-	-	-	46.33 KB	1.00
ReadStateStreamAsync	65536	Orleans	1.370 ms	0.2190 ms	0.0569 ms	1.360 ms	0.97	0.13	-	-	-	34.85 KB	0.75

ReadStateBinaryAsync	65536	NewtonsoftJson	2.658 ms	3.7338 ms	0.9697 ms	2.411 ms	1.11	0.53	5.0000	-	-	448.97 KB	1.00
ReadStateStreamAsync	65536	NewtonsoftJson	2.281 ms	3.0020 ms	0.7796 ms	1.808 ms	0.96	0.44	4.0000	-	-	340.7 KB	0.76

ReadStateBinaryAsync	65536	SystemTextJson	2.520 ms	3.6914 ms	0.9586 ms	2.320 ms	1.12	0.54	-	-	-	122.21 KB	1.00
ReadStateStreamAsync	65536	SystemTextJson	1.826 ms	1.6279 ms	0.2519 ms	1.927 ms	0.81	0.28	-	-	-	83.08 KB	0.68

ReadStateBinaryAsync	131072	Orleans	1.467 ms	0.4653 ms	0.1208 ms	1.479 ms	1.01	0.11	-	-	-	70.33 KB	1.00
ReadStateStreamAsync	131072	Orleans	2.084 ms	3.2107 ms	0.8338 ms	1.637 ms	1.43	0.53	-	-	-	50.85 KB	0.72

ReadStateBinaryAsync	131072	NewtonsoftJson	2.390 ms	1.0338 ms	0.2685 ms	2.491 ms	1.01	0.15	16.0000	12.0000	12.0000	906.9 KB	1.00
ReadStateStreamAsync	131072	NewtonsoftJson	2.045 ms	0.4543 ms	0.1180 ms	2.086 ms	0.87	0.11	16.0000	8.0000	8.0000	2041.11 KB	2.25

ReadStateBinaryAsync	131072	SystemTextJson	1.724 ms	0.8376 ms	0.2175 ms	1.717 ms	1.01	0.16	3.3333	-	-	278.56 KB	1.00
ReadStateStreamAsync	131072	SystemTextJson	1.712 ms	0.5361 ms	0.1392 ms	1.713 ms	1.01	0.14	-	-	-	147.42 KB	0.53

egil · 2026-01-15T23:38:53Z


BenchmarkDotNet v0.15.6, Windows 11 (10.0.26200.7623)
13th Gen Intel Core i7-13800H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.102
  [Host]     : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3
  Job-WRIKEU : .NET 10.0.2 (10.0.2, 10.0.225.61305), X64 RyuJIT x86-64-v3

IterationCount=5  LaunchCount=1  WarmupCount=2

Method	PayloadSize	SerializerKind	Mean	Error	StdDev	Ratio	RatioSD	Allocated	Alloc Ratio
WriteStateBinaryAsync	65536	Orleans	3.265 ms	0.9563 ms	0.2484 ms	1.00	0.10	43.99 KB	1.00
WriteStateStreamAsync	65536	Orleans	3.303 ms	1.2792 ms	0.1980 ms	1.02	0.09	20.53 KB	0.47

WriteStateBinaryAsync	65536	NewtonsoftJson	5.659 ms	1.1809 ms	0.1827 ms	1.00	0.04	197 KB	1.00
WriteStateStreamAsync	65536	NewtonsoftJson	3.688 ms	0.5972 ms	0.1551 ms	0.65	0.03	27.68 KB	0.14

WriteStateBinaryAsync	65536	SystemTextJson	3.670 ms	1.1722 ms	0.1814 ms	1.00	0.06	51.52 KB	1.00
WriteStateStreamAsync	65536	SystemTextJson	3.744 ms	0.7988 ms	0.2075 ms	1.02	0.07	21.64 KB	0.42

WriteStateBinaryAsync	131072	Orleans	3.621 ms	1.4803 ms	0.3844 ms	1.01	0.14	67.96 KB	1.00
WriteStateStreamAsync	131072	Orleans	3.891 ms	1.2095 ms	0.1872 ms	1.08	0.11	20.54 KB	0.30

WriteStateBinaryAsync	131072	NewtonsoftJson	7.757 ms	6.4783 ms	1.0025 ms	1.01	0.17	361.02 KB	1.00
WriteStateStreamAsync	131072	NewtonsoftJson	6.566 ms	1.5394 ms	0.2382 ms	0.86	0.11	27.67 KB	0.08

WriteStateBinaryAsync	131072	SystemTextJson	5.252 ms	4.9974 ms	0.7734 ms	1.02	0.21	83.52 KB	1.00
WriteStateStreamAsync	131072	SystemTextJson	6.462 ms	5.5679 ms	1.4460 ms	1.25	0.32	22.11 KB	0.26

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 7 comments.

egil · 2026-01-16T17:05:02Z

Not sure why this test is failing, and if it has anything to do with the changes on this PR:

  Failed UnitTests.General.AllowCallChainReentrancyTests.CallChainReentrancy_WithSuppression [2 m]
  Error Message:
   System.OperationCanceledException : The operation was canceled.
  Stack Trace:
     at System.Threading.Channels.AsyncOperation`1.GetResult(Int16 token)
   at UnitTests.General.CallChainReentrancyTestHelper.CallChainObserver.WaitForOperationAsync(CallChainOperation operationType, String grain, Int32 callIndex, CancellationToken cancellationToken) in /_/test/TesterInternal/MessageScheduling/CallChainReentrancyTestHelper.cs:line 208
   at UnitTests.General.CallChainReentrancyTestHelper.CallChainReentrancy_WithSuppression() in /_/test/TesterInternal/MessageScheduling/CallChainReentrancyTestHelper.cs:line 175
   at UnitTests.General.AllowCallChainReentrancyTests.CallChainReentrancy_WithSuppression() in /_/test/TesterInternal/MessageScheduling/AllowCallChainReentrancyTests.cs:line 75
--- End of stack trace from previous location ---

egil · 2026-04-01T09:15:10Z

If I remember correctly, there is another change to the api surface you had I mind @ReubenBond? Or so you think this is ready to be merged in?

Ahh yes, this: #9879 (comment)

egil · 2026-04-02T17:21:26Z

Thanks for the help getting this ready @ReubenBond. Let me know if there are anything else I can help with at this point.

Introduce IGrainStorageStreamingSerializer and stream overloads for OrleansJsonSerializer plus Orleans/Json grain storage serializers. Azure blob storage now supports pooled read buffers, a buffered stream write mode, and logs a warning when large payloads force a fallback to DownloadContentAsync. Add Azure Blob storage tests for pooled reads and streaming serializer behavior, and add focused grain storage benchmarks (binary vs streaming) across Orleans, Newtonsoft.Json, and STJ. Alternatives considered: full streaming OpenWriteAsync with separate ETag readback (rejected due to concurrency race), and IBufferWriter/ReadOnlySequence buffer paths (explored, but didnt improve performance or allocation in a meaningful way, so but dropped for now). Buffered stream uploads and pooled reads were kept as the best allocation/compatibility tradeoff while keeping BinaryData writes as the default.

…zeAsync return type

Delay ETag assignment until Azure Blob state deserialization succeeds so failed reads do not mutate the in-memory grain state. Also ensure pooled read buffers are always returned and add regression tests for failed stream, pooled, and binary deserialization paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use IGrainStorageStreamingSerializer as the derived serializer contract so AzureBlobGrainStorage can work from a single serializer field after the PR 9879 rebase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

gfoidl reviewed Jan 14, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs Outdated

egil force-pushed the pooled-azure-blob-storage-read branch from 7a23245 to 7ec8b60 Compare January 14, 2026 16:49

egil marked this pull request as ready for review January 14, 2026 16:51

Copilot AI review requested due to automatic review settings January 14, 2026 16:51

Copilot started reviewing on behalf of egil January 14, 2026 16:51 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs Outdated

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs Outdated

Comment thread test/Benchmarks/run_test.cmd Outdated

egil force-pushed the pooled-azure-blob-storage-read branch from 7ec8b60 to 99a2f67 Compare January 14, 2026 17:04

ReubenBond reviewed Jan 14, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs Outdated

egil force-pushed the pooled-azure-blob-storage-read branch from 99a2f67 to b8fb3bc Compare January 14, 2026 18:04

egil requested review from ReubenBond and Copilot January 14, 2026 23:57

Copilot started reviewing on behalf of egil January 14, 2026 23:57 View session

Copilot AI reviewed Jan 15, 2026

View reviewed changes

egil marked this pull request as draft January 15, 2026 11:00

egil force-pushed the pooled-azure-blob-storage-read branch 2 times, most recently from e4321a9 to 370139e Compare January 15, 2026 22:26

egil changed the title ~~Azure blob reads: add optional pooled buffers for streaming downloads~~ Azure Blob storage: pooled reads, streaming serializer, buffered writes Jan 15, 2026

egil force-pushed the pooled-azure-blob-storage-read branch 3 times, most recently from cee520a to 7793d64 Compare January 15, 2026 23:36

egil marked this pull request as ready for review January 15, 2026 23:40

egil requested a review from Copilot January 15, 2026 23:40

Copilot started reviewing on behalf of egil January 15, 2026 23:41 View session

Copilot AI reviewed Jan 15, 2026

View reviewed changes

egil force-pushed the pooled-azure-blob-storage-read branch from 7793d64 to 59a6eac Compare January 16, 2026 09:16

egil commented Jan 17, 2026

View reviewed changes

Comment thread src/Orleans.Core/Providers/IGrainStorageSerializer.cs Outdated

oleg-baulo reviewed Jan 21, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorageOptions.cs Outdated

egil force-pushed the pooled-azure-blob-storage-read branch from 77c0d03 to 485d518 Compare January 22, 2026 10:18

egil commented Apr 1, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorageOptions.cs Outdated

ReubenBond force-pushed the pooled-azure-blob-storage-read branch from 386f633 to 6324e87 Compare April 2, 2026 15:10

ReubenBond reviewed Apr 2, 2026

View reviewed changes

Comment thread src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorageOptions.cs Outdated

ReubenBond approved these changes Apr 2, 2026

View reviewed changes

ReubenBond added this pull request to the merge queue Apr 2, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 2, 2026

egil and others added 7 commits April 2, 2026 10:23

Add nullable annotation to IGrainStorageStreamingSerializer.Deseriali…

03bf133

…zeAsync return type

Use proper word in code comments

58b14c8

Fix Redis storage serializer nullability

1aaec03

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove Azure blob serializer mode options

98a27f5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ReubenBond force-pushed the pooled-azure-blob-storage-read branch from 9dbbe81 to 98a27f5 Compare April 2, 2026 17:23

ReubenBond enabled auto-merge April 2, 2026 17:23

ReubenBond added this pull request to the merge queue Apr 2, 2026

Merged via the queue into dotnet:main with commit 36a80a4 Apr 2, 2026
58 of 59 checks passed

Conversation

egil commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests/Benchmarks

Experiments/Decisions

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ReubenBond commented Jan 14, 2026

Uh oh!

egil commented Jan 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egil commented Jan 15, 2026

Uh oh!

egil commented Jan 15, 2026

Uh oh!

egil commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egil commented Jan 16, 2026

Uh oh!

Uh oh!

Uh oh!

egil commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egil commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

egil commented Jan 13, 2026 •

edited

Loading

egil commented Apr 1, 2026 •

edited

Loading