Allow multipart downloading when pulling from remote

[eric-higgins-ai]

This is just an illustration of my idea for addressing #340

[alex-aizman]

Eric,

Thanks for putting in the effort. As an illustration, this did its job: it helped clarify the direction I’ll take. I don’t yet know the timeline.

The next step will be per-request selection via api.GetArgs.

Most of the mechanism already exists: apc.HdrBlobDownload, apc.HdrBlobWorkers, and apc.HdrBlobChunk. To keep the development incremental and consistent with the existing code, I may expose those headers directly through GetArgs, or add dedicated typed fields. That remains to be seen.

The advantage is that this can be one self-contained commit, with no additional persistent configuration to validate, propagate, and maintain. The tradeoff is that the caller must know it is about to read a large object.

A subsequent step may add a server-side default: cluster config would provide the default, while bucket properties would inherit and optionally override it.

One additional motivation for that second step is support for third-party clients such as Boto3/Botocore.

The threshold decision belongs at the bucket level. A bucket containing 20GB shards and one containing 100KB JSON objects want opposite defaults, and a single cluster-wide setting cannot express that.

Again, I appreciate the push on this one!

[alex-aizman]

PS. Speaking of config, we actually already have it. But let us do part-one first, and then due diligence on the second one.

[eric-higgins-ai]

Some questions:

The next step will be per-request selection via api.GetArgs.

What's still needed here? I think the existing headers already expose the things that we want?

PS. Speaking of config, we actually already have it

Do you have a pointer to it? I didn't see anything like this when I looked at it

[alex-aizman]

Do you have a pointer to it?

https://github.com/NVIDIA/aistore/blob/main/cmn/api.go#L68

and further:

$ git grep "Bprops.*Chunks" ais/t*
ais/tgtobj.go:252:      maxMonoSize := int64(poi.lom.Bprops().Chunks.MaxMonolithicSize)
ais/tgtobj.go:259:              return poi.chunk(int64(poi.lom.Bprops().Chunks.ChunkSize))
ais/tgtobj.go:867:      case goi.lom.Bprops().Chunks.AutoEnabled():
ais/tgtobj.go:1808:     dstMaxMonoSize := dst.Bprops().Chunks.MaxMonolithicSize
ais/tgtobj.go:1831:     case lom.Bprops().Chunks.MaxMonolithicSize != dstMaxMonoSize && lom.Lsize() > int64(dstMaxMonoSize):
ais/tgtobj.go:1833:             res = coi._chunk(t, lom, dst, int64(dst.Bprops().Chunks.ChunkSize))
$ 

[eric-higgins-ai]

is it really desirable to couple multipart download config to chunking config for uploads? Intuitively I would think people may want different configurations for these.

The next step will be per-request selection via api.GetArgs.
What's still needed here? I think the existing headers already expose the things that we want?

bump on this question

I'm also happy to implement this if that's helpful - I'd like to get this resolved in a somewhat timely fashion

[Nahemah1022]

Hi @eric-higgins-ai, feel free to proceed with part one. Please keep it focused on the Go client API - the server path already exists. Add an optional typed blob-download setting to api.GetArgs: non-nil enables blob download.
Translate those values into the existing HdrBlobDownload, HdrBlobChunk, and HdrBlobWorkers headers in the shared GetArgs request-building path.
Make sure it handles potential conflicts with values already supplied through GetArgs.Header. Add focused tests for defaults, explicit values, and conflicts. Once it’s ready, I’ll review it.

[eric-higgins-ai]

@Nahemah1022 it's still not clear to me why we're doing this - why do y'all want HdrBlobDownload, HdrBlobChunk, and HdrBlobWorkers to be specified via query params (?) instead of headers? It doesn't seem really related to my request, which was to configure these headers on the server side