Expose sketch memory footprint.

[tabac]

In some scenarios it is useful to know how much memory a sketch is using (stack + heap). This is something not currently supported.

I could work on a PR that adds a size() function starting from HLL, CPC sketches. Would that be something useful to add? Is there an approach that's more preferable in dealing with this?

[tisonkun]

Feel free to send a PR for review.

BTW, do you find similar functions on datasketches-java or datasketches-cpp? If so, we'd better keep the logic the same, or else it can be a candidate addition to those libraries.

[tabac]

BTW, do you find similar functions on datasketches-java or datasketches-cpp? If so, we'd better keep the logic the same, or else it can be a candidate addition to those libraries.

Sorry forgot to mention that. I didn't find this functionality in the Java, C++ implementations. What I was able to find is functions that estimate the sketch's serialised size, or the the maximum size a sketch can grow to.

I will take a closer look and incorporate any findings in my PR. Thanks!

[tisonkun]

@tabac Thanks for your information! You can ping me as a reviewer when the PR is ready or if there is anything uncertain.

[leerho]

In languages with Garbage Collection, getting the dynamic memory allocation at a point in time is rather nebulous. This is the main reason we have never provided that functionality in Java.

Could you describe the scenarios where having this information would be critical?

Nonetheless, if a particular language finds it useful and easy to add I don't see any objection to it. We don't require that the same exact APIs exist across all languages. Having the same basic functionality for the sketching itself would be important, but auxiliary functions like size(), not so much. :)

[tisonkun]

#136 merged. All existing sketches are covered, IIRC.

Now tracked at #137.

[tisonkun]

@leerho

In languages with Garbage Collection, getting the dynamic memory allocation at a point in time is rather nebulous. This is the main reason we have never provided that functionality in Java.

Ah. That's it. In Rust (and C++, I believe), memory allocation can be audited, so we can try to provide such information.

Could you describe the scenarios where having this information would be critical?

I can foresee two scenarios:

1. For observability, like understanding which parts take the most memory, alerting, etc.
2. For capacity estimation, and possibly a quota limiter.

Nonetheless, if a particular language finds it useful and easy to add I don't see any objection to it. We don't require that the same exact APIs exist across all languages. Having the same basic functionality for the sketching itself would be important, but auxiliary functions like size(), not so much. :)

For sure.