json destructor quite slow

[wolfv]

Description

I was running patch over a ~100Mb repodata file and noticed that it was slow due to copying and destroying the entire object multiple times.

I am quite curious if the performance of destroy() could be improved? On my computer, it takes roughly 1 second to parse the file and create the json object, but also 0.5 seconds to destroy the json object.

Reproduction steps

Download a large JSON file, such as https://conda.anaconda.org/conda-forge/linux-64/repodata.json (curl --compressed https://conda.anaconda.org/conda-forge/linux-64/repodata.json)

and run the following code:

#include <iostream>
#include <chrono>
#include <nlohmann/json.hpp>

int
main()
{
    std::ifstream rdata("repodata.json");
    std::unique_ptr<nlohmann::json> j = std::make_unique<nlohmann::json>();
    
    {
        auto t0 = std::chrono::high_resolution_clock::now();
        rdata >> (*j);
        auto t1 = std::chrono::high_resolution_clock::now();
        std::cout << "took " << std::chrono::duration_cast<std::chrono::milliseconds>(t1-t0).count() <<" ms." << std::endl;
    }

    {
        auto t0 = std::chrono::high_resolution_clock::now();
        j.reset();
        auto t1 = std::chrono::high_resolution_clock::now();
        std::cout << "took " << std::chrono::duration_cast<std::chrono::milliseconds>(t1-t0).count() <<" ms." << std::endl;
    }
}

Expected vs. actual results

destruction should have minimal runtime cost

Minimal code example

No response

Error messages

No response

Compiler and operating system

clang 12, macOS

Library version

3.10.5

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[falbrechtskirchinger]

Performance is not a strong focus of this library, so generally, the answer to "Can X be done faster?" is yes.

I halved the runtime of the destructor without too much effort, but I have to think about this more carefully before submitting a PR.

[falbrechtskirchinger]

I'm going to wait for #3575 to be closed, as that PR touches the same functions and I don't want to deal with the inevitable merge conflicts.

As a preview, here are the respective call graphs of the current destruction implementation (destroy by flattening) and a simple and obvious optimization (destroy by iterative traversal). The number of calls is reduced from ~14e6 to ~6e6, about halving the time taken to destroy repodata.json.

[wolfv]

impressive!

[gregmarr]

As a preview, here are the respective call graphs of the current destruction implementation (destroy by flattening) and a simple and obvious optimization (destroy by iterative traversal). The number of calls is reduced from ~14e6 to ~6e6, about halving the time taken to destroy repodata.json.

Just checking, is this still preserving the non-recursive nature of the destroy operation?

[falbrechtskirchinger]

Of course. I hate recursion with passion. :-)

[gregmarr]

Of course. I hate recursion with passion. :-)

I assumed so based on the call counts, but I just wanted to check before you got too far down the path, because the current non-recursive implementation is due to a past stack overflow issue destroying deeply nested json objects.

[TheZoc]

@falbrechtskirchinger Do you happen to (still?) have a branch with this modification?
I'd like to give it a try