PEDS-1486: (BE) - Optimization changes to improve delivery speed

[kellcuc]

JIRA: https://pcdc.atlassian.net/browse/PEDS-1486

After changes

Before changes

Locally, the file changed in size, but we dropped 271ms which should be a savings in prod. One of the big items was the logging we were outputting, as well as a circle loop, that seemed over repetitive. An option for more speed, would be a caching layer, and we also look at 100000 at one time, maybe lower this, unless there is a reason for such a large pull?

[kellcuc]

Note that we can add caching

[kellcuc]

We were doing double work with json.loads(json.dumps(args.get("filterSets"))), and so we use what we already had in args args = utils.parse.parse_request_json() and parse that.

[kellcuc]

We go for a shallow vs deep copy of the list here, because i could not find that we needed inner dicts.

[paulmurdoch19]

was this ticket not about the table one? If not we should add that to this task since it has the same issue and I believe you can use the graphql endpoint there since we just need to know which consortiums exist

[kellcuc]

@paulmurdoch19 this was actually a two part ticket.

1. ui
2. the performance

@paulmurdoch19 are you talking about this table: ./analysis/tools/stats/consortiums

[paulmurdoch19]

I thought we created this ticket because this was to slow but I might be thinking of something else

PcdcAnalysisTools/PcdcAnalysisTools/blueprint/routes/views/tableOne/__init__.py

Line 39 in d063e15

def get_result():

[paulmurdoch19]

I think the big reason why it takes so long is because we are using the /download endpoint which attempts to transmit 40000 dictionaries with 5 key pairs in prod for the all filter. I am not sure but if its possible to get the needed data but we could try and use the /graphql endpoint.

[kellcuc]

graphql endpoint option, I will test (thanks for endpoint Paul).
https://github.com/chicagopcdc/guppy/blob/0751949be68b4936127e5478e757c8462a098a7a/src/server/server.js#L51

[kellcuc]

using the path=capp.config['GUPPY_API'] + "/graphql", - 186 ms

{
    "3": {
        "count": {
            "fitted": 2,
            "total": 23
        },
        "name": "1. instruct",
        "risktable": [
            {
                "nrisk": 2,
                "time": 0
            },
            {
                "nrisk": 0,
                "time": 1
            }
        ],
        "survival": [
            {
                "prob": 1,
                "time": 0
            },
            {
                "prob": 0.5,
                "time": 0.03174112025878608
            },
            {
                "prob": 0,
                "time": 0.05307142996156295
            }
        ]
    }
}

Using the path=capp.config['GUPPY_API'] + "/download", - 161ms

{
  "3": {
    "count": {
      "fitted": 2,
      "total": 23
    },
    "name": "1. instruct",
    "risktable": [
      {
        "nrisk": 2,
        "time": 0
      },
      {
        "nrisk": 0,
        "time": 1
      }
    ],
    "survival": [
      {
        "prob": 1,
        "time": 0
      },
      {
        "prob": 0.5,
        "time": 0.03174112025878608
      },
      {
        "prob": 0,
        "time": 0.05307142996156295
      }
    ]
  }
}

[paulmurdoch19]

that's going to change at scale in prod there's 40k dictionaries that have to be downloaded in the worst case and that's what's causing it to take 15 sec. When a user selects a filter-set to use it checks to see if the filter-set is in scope using this endpoint /analysis/tools/stats/consortiums which uses the guppy download endpoint. Since we're only finding the consortium counts we can have the /analysis/tools/stats/consortiums use the graphql endpoint instead similar to how the summary page displays the counts.

[grugna]

the display counts is actually reading from the same download endpoint and cache the result since they are static between data release.

[paulmurdoch19]

yes not the display counts the consortium count in the charts portion of the explorer page.