Skip to content

When running at scale, sleep in code can have dramatic affect. #2

@mike-gangl

Description

@mike-gangl

https://github.com/SWOT-Confluence/ssc_input/blob/e9930c51e78673d2b09bd569674d6ea30e8c5d78/ssc_input.py#L133C9-L137C39

These sleep commands, which will almost certainly happen as we are hitting CMR with many queries, cause constant delays (up to two minutes). For reference, we have ~200k reaches to process, so at 1% of those hitting a CMR Error, we might be looking at (assuming average) a 2000 minutes of delay in SSC_input processing.

We can respect the 'retry-after' header in the CMR Queries:
https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#request-moderation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions