Render and serve fully static result pages from S3 to improve performance

[vlatada]

Currently, result pages are rendered by the API server and returned as HTML. Static assets including cutout images and SED fit data are fetched from S3 during this process server-side. We recently improved the efficiency of this rendering by generating compressed images (a.k.a "thumbnails") for the cutout images and SED fit plots. However, the Django database is queried multiple times during the rendering process, both to determine the location of these static images and to extract the other scientific data values displayed on the results page.

We want to explore the feasibility of rendering a complete static results page during the workflow that is stored alongside other static assets like the thumbnail files in the S3 bucket. The results() rendering function would first check if this static page file existed at a canonical location (e.g. .../result_page/2026abc/2026abc.results.html), streaming it directly from the bucket if so; otherwise rendering it dynamically as is currently done.

[manning-ncsa]

You're probably going to want to take a hybrid approach, where we generate static subcomponents similar to the thumbnails, that are downloaded and passed to the Django rendering function. This will necessarily be true to some extent because you cannot export a complete static page that includes the page components that rely on the authenticated user state. There are tables of calculated data values that are constructed from several database queries (local and global photometry, for example) that could be rendered once and exported as HTML files that contain only these subcomponents. The question is whether it is faster to make these db queries as we are now or to check the existence of and download the prerendered HTML components. Another problem I foresee with pre-rendering page sections is that changing page styling will be more challenging if we introduce or delete CSS classes that are baked into the static component files.

There is a good chance that the db queries are fastest, in which case this issue may pivot to our other idea, which is to experiment with static assets being loaded client-side using an S3-to-HTTP proxy that we add to the application deployment. This approach would significantly alleviate the load on the Django API server replicas and be the simplest to implement on the Django side, because the image data download-and-HTML-injection procedure would be replaced with populating <img src=...> tags with the URLs of the images.

[manning-ncsa]

Performance in production has been adequate, so the motivation to pursue this has diminished. Closing until it becomes relevant again.