Add notebook-gallery plugin to auto-generate gallery style cards on index pages

[jaladh-singhal]

Fixes #233

Added a custom local MyST plugin (notebook-gallery directive) that auto-generates a notebook gallery we have for the Euclid index page and used it for all other mission pages.

Design decisions:

@bsipocz you probably care about this more (since we had past conversations about this):

• Implemented this as a MyST JS plugin (the .mjs file) rather than a Python script, as the MyST plugin system integrates JS plugins cleanly at build time and avoids a separate generation step for executable plugins. Also if we want more complexity we can easily migrate to widgets from JS route - right now we don't need it because we are not working with HTML DOM directly
• Took inspiration from JB example-js-plugin
• The plugin depends on the yaml npm package for parsing notebook_metadata.yml. So a build step is required and I had to add package.json (+package-lock.json) and a bundling step via esbuild in tox.ini (as opposed to a single self-contained .mjs file).
  • This has an added benefit: we can eventually move plugins/ to a separate irsa-myst-plugins repo and publish it as npm/GH package if we decide to reuse it across projects or to contribute upstream.

Things to check:

• Test mission index pages: WISE, SPHEREx, Simulated Data, Techniques & Tools, Euclid
• Title differences in the Euclid gallery cards at ops and this PR build
  • Can be fixed if we add short titles from toc.yml to notebook_metadata.yml (they should be reconciled)
  • If not, I can try modifying the plugin to override title for some notebooks but it will make code a bit more complex
• Exception cases in euclid.md will be removed before merging

[jaladh-singhal]

This is relevant for build step changes in tox: https://app.circleci.com/pipelines/github/Caltech-IPAC/irsa-tutorials/849/workflows/b6cd0cd5-fd7d-43d6-9d30-86bd8a2c1fc6/jobs/826?invite=true#step-103-10522_83

Also note L265: the plugin is loaded successfully (right now we just have 1 directive and that's what it tells us)

[bsipocz]

This looks great, thank you so much.

I think 1) this should go in as first version before we jump into with new ideas for improvements (with the removal of the test sections of course -- those could maybe go into a new test md file in the plugin itself, or added as a test md file if/when the plugin will live in its separate repo)

[bsipocz]

This is OK for now, but ultimately we may want to ship the plugin separately from this repo.

[jaladh-singhal]

I agree - and note the plural in "plugins", I envision it as a suite of plugins that we can maintain, ship and potentially share with MyST community.

[bsipocz]

I am definitely thinking of multiple plugins, whether to ship them one per repo or multiple in a plugins one is a separate question :)

[bsipocz]

It's OK here, but we will need to make sure npm is available for tox? I suppose we can kind of rely on that it's available already for JB so we can just piggy-back?

[jaladh-singhal]

Yes, it seemed to work locally as well as on the CircleCI/GA runner had npm pre-installed (which is common for most of their runner images). I don't think we particularly piggybacked on JB because we install JB via pip as python package not as an npm package.

That being said, I didn't think this through and we should be aware of it if we run into npm missing issues. Another reason why plugins/ should be it's own repo and we don't maintain the build step here at irsa-tutorials.

[bsipocz]

If this goes in with the merge we will need to open a follow-up issue to ensure that eventually we turn back on the checks.

[jaladh-singhal]

I will revert it before merging.

The first build failed (when strict was on) not because of plugin installation step but because of some notebook or something (I couldn't tell because the logs are way more verbose than my attention span). But yeah, I'm afraid it's becoming more common to remove strict for these kind of PR builds, maybe we should make it a label like skip testing GH label?

[bsipocz]

Haha, yes, we do use debug level logging to have more info available when there are issues.

Ideally no content executing testing was done with these changes as you haven't touched the notebooks themselves; so I suppose we should go back and check what was going on with that failure.

[bsipocz]

I like this approach very much; thanks Jaladh.

However; I already start seeing a list of options for improvements. What would work best, comment on the plugin itself or here argue for the usecase?

[jaladh-singhal]

I mean it's upto you. If you wish to get this merged, then you can open an issue listing all improvements you see. Else you can comment here and I can iterate within the PR

[jaladh-singhal]

those could maybe go into a new test md file in the plugin itself, or added as a test md file if/when the plugin will live in its separate repo)

@bsipocz Agreed. I plan to add a readme when it becomes more mature and independent. That readme will be a good place for these test cases.

[bsipocz]

OK, so features that comes to mind, but I don't think we should wait for any of them and just get this PR merged as is already:

• being able to wildcard the list of notebooks for the gallery. I like the current explicit approach, but I think it would make sense to be able to also say tutorials/euclid/*md

• alternatively to the wildcard, and this is very IRSA specific, we could also match on the field 'section'.

• we could raise an error for the unrecognised notebooks when using --strict. I'm not sure how complicated this would be.

• have the test cases moved into the plugin directory/future repository; as you suggested a README would suffice to start with as it could both serve as documentation and test

• probably a separate plugin/widget idea; or maybe an add-on feature here: do add a filtering on top of the gallery view. This would definitely be a feature that would also be super useful for upstream.

[troyraen]

This is great, thank you!