Allow packages from sources other than GitHub

[f-f]

Moving over here from purescript/registry#15, which mentions that the only problems of allowing packages to be published from sources other than GitHub come from Pursuit relying on GitHub for some things. Quoting from there:

Right now Pursuit only supports packages hosted on GitHub, and nobody has complained [..]. The reasons for this are that allows us to use the GitHub API to get a rendered HTML readme (although I now consider that a misfeature and I'd like to remove it), and also it allows us to construct source links, since the source links take you to GitHub. If we allow non-GH packages in the registry, this means we won't be able to upload those packages to Pursuit right now (which is not a dealbreaker but probably worth considering). I'd quite like to have package sources hosted in the same place as the HTML API docs for source links though, which will probably be easier to do now with our shiny new registry; if we did that we'd be able to allow non-GH packages without losing source links.

[toastal]

For Sourcehut you can get users and rendered README in one fell swoop through their GraphQL API (schema).

query {
 repositoryByName(name: "purescript-web-page-visibility") {
    name
    readme
    owner { id }
    description
  }
  user(username: "toastal") { id username }
}

Which returns something like

{
  "data": {
    "repositoryByName": {
      "name": "purescript-web-page-visibility",
      "readme": "$ESCAPED_README_AS_HTML",
      "description": "Type definitions and low level interface implementations for W3C Page Visibility API",
      "owner": {
        "id": 11755
      }
    },
    "user": {
      "id": 11755,
      "username": "toastal"
    }
  }

Querying through cURL is something like

$ oauth_token=your oauth token
$ curl \
  --oauth2-bearer "$oauth_token" \
  -H 'Content-Type: application/json' \
  -d '{"query": "{ version { major, minor, patch } }"}' \
  https://meta.sr.ht/query

with full documentation.

If I was more familiar with Haskell (and by that I mean getting tooling set up and working), I would work on this.

[hdgarrood]

Rather than maintaining numerous integrations, I’d rather have Pursuit know as little as possible about the various code hosting services there are. In particular I would rather move in the direction of having the JSON upload contain Markdown which will be rendered by Pursuit itself instead of using APIs to render readmes. See #150 (comment)

[toastal]

Oh, I wouldn't disagree, but now you definitely need to have your own sanitizer. One of my quibbles with GitHub's Markdown renderer, is that it removes <abbr> tags that add to accessibility, but if you let in script tags and who knows what else, that could be a concern (I assume this was 100% already thought about though).

---

One thing I appreciate about Sourcehut is that through their GraphQL API, you can POST any sort of HTML -- BYOR, bring your own renderer. Similar to not relying on a singular Git forge, you're also not relying on a markup language. Markdown has no specific spec really and is quite limiting for many application. In my own case, I ran asciidoctor from my Nix Flake package and had the build system cURL POST the rendered HTML output. I would personally say BYOR is preferable -- use reStructuredText or Org Mode or whatever lightweight markup suits you.

[hdgarrood]

Yes, we already have a sanitiser, for rendering the markdown in doc-comments. The problem with allowing people to bring their own renderer is that you then have to account for any kind of dodgy HTML they might produce. We can already sanitise markdown, but sanitising HTML is harder.

[toastal]

Then will you bring a Markdown flavor that supports tables, abbreviations, admonitions, imports, and more out-of-the-box features? That's my biggest concern more than syntax A vs syntax B.

[hdgarrood]

No - this is just for the package’s readme on Pursuit. If you need all those things, I’d suggest setting up a separate documentation site.

[toastal]

Ah, that is reasonable 🖖 (this said, abbreviation support I think is a good accessibility feature)