respect robots.txt

[eriktks]

Add respect for robots.txt rules before visiting websites, to default browser plugin.

Notes:

1. Allow users to provide list of URLs, one per line
2. If line does not contain URL, assume DOI and convert to: https://doi.org/DOI
3. Assume opening url list is user-generated action, equivalent "Open All Bookmarks" in Firefox: no robots.txt checks required
4. Assume opening detected PDF links found by software as bot action: robots.txt check necessary
5. Offer blocked PDF links as clickable link on status page of browser plugin, see issue build extra tab for control and status #2