Pull data for hydrus PTR

[leqwin]

• Fetch tags for a given image by hash
• Fetch implications and aliases for tags that already exist in monbooru

[gary-host-laptop]

How is Monbooru going to handle the extra categories/namespaces present in the PTR? I think you favoured copyright: instead of series: / studio: but once PTR integration happens I think those would need to be added as defaults to avoid breakage. I think something sound would be to keep all of them. I imagine the workflow like this: user adds post from danbooru with a bunch of tags, category is copyright:, user calls ptr rules, tags get implications/aliases that transform copyright: tags into series: studio: etc.

[leqwin]

There are two cases to consider :
1°) When fetching tags for a given image hash from the PTR into monbooru : we can just do the same mapping that exists for hydrus imports (https://github.com/leqwin/monbooru/blob/main/docs/MIGRATING.md#hydrus-network). so series:xxx becomes copyright:xxx, there is no issue.
2°) For reverse lookups (we have a tag in monbooru and want to get its implications/aliases) I think your idea is the right one if I understand it correctly : we reverse the mapping table and search the best solution. (so if monbooru asks copyright:xxx we can search in the PTR for copyright:xxx, series:xxx, and studio:xxx and return the one that gets matches).

The real issue are other custom namespaces that exists in the PTR that we have no way to know, and that gets flattened into the general category when imported. (A cake:chocolate tag in PTR becomes general:chocolate in monbooru and the lookup would have no way to find it back in the PTR; in the same way if a monbooru user create a general:chocolate tag directly it won't reconcile).

I've queried the PTR to see what the distributions of namespaces are. It turns out that 90% of the tags are in the common mapped categories that should be handled correctly, 5% are book keeping namespaces (page: , source:, title:) that we don't need for aliases/implications, and 5% are other namespace like clothes: gender: color:, bodypart: patreon_id: etc... . These would get flattened in general and wouldn't get aliases/implications. They represent only 1.5% of aliases/implications though so it's not that much of a deal.

We could imagine an index that record the tag in monbooru <-> tag in PTR matching when the tag is pulled from the PTR into monbooru, but it would be weird to manage, and wouldn't solve the case where the tag has been created in monbooru and not imported from the PTR and we want to do a lookup in the PTR, since there would be no mapping.

[gary-host-laptop]

I guess I was thinking PTR would become the source of truth, I imagined that it's better maintained than Danbooru in terms of having more separation of namespaces. Like the cases I mentioned were Danbooru uses tags like copyright:mario_series but maybe it's not as tidy? If a user fetches the PTR for an item (the item currently has only copyright namespace) then series: and studio: are added along with copyright? that's what you're proposing? If so, kind of what I proposed was to have some sort of automated process where copyright namespaces are aliased into series/studio if the user prefers to use PTR as the source of truth or instead use Danbooru, alternatively keep all of them?

From my understanding of Hydrus docs, the PTR has a set of maintained namespaces, the rest that might show up probably are used by some users but not actively maintained by the PTR janitors. title: is used for the title of an image, let's say a painting, the rest of the "book keeping" namespaces are managed by what Monbooru calls metadata right? I've used title: a few times in monbooru but it doesn't feel very nice since you have the restrictions of a tag and titles could have expressive titles that can't be represented.

We could imagine an index that record the tag in monbooru <-> tag in PTR matching when the tag is pulled from the PTR into monbooru, but it would be weird to manage, and wouldn't solve the case where the tag has been created in monbooru and not imported from the PTR and we want to do a lookup in the PTR, since there would be no mapping.

Regarding this, would it be simpler if the indexing is handled separately of the PTR fetching? Monloader only fetches tags from PTR, they get added into Monbooru (copyright, series and studio all live together now), and then you can manage them independently without having to deal with it in the middle of the import. Here you could alias all copyright tags into series/studio based on the tag name. This would also be beneficial since at some point, in my opinion, the tags page could use an improvement with tag management automation. For example, currently batch operations are not possible (select a few tags and add X tag as alias), or batch change of namespace, batch delete, etc. It's a rough idea but I think it could work...

[leqwin]

I think that the PTR should be one source of truth not the unique source of truth. One of the main reason for this is that I will make enabling the PTR in monloader opt-in, since it will require to download and sync a 70GB database, the other is that I think that managing multiple providers would give the best experience, since they have forces and weaknesses. From what I've seen I wouldn't say that the PTR is significantly more maintained than danbooru, but it does have a bigger database. On the other side danbooru provides metadata (original source, commentary, annotations...) that the PTR doesn't have.
I'm thinking of the the following design : Instead of pulling tags from the PTR only, you would have a lookup button that fetch and merge tags and other data from :

• the PTR if its enabled
• online boorus (danbooru, gelbooru, konachan, etc... have a md5: search that allows you to reverse search an image from its hash)
• similarity services like iqdb and saucenao. The weakness of both the PTR and booru's is that reverse search is by exact hash so it doesn't survive image compression or cropping. Using these service we could with a bit of luck find the source (even the danbooru original) of an image that has been modified.

I think a setup like this would be superior to a simple PTR search as it would allow you to browse in a single go almost all of the curated booru surface of the web.
However, doing this requires to reconcile different tagging system, that's why I think the mapping is unavoidable since danbooru and hydrus have different mapping system anyway.

For the batch tags management, I think its a good feature request idea for monbooru. And yeah, this is what will be required if the user really wants to map the tags that are in the uncommon hydrus namespaces.

[leqwin]

To answer your earlier question directly: for me a PTR fetch shouldn't add series: and studio: alongside copyright:. All three hydrus namespaces should collapse into monbooru's copyright:, the same as a danbooru import, and the reverse lookup (tag -> aliases/implications) treats copyright/series/studio as equivalent in both directions. That's less a PTR choice than a danbooru one: danbooru already files both series and studio under copyright, so as long as danbooru is one of the merged providers, monbooru's single copyright category is what keeps them consistent. Giving series/studio their own categories would put monbooru out of parity with danbooru (which is the other reason I'd keep the mapping rather than chase per-namespace fidelity.)

[gary-host-laptop]

Oof, 70GB, that's a lot, recently my biggest HDD died so I'm very low on storage, I don't even know if I'll be able to use/test it then... But I understand and agree with what you said.

Would the workflow be something like this? User presses fetch button and picks which sources to fetch from (or configures the fetch sources from the settings). Tags are fetched. If user wants to cherry pick namespaces this is done in the tags page through manual/semi-automated process of aliasing/implications.

One more thing, what are your thoughts on data pushing to the PTR? Pulling is a good start to solidify the experience and it's the feature most users would use, but ultimately pushing would be the nice thing to do to contribute to the community. Just asking to know what you think about it, it's not a priority by any means.

[leqwin]

Oof, 70GB, that's a lot, recently my biggest HDD died so I'm very low on storage, I don't even know if I'll be able to use/test it then... But I understand and agree with what you said.

It also take a while to download, had to sync for 2 whole days to download it fully. The database is really huge however (196M hashes, 52M tags and 3616M mappings) and it's nice to be able to pull from it locally once its downloaded, but it's quite a commitment. Also since it works by exact hashes you can't grab tags for images that have been re-encoded, cropped, etc... That's why I'm thinking a future could be to replace the current image with the original found in iqdb/saucenao to be able to pull the images from the PTR in these cases.

Would the workflow be something like this? User presses fetch button and picks which sources to fetch from (or configures the fetch sources from the settings). Tags are fetched. If user wants to cherry pick namespaces this is done in the tags page through manual/semi-automated process of aliasing/implications.

Pretty much: The user get's a "Lookup" button similar to the "Auto-tag" button and can pull from (individual or batch) :

• The PTR (exact hash match)
• An online booru, with multiple candidates that can be ranked in monloader (first match wins)
  danbooru (exact match)
  iqdb (similarity)
  saucenao (similarity)
  rule34 (exact match)
  e621 (exact match)
  yandere (exact match)
  konachan (exact match)
  If the exact match triggers, the tag are pulled. If the similarity gives an image with similarity > 80% and returns a booru source the tags are pulled from it, if it does not return a boory source and none of the other booru source match then only the source URL is pulled.
• Both the PTR and an online booru

For the PTR each tag get's a lookup button in the tags page to pull its aliases and implied tags. This can be done individually or by batch action.

One more thing, what are your thoughts on data pushing to the PTR? Pulling is a good start to solidify the experience and it's the feature most users would use, but ultimately pushing would be the nice thing to do to contribute to the community. Just asking to know what you think about it, it's not a priority by any means.

I haven't look at how writing to the PTR works but I'd love to add this when the read-only part is completely stable. It seems natural that an alternative client to a community project such as the PTR should be able to contribute.

[leqwin]

Done in v1.4.0 and can be used with monbooru >=v1.14.0. Contribution will be for a future version.

[gary-host-laptop]

hey, do you think there's something wrong with this or is it just incredibly super slow? I started downloading the day of the release and it's only there, the last 2 days it barely moved.

[leqwin]

Interesting. I checked the hydrus doc and it seems there is a real difference whether you sync on HDD or SSD.

It took me about 2 days on SSD but apparently on HDD you can expect weeks if you are using it.
I see 3 things to fix on the monloader side :

• The sentence "it can take several hours" is too optimistic
• The download speed (3.9 KiB/s) is underestimated because it does not exclude disk processing time
• The progress should be non-linear. Updates contain a lot more mappins after 2018 so it's showing you're almost 50% in update but you're only 10% of the total mapping.

Hydrus suggest leaving the sync running in the background, they also have an option to download a pre-synced client :

Maybe it could be an option for monbooru as well to direct download a pre-seeded db up to 2026 to speed up this step.

Edit : Also, are you completely stuck on update 2169 or are you (slowly) moving? There may be some improvements to do on how some of the largest blobs are handled and the 2169 is one of them.