Conversation
included a full size embedded jpeg so generating mipmap cache for anything larger than mipmap 3 requires the raw to be loaded and processed. This script copies the paired jpeg from the RAW+JPEG pair to the full resolution mipmap. This can then be used to generate the remaining mipmap cache. The jpeg image can be discarded after being copied to the cache or retained based on a preference in Lua options.
| if not pmj.keep_jpgs then | ||
| refresh_collection() | ||
| end | ||
| dt.util.message(MODULE, "responsive_cache", "build cache") |
There was a problem hiding this comment.
It looks like "responsive_cache" needs to be updated to this script name.
There was a problem hiding this comment.
I misunderstood what the message function does, so now I see that this is interprocess communication, not a message to the user. So my previous comment should be ignored. Is the responsive_cache script available somewhere? What happens if the user doesn't have it installed?
There was a problem hiding this comment.
Once upon a time long, long ago in a...
Before the crawler existed the only way to generate cache was darktable-generate-cache. The problem was it generated all the cache sizes for all of the images in the database. What I wanted was a way to generate just the cache I needed, when I needed it (right after import). So I added dt_lua_image_t:generate_cache() to the Lua API and started working on a script, gen_cache, to generate just the cache I needed (at that time HD and 2K) right after an import. The next iteration was import_cache, which was smarter with more features and options. The third iteration is responsive_cache which finally satisfies all my needs. It runs after import generating size 3 and 6 cache. When I edit I tend to do long sessions hitting the space bar to go to the next image. I hated waiting when I returned to lighttable for all the thumbnails to reload, so responsive_cache regenerates the updated thumbnail in background after I change to the next image and when I return to lighttable all the thumbnails are up to date.
I have an R7 and the largest embedded JPG is mipmap size 3. So I could generate cache for size 3 at 100+ images/sec. When I did size 6 it was a little more than 3/sec. I shoot sports so most imports are 1000+ images and generating the cache took 15 minutes or so. While responsive cache was running, no other scripts could run because Lua is single threaded. So, I added dt.util.message() so that I could have "cooperative multitasking". With the addition of a simple FIFO scheduler I could "pause" responsive_cache and let other scripts run then resume. When I added use_paired_jpg_as_mipmap, responsive_cache would try and run before use_paired_jpg_as_mipmap, so then I changed responsive_cache to not run on import and wait for a message from use_paired_jpg_as_mipmap to run. Now cache generation after import is around a minute.
The crawler works, but you have to wait for it and it generates way too much cache. I tried it as Hanno was implementing it and let it run for awhile and it ran me out of disk space. It will be even worse now since the new max mipmap size is 10, so it will generate size 8 and 9 too.
I hadn't thought to add responsive_cache to the "merge fest" but I guess I could...
| -- local ds = require "lib/dtutils.string" | ||
| -- local dtsys = require "lib/dtutils.system" | ||
| local log = require "lib/dtutils.log" | ||
| -- local debug = require "darktable.debug" |
There was a problem hiding this comment.
Should the commented out lines be removed?
There was a problem hiding this comment.
Probably. When I write a script I use a boilerplate and a bash script that fills in some of the blanks. I tend to leave the above lines commented out because too often I found myself adding something and needing another library. But, when I'm done I'm sometimes lazy and don't clean up after myself.
There was a problem hiding this comment.
It looks like there are some other functions defined that aren't used. I read through all of them when learning how the script worked, only to find that some of them were unused, so it would be helpful to trim to just what is needed.
|
|
||
| use_paired_jpg_as_mipmap looks for RAW+JPEG image pairs as images are imported. After | ||
| import the JPEG image is copied to the mipmap cache as the full resolution mipmap. Requests | ||
| for smaller mipmap sizes can be satisfied by down sampling the full resolution mipmap. User's can |
There was a problem hiding this comment.
| for smaller mipmap sizes can be satisfied by down sampling the full resolution mipmap. User's can | |
| for smaller mipmap sizes can be satisfied by down sampling the full resolution mipmap. Users can |
There was a problem hiding this comment.
I tend to use apostrophes way too often... :-(
|
I just tested this on Darktable 5.4.1, importing 96 raw files with 96 paired jpg files (in the same folder, with deletion enabled). After import, I hit F for a full screen view and then zoomed to 100% and started quickly moving through photos. Darktable started fully using all 8 cpu cores (but no gpu), and the UI became frozen for a bit with a "working..." message on the screen. After about two minutes it became responsive again. ... Oh, I now see that I accidentally had OpenCL disabled. (It's glitchy for me after a suspend-resume cycle.) So it looks like DT was generating the mipmap cache from the raw files, which I didn't expect to happen given that this script had already created full size mipmaps. [Edit: Based on the tests below, I don't think it was using the raw files to generate the mipmap cache.] Relevant settings:
Maybe the last one caused an issue? But I thought it would have just downscaled the jpgs already in the cache. And I thought that this only ran when darktable was idle for at least 5s. |
|
Let me know what log options I can use to provide more info. (And of course thanks so much for providing this!) |
|
I started over, removing the folder from darktable, and restoring the jpgs. I changed the start time for the background thumbnails to 25s, and made sure to not be idle that long. I also enabled OpenCL. Still, darktable used all 8 cpu cores right after import. I started over again, this time completely disabling the background thumbnail generation ("never"). Still, after I had scrolled through a few of the images, darktable started using all 8 cpu cores and no GPU. Another issue: for some reason, after each attempt, between 10 and 20 of the jpg files were not deleted (out of the 96 I had). |
|
As another experiment, I wrote a version that works when you import a folder that just contains raw files, with the corresponding jpg files available in a jpg subfolder (or really, a symlink to the right place). This avoids having to import the jpg and then remove it from darktable. Also, I hardlink the jpg to the cache dir, so no data needs to be written to disk, and therefore this is very fast. (This only works because my cache directory is on the same filesystem as my images. If you want to get rid of the jpg files, then instead a "mv" is what would accomplish this quickly.) It runs as each image is imported, rather than at the end. I verified that it correctly and quickly makes the hardlinks. However, again I found that when I start viewing the images at 100%, darktable uses all 8 cores and the UI gets very laggy. Is it possible that darktable doesn't know about these files that got added to the cache, and we need to tell it to update its view of the cache? I also noticed that no cache files got created at different sizes, besides the ones at size 8. (I'm on 5.4.1.) -- Rough draft. Needs checking of return codes, etc.
dt.register_event(MODULE, "post-import-image",
function(event, image)
dt.print_log("Importing " .. image.filename)
local extension = string.lower(df.get_filetype(image.filename))
if not string.match(extension, "jpg") then
local mipmap_dir = get_mipmap_dir() .. PS .. MAX_MIPMAP_SIZE .. PS
local rc = df.mkdir(mipmap_dir)
dt.print_log("rc from mkdir is " .. rc)
local basename = df.get_basename(image.filename)
local raw_image_id = image.id
-- todo: check existence; handle JPG vs jpg
local fname = image.path .. PS .. "jpg" .. PS .. basename .. ".JPG"
local mipname = mipmap_dir .. raw_image_id .. ".jpg"
-- todo: quote filenames:
local command = "ln " .. fname .. " " .. mipname
dt.print_log("running: " .. command)
rc = dtsys.external_command(command)
dt.print_log("rc: " .. rc)
-- If successful, could also trigger creation of size 5 here.
-- Or could do via responsive_cache in the background after import.
end
end
) |
|
The above code will work on Linux and probably MacOS, but not on Windows.
I'll add some debug logging so that we can see what is going on.
I usually avoid that because on a slow filesystem (or if you're doing a lot) the import slows to a crawl. It's safer to buffer up the images and process them in a job after so darktable isn't "hung" waiting on the import to complete. |
Yes, it's very specific to my situation, so I don't think it's useful for others as it is. But the idea of using "mv" rather than "copy and later delete" in the case the user wants the jpg files deleted might be a good one. On some filesystems, "mv" will be instantaneous, but it will fall back to doing the right thing. And there could be a "hardlink_or_copy" function that does a hardlink on systems that support it, and falls back to a copy when needed.
Great, I'll definitely test more. Let me know if I should test with an empty config dir, and if so, if there are some settings I should change before doing the import. The fact that some jpg files remained at the end is really suspicious, so some debugging related to that might be helpful. Maybe the issue has to do with the jpg files getting imported, and then removed one by one after the import is done? Can we instead use the post-import-image hook to prevent them from being imported in the first place (but still keeping a list of them to use in the post-import-film hook)?
In general, I agree, but hardlinks are instantaneous, so this is really fast in my situation, and I know when the import is done that the script has finished. |
|
Ok, I think I figured out what is going on. The way I usually import a folder is to start darktable with "darktable /path/to/folder" or "darktable ." when I'm already in the correct directory. This avoids me having to navigate to the right folder in darktable's interface. But with some extra logging, I'm now seeing that only some of the imported files have the associated jpg copied to the mipmap cache. For example, one time when I imported 96 files, only the last 16 were in the cache. So I think there is a race condition: darktable starts doing the import before the lua script has a chance to register itself to handle imports. This also explains why some of the jpg files weren't deleted. If I instead add the folder from an already started darktable process, then all of the jpgs get added to the cache, and it's reasonably fast to flip between them (although not as fast as I'd like, so I'm going to do something like your responsive_cache approach to precompute smaller sizes as well). Can darktable be adjusted to finish loading lua modules before importing a folder specified on the command line? |
That's a really good question. I'll have to do some testing I added some more logging to the script, I'll push an update in a little bit Here's responsive_cache. I changed the trigger message from use_paired_jpg_as_mipmap in the update I'm going to post and in responive_cache, so get the update and they should work fine together. |
trigger message for cache building scripts
|
Did you forget to update the message in this PR to match the one in the zip file? Also, I'm curious about the hardcoded cache widths in the zip file, as they don't exactly match the sizes given here: https://docs.darktable.org/lua/stable/lua.api.manual/types/dt_lua_image_t/ Do you know which is right? |
|
I confirmed that your cache widths match the 5.4.1 source code, so the web page is incorrect. The web page also has a "tiny" one at the beginning that I think is wrong. And it doesn't mention that these are the widths. Should I open an issue about this at https://github.com/darktable-org/luadocs ? In master, the new widths are 6144 and 7680, and I guess they could be added to the responsive_cache script based on a version check, as is done in this PR. |
|
I have attached an updated version of responsive_cache.lua. First, it handles the extra mipmap sizes for > 5.4.1. Second, it simplifies the logic in the loop for computing the correct mipmap. I tested before and after using local test_widths = {179, 180, 181, 359, 360, 361, 1919, 1920, 1921, 5121, 7681}
for i = 1, #test_widths do
local w = test_widths[i]
local m = cache_size(w)
dt.print_log("For width " .. w .. " got mip " .. m .. " with width " .. CACHE_SIZES[m+1])
endand both versions produced the same output. The diff showing my changes is: --- responsive_cache.lua 2026-03-29 18:31:48.000000000 -0400
+++ responsive_cache_jdc.lua 2026-03-30 19:20:58.093389958 -0400
@@ -62,7 +62,8 @@
-- command separator
local CS <const> = dt.configuration.running_os == "windows" and "&" or ";"
-local CACHE_SIZES <const> = {180, 360, 720, 1440, 1920, 2560, 4096, 5120, 99999999}
+local CACHE_SIZES <const> = (dt.configuration.version > "5.4.1") and {180, 360, 720, 1440, 1920, 2560, 4096, 5120, 6144, 7680, 99999999} or {180, 360, 720, 1440, 1920, 2560, 4096, 5120, 99999999}
+
-- - - - - - - - - - - - - - - - - - - - - - - -
-- A P I C H E C K
@@ -168,21 +169,24 @@
log.log_level(rc.log_level)
- for i = 0, 7 do
- log.msg(log.debug, "cache size " .. i .. " is " .. CACHE_SIZES[i + 1])
- log.msg(log.debug, "cache size " .. i + 1 .. " is " .. CACHE_SIZES[i + 2])
- log.msg(log.debug, "width is " .. width)
- if width >= CACHE_SIZES[i + 1] and width < CACHE_SIZES[i + 2] then
- if width > CACHE_SIZES[i + 1] then
- i = i + 1
- end
- log.msg(log.info, "returning cache size " .. i)
+ log.msg(log.debug, "width is " .. width)
+ for i = 0, #CACHE_SIZES-1 do
+ if width <= CACHE_SIZES[i + 1] then
+ log.msg(log.info, "returning cache size " .. i .. " of width " .. CACHE_SIZES[i+1])
return i
end
end
return 0
end |
|
responsive_cache is working for me, but there are a couple of minor issues:
|
Tiny is the small thumbnails in the filmroll. Dan may have changed it when he added size 8 and 9. IIRC 8 and 9 are 6K and 8K screens. The luadocs website is currently not updatable due to technical difficulties. I pushed the update with the changed message. I also added more debugging. You can set the DEFAULT_LOG_LEVEL to log.debug and see what is going on with the JPGs.
Add a destroy event call for post-import-film in the destroy function.
I haven't run into this. I'll do a run in a little while and check. I usually cull right after building cache completes, so that may be enough to force the write though I'm pretty sure it writes when it runs. If not, then darktable should write it when it exits. |
so that any cache building script could catch it and then execute.
|
Oops, the message didn't get changed like I thought it had. Fixed now |
|
And I had a typo in the logging, so pushed that The responsive_cache restart was probably the shortcut events trying to register again. There's a fix for that 😄 , but I never incorporated it. You can wrap the shortcuts in if not dt.query_event(MODULE .. _<extension>, "shortcut) then
...
endso if the shortcut event exists it wont try and register it again |
|
I did run a check on cache writing. I imported 634 pairs and generated the cache for them (3 and 6). Most of the cache got written to disk while darktable was running, but not all. I exited darktable and the rest of the cache got written. |
Can you try a smaller import and exit as soon as the smaller cache sizes are generated? I'll bet that the cache doesn't get written. (It doesn't for me, on 5.4.1. If it does for you, then maybe it's been changed since then?) |
In my case I generate size 6 then size 3. So you want me to exit after size 3? |
Yes. In my case, when I import 96 raw files, the size 8 cache files are created with hardlinks, so they are on disk, and then responsive_cache creates sizes 2 and 5. In one test after exiting immediately, there were 67 size 2 files on disk, and 0 size 5 files. I have 32GB of RAM and set darktable resources to be "large" in the preferences, so it has little memory pressure. |
|
I can confirm. I also noticed while testing the the import list isn't sorted. I had assumed it was but when I went to pick a small slice I noticed it was a random order. I may want to sort the imported image list before processing. If you look in src/common/cache.h it lists a couple of evict functions that ensure the cache is written to disk. I don't think they get called with darktable exits. I'll look at what it takes to add it. |
Great, thanks for looking into this. The issue that darktable starts to import photos before the lua scripts have been loaded (when given a folder on the command line) is more bothersome. Should I open an issue on the darktable repo for this? |
Yes. I'm not sure how hard that will be to fix, but we're coming up on a lot of "under the hood" changes so it might get picked up in there. |
|
darktable-org/darktable#20719 fixes the write to disk issue. |
2 participants
Canon R series cameras don't include a full size embedded jpeg so generating mipmap cache for anything larger than mipmap 3 requires the raw to be loaded and processed. This script copies the paired jpeg from the RAW+JPEG pair to the full resolution mipmap. This can then be used to generate the remaining mipmap cache. The jpeg image can be discarded after being copied to the cache or retained based on a preference in Lua options.
Fixes darktable-org/darktable#19470