[AI] Add Lua AI API with model inference, tensor ops, and raw CFA support

[andriiryzhkov]

This adds a Lua scripting API for the AI subsystem, enabling Lua scripts to load ONNX models, run inference, and process images - including raw CFA sensor data. The goal is to let script authors build custom AI workflows (denoise, upscale, tagging, raw processing) without C code and external dependances.

@TurboGit any feedback on the Lua integration would be appreciated, especially the tensor type approach.

Summary

• Add darktable.ai Lua namespace for AI model inference and tensor manipulation
• Provides model discovery, loading with GPU provider selection, and two inference calling conventions (pre-allocated and auto-allocated outputs)
• Tensor type with get/set, crop/paste, dot product, shape introspection, and sRGB/linear conversion
• Image I/O: load from file or darktable library (full pipeline export), raw CFA sensor data access, DNG output with EXIF preservation, 16/32-bit TIFF with ICC profile embedding, PNG/JPEG save
• Includes model_for_task() to get the enabled model for a given task without manual iteration
• Follows darktable Lua conventions: dt_lua_init_singleton for namespace, luaL_checkudata for type safety, dt_lua_image_t for image arguments
• Add Lua AI API to release notes

API overview

-- model discovery and inference
darktable.ai.models()
darktable.ai.model_for_task("denoise")
local ctx = darktable.ai.load_model(model_id)
local output = ctx:run(input)            -- auto-allocate
ctx:run({input}, {output})               -- pre-allocated
ctx:close()

-- tensors
local t = darktable.ai.create_tensor({1, 3, H, W})
t:get({0, 0, y, x})
t:set({0, 0, y, x}, value)
t:shape()  t:ndim()  t:size()
t:crop(y, x, h, w)
t:paste(src, y, x)
t:dot(other)
t:linear_to_srgb()
t:srgb_to_linear()

-- image I/O
darktable.ai.load_image("/path/to/file.png")
darktable.ai.load_image(dt_image_object)
darktable.ai.load_raw(dt_image_object)
darktable.ai.save_dng(tensor, metadata, path)
tensor:save_tiff(path, 16, source_image)
tensor:save(path)

[wpferguson]

@TurboGit, @andriiryzhkov this violates the constraint(understanding?) that Lua isn't supposed to be used to process images in darkroom (though we kind of bend that by manipulating the GUI). The reasoning, AFAIK, was Lua was too slow to provide a decent user experience.

This is going to take some thought...

[andriiryzhkov]

@wpferguson, @TurboGit: I would really appreciate your advice on this.

My motivation comes from the fact that I've seen many attempts to bring AI features to darktable through Lua scripts that call external Python inference. People are already running AI models on their photos - they just do it outside of darktable with manual export/import cycles, and no color management.

This Lua API extension brings model inference inside darktable, which simplifies things significantly: images are loaded through the pipeline with edits applied, color profiles are handled correctly, and there's no external file juggling.

I might not be aware of prior decisions regarding Lua's positioning in darktable, so please let me know if this is the right direction.

[wpferguson]

I might not be aware of prior decisions regarding Lua's positioning in darktable, so please let me know if this is the right direction.

@andriiryzhkov I agree with your arguments about working with external files and hacking on the xmp files.

I don't have a position right now, hence the "This is going to take some thought..".

If the AI takes longer than Lua, then that mitigates the argument about slow processing (i.e. AI is the limiting factor)

I guess I'm going to have to learn AI...

[andriiryzhkov]

@wpferguson:

I don't have a position right now, hence the "This is going to take some thought..".

Sure. No problems.

If the AI takes longer than Lua, then that mitigates the argument about slow processing (i.e. AI is the limiting factor)

To clarify - the proposed Lua AI API does not inject AI processing into the pixel pipeline. There's no Lua callback running per-pixel or per-module in the darkroom pipe. That would indeed be unacceptably slow.

Instead, the AI API works as a batch/offline workflow: the script exports a fully processed image from the library (using the existing export pipeline), passes it as a tensor to an ONNX model for inference, and saves the result as a new file (TIFF/DNG). It's the same workflow people already do with external Python scripts, just without leaving darktable. The AI inference runs on its own thread - it doesn't block the pipeline or the UI.

This would be useful for things like alternative raw denoise, finding and grouping similar images, auto-tagging, and other tasks that work on images as a whole rather than inside the editing pipeline.

I guess I'm going to have to learn AI...

You can count on my help with this.

[wpferguson]

Instead, the AI API works as a batch/offline workflow: the script exports a fully processed image

That's that part that I didn't understand from reading the PR, though in your defense I was a little brain dead yesterday 😄

It's the same workflow people already do with external Python scripts, just without leaving darktable.

That would be an improvement

---

It will take me a little while to get to this, maybe a week or so while I finish up all the other stuff I've got going on.

[TurboGit]

@wpferguson @andriiryzhkov : Clearly Bill is the Lua expert here. So the last word will be from him. On my side I see no issue with that as it is not on darkroom but just scripting using AI feature to create new images (denoised or upsclaled). I don't see what different with a Lua script calling enfuse for example. Well I do see it in the implementation side, one is calling an API the other spawn a process.

My point is that from a user point of view it would be nice to have such an API, so if we can arrange to have a safe approach I'm all for it.

@wpferguson : As you said we already deal with widgets on the Lua side, so maybe not such a big issue :)