Move to a `pixi`-based installation by hameerabbasi · Pull Request #1244 · Nerogar/OneTrainer

hameerabbasi · 2026-01-04T11:33:47Z

Disdvantages:

source venv/bin/activate is replaced by eval "$(pixi -e [cpu|cuda|rocm] shell-hook)" to activate inside an existing shell or pixi -e [cpu|cuda|rocm] shell for a new, pre-activated shell. Or by prefixing each command with pixi run -e [cuda|rocm|cpu]. pip list is replaced by pixi list.

Advantages:

If you have pixi, you can run pixi run -e [cuda|rocm|cpu] <command>, it will take care of installation running the command from beginning to end.
All your intermittent deps are fixed as well as the direct deps inside pixi.lock. Works wonders for reproducibility. To update, just use pixi update.
You can automatically get libgl and tk for linux: No more need for sudo anymore, only hard dep is either wget or curl, and GPU driver (no CUDA toolkit necessary either).
Installation is MUCH faster. This makes creating a docker image or updating a container easy and fast.
Installation/launch scripts are simplified and aligned now.

Decisions to make:

Keep or remove requirements*.txt

ToDos for this PR:

Update the install/run scripts to install/detect pixi and then do pixi install -e ... or pixi run -e ... (or we can get rid of the install script entirely).
Update CLI docs
Update the Dockerfiles (which will now be smaller, yay).

After merging:

Publish a new Docker image

hameerabbasi · 2026-01-05T04:50:31Z

@dxqb As promised, the +/- are in the description of the PR.

requirements-global.txt

hameerabbasi · 2026-01-14T15:53:40Z

prefix-dev/pixi#1259 was resolved via prefix-dev/pixi#5241 and included in https://github.com/prefix-dev/pixi/releases/tag/v0.63.0

… windows signing

scripts/generate_debug_report.py

O-J1

This PR adds 440 LoC. Ignore the lock file, its autogenerated and is 7.2k. We dont manage or touch the lockfile.

Install, update, starting the UI, training (UI & CLI) work on both Linux, Windows (and Mac where appropriate) work.

I didnt have to worry about wtf python I had installed, I didnt have to worry about install location or permissions, I didnt have to worry about already installing pixi, we can automagically install signed binaries as of version v0.65.0.

Overall LGTM and worked. We still need some other users to test just to be absolutely sure, given its scope.

dxqb

thanks for your patience

I've left many comments on the code. Here is only what didn't fit anywhere else.

I've tested this PR on a RunPod host. When I run install.sh, pixi isn't found:

This script will automatically download and install Pixi (latest) for you.
Getting it from this url: https://github.com/prefix-dev/pixi/releases/latest/download/pixi-x86_64-unknown-linux-musl.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 28.6M  100 28.6M    0     0  25.4M      0  0:00:01  0:00:01 --:--:-- 25.4M
The 'pixi' binary is installed into '/root/.pixi/bin'
Updating '/root/.bashrc'
Please restart or source your shell.
[OneTrainer] + pixi install --locked -e cuda
./lib.include.sh: line 132: pixi: command not found

adding /root/.pixi/bin to PATH helps as a workaround
Install is fast then.

When I install OneTrainer through the cloud tab, all output of pixi is suppressed it seems. This is the only output:

[OneTrainer] + pixi self-update
✔ pixi is already up-to-date (version 0.66.0)
[OneTrainer] + pixi install --locked -e cuda
✔ The cuda environment has been installed.

This could be a problem if you have a slow downstream remote host - you wait and nothing happens or is showing progress.

Overall, this speeds up the installation process by a lot, so it is useful. However, it throws away completely a tried-and-proven installation process. Therefore I still think this should be an option.
It can be an option that we make the default very soon (or even from the beginning?). But at the very least, when a user complains on Discord that the new installation process fails for them, I want us to be able to answer: "run it with OT_PREFER_PIP" while we fix it.

It also removes the possibility to use conda. Are we sure this isn't needed anymore?
I haven't been using it so I can't answer, but I remember people asking if conda is supported etc.

dxqb · 2026-03-21T10:10:35Z

modules/trainer/BaseTrainer.py

        )
    def _start_tensorboard(self):
-        tensorboard_executable = os.path.join(os.path.dirname(sys.executable), "tensorboard")
+        tensorboard_executable = shutil.which("tensorboard")


why is this necessary?
shutil.which() returns the path to the executable in PATH

but if tensorboard is in PATH anyway, then why use a path at all and not just execute "tensorboard"?

dxqb · 2026-03-21T10:14:08Z

resources/docker/RunPod-NVIDIA-CLI.Dockerfile

-#    docker tag <image-name> <dockerhub-username>/<repository-name>:<tag>
-#    docker push <dockerhub-username>/<repository-name>:<tag>
-
-FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04


the RunPod and vast.ai both use a base image file that is in wide use on these platforms.

By doing that, these are already in the local docker image caches and only the added layers (containing OneTrainer) have to be downloaded.
By switching to a generic dockerfile based on ubuntu, you loose that benefit.

Not using their image files probably breaks many other things, e.g. access to Jypiter, updating SSH keys through their web API, etc.

dxqb · 2026-03-21T10:17:55Z

scripts/powershell/lib.include.ps1

+
+#region Platform Detection
+
+function Get-Platform {


is this platform detection new or different, or was it done the same way before - just rewritten in PS?

dxqb · 2026-03-21T10:19:29Z

resources/docker/Dockerfile

@@ -0,0 +1,29 @@
+#To build, run


please see my comment about the NVIDIA Dockerfile first.
does this Dockerfile have a purpose beyond RunPod and Vast? Is it also meant to be run locally?

dxqb · 2026-03-21T10:21:03Z

resources/docker/RunPod-NVIDIA-CLI-start.sh.patch

Note, for any modifications you make on the Dockerfiles:
the purpose is that OneTrainer can already be installed, in case you use a persistent disk volume on RunPod. It uses /workspace/OneTrainer if already installed, and if not it links to the OneTrainer in the image, which is at /OneTrainer

dxqb · 2026-03-21T10:35:08Z

LAUNCH-SCRIPTS.md

-
- `OT_PREFER_VENV`: If set to `true`, Conda will be ignored even if it exists on the system, and Python Venv will be used instead. This ensures that people who use `pyenv` (to choose which Python version to run on the host) can easily set up their desired Python Venv environments. Defaults to `false`.
-
- `OT_LAZY_UPDATES`: If set to `true`, OneTrainer's self-update process will only update the Python environment's dependencies if the OneTrainer source code has been modified since the previous dependency update. This speeds up executions of `update.sh`, and is generally safe, but may miss some updates and important bugfixes for external third-party dependencies. If you use this option, you must set it permanently for *every* script (not just `update.sh`). Defaults to `false`.


if LAZY_UPDATES is not supported and not necessary anymore, it should be removed in the cloud training code, where it is used.
However, please see me comments regarding keeping pip for now first

dxqb · 2026-03-21T10:40:09Z

lib.include.sh

+            curl -fsSL https://pixi.sh/install.sh | sh
+        elif can_exec wget; then
+            print_debug '`wget` found, installing `pixi` with `wget`.'
+            wget -qO- https://pixi.sh/install.sh | sh


while there are many attack vectors in a python app with many dependencies, I don't think we have trusted a host so far completely without any checks.

What prevents pixi.sh to serve malicious code randomly every 10000 requests?

dxqb · 2026-03-21T10:41:11Z

pyproject.toml

+    "pytorch-lightning==2.6.1",
+
+    # diffusion models
+    "diffusers @ git+https://github.com/huggingface/diffusers.git@99daaa802da01ef4cff5141f4f3c0329a57fb591",


could you please check that this doesn't cause this issue again:
#1133

dxqb · 2026-03-21T10:42:23Z

README.md

+3. Navigate into the cloned directory `cd OneTrainer`
+4. Perform the installation: `pixi install --locked -e cuda` (Replace `cuda` by `rocm` or `cpu` if needed).
+
+**Note:** We don't support ROCm on Windows currently.


is that new?
weren't there several users on Discord who successfully used ROCm? were these all on linux?

dxqb · 2026-03-21T10:44:27Z

README.md

+1. Install `pixi`: [Guide](https://pixi.prefix.dev/latest/installation/)
+2. Clone the repository `git clone https://github.com/Nerogar/OneTrainer.git`
+3. Navigate into the cloned directory `cd OneTrainer`
+4. Perform the installation: `pixi install --locked -e cuda` (Replace `cuda` by `rocm` or `cpu` if needed).


manual installation with pip should remain possible.
Currently it fails:

INFO: pip is looking at multiple versions of onetrainer to determine which version is compatible with other requirements. This could take a while. ERROR: Cannot install -r requirements-global.txt (line 1) and diffusers 0.37.0.dev0 (from git+https://github.com/huggingface/diffusers.git@99daaa8#egg=diffusers) because these package versions have conflicting dependencies. The conflict is caused by: The user requested diffusers 0.37.0.dev0 (from git+https://github.com/huggingface/diffusers.git@99daaa8#egg=diffusers) onetrainer 0.1.0 depends on diffusers 0.37.0.dev0 (from git+https://github.com/huggingface/diffusers.git@99daaa802da01ef4cff5141f4f3c0329a57fb591) To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

hameerabbasi force-pushed the modernize-install branch from 3c0bb01 to 84a617b Compare January 4, 2026 11:34

This comment was marked as resolved.

Sign in to view

hameerabbasi force-pushed the modernize-install branch 4 times, most recently from 63ef5d0 to f390904 Compare January 4, 2026 12:38

hameerabbasi changed the title ~~Modernize requirements*.txt.~~ Move to a pixi-based installation Jan 5, 2026

hameerabbasi commented Jan 5, 2026

View reviewed changes

requirements-global.txt Outdated Show resolved Hide resolved

hameerabbasi force-pushed the modernize-install branch 2 times, most recently from 5685e79 to 5fc122f Compare January 5, 2026 06:02

This comment was marked as resolved.

Sign in to view

hameerabbasi force-pushed the modernize-install branch 2 times, most recently from 7211167 to 5ff0085 Compare January 5, 2026 15:02

dxqb added the waiting waiting for something / cannot fix currently label Jan 5, 2026

hameerabbasi force-pushed the modernize-install branch from 5ff0085 to 1b993b1 Compare January 14, 2026 15:41

hameerabbasi force-pushed the modernize-install branch 3 times, most recently from 0a4fd4b to fc392c2 Compare January 19, 2026 07:45

dxqb removed the waiting waiting for something / cannot fix currently label Jan 19, 2026

dxqb requested a review from O-J1 February 27, 2026 18:25

hameerabbasi added 8 commits March 3, 2026 08:57

Modernize requirements*.txt.

5dd139e

Move to pixi from pip.

0b1c4c1

Move to pyproject.toml from pixi.toml.

e2b5664

Make everything pip-installable.

46d7833

Require minimum pixi.

6634864

Remove unnecessary parts.`

b41658e

Make it *actually* work without a pixi.toml.

12b7309

Add back --extra-index-url.

f7cb8c6

hameerabbasi added 3 commits March 3, 2026 09:18

Don't install unnecessary packages.

2c24690

Modify format of requirements*.txt.

9426b27

Make OneTrainer package editable for easier dev.

00f6d52

hameerabbasi force-pushed the modernize-install branch from 18735ee to 00f6d52 Compare March 3, 2026 08:21

Repeat git requirements in requirements-global.txt.

e76e790

hameerabbasi force-pushed the modernize-install branch 3 times, most recently from ae69116 to 0ad5150 Compare March 3, 2026 15:53

Update the Linux/macOS installation scripts.

6fc0455

hameerabbasi force-pushed the modernize-install branch from 0ad5150 to 6fc0455 Compare March 3, 2026 15:55

Initial Windows 'it works' pixi scripts and bump min pixi version for…

0f7a373

… windows signing

hameerabbasi commented Mar 9, 2026

View reviewed changes

scripts/generate_debug_report.py Outdated Show resolved Hide resolved

O-J1 added 3 commits March 10, 2026 00:24

Tweak platform_env

e6b83c3

Fix still incorrect platform_env

101cc26

Fix Tensorboard breaking, swap to shutil.which

b897bcc

O-J1 reviewed Mar 9, 2026

View reviewed changes

hameerabbasi and others added 2 commits March 9, 2026 15:48

Update README.md with new install instructions.

fbeec51

Further readme tweaks

1c50e81

hameerabbasi force-pushed the modernize-install branch 2 times, most recently from 511012a to 81ebce1 Compare March 10, 2026 06:44

Add run-cmd.ps1 and update README.md accordingly.

c040fb9

hameerabbasi force-pushed the modernize-install branch from 81ebce1 to c040fb9 Compare March 10, 2026 06:45

Fix run-cmd.ps1 syntax

afe6b42

hameerabbasi force-pushed the modernize-install branch from c2115d3 to 9406403 Compare March 18, 2026 14:50

Minimal Dockerfile.

fbc6991

hameerabbasi force-pushed the modernize-install branch from 9406403 to fbc6991 Compare March 18, 2026 14:51

dxqb reviewed Mar 21, 2026

View reviewed changes

O-J1 mentioned this pull request Mar 21, 2026

fix: update requirements-rocm.txt for Windows support via AMD TheRock nightly #1385

Open

titansonico10 mentioned this pull request Mar 21, 2026

better AMD support #1381

Open

dxqb mentioned this pull request Mar 24, 2026

Improve Docker Stack #963

Draft

4 tasks


		- `OT_PREFER_VENV`: If set to `true`, Conda will be ignored even if it exists on the system, and Python Venv will be used instead. This ensures that people who use `pyenv` (to choose which Python version to run on the host) can easily set up their desired Python Venv environments. Defaults to `false`.

		- `OT_LAZY_UPDATES`: If set to `true`, OneTrainer's self-update process will only update the Python environment's dependencies if the OneTrainer source code has been modified since the previous dependency update. This speeds up executions of `update.sh`, and is generally safe, but may miss some updates and important bugfixes for external third-party dependencies. If you use this option, you must set it permanently for every script (not just `update.sh`). Defaults to `false`.

Uh oh!

Conversation

hameerabbasi commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

hameerabbasi commented Jan 5, 2026

Uh oh!

Uh oh!

This comment was marked as resolved.

hameerabbasi commented Jan 14, 2026

Uh oh!

Uh oh!

O-J1 left a comment

Choose a reason for hiding this comment

Uh oh!

dxqb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dxqb Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hameerabbasi commented Jan 4, 2026 •

edited

Loading

dxqb Mar 21, 2026 •

edited

Loading