Skip to content

WIP: Add Python runtime environments#2704

Draft
TomCC7 wants to merge 14 commits into
mainfrom
cc/feat/venv-worker
Draft

WIP: Add Python runtime environments#2704
TomCC7 wants to merge 14 commits into
mainfrom
cc/feat/venv-worker

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jul 2, 2026

Copy link
Copy Markdown
Member

Problem

Python modules currently run in the coordinator environment, which forces module-specific dependency stacks into dimos run and makes modules with complex or conflicting dependencies difficult to deploy safely.

Closes N/A

Solution

Adds Python runtime environments for isolated module workers:

  • Runtime environment registration and class-keyed runtime placements on blueprints.
  • Runtime reconciliation before deployment slices launch workers, using locked/non-mutating Runtime Projects.
  • Python Runtime Worker launchers for venv/project-backed workers while preserving normal DimOS Python module semantics.
  • Contract/implementation split so the coordinator imports dependency-light Module Contracts and workers import Runtime Implementations.
  • Runtime project example under examples/dimos-demo-worker-module/ with a runnable script showing main venv → isolated worker venv execution.
  • OpenSpec artifacts and docs for the runtime environment model.

Key tradeoffs:

  • Runtime Implementations must subclass Module Contracts for this extraction; descriptor/structural compatibility is deferred.
  • Runtime Project reconciliation is automatic on deployment, but source-controlled project files/lockfiles are not mutated.
  • Native module runtime unification and remote deployment are out of scope.

How to Test

uv run pytest dimos/core/test_runtime_environment.py dimos/core/test_runtime_reconciliation.py dimos/core/test_venv_module_demo.py dimos/core/coordination/test_worker.py dimos/core/coordination/test_module_coordinator.py dimos/core/coordination/test_blueprints.py dimos/robot/cli/test_dimos.py -q

Result: 96 passed.

uv run python examples/dimos-demo-worker-module/demo_run_blueprint.py

Expected: prints the repo/main Python executable, a worker Python executable under examples/dimos-demo-worker-module/.venv/, runtime-only dependency output RuntimeDependency, and worker result: demo-runtime:HELLO FROM MAIN VENV.

openspec validate python-project-runtime-environments --type change --strict --no-interactive

Result: valid.

Contributor License Agreement

  • I have read and approved the CLA.

@mintlify

mintlify Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
dimensional 🟢 Ready View Preview Jul 2, 2026, 11:22 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@TomCC7 TomCC7 changed the title Add Python runtime environments WIP: Add Python runtime environments Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
2137 1 2136 160
View the top 1 failed test(s) by shortest run time
dimos.core.test_venv_module_demo::test_demo_runtime_project_executes_with_project_worker
Stack Traces | 1.31s run time
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xffe238e90dd0>

    @pytest.mark.skipif_macos_bug
    def test_demo_runtime_project_executes_with_project_worker(monkeypatch) -> None:
        monkeypatch.syspath_prepend(str(EXAMPLE_SRC))
        contract_module = importlib.import_module("dimos_demo_worker_module.contract")
        demo_contract = contract_module.DemoWorkerModule
        runtime = PythonProjectRuntimeEnvironment(
            name="demo-worker-test-runtime",
            project=EXAMPLE_ROOT,
        )
>       subprocess.run(
            ("uv", "sync", "--locked"),
            cwd=EXAMPLE_ROOT,
            check=True,
            capture_output=True,
            text=True,
            timeout=180,
        )

contract_module = <module 'dimos_demo_worker_module.contract' from '.../src/dimos_demo_worker_module/contract.py'>
demo_contract = <class 'dimos_demo_worker_module.contract.DemoWorkerModule'>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0xffe238e90dd0>
runtime    = PythonProjectRuntimeEnvironment(name='demo-worker-test-runtime', project=PosixPath('.../dimos/examples/dimos-demo-worker-module'), env={})

dimos/core/test_venv_module_demo.py:104: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = True, timeout = 180, check = True
popenargs = (('uv', 'sync', '--locked'),)
kwargs = {'cwd': PosixPath('.../dimos/examples/dimos-demo-worker-module'), 'stderr': -1, 'stdout': -1, 'text': True}
process = <Popen: returncode: 1 args: ('uv', 'sync', '--locked')>, stdout = ''
stderr = 'warning: `VIRTUAL_ENV=.../dimos/dimos/.venv` does not match the project environment path `.venv` and wi...\nThe lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`.\n'
retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.
    
        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '('uv', 'sync', '--locked')' returned non-zero exit status 1.

capture_output = True
check      = True
input      = None
kwargs     = {'cwd': PosixPath('.../dimos/examples/dimos-demo-worker-module'), 'stderr': -1, 'stdout': -1, 'text': True}
popenargs  = (('uv', 'sync', '--locked'),)
process    = <Popen: returncode: 1 args: ('uv', 'sync', '--locked')>
retcode    = 1
stderr     = 'warning: `VIRTUAL_ENV=.../dimos/dimos/.venv` does not match the project environment path `.venv` and wi...\nThe lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`.\n'
stdout     = ''
timeout    = 180

.../usr/lib/python3.12/subprocess.py:571: CalledProcessError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

pass


class MissingPreparedPythonProjectError(PythonProjectRuntimeEnvironmentError):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer simpler error - only define a single one


@property
def convention(self) -> str:
return "pixi-backed-uv" if self.pixi_toml_path.exists() else "uv"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can just use a has_pixi flag since all projects have pyproject.toml

for atom in blueprint.blueprints:
placement_owner_by_module[atom.module] = blueprint_index
all_disabled_modules = tuple(
module for bp in blueprints for module in bp.disabled_modules_tuple

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is disabled modules? also this whole new block seems to be adding a bunch of things together... might need some comment/cleanup

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and what's the usecase of disabled modules in the codebase?

Comment on lines +202 to +203
runtime_environment_registry: RuntimeEnvironmentRegistry | None = None,
runtime_placement_map: Mapping[type[ModuleBase], RuntimePlacement] | None = None,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deployment/coordinator should stay more highlevel, too dirty to put all venv specific logic here. need better abstraction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant