Python SDK: Developer Experience and Development Environment Improvements

[mmmmxa]

Description

The current documentation and testing configurations in Python SDK have a number of problems:

• README file doesn't offer a reliable and easy to follow guide for development workflow. We are presented with a list of commands that get some of the jobs done, some of them with side effects. More on that in next items.
• Using uv run without --no-sync is performing implicit syncing that overwrites the installation made by maturin develop, reverting the newest changes that were just introduced by installing the package via maturin develop.
• Command to generate stubs is not documented.
• Formatting commands required for the hooks to let your commits and pushes through specifically for Python code are not documented.
• Docker compose and the Dockerfile seem to not have been updated in a while. Iggy server build is failing without seccomp:unconfined option, the overall compose command is failing due to problems with connectivity between containers, how this connectivity is established or checked (I'm still in the process of figuring all the root issues out here).
• Dockerfile can be improved by adding caching and other changes.
• Any other fixes for the development features that Python SDK is providing but which actually don't work out of the box.

In general, this set of changes is intended to make sure that a developer can follow the instructions which cover all necessary steps and all of which work as intended and serve a clear purpose.

Affected area / component

Python SDK

Proposed solution

• Provide a development workflow in README, covering all steps of developing every feature and providing commands that are most reliable and easiest to use. Important action items as a part of this:
  • Add --no-sync to some uv run commands to get rid of undesired side effects.
  • Document the command to generate stubs
  • Provide ruff formatting commands that are required by hooks.
• Update docker-related files to make sure docker-compose runs tests out of the box.
  • Improve test Dockerfile by switching to Rust-based image, installing uv and a specific python version. This will pin specific versions as arguments and will allow to perform checks for different python or rust versions if needed. Add caching for uv and --locked option for the uv sync so that specifically the container build could trigger a developer to intentionally update the lock file. (The latter is subject to change in the process of developing the workflow).
  • Fix issues with entrypoint script of the image and / or the configurations inside of the docker-compose file (there are interconnected and present as a group of issues).
  • Resolve the problem that test_tls.py introduces -- to be updated
• Any other fixes for the development features that Python SDK is providing but which actually don't work out of the box and / or for the documentation.

I have already implemented almost all of these locally and the list may grow further as I discover the algorithm that gets a developer through the whole development process without issues.

Alternatives considered

No response

Contribution

• I'm willing to submit a pull request to implement this feature

Good first issue

• I think this could be a good first issue for a new contributor

[ethanlin01x]

I ran into similar problems getting set up, so clearer docs here would help a lot of newcomers.

One small suggestion: could this be split into separate issues, or a checklist? It currently mixes plain bugs, doc rewrites, and workflow design choices, and I suspect the easy fixes will end up stuck behind the debatable ones.

About the move to .venv/bin/activate: I'd lean the other way, keeping uv and adding --no-sync. CI runs uv end to end and already uses it, so documenting a different path means a green local run no longer predicts CI. If the concern is having to remember the flag, export UV_NO_SYNC=1 once solves that. Happy to go into more detail if that's useful.

Glad to help with any of it.

[mmmmxa]

Hi @ethanlin01x,

thanks a lot for your input.

• Your argument about venv makes sense and I'll follow that.
• Regarding splitting this issue into separate ones. For me as a newcomer here, I feel like this is directed more by the established practices and workflows in the project, so maybe you can suggest the split that you see here (and I'd also like to know what kind of checklist are you talking about). My intuition about putting it all together is to reduce the paperwork and put lots of comparatively tiny changes together to reduce the mental load of having more issues for each. I also understand your intuition and the valid split that I see is basically two issues: first issue is for everything docker-related ("docker files update and fixes" -- the point of it is it was basically not maintained for a while) and second for everything else, which is the documentation, the human development workflow being a part of it. We can do that for example.
• So far I think help-wise I would benefit from the other pairs of eyes to discuss the solutions to these issues, review and verify that the steps produce the desired results on their computer. The changes themselves seem to not be that big and pretty straightforward.

Thank you.

[ethanlin01x]

Hi @mmmmxa,

I'm a newcomer here too, so please read the below as a suggestion rather than established practice. How this actually gets split is the maintainers' call, and it's probably worth waiting for one of them to weigh in before you reorganize anything.

By checklist I just meant keeping this issue as the umbrella and turning the description into - [ ] boxes, so each item gets closed by its own small PR without the issue count growing. Nothing more than that.

On the split, your two-way version already works. If you want a finer cut, I'd split the docker half by what a reviewer has to agree with before approving. The fixes for what's broken today (seccomp, container connectivity) only need one answer: does it work now. The restructuring (Rust base image, uv cache mounts, --locked) needs agreement about build time and image size, which is a longer conversation. Put them in one PR and the fixes wait for that conversation to finish, which is the only reason I'd separate them. The README work is pure addition and can land whenever.

Two small things worth folding in if useful: the README links to foreign/python/CONTRIBUTING.md, which doesn't exist in the repo, and changing the Python version anywhere trips scripts/ci/sync-python-interpreter-version.sh in pre-commit until every file agrees (it has a --fix mode).

Happy to take the verification side. I'm on macOS/arm64, so I can run whatever you write from a clean clone and tell you where it breaks. Just ping me when there's a branch.

[slbotbm]

@mmmmxa Could you edit the original message to make a checklist?

Other than that, I'd like to use the uv run --no-sync option as Ethan said. Plus if all of the changes only affect developer workflow and not any of the artifacts we publish in a material way, we can probably get the changes merged fast.

[mmmmxa]

@slbotbm @ethanlin01x I will suggest / interpret our consensus as creating and organizing a checklist in the original message and not creating other issues. I feel like it will be the simplest and the changes introduced are not that big at the end of the day, and they still make sense together. I will create one PR for this and we can get back to the discussion about splitting if it feels overwhelming for reviewers. Correct me if I'm wrong on anything, thanks.

[mmmmxa]

Hi @slbotbm! I have a few things to discuss.

After tinkering with the docker part I stumbled upon test_tls.py. In this test, testcontainers spins up a container with apache/iggy:edge (btw this can cause an inconsistency in Iggy versions tested when running pytest) with TLS enabled. This is different from the rest of the tests that are connecting to the non-TLS Iggy running from a user command in case of regular tests or non-TLS Iggy running in the docker-compose. The testcontainers approach and pure Docker approach are not very compatible since for test_tls.py the composition process is happening inside of the test and for the rest of the tests the composition is happening by user's command or from the compose file. To me, this mix is not very robust and eventually will require a redesign. I have the following suggestion of how we can go from here:

1. Create a new issue specifically for the redesign of the tests. This issue will essentially cover only the README update without changing anything else.
2. I think that for test redesign I'd simply propose to move all tests to testcontainers and getting rid of two ways that exist now (manual iggy server start + pytest or everything in docker compose). We will essentially configure all the tests to run the Iggy in the container and user will only have to follow the installation requirements to make sure those containers are spinning up correctly. It makes sense because:

• all tests are integration tests which require a running container;
• we won't have to manage two ways of testing;
• the composition process will be designed in a way that allows for more sophisticated integration scenarios if needed.

If you are okay with this idea, I can take up this task of porting all tests to testcontainers. Please let me know what you think.

[ethanlin01x]

Hi @mmmmxa,

Splitting the test redesign into its own issue sounds right.

One thing to check before committing to testcontainers everywhere: CI doesn't use apache/iggy:edge. It builds iggy-server from the PR's checkout with cargo (.github/actions/utils/server-start), and sdk-python has rust-server in its depends_on, so server changes deliberately run the Python suite against the freshly built server. Pulling a published image would test yesterday's edge instead of the change itself, which is the same version skew you flagged for test_tls.py, just applied to the whole suite.

A middle path: conftest already reads IGGY_SERVER_HOST/IGGY_SERVER_TCP_PORT, so tests are connection-agnostic. Make testcontainers an opt-in fixture (spin up a container when no address is set), keep CI on the source-built server, and test_tls.py stops being a special case.