Convert Docker startup scripts to python#375

Closed

MoralCode wants to merge 68 commits into

mainfrom

env/entrypoints

MoralCode commented Jun 10, 2026

Contributor

Description
This is a followup to #311 to address the uses of environment variables that are present in scripts like entrypoint.sh init.sh and config.sh that are part of the docker startup process for CollectOSS.

These bash scripts would be very difficult to hack in support for reading from two different variable prefixes, and I decided it was a better use of effort to move the behaviors of these scripts into python.

This PR is that last piece last piece of fixing the environment variable naming subtask of #279

Notes for Reviewers

Signed commits

Yes, I signed my commits.

MoralCode and others added 30 commits

June 10, 2026 09:02


          add a deprefix helper and some unit tests

c037dcc

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          refactor into a better prefix extraction helper

d62214b

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          add first pass SystemEnv

4a3d431

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          allow different prefixes to be passed in for testing purposes

61a564f

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          update deprecation message

e8cc02f

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          basic functionality unit tests

b2f99ca

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          handle cases with no known prefix to avoid breaking stuff

7a737fc

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          factor out bool fetching class to deduplicate logic

f6109aa

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          Test to make sure we can potentially migrate the AUGUR_ vars in the c…

67ddbfd

…ode slowly over time

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          replace all references to os.getenv with new class

ddb0ef5

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          deprecate older Environment class that is buried in the module tree

5b04de6

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          get_bool docstring

827dfb9

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          basic setter

1963d35

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          replace references to os.environ[] with new class

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          remove some redundant wrapping code

dc16079

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          replace references to os.environ.get

1bb1de2

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          Swap out only usage of deprecated Environment class

f5b5019

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          Remove no-longer-used Environment class in API

202465c

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          Refactor extract_prefix

da765da

Co-authored-by: Shlok Gilda <gildashlok@hotmail.com>
Signed-off-by: Adrian Edwards <17362949+MoralCode@users.noreply.github.com>


          refactor get_bool

b3e92a2

Co-authored-by: Shlok Gilda <gildashlok@hotmail.com>
Signed-off-by: Adrian Edwards <17362949+MoralCode@users.noreply.github.com>


          fix ~ path expansion in default scorecard value

21a34f1

Co-authored-by: Shlok Gilda <gildashlok@hotmail.com>
Signed-off-by: Adrian Edwards <17362949+MoralCode@users.noreply.github.com>


          replace a bunch of env var names the application accesses

8945ef8

impact: low due to the new env variable interpretation layer

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          update env var names in RST docs

27fd27e

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          update vars in example env

bef5639

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          update variable names in docker compose

6fd7ee0

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          hard change env var prefix for CLI commands

0aa0d8a

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          attempt to add transitional variables for the specific existing place…

110f42b

…s where env vars are explicitly needed in the CLI

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          factor tests into a class

f3ff5a3

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          import SystemEnv into celery_app

75345cb

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          move test next to the actual known good config tests

bcff419

Signed-off-by: Adrian Edwards <adredwar@redhat.com>

MoralCode added 25 commits

June 10, 2026 09:02


          provide default facade repo directory

32033a0

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          ensure facade repo directory is resolveable/absolute

5fc3ecc

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          ensure facade repo directory has a trailing slash

4747b8c

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          oops

1d2910b

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          handle swapping localhosts for redis connection string too

ba802c2

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          entrypoint is now empty

7e5fc18

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          provide excessive user feedback with verbose logging

e4c13ef

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          port over the facade config file handling into python

f7f7cb7

itll start as deprecated since there are already plans to make this better.

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          refactor config merging into a separate function so it could be reuse…

ac5a57d

…d from two places in the CLI

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          config script no longer needed

a8280c2

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          enable the basic schema update subshell for now

553f8a6


          add a platform information function

62a8a16


          include new environment file in linted files list

85acff2

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          add function to detect and warn if files are present at the previous …

de12791

…autoimport locations

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          move remaining command in init into the docker init directive

52f8fe6

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          remove old dockerfile entrypoint and init script infrastructure

ce25502

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          revert keyman orchestrator back to regular environment variables

fcde27a

so it doesn't import CollectOSS

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          logger.verbose -> logger.debug

f3fe0e0

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          ensure logger is correctly passed and all functions are imported

aa02bda

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          use logger to output version information

a555f43

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          open facade directory file in write mode

1ff171f

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          fix config bug with new startup process

faca04e

github keys being put in wrong places

Signed-off-by: Adrian Edwards <adredwar@redhat.com>


          improve robustness of logger against missing log directories in config

bcad37c


          include logs directory environment variable in default compose

1f59f00


          correct translation error in git config command

e4ff788

Signed-off-by: Adrian Edwards <adredwar@redhat.com>

github-advanced-security AI found potential problems

View reviewed changes

collectoss/util/startup.py

+              from pathlib import Path
+              import os
+              import getpass
+              import subprocess

collectoss/util/startup.py

+              import os
+              import getpass
+              import subprocess
+              from subprocess import check_call

collectoss/util/startup.py

+                  """
+                  pass
+                  # alembic upgrade head, unless theres an env var preventing automatic migration
+                  check_call(["alembic", "upgrade", "head"])

collectoss/util/startup.py

+                  """
+                  pass
+                  # alembic upgrade head, unless theres an env var preventing automatic migration
+                  check_call(["alembic", "upgrade", "head"])

collectoss/util/startup.py

+                  with git_credentials.open("w", encoding="utf-8") as c:
+                      c.writelines(credentials)
+                  subprocess.call(["git", "config", "--global", "credential.helper", f"store --file {str(git_credentials)}"])

collectoss/util/startup.py

+                  with git_credentials.open("w", encoding="utf-8") as c:
+                      c.writelines(credentials)
+                  subprocess.call(["git", "config", "--global", "credential.helper", f"store --file {str(git_credentials)}"])

github-actions Bot reviewed

View reviewed changes

collectoss/application/cli/backend.py

		@@ -130,7 +147,7 @@ def start(ctx, disable_collection, development, pidfile, port):
		processes = start_celery_worker_processes((core_worker_count, secondary_worker_count, facade_worker_count), disable_collection)

github-actions Bot Jun 10, 2026

[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'processes' from outer scope (line 412) (redefined-outer-name)

collectoss/application/cli/backend.py

		@@ -15,6 +15,7 @@
		import requests
		from redis.exceptions import ConnectionError as RedisConnectionError

github-actions Bot Jun 10, 2026

[pylint] _{reported by reviewdog 🐶}
W0611: Unused ConnectionError imported from redis.exceptions as RedisConnectionError (unused-import)

collectoss/application/cli/backend.py

               import sqlalchemy as s
+              from collectoss.util.startup import check_init_schema, check_update_schema, collect_env_variables, print_platform_information, setup_facade_directory, warn_import_repos, merge_config
               from keyman.KeyClient import KeyClient, KeyPublisher

github-actions Bot Jun 10, 2026

[pylint] _{reported by reviewdog 🐶}
W0611: Unused KeyClient imported from keyman.KeyClient (unused-import)

collectoss/util/startup.py

+              import platform
+              import sys
+              from sqlalchemy.orm.attributes import get_history

github-actions Bot Jun 10, 2026

[pylint] _{reported by reviewdog 🐶}
W0611: Unused get_history imported from sqlalchemy.orm.attributes (unused-import)

MoralCode commented Jun 10, 2026

Contributor Author

Just kidding. Without this PR, the CI is failing for the PR this depends on. ill just do them together

MoralCode closed this

MoralCode deleted the env/entrypoints branch

June 10, 2026 19:30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet