Cache not working self-hosted kubernetes runners

[MarcHagen]

Hi! So I'm a bit clueless how to set this up.
I'm not quite sure if you are familiar with the GitHub ARC concept.

Essentially, you have a runner pod/container with an extra runner pod for Docker (aka dind)

Because this repo is running a docker container, it will be running on the extra dind pod.
This pod has three volumes:

• /home/runner/_work
• /run/docker
• /home/runner/externals

Meaning the mount to /tmp/composer-cache in in the dind container.
So the actions/cache script is running in the main runner pod, had thus has no clue about this /tmp/composer-cache

flowchart TD
    Startup

    Init[Init container]
    Runner(Runner)
    DinD(DinD)

    VolumeWork[(/home/runner/_work)]
    VolumeDinDSockWork[(/run/docker)]

    Startup -->|Create POD| Init
    Startup -->|Create POD| Runner

    Init -->|copy /home/runner/externals | DinD

    DinD-- volume mount ---VolumeWork
    DinD-- volume mount ---VolumeDinDSockWork

    Runner-- volume mount ---VolumeWork
    Runner-- volume mount ---VolumeDinDSockWork

Loading

---

So what I've done now it changed the bash script to point composer to /app/composer-cache
Where /app is the _work directory, aka GITHUB_WORKSPACE

mkdir -p "${GITHUB_WORKSPACE}/composer-cache"
export COMPOSER_CACHE_DIR="/app/composer-cache"
...
docker run --rm \
    --user "$(id -u):$(id -g)" \
    --volume "${github_action_path}/composer.phar":/usr/local/bin/composer \
    --volume ~/.gitconfig:/root/.gitconfig \
    --volume ~/.ssh:/root/.ssh \
    --volume "${GITHUB_WORKSPACE}":/app \
    --workdir ${container_workdir} \
    --env-file ./DOCKER_ENV \
    --network host \
    ${memory_limit} \
    ${docker_tag} /bin/sh -c "${command_string}"

Removed the /tmp/composer-cache mount.
And set the action/cache to use ${{ github.workspace }}/composer-cache

This works, but is a bit modified. So I'm making this issue to open a discussion on how we can improve this.

Another side effect is that the ~/.gitconfig and ~/.ssh will never be mounted, because they dont exist on the DinD pod.

How can we make this action work better for self-hosted ARC runners?

---

Resource links:

• Introducing Autoscaling Runner Scale Sets mode actions/actions-runner-controller#2174
• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller
• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#using-docker-in-docker-or-kubernetes-mode-for-containers
• Good video: https://www.youtube.com/watch?v=_F5ocPrv6io