Skip to content

Update base image driver to 580 in a3m and a4x blueprint#5484

Open
saara-tyagi27 wants to merge 5 commits into
GoogleCloudPlatform:developfrom
saara-tyagi27:update-base-580
Open

Update base image driver to 580 in a3m and a4x blueprint#5484
saara-tyagi27 wants to merge 5 commits into
GoogleCloudPlatform:developfrom
saara-tyagi27:update-base-580

Conversation

@saara-tyagi27
Copy link
Copy Markdown
Contributor

@saara-tyagi27 saara-tyagi27 commented Apr 13, 2026

Summary

This pull request migrates the A3 Mega and A4x high GPU infrastructures to Ubuntu 24.04 and NVIDIA driver 580.

Key changes

  • Base Image Upgrade: Updated the base image for A3 Mega GPU configurations from Ubuntu 22.04 to Ubuntu 24.04.
  • NVIDIA Driver and Toolkit Updates: Upgraded NVIDIA components to version 580 and updated the CUDA toolkit installation to version 13.0.
  • Compatibility Fixes: Added shell scripts to handle Enroot environment requirements, patched the DMABUF import helper for kernel 6.17 compatibility, and implemented a mechanism to signal early completion of the packer process.

@saara-tyagi27 saara-tyagi27 added the release-breaking-changes Prevents "smooth" re-deploy across versions label Apr 13, 2026
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request migrates the A3 Mega GPU infrastructure to Ubuntu 24.04. It includes necessary updates to NVIDIA software stacks, introduces specific workarounds for Enroot and DMABUF compatibility on newer kernels, and optimizes the deployment process by adding an early completion signal.

Highlights

  • Base Image Upgrade: Updated the base image for A3 Mega GPU configurations from Ubuntu 22.04 to Ubuntu 24.04.
  • NVIDIA Driver and Toolkit Updates: Upgraded NVIDIA components to version 580 and updated the CUDA toolkit installation to version 13.0.
  • Compatibility Fixes: Added shell scripts to handle Enroot environment requirements, patched the DMABUF import helper for kernel 6.17 compatibility, and implemented a mechanism to signal early completion of the packer process.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@saara-tyagi27 saara-tyagi27 marked this pull request as ready for review April 13, 2026 05:34
@saara-tyagi27 saara-tyagi27 requested review from a team and samskillman as code owners April 13, 2026 05:34
@saara-tyagi27 saara-tyagi27 changed the title update base image for a3m to 580 Update base image driver to 580 in a3m blueprint Apr 13, 2026
Comment thread examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml
Comment thread examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml Outdated
@saara-tyagi27 saara-tyagi27 changed the title Update base image driver to 580 in a3m blueprint Update base image driver to 580 in a3m and a4x blueprint Apr 13, 2026

- type: ansible-local
destination: install_dmabuf.yml
- type: shell
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we switching this to a shell script?

-i localhost, --limit localhost --connection=local \
-e @/var/tmp/slurm_vars.json \
ansible/playbook.yml
- type: ansible-local
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please confirm that this can be removed and that a gve-dkms package exists already in the image.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed the gve driver is built-in (intree: Y) on both images. I wonder if the install step was originally there because older kernels needed it for missing features.

Since kernel 6.17 likely (assuming here) has these features built-in and the old 1.4.3 package fails to compile on this new kernel, I propose removing the step. Does that sound correct, or was there a specific reason for version 1.4.3 I'm missing?

Comment thread examples/machine-learning/a3-megagpu-8g/a3mega-slurm-blueprint.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-breaking-changes Prevents "smooth" re-deploy across versions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants