Skip to content

Commit 84e22e2

Browse files
SikaGrrSikaGrr
authored
Enable Crane (#1120)
* Enable Crane. Keep a golden and integration test for Docker (to be cleaned up later) * Remove non-crane tests
1 parent 6f388ae commit 84e22e2

8 files changed

Lines changed: 26 additions & 253 deletions

‎.github/workflows/integration_basic_cluster_create.yaml‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,6 @@ jobs:
9292
env:
9393
TPU_CLUSTER_NAME: nightly-xpk-2-v5p-8
9494
WORKLOAD_NAME: xpktest-nightly-${{ github.run_attempt }}
95-
CRANE_WORKLOAD_NAME: xpktest-nightly-crane-${{ github.run_attempt }}
9695
steps:
9796
- uses: actions/download-artifact@v4
9897
with:
@@ -121,16 +120,6 @@ jobs:
121120
run: xpk info --cluster $TPU_CLUSTER_NAME --zone=us-central2-b
122121
- name: Delete the workload on the cluster
123122
run: xpk workload delete --workload $WORKLOAD_NAME --cluster $TPU_CLUSTER_NAME --zone=us-central2-b
124-
125-
- name: Run a crane workload
126-
run: CRANE_WORKLOADS_ENABLED=True xpk workload create --cluster $TPU_CLUSTER_NAME --workload $CRANE_WORKLOAD_NAME --command "bash workload.sh" --tpu-type=v4-8 --num-slices=2 --zone=us-central2-b
127-
- name: Run xpk inspector with the workload created above
128-
run: CRANE_WORKLOADS_ENABLED=True xpk inspector --cluster $TPU_CLUSTER_NAME --zone=us-central2-b --workload $CRANE_WORKLOAD_NAME
129-
- name: Wait for workload completion and confirm it succeeded
130-
run: CRANE_WORKLOADS_ENABLED=True xpk workload list --cluster $TPU_CLUSTER_NAME --zone=us-central2-b --wait-for-job-completion $CRANE_WORKLOAD_NAME --timeout 600
131-
- name: Delete the workload on the cluster
132-
run: CRANE_WORKLOADS_ENABLED=True xpk workload delete --workload $CRANE_WORKLOAD_NAME --cluster $TPU_CLUSTER_NAME --zone=us-central2-b
133-
134123
- name: Delete the cluster created
135124
if: always()
136125
run: xpk cluster delete --cluster $TPU_CLUSTER_NAME --zone=us-central2-b --force

‎recipes/Workload_create.md‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -24,25 +24,12 @@ kubectl get configmap golden-cluster-resources-configmap -o=custom-columns="Conf
2424
[XPK] No gcp parallelstore instances to add detected.
2525
[XPK] No gce persistent disk instances to add detected.
2626
[XPK] No managed lustre instances to add detected.
27-
[XPK] Temp file (4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688) content:
28-
FROM python:3.10
27+
[XPK] Temp file (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) content:
2928
30-
# Set the working directory in the container
31-
WORKDIR /app
32-
33-
# Copy all files from local workspace into docker container
34-
COPY . .
35-
36-
WORKDIR /app
37-
38-
[XPK] Building /tmp into docker image.
39-
[XPK] Task: `Building script_dir into docker image` is implemented by the following command not running since it is a dry run.
40-
docker buildx build --platform=linux/amd64 -f 4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688 -t dry-run-runner /tmp
41-
[XPK] Adding Docker Image: gcr.io/golden-project/dry-run-runner:prefix-current to golden-project
42-
[XPK] Task: `Tag Docker Image` is implemented by the following command not running since it is a dry run.
43-
docker tag dry-run-runner gcr.io/golden-project/dry-run-runner:prefix-current
44-
[XPK] Task: `Upload Docker Image` is implemented by the following command not running since it is a dry run.
45-
docker push gcr.io/golden-project/dry-run-runner:prefix-current
29+
[XPK] Adding /tmp to container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
30+
[XPK] Task: `Upload Container Image` is implemented by the following command not running since it is a dry run.
31+
crane mutate python:3.10 --append e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 --platform linux/amd64 --tag gcr.io/golden-project/dry-run-runner:prefix-current --workdir /app
32+
[XPK] Deleting container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
4633
[XPK] Temp file (39eda1549f4c0d68a4f11e6cbd89ba655d49d2faeef6898a140f476e6e70ae0e) content:
4734
apiVersion: jobset.x-k8s.io/v1alpha2
4835
kind: JobSet

‎recipes/Workload_create_Crane.md‎

Lines changed: 0 additions & 151 deletions
This file was deleted.

‎recipes/Workload_create_pathways.md‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -23,25 +23,12 @@ kubectl get configmap golden-cluster-resources-configmap -o=custom-columns="Conf
2323
gcloud container clusters list --project=golden-project --filter=name=golden-cluster --format="value(location)"
2424
[XPK] Task: `Get All Node Pools` is implemented by the following command not running since it is a dry run.
2525
gcloud beta container node-pools list --cluster golden-cluster --project=golden-project --location=us-central1 --format="csv[no-heading](name)"
26-
[XPK] Temp file (4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688) content:
27-
FROM python:3.10
26+
[XPK] Temp file (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) content:
2827
29-
# Set the working directory in the container
30-
WORKDIR /app
31-
32-
# Copy all files from local workspace into docker container
33-
COPY . .
34-
35-
WORKDIR /app
36-
37-
[XPK] Building /tmp into docker image.
38-
[XPK] Task: `Building script_dir into docker image` is implemented by the following command not running since it is a dry run.
39-
docker buildx build --platform=linux/amd64 -f 4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688 -t dry-run-runner /tmp
40-
[XPK] Adding Docker Image: gcr.io/golden-project/dry-run-runner:prefix-current to golden-project
41-
[XPK] Task: `Tag Docker Image` is implemented by the following command not running since it is a dry run.
42-
docker tag dry-run-runner gcr.io/golden-project/dry-run-runner:prefix-current
43-
[XPK] Task: `Upload Docker Image` is implemented by the following command not running since it is a dry run.
44-
docker push gcr.io/golden-project/dry-run-runner:prefix-current
28+
[XPK] Adding /tmp to container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
29+
[XPK] Task: `Upload Container Image` is implemented by the following command not running since it is a dry run.
30+
crane mutate python:3.10 --append e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 --platform linux/amd64 --tag gcr.io/golden-project/dry-run-runner:prefix-current --workdir /app
31+
[XPK] Deleting container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
4532
[XPK] Temp file (8e311bca9f9f54ee09e88dbb12e7d20d536478aa69d990a59142ee6fb70da079) content:
4633
apiVersion: jobset.x-k8s.io/v1alpha2
4734
kind: JobSet

‎recipes/Workload_create_sub-slicing.md‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -28,25 +28,12 @@ kubectl get configmap golden-cluster-resources-configmap -o=custom-columns="Conf
2828
[XPK] No gce persistent disk instances to add detected.
2929
[XPK] No managed lustre instances to add detected.
3030
[XPK] Workload will be scheduled using the Sub-slicing feature.
31-
[XPK] Temp file (4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688) content:
32-
FROM python:3.10
31+
[XPK] Temp file (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) content:
3332
34-
# Set the working directory in the container
35-
WORKDIR /app
36-
37-
# Copy all files from local workspace into docker container
38-
COPY . .
39-
40-
WORKDIR /app
41-
42-
[XPK] Building /tmp into docker image.
43-
[XPK] Task: `Building script_dir into docker image` is implemented by the following command not running since it is a dry run.
44-
docker buildx build --platform=linux/amd64 -f 4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688 -t dry-run-runner /tmp
45-
[XPK] Adding Docker Image: gcr.io/golden-project/dry-run-runner:prefix-current to golden-project
46-
[XPK] Task: `Tag Docker Image` is implemented by the following command not running since it is a dry run.
47-
docker tag dry-run-runner gcr.io/golden-project/dry-run-runner:prefix-current
48-
[XPK] Task: `Upload Docker Image` is implemented by the following command not running since it is a dry run.
49-
docker push gcr.io/golden-project/dry-run-runner:prefix-current
33+
[XPK] Adding /tmp to container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
34+
[XPK] Task: `Upload Container Image` is implemented by the following command not running since it is a dry run.
35+
crane mutate python:3.10 --append e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 --platform linux/amd64 --tag gcr.io/golden-project/dry-run-runner:prefix-current --workdir /app
36+
[XPK] Deleting container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
5037
[XPK] Temp file (2018fe16498f36301979a10667302a0aff6beb09956705b64ff396373af777ba) content:
5138
apiVersion: jobset.x-k8s.io/v1alpha2
5239
kind: JobSet

‎recipes/Workload_create_super-slicing.md‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -28,25 +28,12 @@ kubectl get configmap golden-cluster-resources-configmap -o=custom-columns="Conf
2828
[XPK] No gce persistent disk instances to add detected.
2929
[XPK] No managed lustre instances to add detected.
3030
[XPK] Workload will be scheduled using the Super-slicing feature.
31-
[XPK] Temp file (4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688) content:
32-
FROM python:3.10
31+
[XPK] Temp file (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) content:
3332
34-
# Set the working directory in the container
35-
WORKDIR /app
36-
37-
# Copy all files from local workspace into docker container
38-
COPY . .
39-
40-
WORKDIR /app
41-
42-
[XPK] Building /tmp into docker image.
43-
[XPK] Task: `Building script_dir into docker image` is implemented by the following command not running since it is a dry run.
44-
docker buildx build --platform=linux/amd64 -f 4b6736a12db8ea0f78ce793fd0d4ee0c94c652303f1dc0fecad085ea0993f688 -t dry-run-runner /tmp
45-
[XPK] Adding Docker Image: gcr.io/golden-project/dry-run-runner:prefix-current to golden-project
46-
[XPK] Task: `Tag Docker Image` is implemented by the following command not running since it is a dry run.
47-
docker tag dry-run-runner gcr.io/golden-project/dry-run-runner:prefix-current
48-
[XPK] Task: `Upload Docker Image` is implemented by the following command not running since it is a dry run.
49-
docker push gcr.io/golden-project/dry-run-runner:prefix-current
33+
[XPK] Adding /tmp to container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
34+
[XPK] Task: `Upload Container Image` is implemented by the following command not running since it is a dry run.
35+
crane mutate python:3.10 --append e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 --platform linux/amd64 --tag gcr.io/golden-project/dry-run-runner:prefix-current --workdir /app
36+
[XPK] Deleting container image archive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
5037
[XPK] Temp file (608e1382aabe2b0335855e5e99876a2e67de954453ebfa4cf12eb82c966f85da) content:
5138
apiVersion: jobset.x-k8s.io/v1alpha2
5239
kind: JobSet

0 commit comments

Comments
 (0)