fix(kubeflow): tail only first and last node logs

[ko3n1g]

Background & motivation

A multi-node Kubeflow TrainJob runs one pod per node. The CI console should show rank 0 (setup/config + per-step throughput) and the last global rank (Megatron's print_rank_last loss/step), while every rank is persisted for golden-value parsing. Plain kubectl logs -l resolves its pod set once and never re-attaches, so restarts or late-scheduled pods silently drop output.

What changed

• fetch_logs: tail every rank to log-allranks_0.out; forward only rank 0 + the true last global rank to stdout.
• Rank resolution: map pod → GROUP_RANK from the worker's environ (c10d assigns ranks by join order, not the JobSet completion-index). Resolve rank-0 and the last rank before forwarding, and never fall back to the completion-index heuristic — keep waiting while the job is alive (Pending / mid-rendezvous), stop only on a terminal state.
• Robust streaming: periodic re-attach via --since-time to pick up restarted/late pods; timestamp-ordered forwarding with a small reorder buffer; idempotent launch on 409; per-job code_dir; kube-client reload across cert rotation.
• Pod-template metadata: support labels / annotations on the TrainJob and its pods (podTemplateOverrides[].metadata).
• Tests: updated the kubeflow unit tests for the new behavior (uuid-suffixed TrainJob names, idempotent-409 recreate, all-ranks log write + rank-0/last forwarding); all green, ruff-clean.

Streaming decision

flowchart TD
  S[fetch_logs stream] --> R{rank-0 and last GROUP_RANK resolved?}
  R -->|no, job still alive| W[wait + re-list pods] --> R
  R -->|yes| A[kubectl logs -l -f --tail=-1]
  A --> D[all ranks -> log-allranks_0.out]
  A --> F[forward rank-0 + last rank -> stdout]

Loading

Technical notes

• First attach replays history (--tail=-1), so waiting for ranks to resolve loses nothing.
• Used by NVIDIA-NeMo/Megatron-Bridge#4061 and the CI in dl/joc/nemo-ci!2422.