Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
291 changes: 213 additions & 78 deletions .github/actions/rust/pre-merge/action.yml

Large diffs are not rendered by default.

32 changes: 20 additions & 12 deletions .github/actions/utils/setup-rust-with-cache/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,14 @@ name: setup-rust-with-cache
description: Setup Rust toolchain with Swatinem/rust-cache

inputs:
install-nextest:
description: "Whether to install cargo-nextest"
required: false
default: "false"
install-system-dependencies:
description: "Whether to install system packages required to build or run Rust binaries"
required: false
default: "true"
read-cache:
description: "Whether to read from cache"
required: false
Expand Down Expand Up @@ -54,15 +62,15 @@ runs:
with:
aggressive: ${{ inputs.free-disk-space-aggressive }}

- name: Install system dependencies (Linux)
if: runner.os == 'Linux'
- name: Install Rust build and runtime dependencies (Linux)
if: runner.os == 'Linux' && inputs.install-system-dependencies == 'true'
run: |
sudo apt-get update
sudo apt-get install -y libhwloc-dev pkg-config libudev-dev
shell: bash

- name: Install system dependencies (macOS)
if: runner.os == 'macOS'
- name: Install Rust build and runtime dependencies (macOS)
if: runner.os == 'macOS' && inputs.install-system-dependencies == 'true'
run: |
# Pin version of hwloc to 2.12.2_1
# brew extract doesn't have this version, so we fetch the formula directly
Expand Down Expand Up @@ -117,10 +125,11 @@ runs:
shell: bash

- name: Install cargo-nextest
if: runner.os == 'Linux'
if: runner.os == 'Linux' && inputs.install-nextest == 'true'
run: |
if command -v cargo-nextest &> /dev/null; then
echo "cargo-nextest already installed"
NEXTEST_VERSION="0.9.137"
if cargo nextest --version 2>/dev/null | grep -q "^cargo-nextest ${NEXTEST_VERSION}"; then
echo "cargo-nextest ${NEXTEST_VERSION} already installed"
cargo nextest --version
exit 0
fi
Expand All @@ -130,18 +139,17 @@ runs:
x86_64) NEXTEST_PLATFORM="linux" ;;
aarch64) NEXTEST_PLATFORM="linux-arm" ;;
*)
echo "Unsupported architecture: $ARCH, skipping nextest"
exit 0
echo "::error::Unsupported architecture for cargo-nextest: $ARCH"
exit 1
;;
esac

curl -LsSf "https://get.nexte.st/latest/${NEXTEST_PLATFORM}" | tar xzf - -C ${CARGO_HOME:-~/.cargo}/bin
curl -LsSf "https://get.nexte.st/${NEXTEST_VERSION}/${NEXTEST_PLATFORM}" |
tar xzf - -C ${CARGO_HOME:-~/.cargo}/bin
cargo nextest --version
shell: bash
continue-on-error: true

- name: Configure Cargo for CI
if: inputs.read-cache == 'true'
run: |
echo "CARGO_INCREMENTAL=0" >> $GITHUB_ENV
echo "CARGO_PROFILE_DEV_DEBUG=0" >> $GITHUB_ENV
Expand Down
4 changes: 1 addition & 3 deletions .github/config/components.yml
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,7 @@ components:
- "sort"
- "doctest"
- "machete"
- "test-1"
- "test-2"
- "test"
- "compat"
- "build-aarch64-gnu"
- "build-aarch64-musl"
Expand Down Expand Up @@ -497,7 +496,6 @@ components:
paths:
- "core/bench/dashboard/**"
- "scripts/dashboard/**"
tasks: ["build"]

# Core tools component
rust-tools:
Expand Down
12 changes: 0 additions & 12 deletions .github/workflows/_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,18 +62,6 @@ jobs:
task: ${{ inputs.task }}
component: ${{ inputs.component }}

- name: Upload coverage to Codecov
if: startsWith(inputs.component, 'rust') && startsWith(inputs.task, 'test-')
uses: codecov/codecov-action@v7.0.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: codecov.json
disable_search: true
flags: rust
fail_ci_if_error: false
verbose: true
override_pr: ${{ github.event.pull_request.number }}

# Python SDK
- name: Set up Docker Buildx for Python
if: inputs.component == 'sdk-python' && inputs.task == 'test'
Expand Down
125 changes: 125 additions & 0 deletions .github/workflows/_test_rust.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: _test_rust
on:
workflow_call:
inputs:
component:
type: string
required: true
description: "Rust component to test"
task:
type: string
required: true
description: "Rust task to run"
secrets:
CODECOV_TOKEN:
required: false

permissions:
contents: read
security-events: write
pull-requests: write

jobs:
run:
if: inputs.task != 'test'
uses: ./.github/workflows/_test.yml
with:
component: ${{ inputs.component }}
task: ${{ inputs.task }}
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

build-rust-tests:
if: inputs.task == 'test'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this cuts the noise a lot but doesn't remove it. the previous run had 85 *-rust-tests rows, 82 of them skipped, across 41 callers. now that only test-rust reaches these jobs the SDK callers drop to zero, but each of the eleven non-test rust tasks still materialises a skipped build-rust-tests and a skipped run-rust-tests, so about 22 skipped rows remain plus one for the skipped run forward on rust/test.

to get to zero the split has to move a level up - have _detect.yml emit the test task separately from the rest of rust_matrix, then this workflow holds only the build and partition jobs and never needs the run forward at all. not required here, but that's the shape that actually makes the check list clean.

runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v7.0.1

- name: Build and archive Rust tests
uses: ./.github/actions/rust/pre-merge
with:
task: test-build
component: ${{ inputs.component }}

- name: Upload build coverage to Codecov
uses: codecov/codecov-action@v7.0.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: codecov.json
disable_search: true
flags: rust
fail_ci_if_error: false
verbose: true
override_pr: ${{ github.event.pull_request.number }}

- name: Upload build reports
if: always()
uses: actions/upload-artifact@v7
with:
name: rust-test-build-reports-${{ github.run_id }}-${{ github.run_attempt }}
path: |
target/llvm-cov/**
codecov.json
if-no-files-found: ignore
retention-days: 7

run-rust-tests:
if: inputs.task == 'test'
needs: build-rust-tests
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
partition: [1, 2]
steps:
- name: Checkout code
uses: actions/checkout@v7.0.1

- name: Run Rust test partition
uses: ./.github/actions/rust/pre-merge
with:
task: test-run-${{ matrix.partition }}
component: ${{ inputs.component }}
test-partition-count: ${{ strategy.job-total }}

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v7.0.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: codecov.json
disable_search: true
flags: rust
fail_ci_if_error: false
verbose: true
override_pr: ${{ github.event.pull_request.number }}

- name: Upload reports
if: always()
uses: actions/upload-artifact@v7
with:
name: rust-test-${{ matrix.partition }}-reports-${{ github.run_id }}-${{ github.run_attempt }}
path: |
target/llvm-cov/**
codecov.json
if-no-files-found: ignore
retention-days: 7
1 change: 1 addition & 0 deletions .github/workflows/coverage-baseline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ jobs:
- name: Setup Rust with cache
uses: ./.github/actions/utils/setup-rust-with-cache
with:
install-nextest: "true"
# Also warms the GitHub Actions build cache for subsequent PR builds
save-cache: "true"
# llvm-cov instrumentation roughly doubles object sizes; the light
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pre-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
strategy:
fail-fast: true
matrix: ${{ fromJson(needs.detect.outputs.rust_matrix) }}
uses: ./.github/workflows/_test.yml
uses: ./.github/workflows/_test_rust.yml
with:
component: ${{ matrix.component }}
task: ${{ matrix.task }}
Expand Down
4 changes: 3 additions & 1 deletion core/integration/src/harness/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ fn is_cleanup_disabled_by_env() -> bool {
}

static TEST_LOGS_DIR: LazyLock<PathBuf> = LazyLock::new(|| {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
std::env::var_os("CARGO_MANIFEST_DIR")
.map(PathBuf::from)
.expect("CARGO_MANIFEST_DIR should be set when running integration tests")
.parent()
.and_then(|p| p.parent())
.expect("Failed to find workspace root")
Expand Down
18 changes: 14 additions & 4 deletions core/integration/tests/server/http_tls.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,21 @@ const READY_RETRY_INTERVAL: Duration = Duration::from_millis(50);
const REQUEST_TIMEOUT: Duration = Duration::from_secs(30);

/// Absolute path to a repo loopback cert asset. The spawned server's CWD is a
/// temp dir, so relative paths break; `CARGO_MANIFEST_DIR` is the integration
/// crate, whose sibling `../certs` holds the checked-in loopback material.
/// temp dir, so relative paths break; nextest remaps the runtime
/// `CARGO_MANIFEST_DIR` to the integration crate, whose sibling `../certs`
/// holds the checked-in loopback material.
fn cert_asset(file: &str) -> PathBuf {
std::fs::canonicalize(format!("{}/../certs/{file}", env!("CARGO_MANIFEST_DIR")))
.unwrap_or_else(|error| panic!("canonicalize repo cert asset {file}: {error}"))
let manifest_dir = std::env::var_os("CARGO_MANIFEST_DIR")
.map(PathBuf::from)
.expect("CARGO_MANIFEST_DIR should be set when running integration tests");
let asset_path = manifest_dir.join("../certs").join(file);

std::fs::canonicalize(&asset_path).unwrap_or_else(|error| {
panic!(
"canonicalize repo cert asset {}: {error}",
asset_path.display()
)
})
}

/// Boot iggy-server-ng with `[http.tls]` enabled against the repo loopback
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/python-sdk-version-sync.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/sync-python-interpreter-version.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated to either commit in this PR, and incomplete - four other scripts under scripts/ci still use #!/bin/bash. either split it out or convert all of them.

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/sync-rustc-version.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/third-party-licenses.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/uv-lock-check.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down
Loading