[Adroid v4 API] Add Android Build API V4 REST client and tests by IvanBM18 · Pull Request #5370 · google/clusterfuzz

IvanBM18 · 2026-07-21T06:21:49Z

Changes

We previously migrated the v3 api to use v4 in fetch_artifact.py but after testing the implementation we discovered that we can't continue the same approach of using a client created by service discovery url , if we want to continue using the android build api we needed to change our strategy and start using raw http requests.

Creates the new android v4 api class
- Instantiated trough a static method that recieves service account credentials to avoid silent failures
Updates the utils.fetch_url to unify requests and reuse it in this new api while maintaining previous functionality

For more info in the v4 api see: go/build-api-v4-clients-migration-guide & b/422775458

Tests

Created unit tests for this new class
Tested this class implementation using a custom script

Testing script details

The custom script performs various tests that imitate how clusterfuzz would use the fetch_artifact.py module in conjunction with our new api.

The script test the following:

Correct API Version selection
Correct API instantiation by checking that the credentials in the db are valid
Queries latest artifact info(like build_id) for the given branch and target
- branch and target come from a adb device (see where we got this from)
Uses the retrieved build_id to get android builds artifact metadata
Downloads the given artifact

Test steps:

Chery-pick the scripot commit into my branch
Run the script using butler:

pipenv run python butler.py run -c /usr/local/google/home/ibarba/projects/clusterfuzz-config/configs/chrome-staging test_fetch_artifact \
 --script_args \
 --branch=git_main \
 --target=cf_x86_64_phone-next-userdebug

Dev verification

After the initial tests ^ i tested this changes once more in dev(including all 3 prs), looking specifically for utils.fetch_url or errors inherent to the return value from this method. Basically since my changes landed in dev the following error groups where seen:

Discarded the errors following this logic:

403 Errors are expected to appear as per the changes that were previously there to avoid them are no longer in dev
Some error groups only appeared on previous clusterfuzz versions
Other errors are expected(like the queue size limit or bad revision errors)
There are errors from corpus pruning & fuzzer not found, this have been seen for a while now, not related to my changes
Lastly there are some errors in a cron_job, specifically in this file but this one doesn't make usage of fetch_url and nor does the method that triggered the error

Note:

PR review Stack

👉 [Adroid v4 API] Add Android Build API V4 REST client and tests #5370 Add Android Build API V4 REST client and tests (This PR)
#5371 Migrate fetch_artifact to use new V4 API
#5373 Creates Feature Flag for Android V4 API

Please only approve, I'll merge this PR's to avoid any possible merge conflict that can appear in the process :D

IvanBM18 · 2026-07-21T06:22:18Z

+HTTP_TIMEOUT = 60
+
+# 20 MB default chunk size for file downloads.
+DEFAULT_CHUNK_SIZE = 20 * 1024 * 1024


Values taken from https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/platforms/android/fetch_artifact.py

letitz · 2026-07-21T08:37:19Z

I'll let @dylanjew handle this as I'm about to go OOO for a while.

vitaliset · 2026-07-21T14:31:01Z

I'm not super familiar with this part of the code, so I'll trust your judgment: is this something we should test on dev? If so, can you provide some evidence that it is working as expected? Same applies for #5371.

dylanjew

LGTM! just a couple small questions

dylanjew · 2026-07-21T14:14:59Z

+
+  @retry.wrap(
+      retries=5, delay=1, backoff=2, function='android_build_v4_api._request')
+  def _request(self,


could we just use fetch_url?

Actually, thats a good idea, Ill adapt the fetch_url util to accept a bunch of additional params without compromising its behavior

dylanjew · 2026-07-21T14:33:07Z

+    except (requests.exceptions.RequestException,
+            google.auth.exceptions.GoogleAuthError, ValueError) as e:
+      logs.error(f'V4 get_artifact_metadata failed for artifact {name}: {e}')
+      return None


Is there a reason we catch and log instead of just allowing the error to raise? Was this the previous behavior?

Yes, in the V3 API implementation, all HTTP requests were wrapped in _execute_request_with_retries(request)

When failed, it catched the exception logged an error and then returned None

dylanjew · 2026-07-21T14:38:00Z

+    self.mock.get.return_value = mock_response
+
+    result = self.api.list_builds('git_main', 'target_name')
+    self.assertIsNone(result)


this is why it's a bit confusing to swallow the error and return None, we can't tell whether the error was due to an intermittent connection issue or because some assumption about the API has changed and we need to parse the response differently

True, we can let the exceptions be thrown instead(by encapsulating them) in a custom exception, like we do in the swarming api But for this api use case, we will end up catching the exception and logging an error and returning None(as done previously)

We can also create another PR that focuses on throwing exceptions instead, WDYT?

Thanks! I think it's reasonable to preserve the existing behavior to reduce the blast radius of the API change.

dylanjew · 2026-07-21T14:44:55Z

I'm not super familiar with this part of the code, so I'll trust your judgment: is this something we should test on dev? If so, can you provide some evidence that it is working as expected? Same applies for #5371.

@IvanBM18 Can you add your testing to the description? This PR doesn't exercise the new code, but #5371 will. The codepath is Chrome specific.

dylanjew · 2026-07-21T14:47:25Z

I'm not super familiar with this part of the code, so I'll trust your judgment: is this something we should test on dev? If so, can you provide some evidence that it is working as expected? Same applies for #5371.

@IvanBM18 Can you add your testing to the description? This PR doesn't exercise the new code, but #5371 will. The codepath is Chrome specific.

Oh, I see that you included it in the script! Could you add a brief summary of what the script tests to the description? e.g. ensures each API returns successfully, the artifacts are downloaded etc.

dylanjew

LGTM, thanks!

javanlacerda

Overall LGTM, but as it changes the fetch_url, it would be great to run it in dev to check any side effects.

IvanBM18 · 2026-07-22T20:15:22Z

Overall LGTM, but as it changes the fetch_url, it would be great to run it in dev to check any side effects.

Hi @javanlacerda pushed the changes from this pr(and its dependants) to dev and reported the errors found in this description, everything LGTM please check it out :D

## Overview As per requested, we created a new feature flag to disable definitely the API build use cases(used trough `fetch_artifact.py`), this feature flag is enabled by default, because that's the current API & ClusterFuzz workflow. But can be disabled by adding the feature flag to the DB and marking it as enabled=false. ## Changes - Creates new feature_flag, and enables it by default in `fetch_artifact.py` - Logs warnings when the feature flag is disabled in `fetch_artifact.py` - Safely exists the related methods in `symbols_downloader` and `flash.py` which consume the `fetch_artifact.py` - Moved up the `adb.bad_state_reach()` method from the `fetch_artifact.py` module to its callers, this way we can now handle better the None values and raised errors from the API being turned off by the feature flag. ## Tests After the initial tests in the PR parents, i tested this changes once more in dev(including all 3 prs), looking specifically for `utils.fetch_url` or errors inherent to the return value from this method. Basically since my changes landed in dev the following error groups where seen: <img width="1735" height="1240" alt="image" src="https://github.com/user-attachments/assets/0591fb5c-d728-406a-b888-4f2534b2daab" /> Discarded the errors following this logic: - 403 Errors are expected to appear as per the changes that were previously there to avoid them are no longer in dev - Some error groups only appeared on previous clusterfuzz versions - Other errors are expected(like the queue size limit or bad revision errors) - There are errors from corpus pruning & fuzzer not found, this have been seen for a while now, not related to my changes - Lastly there are some errors in a cron_job, specifically in [this file ](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/cron/load_bigquery_stats.py)but this one doesn't make usage of `fetch_url` and nor does the method that triggered the error #### PR review 1. [**#5370** Add Android Build API V4 REST client and tests ](#5370) 2. [#5371 Migrate fetch_artifact to use new V4 API](#5371) 3. 👉 [#5373 Creates Feature Flag for Android V4 API](#5373) *(This PR)* Please only approve, I'll merge this PR's to avoid any possible merge conflict that can appear in the process :D

## Overview Bug: b/422775458 Our previous implementation of the android builv v4 api was wrong, it failed always because the api no longer worked on discovery service url. In other [PR](#5370) we focused on creating the new api to interact with the v4 requirements. In this PR we focus on using this api class in the `fetch_artifact.py` module, ## Changes - Made private all the methods that were only used internally in `fetch_artifact.py` - Refactors previous usage of the v4 api to now use the current version with minimal changes in the `fetch_artifact.py` script to avoid breaking anything ## Tests performed - Created unit tests for `feth_artifact.py` - Tested this script using [a custom script](https://github.com/google/clusterfuzz/compare/android-build-v4-testing) (test details in [this PR description](#5370) ) #### PR review 1. [**#5370** Add Android Build API V4 REST client and tests ](#5370) 2. 👉 #5371 Migrate fetch_artifact to use new V4 API *(This PR)* 3. [#5373 Creates Feature Flag for Android V4 API](#5373) Please only approve, I'll merge this PR's to avoid any possible merge conflict that can appear in the process :D

IvanBM18 requested review from dylanjew and letitz July 21, 2026 06:21

IvanBM18 requested a review from a team as a code owner July 21, 2026 06:21

IvanBM18 commented Jul 21, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/platforms/android/android_build_v4_api.py Outdated

IvanBM18 commented Jul 21, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/platforms/android/android_build_v4_api.py Outdated

Add Android Build API V4 REST client and tests

b185fc1

IvanBM18 force-pushed the feature/creates-android-v4-api branch from 220c2da to b185fc1 Compare July 21, 2026 06:24

Fix Unit test failures

21c1c34

IvanBM18 mentioned this pull request Jul 21, 2026

[Android v4 API] Migrate fetch_artifact to use new V4 API #5371

Merged

letitz removed their request for review July 21, 2026 08:37

dylanjew reviewed Jul 21, 2026

View reviewed changes

IvanBM18 added 2 commits July 21, 2026 18:27

adopt utils.fetch_url in AndroidBuildV4Api

8356587

Fix UT failure

a595008

dylanjew approved these changes Jul 21, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/platforms/android/android_build_v4_api.py

IvanBM18 mentioned this pull request Jul 21, 2026

[Android V4 api Migration] Creates Feature Flag #5373

Merged

IvanBM18 changed the title ~~Add Android Build API V4 REST client and tests~~ [Adroid v4 API] Add Android Build API V4 REST client and tests Jul 21, 2026

IvanBM18 requested a review from javanlacerda July 21, 2026 21:13

javanlacerda reviewed Jul 22, 2026

View reviewed changes

IvanBM18 added a commit that referenced this pull request Jul 22, 2026

[#5370] Add Android Build API V4 REST client and tests

94bc18a

javanlacerda approved these changes Jul 23, 2026

View reviewed changes

IvanBM18 added 2 commits July 23, 2026 10:10

Nit: simplify return in _call_android_api_enabled

89d6ea2

IvanBM18 merged commit e36152a into master Jul 23, 2026
14 of 15 checks passed

IvanBM18 deleted the feature/creates-android-v4-api branch July 23, 2026 17:06

Uh oh!

Conversation

IvanBM18 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Testing script details

Dev verification

Note:

PR review Stack

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

letitz commented Jul 21, 2026

Uh oh!

vitaliset commented Jul 21, 2026

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dylanjew commented Jul 21, 2026

Uh oh!

dylanjew commented Jul 21, 2026

Uh oh!

dylanjew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

javanlacerda left a comment

Choose a reason for hiding this comment

Uh oh!

IvanBM18 commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

IvanBM18 commented Jul 21, 2026 •

edited

Loading