[Android V4 api Migration] Creates Feature Flag

[IvanBM18]

Overview

As per requested, we created a new feature flag to disable definitely the API build use cases(used trough fetch_artifact.py), this feature flag is enabled by default, because that's the current API & ClusterFuzz workflow.

But can be disabled by adding the feature flag to the DB and marking it as enabled=false.

Changes

• Creates new feature_flag, and enables it by default in fetch_artifact.py
  • Logs warnings when the feature flag is disabled in fetch_artifact.py
• Safely exists the related methods in symbols_downloader and flash.py which consume the fetch_artifact.py
• Moved up the adb.bad_state_reach() method from the fetch_artifact.py module to its callers, this way we can now handle better the None values and raised errors from the API being turned off by the feature flag.

Tests

After the initial tests in the PR parents, i tested this changes once more in dev(including all 3 prs), looking specifically for utils.fetch_url or errors inherent to the return value from this method. Basically since my changes landed in dev the following error groups where seen:

Discarded the errors following this logic:

• 403 Errors are expected to appear as per the changes that were previously there to avoid them are no longer in dev
• Some error groups only appeared on previous clusterfuzz versions
• Other errors are expected(like the queue size limit or bad revision errors)
• There are errors from corpus pruning & fuzzer not found, this have been seen for a while now, not related to my changes
• Lastly there are some errors in a cron_job, specifically in this file but this one doesn't make usage of fetch_url and nor does the method that triggered the error

PR review

1. #5370 Add Android Build API V4 REST client and tests
2. #5371 Migrate fetch_artifact to use new V4 API
3. 👉 #5373 Creates Feature Flag for Android V4 API (This PR)

Please only approve, I'll merge this PR's to avoid any possible merge conflict that can appear in the process :D

[dylanjew]

thanks! just one question

[dylanjew]

what happens if this returns without the build? Does the device get stuck in a flash cycle?

[IvanBM18]

Trueeeee forgot about what i told you in this doc's appendix.

I performed an extensive investigation with jetski(TMI to digest by myself), and heres what i found out:

• All non uworker bots can update their codebase themselves trough update_task
• When a PHYSICAL android bot needs to flash and for some reason, the artifact/build is not found, we reach this bad_state method, which causes the android device to sleep for 1 hour.
• After that the bot wakes up and since we are still in a loop it updates itself and tries to work again, so if lets say we introduce a defect in the code then the bot will just update itself each hour till we eventually publish the fix
• If there's a none code issue(like a failing api) or something is inherently wrong with the bot then the bot will just continue to operate till it hits another timeout

So the way i see it here is to ensure that if a bot fails it calls the bad_state function to trigger the whole self-healing process, which we already do(see changes in flash.py)

For emulated uworker devices(like in swarming), we are 100% sure that they wont try to flash themselves, so theres no worry in here. The other place where we could run into an android api v4 issue is at symbols_downloader.py where it tries to get artifacts to process a stack-trace of errors, but i also investigated if this could cause the bot to loop, and the validations that we have in place avoid the uworker bot from crashing and just ignores the stack_trace symbolization.

[dylanjew]

Thanks! This flag is fine as long as the bot is in the retry loop but isn't actually making a bunch of calls that we know will fail.

[dylanjew]

Thanks! This flag is fine as long as the bot is in the retry loop but isn't actually making a bunch of calls that we know will fail.

[dylanjew]

LGTM! last question is more of a nit

[javanlacerda]

Suggested change

	if flag is None:
	return True
	return flag.enabled
	return flag.enabled if flag else True