fix: don't count DNS resolution errors against consecutive failure limit

When crawling large domain lists (e.g. Tranco top 100k), many domains
cannot be resolved via DNS. These dnsNotFound neterrors are expected and
do not indicate a browser or instrumentation failure. Previously they
were counted against MAX_CONSECUTIVE_FAILURES, eventually crashing the
crawl.

Skip the failure counter increment and browser restart for dnsNotFound
neterrors so that DNS resolution failures no longer crash large crawls.

Fixes #1116

Author: vringar

openwpm/browser_manager.py

@@ -461,7 +461,18 @@ def execute_command_sequence(
                 )
                 return
 
-            if command_status != "ok":
+            # DNS resolution errors are expected when crawling large domain
+            # lists (e.g. Tranco top 100k) and don't indicate a browser or
+            # instrumentation failure, so we skip the failure counter and
+            # browser restart for them.  See:
+            # https://github.com/openwpm/OpenWPM/issues/1116
+            is_dns_error = (
+                command_status == "neterror"
+                and error_text is not None
+                and error_text == "dnsNotFound"
+            )
+
+            if command_status != "ok" and not is_dns_error:
                 with task_manager.threadlock:
                     task_manager.failure_count += 1
                 if task_manager.failure_count > task_manager.failure_limit: