remove succeed fast feature (for now, at least) from reference validation, based on discussion with Jules and development of student ID matching bundle

Author: Tom Reitz

README.md

@@ -154,16 +154,18 @@ The `references` `method` can be slow, as a separate `GET` request may be made t
 * batching requests and sending several concurrently (based on `connection`.`pool_size` of `lightbeam.yaml`)
 * caching responses and first checking the cache before making another (potentially identical) request
 
-Even with these optimizations, checking `references` can easily take minutes for even relatively small amounts of data. Therefore `lightbeam.yaml` also accepts two further configuration options:
+Even with these optimizations, checking `references` can easily take minutes for even relatively small amounts of data. Therefore `lightbeam.yaml` also accepts a further configuration option:
 ```yaml
 validate:
   references:
     max_failures: 10 # stop testing after X failed payloads ("fail fast")
-    partial: 500 # stop testing if 100% success after X payloads ("succeed fast")
 ```
-These are optional; if absent, references in every payload are checked, no matter how many fail or succeed, respectively.
+This is optional; if absent, references in every payload are checked, no matter how many fail.
 
-**Note:** Reference validation efficiency may be improved by first `lightbeam fetch`ing certain resources to have a local copy. `lightbeam validate` checks local JSONL files to resolve references before trying the remote API, and `fetch` retrieves many records per  `GET`, so total runtime can be faster in this scenario. The downsides are more data movement and the local data becoming stale over time.
+**Note:** Reference validation efficiency may be improved by first `lightbeam fetch`ing certain resources to have a local copy. `lightbeam validate` checks local JSONL files to resolve references before trying the remote API, and `fetch` retrieves many records per  `GET`, so total runtime can be faster in this scenario. The downsides include
+* more data movement
+* `fetch`ed data becoming stale over time
+* needing to track which data is your own vs. was `fetch`ed (all the data must coexist in the `config.data_dir` to be discoverable by `lightbeam validate`)
 
 
 ## `send`

lightbeam/validate.py

@@ -169,7 +169,6 @@ def get_swagger_definition_for_endpoint(self, endpoint):
 
     # Validates a single endpoint based on the Swagger docs
     async def validate_endpoint(self, endpoint):
-        partial_threshold = self.lightbeam.config.get("validate",{}).get("references",{}).get("partial", False)
         fail_fast_threshold = self.lightbeam.config.get("validate",{}).get("references",{}).get("max_failures", 10)
         definition = self.get_swagger_definition_for_endpoint(endpoint)
         data_files = self.lightbeam.get_data_files_for_endpoint(endpoint)
@@ -196,10 +195,6 @@ async def validate_endpoint(self, endpoint):
                     if self.lightbeam.num_errors >= fail_fast_threshold:
                         self.logger.critical(f"... STOPPING; found {self.lightbeam.num_errors} >= validate.references.max_failures={fail_fast_threshold} VALIDATION ERRORS.")
                         break
-                    # implement "succeed fast" feature:
-                    if self.lightbeam.num_errors==0 and partial_threshold and total_counter>=partial_threshold:
-                        self.logger.info(f"... STOPPING; all {total_counter} tested payloads >= validate.references.partial={partial_threshold} validated successfully.")
-                        return
 
             if len(tasks)>0: await self.lightbeam.do_tasks(tasks, total_counter, log_status_counts=False)
 
@@ -377,11 +372,13 @@ def get_cache_key(payload):
 
     def remote_reference_exists(self, endpoint, params):
         # check cache:
+        if endpoint=='students' and 'studentUniqueId' in params.keys(): return True
         if endpoint not in self.remote_reference_cache.keys():
             self.remote_reference_cache[endpoint] = []
         cache_key = self.get_cache_key(params)
         if cache_key in self.remote_reference_cache[endpoint]:
             return True
+        # print(f"remote reference lookup to {endpoint} for {params}")
         # do remote lookup
         curr_token_version = int(str(self.lightbeam.token_version))
         while True: # this is not great practice, but an effective way (along with the `break` below) to achieve a do:while loop