Merge pull request #61 from edanalytics/feature/validate_references_selector

implementing validate references selector, behavior and remote switch, plus update docs

Author: tomreitz

README.md

@@ -64,6 +64,19 @@ count:
   separator: ,
 fetch:
   page_size: 100
+validate:
+  methods:
+    - schema # checks that payloads conform to the Swagger definitions from the API
+    - descriptors # checks that descriptor values are either locally-defined or exist in the remote API
+    - uniqueness # checks that local payloads are unique by the required property values
+    - references # checks that references resolve, either locally or in the remote API
+  # or `methods: "*"`
+  references:
+    selector:
+      - studentAssessments.studentReference
+      - studentSchoolAssociations.schoolReference
+    behavior: exclude # or `include`
+    remote: False # default=True
 force_delete: True
 log_level: INFO
 show_stacktrace: True
@@ -94,6 +107,7 @@ show_stacktrace: True
   * (optional) Whether to `verify_ssl`. The default is `True`. Set to `False` when working with `localhost` APIs or to live dangerously.
 * (optional) for [`lightbeam count`](#count), optionally change the `separator` between `Records` and `Endpoint`. The default is a "tab" character.
 * (optional) for [`lightbeam fetch`](#fetch), optionally specify the number of records (`page_size`) to GET at a time. The default is 100, but if you're trying to extract lots of data from an API increase this to the largest allowed (which depends on the API, but is often 500 or even 5000).
+* (optional) for [`lightbeam validate`](#validate), optionally specify the list of validation `methods` to run (from `schema`, `descriptors`, `uniqueness`, and `references`). If validating `references`, specify a list of `selector`s to either `include` or `exclude` (`behavior`) when validating. Also optionally disable `remote` referece validation (enabled by default).
 * (optional) Skip the interactive confirmation prompt (for programmatic use) when using the [`delete`](#delete) command. The default is `False` (prompt).
 * (optional) Specify a `log_level` for output. Possible values are
   - `ERROR`: only output errors like missing required sources, invalid references, invalid [YAML configuration](#yaml-configuration), etc.
@@ -147,6 +161,12 @@ validate:
     - references # checks that references resolve, either locally or in the remote API
   # or
   # methods: "*"
+  references:
+    selector:
+      - studentAssessments.studentReference
+      - studentSchoolAssociations.schoolReference
+    behavior: exclude # or `include`
+    remote: False # default=True
 ```
 Default `validate`.`methods` are `["schema", "descriptors", "uniqueness"]` (not `references`; see below). In addition to the above methods, `lighteam validate` will also (first) check that each payload is valid JSON.
 
@@ -167,6 +187,8 @@ This is optional; if absent, references in every payload are checked, no matter
 * `fetch`ed data becoming stale over time
 * needing to track which data is your own vs. was `fetch`ed (all the data must coexist in the `config.data_dir` to be discoverable by `lightbeam validate`)
 
+You may specify a `selector` list of the form `someEndpoint.path.to.someReference` to include or exclude (according to `behavior`) specific references from reference validation. You may also specity `remote: False` to only validate references against local data in your JSONL files.
+
 
 ## `send`
 ```bash
@@ -209,6 +231,17 @@ Running the `truncate` command will prompt you to type "yes" to confirm. This co
 
 `truncate` is a convenience command which should be used sparingly, as it can generate large numbers of `deletes` records and cause performance issues when pulling from `deletes` endpoints. If you want to wipe an entire Ed-Fi ODS, a better approach may be to drop and recreate the database (and re-send Descriptors and other default resources as needed).
 
+## `create`
+```bash
+lightbeam create -s students,schools,studentSchoolAssociations -c path/to/config.yaml
+```
+Creates a skeleton of an [`earthmover`](https://edanalytics.github.io/earthmover/) project for _creating_ JSONL Ed-Fi data which one can then `lightbeam send` to an Ed-Fi API. It uses the Ed-Fi API's [OpenAPI specification](https://spec.openapis.org/oas/latest.html) to determine the schema of the endpoints you select (`lightbeam create` should usually be used with the `-s` [selector](#selectors)). Then, in the current directory, it:
+* creates (if it doesn't already exist, otherwise adds/overwrites) a partial `earthmover.yml` configuration file with empty `sources` and `transformations` but `destinations` for each selected endpoint, plus comments indicating what column names and data types/values are required, and the required grain of the table (based on `isIdentity` flags in the OpenAPI definitions)
+* creates (or overwrites) `templates/*.jsont` for each selected endpoint, with skeleton Jinja-JSON that includes all the required fields (including nested ones), optional fields wrapped in conditionals, and comments with some of the property metadata from OpenAPI definitions, such as `type`, `description`, `isIdentity`, etc.
+
+The purpose of `lightbeam create` is to save developers time if they want to use `earthmover` to create Ed-Fi-shaped data from other data sources. See the [`earthmover` documentation](https://edanalytics.github.io/earthmover/) for more information.
+
+
 ## Other options
 See a help message with
 ```bash
@@ -222,6 +255,14 @@ lightbeam -v
 lightbeam --version
 ```
 
+Override specific configurations in `lightbeam.yml` from the command-line using the `--set` flag
+```
+lightbeam fetch --set fetch.page_size 1000
+lightbeam validate --set log_level WARN show_stacktrace True
+lightbeam send --set connection.timeout 15
+```
+(`--set` must be followed by a set of key-value pairs.)
+
 
 # Features
 This tool includes several special features:

lightbeam/validate.py

@@ -39,7 +39,17 @@ def validate(self):
         if type(self.validation_methods)==str and (self.validation_methods=="*" or self.validation_methods.lower()=='all'):
             self.validation_methods = self.DEFAULT_VALIDATION_METHODS
             self.validation_methods.append("references")
-        
+        self.validation_references_selector = self.lightbeam.config.get("validate",{}).get("references",{}).get("selector", [])
+        for selector in self.validation_references_selector:
+            if "." not in selector:
+                self.logger.error(f"`config.validate.references.selector` {selector} is incorrectly formatted (should be `someEndpoint.someReference`, such as `studentSchoolAssociation.schoolReference`)")
+        self.validation_references_behavior = self.lightbeam.config.get("validate",{}).get("references",{}).get("behavior", "exclude")
+        if self.validation_references_behavior not in ["exclude", "include"]:
+            self.logger.error(f"`config.validate.references.behavior` must be either `exclude` (default) or `include`)")
+        self.validation_references_remote = self.lightbeam.config.get("validate",{}).get("references",{}).get("remote", True)
+        if "references" in self.validation_methods and not self.validation_references_remote:
+            self.logger.info(f"(references will only be validated against local data, since `config.validate.references.remote: False`)")
+
         self.lightbeam.api.load_swagger_docs()
         self.logger.info(f"validating by methods {self.validation_methods}...")
         if "descriptors" in self.validation_methods:
@@ -305,7 +315,7 @@ async def do_validate_payload(self, endpoint, file_name, data, line_number):
         # check references values are valid
         if "references" in self.validation_methods and "Descriptor" not in endpoint: # Descriptors have no references
             self.lightbeam.api.do_oauth()
-            error_message = self.has_invalid_references(payload, path="")
+            error_message = self.has_invalid_references(endpoint, payload, path="")
             if error_message != "":
                 self.log_validation_error(endpoint, file_name, line_number, "references", error_message)
 
@@ -399,40 +409,46 @@ def has_invalid_descriptor_values(self, payload, path=""):
         return ""
 
     # Validates descriptor values for a single payload (returns an error message or empty string)
-    def has_invalid_references(self, payload, path=""):
+    def has_invalid_references(self, endpoint, payload, path=""):
         for k in payload.keys():
             if isinstance(payload[k], dict) and not k.endswith("Reference"):
-                value = self.has_invalid_references(payload[k], path+("." if path!="" else "")+k)
+                value = self.has_invalid_references(endpoint, payload[k], path+("." if path!="" else "")+k)
                 if value!="": return value
             elif isinstance(payload[k], list):
                 for i in range(0, len(payload[k])):
-                    value = self.has_invalid_references(payload[k][i], path+("." if path!="" else "")+k+"["+str(i)+"]")
+                    value = self.has_invalid_references(endpoint, payload[k][i], path+("." if path!="" else "")+k+"["+str(i)+"]")
                     if value!="": return value
             elif isinstance(payload[k], dict) and k.endswith("Reference"):
+                check_this_reference = (
+                    (f"{endpoint}.{path}{k}" in self.validation_references_selector and self.validation_references_behavior=="include")
+                    or (f"{endpoint}.{path}{k}" not in self.validation_references_selector and self.validation_references_behavior=="exclude")
+                )
+                if not check_this_reference: continue
                 is_valid_reference = False
                 original_endpoint = self.resolve_reference_to_endpoint(k)
 
+                params = payload[k].copy()
+                if "link" in params.keys(): del params["link"]
+                
                 # this deals with the fact that an educationOrganizationReference may be to a school, LEA, etc.:
                 endpoints_to_check = self.EDFI_GENERICS_TO_RESOURCES_MAPPING.get(original_endpoint, [original_endpoint])
-                for endpoint in endpoints_to_check:
+                for endpt in endpoints_to_check:
                     # check if it's a local reference:
-                    if endpoint not in self.local_reference_cache.keys(): break
+                    if endpt not in self.local_reference_cache.keys(): break
                     # construct cache_key for reference
-                    cache_key = self.get_cache_key(payload[k])
-                    if cache_key in self.local_reference_cache[endpoint]:
+                    cache_key = self.get_cache_key(params)
+                    if cache_key in self.local_reference_cache[endpt]:
                         is_valid_reference = True
                         break
-                if not is_valid_reference: # not found in local data...
-                    for endpoint in endpoints_to_check:
+                if not is_valid_reference and self.validation_references_remote: # not found in local data...
+                    for endpt in endpoints_to_check:
                         # check if it's a remote reference:
-                        params = payload[k].copy()
-                        if "link" in params.keys(): del params["link"]
-                        value = self.remote_reference_exists(endpoint, params)
+                        value = self.remote_reference_exists(endpt, params)
                         if value:
                             is_valid_reference = True
                             break
-                    if not is_valid_reference:
-                        return f"payload contains an invalid {k} " + (" (at "+path+"): " if path!="" else ": ") + json.dumps(params)
+                if not is_valid_reference:
+                    return f"payload contains an invalid {k} " + (" (at "+path+"): " if path!="" else ": ") + json.dumps(params)
         return ""
 
     @staticmethod
@@ -497,7 +513,7 @@ def remote_reference_exists(self, endpoint, params):
                     else:
                         pass # await asyncio.sleep(1)
                     curr_token_version = int(str(self.lightbeam.token_version))
-                elif status=='404':
+                elif status=='404' or status=='400':
                     return False
                 elif status in ['200', '201']:
                     # 200 response might still return zero matching records...