Skip to content

Commit eed179d

Browse files
jayckaiserrlittle08alchenist
authored
Rc/3.0 redux (#49)
* Update CHANGELOG in preparation for release. * fix behavior for terminal acronyms like in studentIEPs (#47) * Update changelog with latest commit. * Update CHANGELOG with imminent additions in migrate_from_rc branch. * Feature/migrate session from rc (#40) * Refactor entire library to use EdFiSession from rc/0.3.0 branch; use logging library; remove Ed-Fi 2 functionality. * Fix bugs across repo. * Implements basic plain-text file cache for auth tokens * Make token base path configurable * Move token load/update from cache into separate method * Add test with forced refreshes * Refactor portalocker caching into its own class * Add pure Python soft lock based token cache * Have token caches implement abstract base class * For soft lockfile, impelement staleness threshold * Tweak tests and default timeout * Adds tests for get_token_info and for instance-specific auth endpoint * Add test for max_retries on paged requests * Add test for default behavior of retries off * Fix no-retry default test to add check for total call count * Make access_token a property to force authentication if accessed before first request. * Migrate get_instance_locator() code into instance_locator property. * Improve docstrings to clarify cleanup logic for posted records. * Update get_rows/pages docstrings and reorder arguments for clarity. * Update API year error to install latest 0.2 version. * Fix mock in get_token_info test to chdeck for JSON-encoded POST body * Incorporate from main auth URL fixes for specific API mode. * Feature/persist tokens to disk cleanup (#45) * Add Python 3.10 requirement (and fix indentation). * Clean up comments, whitespace, and method order in Session. * Move snapshot header setting outside internal auth helper method. * Pass max_retries argument into read_lock. * Move time attribute setters and cached token reauthentication into their respective helpers. * Clean up whitespacing, type hints, comments, and variable names; move shared definitions of `exists()` and `get_last_modified()` helpers into BaseTokenCache(). * Move shared abstract methods back into interface to allow future child class that caches non-locally. * Define refresh_at outside the helper call to ensure this argument can also be defined using cached tokens. * Bugfixes and make caller responsible for constructing TokenCache In preparation for removing PortalockerTokenCache (and leaving the pure Python LockfileTokenCache as the only option), shift instantiation and configuration of token caches onto the caller. - Replace `use_token_cache` parameter in EdFiClient with `token_cache`; caller must pass in instantiated token cache in order to use caching feature. - Remove `token_cache_lock_type` parameter and corresponding match/case from EdFiClient. - Add `token_id` property to BaseTokenCache abstract base class. Now that we expect a token cache to be instantiated outside of EdFiSession, sessions will have to pass in instance-client IDs, used to uniquely identify shared tokens, after instantiation. Implement such a setter in LockfileTokenCache that updates cache and lockfile paths upon being passed a new ID. - Update tests to follow this new pattern. Bug introduced in cleanup where EdFiSession.refresh_at was None even though EdFiSession.authenticated_at was not, occurring in non-cache code path. Session would call _make_auth_request, which does not update both timestamps. - Have _make_auth_request set both timestamps, not just authenticated_at. - Factor out token loading into EdFiSession._load_token_from_cache, a method that also updates both timestamps and shares the same signature as _make_auth_request. This should make the invariant more clear. Bug found in testing where LockfileTokenCache.get_write_lock would error out when `os.path.getmtime` was called on a nonexistent lockfile because the lockfile had been deleted in between the `os.path.exists` check and the `getmtime` call. Fixed with an extra branch in an existing try/except catching this FileNotFoundError. Could be addressed by another try/except that would allow us to try to acquire the write lock immediately, but this seems more readable and easier to reason about. * Remove portalocker caching implementation * Move parameters into __init__ to allow for caller to control token caches * Tweak test_multiprocessing_with_forced_refreshes interval to force more invalidations * Push uniqueness responsibility down to token caches instead of EdFiSession * Rename missing variable. * Rename object imported to match typing. * Linting/type checking fixes for token caching - Explicit `typing.cast` to narrow down types for Optional instance attributes - Fixing various timestamps to ints - Missing type hints - Default string/PathLike values to get rid of Optional types - Unused imports, unused variable fixes * Add mocked test for default uncached client fetching own token * Fix EdFiSession.connect type hint * Fix re-auth on stale token from cache * Update setup.py metadata for release. * Add additional comments for test against live ODS * Add token caching info to README --------- Co-authored-by: Alex Chen <achen@edanalytics.org> Co-authored-by: Alex C. <alchenist@users.noreply.github.com> * Update README with new features of the release candidate. * Add instance_locator to Swagger URL. (#48) * Update CHANGELOG. --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> Co-authored-by: Alex Chen <achen@edanalytics.org> Co-authored-by: Alex C. <alchenist@users.noreply.github.com>
1 parent 9bc2cb9 commit eed179d

11 files changed

Lines changed: 1430 additions & 783 deletions

File tree

CHANGELOG.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,22 @@
1+
# edfi_api_client v0.3.0
2+
## New Features
3+
- Remove support for Ed-Fi 2.
4+
- Add optional token caching to persist auth tokens to disk using new `token_cache` argument.
5+
6+
## Under the Hood
7+
- Migrate REST functionality into its own `Session` class:
8+
- Initializes lazily and only authenticates when interfacing with an authenticated API call
9+
- Allow exponential backoff for any Session method call.
10+
- Add internal methods for POSTs, DELETEs, and PUTs.
11+
- Replace `verbose_log` print statements with the built-in `logging` library.
12+
- Deprecate `EdFiEndpoint.total_count()` in favor of more descriptive `get_total_count()`.
13+
- Add authentication and caching tests to package.
14+
15+
## Fixes
16+
- Fix behavior of `camel_to_snake()` util helper to handle more resource names.
17+
- Add instance locator to Swagger endpoint URL in `instance_year_specific` ODS instances.
18+
19+
120
# edfi_api_client v0.2.3
221
## New Features
322
- Add `use_snapshot` flag to `EdFiClient` for making requests against snapshots (default `False`).

README.md

Lines changed: 73 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ api = EdFiClient(BASE_URL, CLIENT_KEY, CLIENT_SECRET, api_version=3)
1010

1111
# Get the total row-count for the 'students' resource in the ODS
1212
students = api.resource('students')
13-
students.total_count()
13+
students.get_total_count()
1414

1515
# Pull all rows for the 'staffs' resource deletes endpoint (setting a custom page-size)
1616
staffs = api.resource('staffs', get_deletes=True)
@@ -50,7 +50,8 @@ Some methods do not require credentials to be called.
5050
| api_mode | The API mode of the ODS (e.g., `shared_instance`, `year_specific`, etc.). If empty, the mode will automatically be inferred from the ODS' Swagger spec (Ed-Fi 3 only). |
5151
| api_year | The year of data to connect to if accessing a `year_specific` or `instance_year_specific` ODS. |
5252
| instance_code | The instance code if accessing an `instance_year_specific` ODS. |
53-
| use_snapshot | Boolean flag for whether connected ODS is a snapshot (default `False`). |
53+
| use_snapshot | Boolean flag for whether connected ODS is a snapshot (default `False`). |
54+
| token_cache | An optional token cache instance, such as `edfi_api_client.token_cache.LockfileTokenCache`, for storing OAuth bearer tokens to be shared among clients. |
5455

5556
If either `client_key` or `client_secret` are empty, a session with the ODS will not be established.
5657

@@ -84,8 +85,6 @@ Authentication with the ODS is not required:
8485
-----
8586

8687
### resources
87-
This method is unavailable in Ed-Fi2.
88-
8988
Retrieve a list of namespaced-resources from the `resources` Swagger payload.
9089

9190
```python
@@ -104,8 +103,6 @@ Retrieve a list of namespaced-resources from the `resources` Swagger payload.
104103
-----
105104

106105
### descriptors
107-
This method is unavailable in Ed-Fi2.
108-
109106
Retrieve a list of namespaced-descriptors from the `descriptors` Swagger payload.
110107

111108
```python
@@ -128,8 +125,6 @@ Authentication with the ODS is not required:
128125
-----
129126

130127
### get_info
131-
This method is unavailable in Ed-Fi2.
132-
133128
Ed-Fi3 provides an informative payload at the ODS base URL.
134129
This contains versioning by suite and build, API mode, and URLs for authentication and data management.
135130

@@ -159,8 +154,6 @@ This contains versioning by suite and build, API mode, and URLs for authenticati
159154
-----
160155

161156
### get_api_mode
162-
This method is unavailable in Ed-Fi2.
163-
164157
Each Ed-Fi3 ODS has a declared API mode that alters how users interact with the ODS. This is a shortcut-method for finding the API mode of the Ed-Fi ODS via the payload retrieved using `EdFiClient.get_info()`, formatted in snake_case.
165158

166159
```python
@@ -176,8 +169,6 @@ This method is called automatically when `api_mode` is left undefined by the use
176169
<summary><code>get_ods_version</code></summary>
177170

178171
### get_ods_version
179-
This method is unavailable in Ed-Fi2.
180-
181172
This is a shortcut-method for finding the version of the Ed-Fi ODS via the payload retrieved using `EdFiClient.get_info()`.
182173

183174
```python
@@ -196,8 +187,6 @@ This is a shortcut-method for finding the version of the Ed-Fi ODS via the paylo
196187
-----
197188

198189
### get_data_model_version
199-
This method is unavailable in Ed-Fi2.
200-
201190
This is a shortcut-method for finding the data model version of the Ed-Fi ODS' 'ed-fi' namespace via the payload retrieved using `EdFiClient.get_info()`.
202191

203192
```python
@@ -216,8 +205,6 @@ This is a shortcut-method for finding the data model version of the Ed-Fi ODS' '
216205
-----
217206

218207
### get_swagger
219-
This method is unavailable in Ed-Fi2.
220-
221208
The entire Ed-Fi API is outlined in an OpenAPI Specification (i.e., Swagger Specification).
222209
There is a separate Swagger defined for each component type (e.g., resources, descriptors, etc.).
223210

@@ -245,28 +232,49 @@ Returns an `EdFiSwagger` class containing the complete JSON payload, as well as
245232
-----
246233

247234
### is_edfi2
248-
This boolean filter returns whether the client-connection to the ODS is via Ed-Fi2.
249-
Ed-Fi3 introduces many new features that are utilized heavily in this package.
235+
Ed-Fi3 introduced many new features that are utilized heavily in this package.
250236

251237
```python
252238
>>> api.is_edfi2()
253239
False
254240
```
255241

242+
Package compatibility with Ed-Fi2 has been deprecated as of version 0.3.
243+
256244
-----
257245

258246
</details>
259247

260248

261249
Authentication with the ODS is required:
262250

251+
<details>
252+
<summary><code>get_token_info</code></summary>
253+
254+
-----
255+
256+
### get_token_info
257+
This method requires a connection to the ODS.
258+
259+
The Ed-Fi API provides a way to get information about the education organization related to a token.
260+
This method returns the `oauth/token_info` payload for the current session.
261+
262+
```python
263+
>>> api.get_token_info()
264+
{'active': True, 'client_id': '', 'namespace_prefixes': [], 'education_organizations': [], 'assigned_profiles': []}
265+
```
266+
267+
-----
268+
269+
</details>
270+
271+
263272
<details>
264273
<summary><code>get_newest_change_version</code></summary>
265274

266275
-----
267276

268277
### get_newest_change_version
269-
This method is unavailable in Ed-Fi2.
270278
This method requires a connection to the ODS.
271279

272280
Starting in Ed-Fi3, each row in the ODS is linked to an ODS-wide "change version" parameter, which allows for narrow time-windows of data to be filtered for delta-ingestions, instead of only full-ingestions.
@@ -288,7 +296,7 @@ This method returns the newest change version defined in the ODS.
288296
-----
289297

290298
### resource
291-
This method requires a connection to the ODS.
299+
This method only requires a connection to the ODS when calling a REST method.
292300

293301
Use this method to initialize an EdFiResource (i.e. EdFiEndpoint).
294302
This object contains methods to pull rows and resource metadata from the API.
@@ -318,7 +326,7 @@ This object contains methods to pull rows and resource metadata from the API.
318326
-----
319327

320328
### descriptor
321-
This method requires a connection to the ODS.
329+
This method only requires a connection to the ODS when calling a REST method.
322330

323331
Use this method to initialize an EdFiResource (i.e. EdFiEndpoint).
324332
This object contains methods to pull rows and descriptor metadata from the API.
@@ -348,7 +356,7 @@ Note that although descriptors and resources are saved at the same endpoint in t
348356
-----
349357

350358
### composite
351-
This method requires a connection to the ODS.
359+
This method only requires a connection to the ODS when calling a REST method.
352360

353361
Use this method to initialize an EdFiComposite (i.e. EdFiEndpoint).
354362
This object contains methods to pull rows and composite metadata from the API.
@@ -453,6 +461,8 @@ This offers a shortcut for verifying claim-set permissions without needing to pu
453461
{'message': 'Ping was successful! ODS data has been intentionally scrubbed from this response.'}
454462
```
455463

464+
Note that `params` and retry-arguments can be passed to override those initialized in the `EdFiEndpoint`.
465+
456466
-----
457467

458468
</details>
@@ -479,6 +489,8 @@ If unspecified, the default limit will be retrieved.
479489
```
480490
Because this GET does not use pagination, the return is a list, not a generator.
481491

492+
Note that `params` and retry-arguments can be passed to override those initialized in the `EdFiEndpoint`.
493+
482494
-----
483495

484496
</details>
@@ -521,26 +533,30 @@ Under the hood, `get_rows()` implements `get_pages()`, but unnests the rows befo
521533
```
522534
To circumvent memory constraints, these methods return generators instead of lists.
523535

536+
Note that `params` and retry-arguments can be passed to override those initialized in the `EdFiEndpoint`.
537+
524538
-----
525539

526540
</details>
527541

528542

529543
<details>
530-
<summary><code>total_count</code></summary>
544+
<summary><code>get_total_count</code></summary>
531545

532546
-----
533547

534-
### total_count
548+
### get_total_count
535549
This method returns the total count of rows for the given endpoint, as declared by the API.
536550
This action is completed by sending a limit 0 GET request to the API with the `Total-Count` header set to `True`.
537551

538552
```python
539-
>>> students.total_count()
553+
>>> students.get_total_count()
540554
4135
541555
```
542556

543-
`total_count()` is currently only implemented for resources, not composites.
557+
`get_total_count()` is currently only implemented for resources, not composites.
558+
559+
Note that `params` and retry-arguments can be passed to override those initialized in the `EdFiEndpoint`.
544560

545561
-----
546562

@@ -692,3 +708,34 @@ However, this row will not be lost.
692708
-----
693709

694710
</details>
711+
712+
713+
## Token caching
714+
[Starting in version 7.3](https://github.com/Ed-Fi-Alliance-OSS/Ed-Fi-ODS-Implementation/commit/80970eef722e31020bd9bb232eb92b5ca4a81f12), EdFi Web API instances limit clients to 15 concurrent bearer tokens by default. In case an application uses multiple concurrent `EdFiClient`s using the same client key and hitting the same API on a single machine or shared filesystem, this library provides a barebones on-disk cache to conserve tokens and avoid this limit.
715+
716+
```python
717+
from edfi_api_client import EdFiClient
718+
from edfi_api_client.token_cache import LockfileTokenCache
719+
720+
api = EdFiClient(BASE_URL, CLIENT_KEY, CLIENT_SECRET, api_version=3, token_cache=LockfileTokenCache())
721+
```
722+
723+
724+
<details>
725+
<summary>Arguments:</summary>
726+
727+
-----
728+
729+
| Argument | Description |
730+
|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
731+
| token_cache_directory | Path to store tokens in. One cache is a JSON file containing an authentication payload, unique by OAuth URL and client key. (default `~/.edfi-tokens/`) |
732+
| write_lock_timeout | Seconds to wait to acquire a write lock if the cache is to be updated. (default 30) |
733+
| write_lock_staleness_threshold | Seconds to wait after the last modified timestamp before forcibly deleting an existing lockfile (default 60). Aggressive by default, because a client won't try to obtain a write lock unless the payload inside is already expired or is corrupt. |
734+
| write_lock_retry_delay | Seconds to wait before retrying to acquire a write lock. (default 0.5) |
735+
736+
737+
-----
738+
739+
</details>
740+
741+
Other kinds of shared caches may be implemented with the interface defined in `edfi_api_client.token_cache.BaseTokenCache`, should more sophisticated functionality be required (encryption, distributed caching, etc.).

0 commit comments

Comments
 (0)