- Update the following TPDM/EPDM base and stage models, to support new edu_wh models:
- base_epdm__certification_exam_results
- stg_epdm__certification_exams
- stg_epdm__certification_exam_results
- Fix invalid reference for relationships test on
stg_ef3__student_program_evaluations
extract_descriptor()macro now correctly handles descriptors in cases where the field name doesn't match the underlying descriptor code (e.g. birthCountryDescriptor -> countryDescriptor)
- Add base/stg models for new SEDM
student_iepsandiep_events - Add Ed-Fi Data Standard v5 special education base/stage models:
base_ef3__program_evaluations,base_ef3__student_program_evaluations,base_ef3__student_special_education_program_eligibility_associations,stg_ef3__program_evaluations,stg_ef3__student_program_evaluations,stg_ef3__student_special_education_program_eligibility_associations - Add columns to
base_ef3__student_special_education_program_associationsthat are new in Ed-Fi Data Standard v5.2 - Add
stg_ef3__sections__programs - Add
k_persontostg_ef3__staffs(useful for TPDM/EPDM domain links)
- Breaking change if your project is using the TPDM/EPDM domain: The variable
edu:tpdm:enabledhas been renamed tosrc:domain:tpdm:enabledto remain consistent with the existing domain variables. - Breaking changes if your project is using the TPDM/EPDM domain: removed
k_survey_response_person_target_association,k_candidate_educator_preparation_program_associationand renamedk_educator_prep_programtok_educator_preparation_program
- Add
stg_tpdm__candidates__other_names,stg_tpdm__candidate_races, and modifystg_tpdm__candidatesto supportdim_candidatemodel
- Fix missing column from test
cfg_student_assessment_score_name_dupes - Minor fixes for Databricks compatibility
- Add
stg_ef3__student_academic_records__academic_honorsto capture academic honors from student academic records - Updated
extract_extension.sqlmacro to allow for extensions to be defined in the edu_edfi_sourcedbt_project.ymland enabled per project.- And added the tpdm/epdm predefined extension to
dbt_project.yml
- And added the tpdm/epdm predefined extension to
- To fix keys that were triggering faulty dbt test failures:
- Updated
gen_skey.sqlmacro. Changed the order ofk_survey_sectioncolumn list. Changed survey_section_title from snake case to camel case. - Corrected
stg_ef3__survey_section_responses.sqlmodelk_survey_section_responsevariable. previously missingsurvey_response_idvariable.
- Updated
- Add base/stage models
base_ef3__student_school_food_service_program_association,stg_ef3__student_school_food_service_program_association,stg_ef3__stu_school_food_service__program_services - Add base/stage models
base_ef3__student_migrant_education_program_associations,stg_ef3__student_migrant_education_program_associations,stg_ef3__stu_migrant_edu__program_services - Add
stg_ef3__students__other_names
- Add
k_assessmenttostg_ef3__student_assessments__score_resultsandstg_ef3__student_assessments__performance_levelsfor use downstream
- Fix deduplication logic of
stg_ef3__course_transcriptsto remove duplicates from across ODS years, by replacingk_course(tied to api_year) withcourse_code, course_ed_org_idin the deduplication key
- Add Databricks platform compatibility
- The only potentially breaking change:
base_ef3__discipline_incidents.v:incidentTimedatatype switched from time to string.
- The only potentially breaking change:
- Add new optional fields in
student_education_organization_associationsandstudent_school_associationsfrom Ed-Fi Data Standard v5:- stuEdOrg:
gender_identity,supporter_military_connection - stuSchAssoc:
is_school_choice,school_choice_basis,enrollment_type,next_year_school_id,next_year_grade_level
- stuEdOrg:
- Add base/stage model for
StudentCTEProgramAssociation - Add stage model
stg_ef3__stu_cte__program_servicesto flatten program services list forStudentCTEProgramAssociation
- Add handling of multiple academic subjects per course for DS 5.0 compatibility; new
stg_ef3__courses__academic_subjectsmodel and newv_academic_subjectscolumns inbase_ef3__coursesandstg_ef3__coursesmodels. - Correctly cast minimum and maximum credits for courses to float instead of int.
- Update unique key of staff section association for DS 5.0 compatibility
- Fix unique key of
k_objective_assessmentto account for recent update to include obj assess subject.
- Add base/stage models for
staffEducationOrganizationContactAssociations - Add (optional) support for incremental materialization of most expensive stg models
- Fix unique key of
k_assessmentinstg_ef3__objective_assessments - Fix unique key
k_student_assessmentinstg_ef3__student_assessmentsto includestudent_unique_id(relevant only wherestudent_assessment_identifieris not on its own unique) - Fix surrogate key generation for References that include Descriptors that utilize EDU's 'replace descriptor' functionality
- Fix deduplication logic in all stg models to handle deleted records correctly before removal. This brings EDU more in sync with ODS true state
- Fix unique key in
stg_ef3__student_objective_assessmentsto include assessment_identifier to handle edge case duplicates
- Add base/stage models for
restraintEvents - Add partial support for TPDM Core and TPDM Community
- Handle invalid timestamp formatting in student_assessments
- Fix coalesce logic for academic subjects in
stg_ef3__objective_assessmentsandstg_ef3__student_objective_assessmentsto hydrate correctly when populated in respective Ed-Fi element'sacademicSubjectfield
- Add missing
program_enroll_end_dateto every stg-stu-program__program_services+stu_spec_ed__disabilities
- Fix surrogate key creation for
stg_ef3__grading_periodsto properly hanlde lowering of alphanumeric column (grading_period_name) that is part of natural key
- Fix surrogate key creation for
stg_ef3__student_contact_associationsto properly hanlde lowering of alphanumeric columns that are part of natural keys
- Add
stg_ef3__stu_ed_org__cohort_yearstracking student cohort designations (flattens Ed-Fi collectioncohort_yearsfor easier downstream use) - Add base/stage models for
contactsandstudent_contact_associations, added due to the rename from parent to contact in Ed-Fi data standard v5.0. - Rename
k_parenttok_contactinstg_ef3__survey_responses. - Add
gender_identity,preferred_first_name,preferred_last_namecolumns tostaffs(Ed-Fi Data Standard v5.0 additions) - Add
section_typedescriptor column tosections(Ed-Fi Data Standard v5.0 addition) - Add
responsible_teacher_staff_reference,v_programs,v_sectionscolumns tocourse_transcripts(Ed-Fi Data Standard v5.0 additions) - Add
v_programscolumn tocourse_transcripts(Ed-Fi Data Standard v5.0 addition)
- Add columns to
base_ef3__parentsto allow data to be unioned into newstg_ef3__contactsmodel - Update package dependency
dbt_utilsto 1.3.0, including alignment to renamedgenerate_surrogate_key()macro. Note, this change now treats nulls and empty strings as distinct values in surrogate key generation. - Make package macro calls to
extract_descriptorexplicit soflatten_arrayscan be used by outer packages
- Fix typo in column name
courses.maxCompletionsForCredit - Fix surrogate key creation for
stg_ef3__student_academic_records,stg_ef3__student_objective_assessments, andstg_ef3__studentsto properly handle lowering of alphanumeric columns that are part of natural keys - Rename
stg_ef3__staff__racestostg_ef3__staffs__racesfor consistency with EDU naming conventions
- Fix data type on base_ef3__student_special_education_program_associations.student_unique_id
- Remove records deleted from Ed-Fi in
student_education_organization_associationsandstudent_parent_associations
- Add
v_exthandling indisciplineandstudent_academic_recordflattened models - Add missing references/surrogate key generation for ed_orgs in
graduation_plansandcourses
- Add base/stage models for student_education_organization_responsibility_associations
- Add column
last_modified_timestampto every base table (viasource_edfi3macro). This includes timestamps of Ed-Fi deletes, which is helpful for tracking down when deletes occurred.
- Improve performance of
stg_ef3__student_assessmentsby fixing a join inint_ef3__student_assessments__identify_subject. Also fixes edge case bug (only applicable where assessments were loaded incorrectly).
- Add explicit namespacing for macro call:
edu_edfi_source.extract_descriptor()withingen_skey()sogen_skey()can be used outside this package
- Force staff_unique_id and student_unique_id to lower in construction of
k_staffandk_student. This is needed for keys to match foreign keys generated usinggen_skey()macro.
- Add
base_ef3__staff_education_organization_employment_associations - Add
stg_ef3__staff_education_organization_employment_associations - Add `stg
- Add
base_ef3__staff_education_organization_employment_associations - Add
stg_ef3__staff_education_organization_employment_associations - Add
stg_ef3__staff__races
- Fix to
stg_ef3__student_assessments: remove deleted records
- Bring SEA and ESC in line with LEA by using abbreviated names and keys
- Fix spelling of
stg_ef3__stu_ed_org__disabilities.disability_designation - Fix unique key of
stg_ef3__student_school_attendance_eventsandstg_ef3__student_section_attendance_eventsto correctly align with Ed-Fi key. This creates new records that are handled in edu_wh v0.3.0
- add k_student_xyear to student assessment models
- Fix deduplication ordering in student academic records
- Add models for SEAs
- Tag cohort models for easy disabling when not in use
- Fix typo on attendance durations
- Add base & stage models for learning standards, and stg_ef3__grades__learning_standards for student-learning standard-grades
- Fix a rare edge case in the grain of the discipline model against multi-year ODSes
- Add studentAcademicRecord diplomas model
- On Assessment records, use timestamps without timezones for better compatibility
- Fix discipline-behavior grain
- Add missing model config for disabilities
- Handle single quotes in Descriptors when swapping from codeValue to shortDescription
- Allow for swapping descriptor codeValues for shortDescriptions or Descriptions
- Fix attribution of Objective Assessments to the correct Academic Subject
- Add models for the main components of the Survey domain
- Add model for disabilities on studentSpecialEducationProgramAssociation
- Change the way foreign key generation works: optional references that are Null will now produce a Null key rather than a valid hash that doesn't join to anything
- Bugfix release: address inconsistent casing in deletes IDs in Ed-Fi deployments
- Bugfix release: correct typo
- Bugfix release: correct typo
- Add to default non-offender codes
- Add models for education service centers
- Add additional discipline/behavior models
- Update
extract_extensionmacro to handle multiple models - Add school_year to staff ed org assignments
- Standardize naming conventions across program models
- Add minimum dbt version requirement
- Add optional domain disabling to all non-core models, using vars in dbt_project.yml.
- Add a single model properties file under each subdirectory, as per DBT recommendation.
- Remove unnecessary deduplication keys for program association staging models
- Add simplified
full_addressfield to models containing addresses
- Simplify
do_not_publish_indicatortodo_not_publish
- Fix incorrect indentation on cohort test
- Add models for studentLanguageInstructionProgramAssociations, studentHomelessProgramAssociations, and studentTitleIPartAProgramAssociations
- Add model for Special Education Program Services
- Add models for cohorts
- Add model for graduation plans
- Remove unnecessary deduplication keys for program association staging models
- Add models for descriptors
- Add model for student languages
- Add model for student telephone numbers
- Add model for Parents
- Add model for studentParentAssociations
- Factor out handling of common sub-objects like addresses and phone numbers to macros
Initial release