Repository navigation
fix(events): keep three exported accounts behind every named organisation - #60
Merged
Merged
Conversation
…tion account_daily counts only accounts created before today (UTC), the accounts the export holds, so a named domain always shows at least 3 of them. The account export fails on an empty user table, replaces the table with no rows when every account is from today, and fails on an empty webmail_domain. Each exported event row carries export_rules, a digest of the export rules, ROW_SHAPE_VERSION and an HMAC under the hashing key, and every day holding an older digest reloads. A day that fails no longer blocks later days, the run stops starting days after a 20-minute budget, and its error names the failed, short, and unreached days.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
account_dailynamed an organisation once its domain held 3 accounts, counting today's, while the view covers days through yesterday. A domain could then appear by name with 2 accounts in the export. Migration0021_account_daily_before_todayrecreates the view so the threshold counts only accounts created before today (UTC), the accounts the export holds, and addsaccount_daily_sourcefor the export's fault checks.db/account-export.tsfails on an emptyusertable or an emptywebmail_domain, and replaces the BigQuery table with no rows when every account is from today, so a deleted account leaves the export.The event export (
db/event-export.ts) writes a NULLABLEexport_rulescolumn on every row: a digest ofPROP_RULES, the path rules,EXPORT_SCHEMA,ROW_SHAPE_VERSIONand an HMAC underEXPORT_ID_HMAC_KEY. Every day holding a row with another digest joins the run, so a rule change or a key rotation reaches every partition already loaded, and a day that fails is retried with the window on the next run. The first run after deploy reloads every day once. A day that fails keeps its partition while later days load, the run starts no new day after a 20-minute budget, and its error names the failed, short, and unreached days. Partition counts come fromINFORMATION_SCHEMA.PARTITIONS(daily partitions only), and day reads page on(created_at, id).SHARE_MODESand the prompt intents are checked against their source unions, and a REQUIRED field added toEXPORT_SCHEMAfails to compile.deploy/README.mdgives the bounds (about 360 bytes an event, 0.5 MB on the busiest day so far against 512 MiB, and a 13.5 million row ceiling under the 1 GB billing cap), anddb/connection.ts's comments describe the export Job's use of the runtime connection.Testing:
typecheck,lintand the db tests pass (155 against Postgres 18.4), and each new test fails without its change.IS DISTINCT FROMwith a named parameter and thepartition_idfilter were run on prod BigQuery.