Repository navigation
fix(export): bound every export call and order current days first - #63
Merged
Merged
Conversation
Every Postgres and BigQuery call in the event and account exports has a client-side bound, and a Postgres client that times out is destroyed. A 28-minute watchdog in runAndExit logs the phase in flight, the days not started and the faults so far, and exits 1 before the Job's 1800 s task timeout; every other failure is logged once with its phase. The window days run before the stale backlog. Day counts come from one exact query over the table, readDay stops on a short page, and the export_rules digest covers the readDay SQL, so a change to the exported row shape reloads history. A golden test pins exported rows to ROW_SHAPE_VERSION. The request log classifies the target at request start, adds no listener for static assets, and logs a placeholder path for any target other than an origin-form path, an http(s) URL or "*".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the faults a review of release #62 found in the event export, the account export and the request log.
Export Job (
db/event-export.ts,db/account-export.ts,db/bounded.ts,db/export-events.ts). Every Postgres and BigQuery call has a client-side bound, listed indeploy/README.md, and a Postgres client that times out is destroyed, never returned to the pool. A 28-minute watchdog inrunAndExitlogs the phase in flight, the days not started and the failed or short days so far, then exits 1, two minutes before the Cloud Run task timeout; any other failure is logged once with its phase. Yesterday and today export before the stale backlog, so a run the watchdog stops leaves only old days for the next hourly run. Day counts come from one exactCOUNT(*)over the table, replacingINFORMATION_SCHEMA.PARTITIONS, and feed both the never-shrink check and the stale days.readDaystops on a short page. Theexport_rulesdigest covers thereadDaySQL, so the first run after deploy reloads every day once. A golden test pins exported rows toROW_SHAPE_VERSION.Request log (
src/lib/proxy/requestLog.ts). The target is classified at request start, and a static asset gets no listener and no caller HMAC.pathis the arrived target for an origin-form path, the pathname of an http(s) URL, or*; any other target logs/:unparsable, so no userinfo, host or IP reaches the line.docs/analytics.mddescribes the rules.Testing: 1,884 tests pass against Postgres 18.4, including hang tests for a page read, the day count, the account read, a load and its cancel, and the watchdog. tsc and eslint are clean. Nothing ran against real BigQuery or Cloud SQL.