fix: add pg8000 graceful shutdown on Cloud Run scale-down#4436
Open
panish16 wants to merge 13 commits into
Open
fix: add pg8000 graceful shutdown on Cloud Run scale-down#4436panish16 wants to merge 13 commits into
panish16 wants to merge 13 commits into
Conversation
Calls setup_pg8000_close_event_listener(engine) after db init in all lear queue services and legal-api. Scoped to long-lived Cloud Run services only — GCP jobs excluded as no pg8000 errors observed in logs. Resolves #33564
panish16
force-pushed
the
fix/pg8000-graceful-shutdown
branch
from
227c2ba to
f48ae14
Compare
…utdown for legal-api
panish16
changed the title
… blocks Remove the DBConfig blocks from all 5 queue services (business-bn, business-emailer, business-pay, business-filer, business-digital-credentials). These services connect via SQLALCHEMY_DATABASE_URI in their configs and don't need Cloud SQL Connector connection pooling. The setup_pg8000_close_event_listener call is now unconditional since the function already returns early for non-pg8000 engines.
…listener DBConfig/SQLALCHEMY_ENGINE_OPTIONS setup was removed from __init__.py without being moved anywhere, breaking Cloud SQL connections on GCP. Move it into each service's config.py alongside the existing CLOUDSQL_INSTANCE_CONNECTION_NAME block. Replace legal-api's local _setup_pg8000_graceful_shutdown reimplementation with setup_pg8000_close_event_listener from cloud_sql_connector.
…s updated setup_pg8000_close_event_listener is not yet available on the cloud-sql-connector-python3.9 branch used by legal-api, so retain the local _setup_pg8000_graceful_shutdown implementation. DBConfig setup remains moved to config.py.
Contributor
Author
|
/retest |
…dentials so coverage.xml is generated and uploaded
panish16
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
connectevent listener on app startup that wraps each pg8000 connection'sclose()to suppresspg8000.exceptions.InterfaceErrorduring Cloud Run scale-downpg8000.exceptions.InterfaceError: network errorlogged when Cloud Run terminates idle instances and SQLAlchemy's connection pool teardown tries to close already-severed socketssetup_pg8000_close_event_listener(engine)(fromsbc-connect-common/cloud-sql-connector) in all affected long-running servicesServices fixed
legal-api/src/legal_api/__init__.pyqueue_services/business-filer/src/business_filer/__init__.pyqueue_services/business-emailer/src/business_emailer/__init__.pyqueue_services/business-pay/src/business_pay/__init__.pyqueue_services/business-bn/src/business_bn/__init__.pyqueue_services/business-digital-credentials/src/business_digital_credentials/__init__.pyTest plan
pg8000.exceptions.InterfaceErrorentries in GCP Log Explorer fora083gt-prodproject after deployment