Multi-Source ETL Framework

Reusable Python connector framework for ingesting data from REST APIs, Google Sheets, Airtable, and Postgres into BigQuery or Snowflake. Define a pipeline in YAML and run it — no custom code needed per source.

Supported Sources → Destinations

Source	Destination
REST API (any, with pagination)	BigQuery (append / replace / upsert)
Google Sheets	Snowflake
Airtable	Local Parquet
Postgres / MySQL

Quick Start

pip install -r requirements.txt
cp config/pipelines.yaml.example config/pipelines.yaml
# fill in credentials

python run.py --pipeline crm_contacts
python run.py --all          # run every pipeline in config

Pipeline Config

pipelines:
  - name: my_pipeline
    source:
      type: rest_api
      base_url: "https://api.example.com"
      endpoint: "/records"
      auth_type: bearer
      auth_value: "${API_TOKEN}"
      results_key: data
      page_size: 200
    destination:
      type: bigquery
      project: "${GCP_PROJECT}"
      dataset: raw_sources
      table: records
      mode: upsert       # append | replace | upsert
      merge_key: id
    schedule: "0 6 * * *"

Adding a New Connector

1. Create connectors/my_source.py subclassing BaseConnector
2. Implement name, test_connection(), and extract() (yields DataFrames)
3. Register it in connectors/__init__.py

Three methods, and your connector works with every destination and schedule in the framework.

Upwork Use Case

Fork this repo for each client. Fill in their credentials and pipeline config. The ingestion logic, error handling, retry, and BigQuery schema evolution are already done — you're delivering working pipelines in hours instead of days.