Skip to content

README.md

Latest commit

 

History

History
134 lines (98 loc) · 3.78 KB

README.md

File metadata and controls

134 lines (98 loc) · 3.78 KB

Tech Trends — Technology Trend Analysis Platform

Project Overview

Challenge Solution Impact
Multi-source tech trends with consistent scoring Canonical ETL pipeline + Trend Score bridge Stable, explainable rankings
Fragile frontend assumptions Bridge-first JSON + CSV fallback Resilient UI under data changes
Weekly refresh without regressions CI gates + data contracts Predictable releases

Key Metrics (Latest Run)

Metric Value
Repos classified 931
StackOverflow questions (annual window) 33,356
Reddit mentions (emerging topics) 366
Technologies in ranking 22
Window 2025-03-13 → 2026-03-13 (UTC)

Pipeline Architecture

Data Pipeline Architecture

Tech Stack

Layer Technologies
Data Python, pandas, duckdb, pandera
ETL Custom pipelines + contracts
Frontend Flutter Web
Hosting GitHub Pages

Quick Start

# backend
pip install -r backend/requirements.txt
python -m pytest -q

# run ETLs
python backend/github_etl.py
python backend/stackoverflow_etl.py
python backend/reddit_etl.py
python backend/trend_score.py
python backend/sync_assets.py

# frontend
cd frontend
flutter pub get
flutter run -d chrome

Automation (GitHub Actions)

  1. ETL Weekly Refresh (etl_semanal.yml)
    • Schedule: Monday 08:00 UTC
    • Runs all ETLs, Trend Score, syncs assets, validates contracts, and publishes data.
  2. Dependency Security (dependency_security.yml)
    • Schedule: Monday 09:00 UTC
    • Runs pip-audit against backend/requirements.txt.
  3. Deploy Frontend (deploy_frontend.yml)
    • Publishes Flutter Web to GitHub Pages.

Project Structure

Technology-trend-analysis-platform/
├── backend/                       # ETLs + Trend Score + contracts
├── datos/                         # CSV outputs (legacy + latest + history)
├── frontend/                      # Flutter Web UI
├── docs/                          # Technical docs
├── scripts/                       # CI / assets validation helpers
└── .github/workflows/             # CI + ETL automation

Author

Samir Caizapasto
Junior Data Engineer & Analyst

If you find this project useful, please give the repository a star.