Skip to content

feat(google_finance): add Google Finance mirror (site 17, port 40016)#73

Open
QiushiSun wants to merge 2 commits into
aiming-lab:mainfrom
QiushiSun:feat/google-finance
Open

feat(google_finance): add Google Finance mirror (site 17, port 40016)#73
QiushiSun wants to merge 2 commits into
aiming-lab:mainfrom
QiushiSun:feat/google-finance

Conversation

@QiushiSun

@QiushiSun QiushiSun commented Jul 26, 2026

Copy link
Copy Markdown

Mirrors https://www.google.com/finance/ as site 17 on port 40016.

Coverage

Home page with seven region tabs (US / Europe / Asia / Latin America /
Currencies / Crypto / Futures); quote pages for stocks, ETFs, indexes, sector
indexes, crypto, FX and futures across eight chart ranges and
Overview / Analysis / Earnings / Financials / Holdings tabs; eight market
boards (most active, gainers, losers, indexes, cryptocurrencies, currencies,
futures, climate leaders); scored token-overlap search with autocomplete; a
news index and article pages; a currency converter; a compare view; and
auth-gated watchlists and portfolios with live gain/loss maths. Sign-in works
through both an email/password form and an accounts.google.com-style profile
picker.

Charts are rendered server-side as SVG — gradient area fill, volume band,
after-hours leg behind a session divider, previous-close line, key-moment
markers, 50/200-day moving averages, and a hover crosshair. JavaScript is
enhancement only; the toolbar menus are <details>/<summary> so they work
with JS off.

Seeded rows — 10,275 total

Model Rows Model Rows
Instrument 182 KeyMoment 2,034
Quote 182 InstitutionalHolder 1,141
PriceHistory 182 NewsArticle 577
FinancialRow 2,938 NewsLink 557
EarningsRow 904 Publisher 41
AnalystRating 1,324 MarketSummary 12
EtfHolding 140 WatchlistItem 27
PortfolioLot 20 User / Watchlist / Portfolio 4 / 5 / 5

The 182 instruments are 113 stocks across 11 GICS sectors (≥6 each), 12 ETFs,
20 indexes (five each for US / Europe / Asia / Latin America, matching the live
home-page tabs), 11 sector indexes, 10 cryptocurrencies, 10 FX pairs and 6
futures. All four benchmark accounts carry pre-existing auth-gated state, and
every portfolio holds at least one losing position so "which holding is down"
phrasings are answerable.

Data policy

Company identities, logos, About blocks and profile prose are real,
harvested from the live site. Every market number — price, OHLC series, key
statistics, financial statements, earnings, analyst targets — is synthetic,
generated from a fixed per-symbol seed and frozen at MARKET_DATE = 2026-07-24,
with magnitudes anchored to a real open price and share count so each
instrument lands in a believable range.

Two benchmark-driven reasons: scraped quotes are stale the day they are
captured, and a frontier model can approximately recall real fundamentals,
which is the "trivially answerable from prior knowledge" shortcut the guide
rejects. Every task therefore targets synthetic values only.

Assets

125 real company logos (inline SVG), 41 real publisher favicons, and Google
Finance's own favicon set — 4.3 MB packed. Google Finance ships almost no
bitmap imagery; its visual identity is typography, colour and charts, so
fidelity here comes from the SVG chart engine rather than photographs.

Tasks

20 in sites/google_finance/tasks.jsonl, contributor keys only
(web_name, id, ques, web, upstream_url) — no answer, verifier_path or
judge_rubric. Every answer is a frozen synthetic value that has to be read
off a detail page.

Coverage spans the home region tabs, quote pages for stocks / ETFs / indexes /
crypto, all five quote tabs, the chart range switcher and key-moment markers,
two market boards, scored search, news, the currency converter, the compare
view, the Google account chooser, auth-gated watchlist and portfolio reads, and
one portfolio write (create + add position).

Audited with four Playwright passes: every answer is present on its target page
and absent from every upstream page; every answer page is reachable by clicking
only; every UI label and control a task's wording names actually exists; and
every "which one is highest/lowest" task has a genuine runner-up with no ties
(closest margin 10.4%).

One note on the hardening guide's ≤50% full-match density target:
/search?q=utility returns 8 rows of which 7 are Utilities-sector stocks. The
discrimination in that task is the dividend-yield comparison across the
matches, not the sector filter — the two non-Utilities results (a sector index
and a Consumer Staples name) are the near-misses, and the QA suite asserts the
yield winner beats the runner-up by between 1% and 60%.

Two deliberate deviations from the current upstream UI

  1. No Gemini "Research" rail. google.com/finance now redirects to
    /finance/beta, whose right column is an LLM chat. Mirroring it would need
    a live model at request time, which breaks the determinism the project rests
    on; a dead textbox is an explicit reject reason. The rail is kept and filled
    with deterministic content instead: market movers and top news on most
    pages, related assets and sector context on a quote page.

  2. A real /search?q= results page. In the beta UI the top search bar
    routes into that same AI panel. WebHarbor requires scored token-overlap
    search and lists "non-functional search" as a reject reason, so the mirror
    implements the classic Google Finance behaviour: an autocomplete dropdown
    plus a results page over instruments and news.

Shared-file changes

The three registration points (websyn_start.sh SITES, control_server.py
SITES, Dockerfile EXPOSE 40000-40016), plus one dependency pin:
MarkupSafe==3.0.3. sites/google_finance/app.py is the only site module that
imports markupsafe directly (for the server-rendered SVG). pip already pulls
it in transitively via Jinja2, but AGENTS.md asks for locked versions, and the
Dockerfile already pins Flask's other transitive deps (Werkzeug, Jinja2) the
same way.

Determinism

The market clock is frozen at 2026-07-24; nothing calls date.today(). All
three seed_*() functions gate at function level.

Pre-PR checks

Run against webharbor:dev built from this branch (5.95 GB image), container
on 8201:8101 and 41000-41016:40000-40016:

  • python3 -m py_compile sites/google_finance/app.py — clean
  • GET /healthok: true, 17/17 sites alive
  • all seventeen ports 4100041016 return 200
  • byte-identical reset, run against a deliberately dirtied database
    (a row was inserted first, so the md5 genuinely diverged to
    4eef20b25fa17283382abce7d2a5e28b before the reset):
POST /reset/google_finance
{"pid":177,"ready":true,"site":"google_finance"}

7520ab730ece421bf95b53755b6d8e7f  /opt/WebSyn/google_finance/instance/google_finance.db
7520ab730ece421bf95b53755b6d8e7f  /opt/WebSyn/google_finance/instance_seed/google_finance.db

Assets pin

.assets-revision is pinned to refs/pr/47 — the paired HF PR — following the
same convention as #53 and #61, so this branch builds end-to-end before the
assets PR is merged. That ref is current main plus this one tarball, so all 17
sites are present. Maintainer: once HF PR #47 is merged, bump the pin to the
merged commit SHA on main; the file carries that note inline.

Verified from a clean git clone of this branch, not from my working tree:

./scripts/fetch_assets.sh          -> 17 site(s) extracted, none missing a seed DB
./scripts/build.sh                 -> webharbor:dev, 5.95 GB
docker run ... 41000-41016         -> /health ok:true, 17/17 alive
                                      all seventeen ports 200
POST /reset/google_finance         -> {"pid":177,"ready":true,"site":"google_finance"}
                                      md5 identical after dirtying the DB first

This matters more than it looks: websyn_start.sh copies instance_seed to
instance for every site under set -e, so a missing directory aborts the
whole entrypoint — without the pin no site comes up, not just this one.

Port slot

The site is registered at index 16 / port 40016, appended after
merriam_webster. Several open PRs are queued on that same slot; happy to
rebase onto whichever lands first — the port appears only in the three
registration points and in tasks.jsonl's web field.

QiushiSun and others added 2 commits July 25, 2026 20:28
Mirrors https://www.google.com/finance/ as site 17 on port 40016.

Registers the site in the three required places (websyn_start.sh SITES,
control_server.py SITES, Dockerfile EXPOSE 40000-40016) and pins
MarkupSafe==3.0.3, which app.py imports directly for its server-rendered
SVG charts.

Covers the home page with seven region tabs, quote pages for stocks, ETFs,
indexes, sector indexes, crypto, FX and futures across eight chart ranges
and five tabs, eight market boards, scored search with autocomplete, news,
a currency converter, a compare view, and auth-gated watchlists and
portfolios. 10,275 seeded rows across 18 models; market data is synthetic
and frozen at 2026-07-24, company identities and logos are real.

20 benchmark tasks in sites/google_finance/tasks.jsonl, contributor keys
only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetch_assets.sh otherwise resolves a revision that predates the paired
assets PR, and websyn_start.sh copies instance_seed to instance for every
site under set -e — so a missing directory aborts the whole entrypoint and
no site comes up, not just this one.

Verified from a clean clone of this branch: fetch_assets.sh, build.sh, and
a container on 41000-41016 give /health ok with 17/17 alive and every port
200, with the seed DB byte-identical after POST /reset/google_finance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@QiushiSun

QiushiSun commented Jul 26, 2026

Copy link
Copy Markdown
Author

Screenshot evidence of visual fidelity vs. the real site, real web

Real Homepage
1a_real_home
Mirror Homepage
1b_mirror_home
Real APPL
2a_real_quote_aapl
Mirror APPL
2b_mirror_quote_aapl
Real APPL Financials
3a_real_aapl_financials
Mirror APPL Financials
3b_mirror_aapl_financials

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant