Skip to content

fix: restore scraping against current X - #1

Merged
mar0ls merged 2 commits into
mainfrom
dev
Sep 1, 2026
Merged

mar0ls merged 2 commits into
mainfrom
dev

Conversation

@mar0ls

@mar0ls mar0ls commented Sep 1, 2026 •

Copy link
Copy Markdown
Owner

Both scraper modes returned zero rows. Four causes, all fixed:

  • twscrape 0.17.0's parser no longer matched X's response shape → twscrape>=0.20.1
  • the local x-tid patch broke every request on 0.20 (create() signature mismatch) → it now defers to twscrape's own maintained generator and stays a fallback
  • the query-id refresh never worked — doubled backslashes made the bundle regex unmatchable
  • ClientTransaction was built from the x.com root, which no longer carries the ondemand manifest → uses the login shell instead

Adds menu option 6 to store/refresh cookies from inside the app. The released binaries don't ship setup_twscrape_account.py, so until now they had no way to configure an account at all. The environment check reports 5 stages and names the one that fails.

Also drops duplicated query helpers and dead code, and declares beautifulsoup4, which was only pulled in transitively. 2 new offline tests, 15 total, CI green.

Verified live: both modes return tweets and write JSON/CSV.

mar0ls added 2 commits July 25, 2026 22:23
Runs the eBPF supply-chain sensor as the first step of jobs that install
third-party deps or hold release secrets. Guarded with runner.os == 'Linux'
on cross-platform matrices; action pinned to v0.0.36 by SHA.
@mar0ls
mar0ls merged commit 388d82c into main Sep 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant