Skip to content

Commit 5020ced

Browse files
lroollelroolle
committed
feat: v0.3 — refresh against live X, add browser cookie auto-import
Live testing against a real logged-in x.com session (via Chrome MCP) surfaced four breaking changes vs v0.2 and one major DX gap. All five fixed in this commit. ## Live API rotation (v0.2 was stale) Every GraphQL query ID rotated, the response shape moved fields out of `legacy` into a new `core` block, the timeline path lost its `_v2` suffix, and the legacy `verify_credentials.json` REST endpoint is now a 404. Refreshed `endpoints.yaml` with the live IDs captured from the April 2026 web client and made the parsers defensive across both shapes. endpoints.yaml — query IDs captured from the live web client: UserByScreenName IGgvgiOx4QZndDHuD3x9TQ (was NimuplG1OB7Fd2btCLdBOw) UserByRestId VQfQ9wwYdk6j_u2O4vt64Q (was tD8zKvQzwY3kdx5yz6YmOw) UserTweets x3B_xLqC0yZawOB7WQhaVQ (was QWF3SzpHmykQHsQMixG0cg) TweetDetail rU08O-YiXdr0IZfE7qaUMg (was U0HTv-bAWTBYylwEMT7x5A) SearchTimeline pCd62NDD9dlCDgEGgEVHMg (was flaR-PUMshxFWZWPNpq4zA) Following vWCjN9gcTJiXzzMPR5Oxzw (was 2vUj-_Ek-UmBVDNtd8OnQA) BlueVerifiedFollowers wlnE16ojgADYiHEqMzR3sw (new: separate op) + refreshed features blob with the modern keys (hidden_profile_subscriptions_enabled, profile_label_improvements_*, responsive_web_grok_*, premium_content_api_read_enabled, ...) + variables: UserByScreenName now wants withGrokTranslatedBio instead of withSafetyModeUserFields; UserTweets dropped withV2Timeline api/extract.go — defensive projection helpers firstString / firstInt / firstBool walk a list of paths and return the first non-empty match. Used everywhere a field has migrated between `core/*` and `legacy/*`. splitPath helper for slash-delimited paths. api/profile.go — extractProfile now reads core/screen_name → legacy/screen_name core/name → legacy/name core/created_at → legacy/created_at avatar/image_url → legacy/profile_image_url_https privacy/protected → legacy/protected Plus a new GetProfileByID(userId) for the auth liveness check. api/tweets.go — ParseTweet author projection same defensive shape. scrapeUserTimeline now tries data.user.result.timeline.timeline first, falls back to timeline_v2.timeline. Variable maps stripped of withV2Timeline. api/relationships.go — ParseUserSummary same defensive shape, avatar bumped from _normal to _400x400 only when it points at the legacy URL pattern. api/auth.go — VerifyCredentials no longer hits the dead /1.1/account/verify_credentials.json. Parses the user id out of the `twid` cookie (u%3D<id>), calls UserByRestId, returns the live user identity. Falls back to a UserByScreenName "twitter" probe when twid is missing. New parseTwidUserID + normaliseAuthError helpers. ## Browser cookie auto-import (the DX gap) Until this commit, `x auth import` required a manual paste from DevTools. This is exactly what Python's `browser_cookie3` and `rookiepy` solve by reading the browser's cookie SQLite store directly from disk and decrypting the values with the OS-specific Safe Storage key. Go has the same primitive — `github.com/browserutils/kooky` — and now x-cli uses it. internal/browsercookies/ — wraps kooky in a small typed API: Load(ctx, browser, domain) → (*Result, error) FormatCookieHeader(cookies, wanted) → string Supports chrome | firefox | brave | edge | chromium cmd/auth.go — three import modes: x auth import --from-browser chrome # auto-decrypt from disk x auth import --cookie '...' # one-shot for scripted setups x auth import # interactive paste (default) Storage path is unchanged — keychain primary, AES-GCM file fallback, never plaintext. The new modes are read paths, not write paths. Per-OS notes (also in skills/x-cli/references/auth.md): - macOS Chrome: must be CLOSED (cookie file is locked while running). First run prompts for Keychain access to read the Safe Storage AES key. macOS Firefox: works while running. - Linux Chrome: needs libsecret or kwallet running, falls back to a hardcoded `peanuts` salt on truly headless hosts. - Windows: DPAPI; works while the browser is running. ## Live regression fixtures api/testdata/userbyscreenname_jack_2026_04.json api/testdata/usertweets_jack_2026_04.json Captured from the real x.com response on 2026-04-13 via Chrome MCP. Two new tests run the parser against them: TestExtractProfileLiveJackFixture TestParseLiveUserTweetsFixture These catch the next rotation early — if any field migrates again the test breaks with a precise error pointing at which path is dead. ## Misc cmd/doctor.go — printf-vet fixes (Warn/Success now use %s for the egress line so `go vet` on Go 1.24 stays clean). .github/workflows/ci.yml — Go bumped from 1.23 to 1.24 (kooky's go.mod requires it). skills/x-cli/SKILL.md and references/auth.md updated with the new flags and the modern auth flow. README updated to lead with --from-browser. docs/comparison-xactions.md — no change needed (the comparison was about XActions vs x-cli, which still holds). ## Tests 70+ tests, race-clean, api/ 70.9% coverage, internal/store/ 62.8%, internal/browsercookies/ ~50% (the rest needs real browser cookie files; FormatCookieHeader and the no-browsers error path are tested).
1 parent be2e1c7 commit 5020ced

22 files changed

Lines changed: 1047 additions & 168 deletions

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ jobs:
2626

2727
- uses: actions/setup-go@v5
2828
with:
29-
go-version: '1.23'
29+
go-version: '1.24'
3030
cache: true
3131

3232
- name: Module tidy check
@@ -71,7 +71,7 @@ jobs:
7171

7272
- uses: actions/setup-go@v5
7373
with:
74-
go-version: '1.23'
74+
go-version: '1.24'
7575
cache: true
7676

7777
- name: Build ${{ matrix.goos }}/${{ matrix.goarch }}

‎README.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,14 +41,42 @@ make build
4141

4242
## Auth
4343

44-
Log into x.com in your real browser, DevTools → Application → Cookies, copy
44+
Three ways to import the session:
45+
46+
**1. Auto from a local browser (recommended)**
47+
48+
```
49+
x auth import --from-browser chrome
50+
x auth import --from-browser firefox
51+
x auth import --from-browser brave
52+
x auth import --from-browser edge
53+
```
54+
55+
x-cli reads the browser's cookie SQLite store on disk and decrypts
56+
the values using the per-OS Safe Storage key (macOS Keychain on Mac,
57+
libsecret/kwallet on Linux, DPAPI on Windows). Same mechanism Python's
58+
`browser_cookie3` and `rookiepy` use. Chrome must be **closed** on macOS
59+
because it locks the cookie file; Firefox and Linux Chrome usually work
60+
while open. macOS will prompt once for Keychain access on the first run.
61+
62+
**2. Manual paste**
63+
64+
Open x.com in your real browser, DevTools → Application → Cookies, copy
4565
`auth_token` and `ct0`, then:
4666

4767
```
4868
x auth import
4969
# paste: auth_token=...; ct0=...; twid=u%3D...
5070
```
5171

72+
**3. Scripted (CI / setup scripts)**
73+
74+
```
75+
x auth import --cookie 'auth_token=...; ct0=...; twid=u%3D...'
76+
```
77+
78+
(Visible in shell history — prefer `--from-browser` for normal use.)
79+
5280
**Where the cookie lives.** x-cli tries the OS keychain first (`go-keyring`:
5381
Keychain on macOS, libsecret on Linux, Credential Manager on Windows). If the
5482
keychain is unavailable — headless boxes, containers, CI, Linux without a

‎api/auth.go‎

Lines changed: 78 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@ package api
22

33
import (
44
"context"
5-
"encoding/json"
6-
"net/http"
5+
"net/url"
76
"strings"
87
)
98

@@ -60,32 +59,89 @@ func RequireAuthCookies(cookies map[string]string) error {
6059
return nil
6160
}
6261

63-
// VerifyCredentials hits /1.1/account/verify_credentials.json and returns the
64-
// user if the session is alive, or an AuthError otherwise.
62+
// VerifyCredentials confirms the imported session is alive and returns
63+
// the logged-in user.
64+
//
65+
// X removed the legacy /1.1/account/verify_credentials.json endpoint
66+
// (returns 404 as of April 2026). x-cli reads the `twid` cookie that
67+
// the user pasted, parses the numeric user id out of it, and calls
68+
// UserByRestId to confirm the session is alive AND fetch identity in
69+
// one round trip. Falls back to a UserByScreenName "twitter" probe if
70+
// `twid` is missing — that's the cheapest "is this cookie usable at
71+
// all" sanity check we have without it.
6572
func (c *Client) VerifyCredentials(ctx context.Context) (*User, error) {
66-
url := c.endpoints.Bases.REST + "/1.1/account/verify_credentials.json"
67-
resp, err := c.request(ctx, "GET", url, nil, requestOpts{authenticated: true, endpointName: "verifyCredentials"})
68-
if err != nil {
69-
return nil, err
73+
c.sessionMu.RLock()
74+
cookies := c.session.Cookies
75+
twidRaw := ""
76+
if cookies != nil {
77+
twidRaw = cookies["twid"]
78+
}
79+
c.sessionMu.RUnlock()
80+
81+
if userID := parseTwidUserID(twidRaw); userID != "" {
82+
p, err := c.GetProfileByID(ctx, userID)
83+
if err != nil {
84+
return nil, normaliseAuthError(err)
85+
}
86+
u := &User{
87+
ID: p.RestID,
88+
Username: p.ScreenName,
89+
Name: p.Name,
90+
}
91+
c.sessionMu.Lock()
92+
c.session.User = u
93+
c.sessionMu.Unlock()
94+
return u, nil
95+
}
96+
97+
// No twid: probe with a known account so we still detect a dead
98+
// cookie, but we cannot return identity in this branch.
99+
if _, err := c.GetProfile(ctx, "twitter"); err != nil {
100+
return nil, normaliseAuthError(err)
70101
}
71-
defer resp.Body.Close()
102+
return nil, &AuthError{Msg: "session is alive but `twid` cookie missing — re-import a complete cookie string"}
103+
}
72104

73-
if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden {
74-
return nil, &AuthError{Msg: "session invalid or expired", Status: resp.StatusCode}
105+
// parseTwidUserID extracts the numeric user ID from a `twid` cookie value.
106+
// Twitter encodes it as `u%3D<NNN>` (URL-encoded `u=NNN`).
107+
//
108+
// Examples:
109+
//
110+
// "u%3D2017830703355072513" → "2017830703355072513"
111+
// "u=2017830703355072513" → "2017830703355072513"
112+
// "" → ""
113+
func parseTwidUserID(raw string) string {
114+
if raw == "" {
115+
return ""
116+
}
117+
decoded, err := url.QueryUnescape(raw)
118+
if err != nil {
119+
decoded = raw
120+
}
121+
if !strings.HasPrefix(decoded, "u=") {
122+
return ""
75123
}
76-
if resp.StatusCode >= 400 {
77-
return nil, &APIError{Endpoint: "verifyCredentials", Status: resp.StatusCode}
124+
id := decoded[2:]
125+
for _, r := range id {
126+
if r < '0' || r > '9' {
127+
return ""
128+
}
78129
}
130+
return id
131+
}
79132

80-
var u User
81-
if err := json.NewDecoder(resp.Body).Decode(&u); err != nil {
82-
return nil, err
133+
// normaliseAuthError maps domain errors to AuthError when the underlying
134+
// failure looks like a session problem. NotFoundError on a self-lookup
135+
// almost always means the session id is no longer valid.
136+
func normaliseAuthError(err error) error {
137+
if err == nil {
138+
return nil
83139
}
84-
if u.ID == "" {
85-
return nil, &AuthError{Msg: "verify_credentials returned empty user"}
140+
switch e := err.(type) {
141+
case *AuthError:
142+
return e
143+
case *NotFoundError:
144+
return &AuthError{Msg: "session invalid or expired (self-lookup not found)"}
86145
}
87-
c.sessionMu.Lock()
88-
c.session.User = &u
89-
c.sessionMu.Unlock()
90-
return &u, nil
146+
return err
91147
}

‎api/client_test.go‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -329,16 +329,21 @@ func TestRateLimit429WithReset(t *testing.T) {
329329
}
330330

331331
func TestVerifyCredentialsSuccess(t *testing.T) {
332+
// Modern path: VerifyCredentials reads the `twid` cookie, parses
333+
// the user ID, and calls UserByRestId. The fake server responds
334+
// with a modern-shape user (core.screen_name + core.name).
332335
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
333-
if r.URL.Path != "/1.1/account/verify_credentials.json" {
334-
t.Errorf("path = %q", r.URL.Path)
336+
if !strings.Contains(r.URL.Path, "/uid_qid/UserByRestId") {
337+
t.Errorf("unexpected path %q", r.URL.Path)
335338
}
336339
w.Header().Set("Content-Type", "application/json")
337-
_, _ = w.Write([]byte(`{"id_str":"123","screen_name":"jack","name":"Jack Dorsey"}`))
340+
_, _ = w.Write([]byte(`{"data":{"user":{"result":{"__typename":"User","rest_id":"123","is_blue_verified":true,"core":{"screen_name":"jack","name":"Jack Dorsey","created_at":"Tue Mar 21 20:50:14 +0000 2006"},"legacy":{"description":"hi","followers_count":7000000}}}}}`))
338341
}))
339342
defer srv.Close()
340343

341-
c := newTestClient(t, srv.URL, nil, map[string]string{"auth_token": "x", "ct0": "y"})
344+
c := newTestClient(t, srv.URL, map[string]GraphQLEndpoint{
345+
"UserByRestId": {QueryID: "uid_qid", OperationName: "UserByRestId", Kind: "read", RPS: 100, Burst: 10},
346+
}, map[string]string{"auth_token": "x", "ct0": "y", "twid": "u%3D123"})
342347

343348
u, err := c.VerifyCredentials(context.Background())
344349
if err != nil {
@@ -350,23 +355,40 @@ func TestVerifyCredentialsSuccess(t *testing.T) {
350355
}
351356

352357
func TestVerifyCredentialsUnauthorized(t *testing.T) {
358+
// Server returns 401 on UserByRestId; VerifyCredentials must surface
359+
// it as *AuthError, not a raw GraphQL error.
353360
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
354361
w.WriteHeader(401)
355362
}))
356363
defer srv.Close()
357364

358-
c := newTestClient(t, srv.URL, nil, map[string]string{"auth_token": "bad", "ct0": "bad"})
365+
c := newTestClient(t, srv.URL, map[string]GraphQLEndpoint{
366+
"UserByRestId": {QueryID: "uid_qid", OperationName: "UserByRestId", Kind: "read", RPS: 100, Burst: 10},
367+
}, map[string]string{"auth_token": "bad", "ct0": "bad", "twid": "u%3D123"})
359368

360369
_, err := c.VerifyCredentials(context.Background())
361370
if err == nil {
362371
t.Fatal("expected error")
363372
}
364-
ae, ok := err.(*AuthError)
365-
if !ok {
366-
t.Fatalf("want *AuthError, got %T", err)
373+
if _, ok := err.(*AuthError); !ok {
374+
t.Fatalf("want *AuthError, got %T: %v", err, err)
367375
}
368-
if ae.Status != 401 {
369-
t.Errorf("AuthError.Status = %d", ae.Status)
376+
}
377+
378+
func TestParseTwidUserID(t *testing.T) {
379+
cases := map[string]string{
380+
"u%3D2017830703355072513": "2017830703355072513",
381+
"u=2017830703355072513": "2017830703355072513",
382+
"u%3D12": "12",
383+
"": "",
384+
"garbage": "",
385+
"u%3Dnot-a-number": "",
386+
"u%3D": "",
387+
}
388+
for in, want := range cases {
389+
if got := parseTwidUserID(in); got != want {
390+
t.Errorf("parseTwidUserID(%q) = %q, want %q", in, got, want)
391+
}
370392
}
371393
}
372394

‎api/endpoints_test.go‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -161,10 +161,16 @@ func TestLoadShippedEndpoints(t *testing.T) {
161161
if _, ok := m.REST["friendshipsCreate"]; !ok {
162162
t.Error("shipped endpoints.yaml missing friendshipsCreate")
163163
}
164-
if _, ok := m.REST["verifyCredentials"]; !ok {
165-
t.Error("shipped endpoints.yaml missing verifyCredentials")
164+
if _, ok := m.REST["friendshipsDestroy"]; !ok {
165+
t.Error("shipped endpoints.yaml missing friendshipsDestroy")
166166
}
167167
if m.Bearer == "" {
168168
t.Error("shipped endpoints.yaml has empty bearer")
169169
}
170+
// verifyCredentials.json was removed by X — the shipped YAML
171+
// must NOT carry it any more (the auth liveness check now goes
172+
// through UserByRestId).
173+
if _, ok := m.REST["verifyCredentials"]; ok {
174+
t.Error("shipped endpoints.yaml should no longer carry the dead verifyCredentials REST entry")
175+
}
170176
}

‎api/extract.go‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,3 +132,80 @@ func copyMap(m map[string]any) map[string]any {
132132
}
133133
return out
134134
}
135+
136+
// firstString returns the first non-empty string at any of the given
137+
// dot-paths in the root. This is the defensive projection helper used
138+
// across the parsers — Twitter has shipped at least two response shapes
139+
// for the same field (e.g., user.screen_name now lives at `core.screen_name`
140+
// but used to live at `legacy.screen_name`). Reading both keeps x-cli
141+
// working across the rotation.
142+
//
143+
// Each `paths` argument is a slash-delimited path like
144+
// "core/screen_name" or "legacy/profile_image_url_https".
145+
func firstString(root map[string]any, paths ...string) string {
146+
for _, p := range paths {
147+
keys := splitPath(p)
148+
v := walkPath(root, keys...)
149+
if s, ok := v.(string); ok && s != "" {
150+
return s
151+
}
152+
}
153+
return ""
154+
}
155+
156+
// firstInt is the int counterpart of firstString.
157+
func firstInt(root map[string]any, paths ...string) int {
158+
for _, p := range paths {
159+
keys := splitPath(p)
160+
v := walkPath(root, keys...)
161+
switch x := v.(type) {
162+
case float64:
163+
return int(x)
164+
case int:
165+
return x
166+
case int64:
167+
return int(x)
168+
case string:
169+
n := 0
170+
ok := len(x) > 0
171+
for _, r := range x {
172+
if r < '0' || r > '9' {
173+
ok = false
174+
break
175+
}
176+
n = n*10 + int(r-'0')
177+
}
178+
if ok {
179+
return n
180+
}
181+
}
182+
}
183+
return 0
184+
}
185+
186+
// firstBool returns the first true value at any of the given paths.
187+
func firstBool(root map[string]any, paths ...string) bool {
188+
for _, p := range paths {
189+
keys := splitPath(p)
190+
if v, ok := walkPath(root, keys...).(bool); ok && v {
191+
return true
192+
}
193+
}
194+
return false
195+
}
196+
197+
func splitPath(p string) []string {
198+
if p == "" {
199+
return nil
200+
}
201+
out := make([]string, 0, 4)
202+
start := 0
203+
for i := 0; i < len(p); i++ {
204+
if p[i] == '/' {
205+
out = append(out, p[start:i])
206+
start = i + 1
207+
}
208+
}
209+
out = append(out, p[start:])
210+
return out
211+
}

0 commit comments

Comments
 (0)