feat(transport): chromedp auto-bootstraps ct0/twid + per-op features

Live e2e against real X session surfaced multiple stacked issues. This
commit lands the auth+transport layer that finally makes the read path
work end to end (profile, tweets list, following, followers, search
were tested live; see WORKS / KNOWN-FAILS at the bottom).

## Browser auth bootstrap (XActions parity)

User pasted only auth_token. Previously x-cli refused to import without
ct0+twid. The chromebrowser transport now mints them itself:

internal/chromebrowser/browser.go
  Browser.Cookies(ctx, domain) — reads the in-memory Chrome cookie
    jar via CDP Network.GetCookies and returns a flat name→value map.
  Fetch JS — reads ct0 from document.cookie at request time and
    injects it as x-csrf-token. The browser populates ct0 itself
    via Set-Cookie on the initial navigate, so the user never has
    to paste a CSRF token. This matches XActions' Puppeteer flow.
  Navigation target switched from /robots.txt to /i/release_notes
    so the navigate triggers the session bootstrap that actually
    issues twid+gt+personalization_id alongside ct0. /robots.txt was
    too lightweight to populate twid.

internal/chromebrowser/transport.go
  RoundTrip reads the browser cookie jar AFTER each Fetch and
  surfaces all cookies as Set-Cookie headers on the synthetic
  http.Response. The api.Client's existing mergeSetCookies path
  folds them into the in-memory Session — same code path as a
  real ct0 rotation. Net effect: VerifyCredentials runs a probe
  call, the response carries Set-Cookies for ct0+twid+gt+...,
  the merge populates session.Cookies, the probe-then-retry path
  in VerifyCredentials re-reads twid and resolves the user.

  X_CLI_BROWSER_DEBUG=1 prints the cookie jar names after each
  fetch (no values, just names) for diagnosing rotation issues.

api/auth.go
  RequireAuthCookies (compat) → calls RequireAuthCookiesFor(true).
  RequireAuthCookiesFor(cookies, needCt0) — explicit form. The
    browser path passes needCt0=false because chromebrowser will
    fetch one itself. The http path still passes needCt0=true.
  VerifyCredentials restructured into two functions:
    verifyByTwid     — the happy path when twid is present
    VerifyCredentials — orchestrates "no twid → probe → retry"
  errMissingTwid sentinel + isMissingTwid for the retry dispatch.

cmd/auth.go
  acquireCookieString / runAuthImport now treat ct0 as optional in
  the browser path. Tighter timeout on the verify call (60s, accounts
  for first-call Chrome startup ~1-2s).

cmd/doctor.go
  verifyLiveSession honors the global useHTTP flag — was hardcoding
  the http+utls path which hits Cloudflare from inside.

## Per-operation features blob (the stale-features bug)

Different X GraphQL ops require different features blobs. Sending the
union-of-all-keys causes the gateway to 404 unknown features for
specific ops. Each op now carries its own override.

api/endpoints.go
  GraphQLEndpoint adds Features map[string]bool. When non-nil it
  REPLACES the global features for that op. Documentation comment
  explains the rotation-recovery workflow.

api/client.go GraphQL
  Looks up ep.Features first, falls back to c.endpoints.Features.

endpoints.yaml
  Global features now matches the timeline operations' set
  (Followers/Following/SearchTimeline/UserTweets all use the same
  ~36-key blob). Profile-specific keys (hidden_profile_subscriptions,
  subscriptions_verification_info_*, highlights_tweets_tab_ui_enabled,
  responsive_web_twitter_article_notes_tab_enabled,
  subscriptions_feature_can_gift_premium) moved to
  UserByScreenName.features as an override.

  Followers query ID refreshed: `-WcGoRt8IQuPm-l1ymgy6g`
  (captured live; gC_lyAxZOptAMLCJX5UhWw is dead).

## Variable shape fixes

api/relationships.go Followers, Following
  + withGrokTranslatedBio: true (required by current schema)

api/search.go SearchPosts, SearchUsers
  + withGrokTranslatedBio: false

api/tweets.go UserTweets
  includePromotedContent: false → true (matches live web client;
  schema rejection on false)

## Diagnostics

api/client.go logRequest
  X_CLI_FULL_URL=1 disables the `?…` truncation so the full GraphQL
  URL with variables and features is logged. Used during this
  session to compare our URLs against captured live URLs.

api/client.go GraphQL/REST
  401/403/404 paths read up to 400 bytes of the response body and
  embed it in the error message via previewBody. Surfaced the
  Cloudflare HTML challenge in the v0.4→v0.5 transition.

## Live e2e results (Eric Wang's session)

WORKS:
  ✓ x auth import --cookie 'auth_token=…'  (no ct0 needed)
  ✓ x doctor                                (verify, ASN, TLS)
  ✓ x profile get jack                      (real data, all fields)
  ✓ x tweets list ericwang42 -n 10          (with new render: ↳, →q,
                                             📷N, RT @author full body)
  ✓ x following jack -n 5                   (Stella_Assange, Snowden,
                                             elonmusk — real data)

KNOWN FAILS (next commit):
  ✗ x followers <user>      → 404 empty body
  ✗ x search posts <query>  → 404 empty body

Root cause identified via CDP header capture: the live web client
sends `x-client-transaction-id` on every request. UserTweets and
Following accept its absence; Followers and SearchTimeline 404 when
it's missing. The transaction ID is computed by an obfuscated
function in X's main JS bundle. Two fixes are possible:

  (a) Reverse-engineer the algorithm and reproduce it in Go. The
      function uses HMAC over URL path + per-session animation hash
      + timestamp. Open-source ports exist (twikit, twitter-api-go).
  (b) Hijack the SPA's fetch wrapper from inside the page. Navigate
      to /jack/followers, let the SPA bootstrap, replace
      window.fetch with a recording wrapper, capture the next
      Followers call's headers including the JS-computed transaction
      ID, then run our own fetch with the captured headers.

(b) is more reliable across rotations; (a) is more performant. Next
commit picks one. Probably (b) for v0.7.

Author: lroolle

api/auth.go

@@ -2,6 +2,7 @@ package api
 
 import (
 	"context"
+	"fmt"
 	"net/url"
 	"strings"
 )
@@ -48,12 +49,28 @@ func ParseCookieString(s string) map[string]string {
 	return out
 }
 
-// RequireAuthCookies returns an AuthError if the required cookies are absent.
+// RequireAuthCookies returns an AuthError if the required cookies are
+// absent. `auth_token` is mandatory always. `ct0` is mandatory ONLY for
+// the http+utls transport — when the browser transport is in use, the
+// browser fetches a fresh ct0 from x.com on the first navigation, so
+// the caller can pass auth_token alone and let chromebrowser do the
+// rest.
+//
+// The caller signals which transport is in play via the boolean.
+// `false` (browser path) accepts an empty ct0; `true` (http path)
+// requires it.
 func RequireAuthCookies(cookies map[string]string) error {
+	return RequireAuthCookiesFor(cookies, true)
+}
+
+// RequireAuthCookiesFor is the explicit form. Pass needCt0 = false if
+// the caller will go through the browser transport (which can mint
+// ct0 on the fly via Set-Cookie from x.com).
+func RequireAuthCookiesFor(cookies map[string]string, needCt0 bool) error {
 	if cookies["auth_token"] == "" {
 		return &AuthError{Msg: "missing auth_token cookie"}
 	}
-	if cookies["ct0"] == "" {
+	if needCt0 && cookies["ct0"] == "" {
 		return &AuthError{Msg: "missing ct0 (CSRF) cookie"}
 	}
 	return nil
@@ -66,42 +83,77 @@ func RequireAuthCookies(cookies map[string]string) error {
 // (returns 404 as of April 2026). x-cli reads the `twid` cookie that
 // the user pasted, parses the numeric user id out of it, and calls
 // UserByRestId to confirm the session is alive AND fetch identity in
-// one round trip. Falls back to a UserByScreenName "twitter" probe if
-// `twid` is missing — that's the cheapest "is this cookie usable at
-// all" sanity check we have without it.
+// one round trip.
+//
+// When the user only provided `auth_token` (no twid — typical when
+// pasting from DevTools or when going through chromebrowser without
+// a stored twid), VerifyCredentials makes a cheap probe call. The
+// browser transport navigates to x.com on the first request, x.com
+// Set-Cookie's a fresh twid (and ct0, gt, etc.), the transport
+// surfaces those cookies back via Set-Cookie response headers, and
+// client.mergeSetCookies folds them into the session. We then
+// re-read twid and retry UserByRestId.
 func (c *Client) VerifyCredentials(ctx context.Context) (*User, error) {
+	if user, err := c.verifyByTwid(ctx); err == nil {
+		return user, nil
+	} else if !isMissingTwid(err) {
+		return nil, err
+	}
+
+	// No twid yet. Run any cheap GraphQL call to make the browser
+	// transport navigate to x.com — that's what triggers the
+	// Set-Cookie response that populates twid + ct0 in the session.
+	if _, err := c.GetProfile(ctx, "twitter"); err != nil {
+		return nil, normaliseAuthError(err)
+	}
+
+	// After the probe, the session should now have twid (merged in
+	// via mergeSetCookies). Retry the proper verify.
+	user, err := c.verifyByTwid(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("after probe: %w", err)
+	}
+	return user, nil
+}
+
+// verifyByTwid is the "have-twid" branch of VerifyCredentials. Returns
+// errMissingTwid when no usable twid is present, or any other error
+// from the underlying UserByRestId call.
+func (c *Client) verifyByTwid(ctx context.Context) (*User, error) {
 	c.sessionMu.RLock()
-	cookies := c.session.Cookies
 	twidRaw := ""
-	if cookies != nil {
-		twidRaw = cookies["twid"]
+	if c.session.Cookies != nil {
+		twidRaw = c.session.Cookies["twid"]
 	}
 	c.sessionMu.RUnlock()
 
-	if userID := parseTwidUserID(twidRaw); userID != "" {
-		p, err := c.GetProfileByID(ctx, userID)
-		if err != nil {
-			return nil, normaliseAuthError(err)
-		}
-		u := &User{
-			ID:       p.RestID,
-			Username: p.ScreenName,
-			Name:     p.Name,
-		}
-		c.sessionMu.Lock()
-		c.session.User = u
-		c.sessionMu.Unlock()
-		return u, nil
+	userID := parseTwidUserID(twidRaw)
+	if userID == "" {
+		return nil, errMissingTwid
 	}
-
-	// No twid: probe with a known account so we still detect a dead
-	// cookie, but we cannot return identity in this branch.
-	if _, err := c.GetProfile(ctx, "twitter"); err != nil {
+	p, err := c.GetProfileByID(ctx, userID)
+	if err != nil {
 		return nil, normaliseAuthError(err)
 	}
-	return nil, &AuthError{Msg: "session is alive but `twid` cookie missing — re-import a complete cookie string"}
+	u := &User{
+		ID:       p.RestID,
+		Username: p.ScreenName,
+		Name:     p.Name,
+	}
+	c.sessionMu.Lock()
+	c.session.User = u
+	c.sessionMu.Unlock()
+	return u, nil
 }
 
+// errMissingTwid is the sentinel returned by verifyByTwid when no
+// usable twid cookie is present in the session. VerifyCredentials
+// uses isMissingTwid to detect it and trigger the probe-then-retry
+// path.
+var errMissingTwid = &AuthError{Msg: "twid cookie not yet present — probe needed"}
+
+func isMissingTwid(err error) bool { return err == errMissingTwid }
+
 // parseTwidUserID extracts the numeric user ID from a `twid` cookie value.
 // Twitter encodes it as `u%3D<NNN>` (URL-encoded `u=NNN`).
 //

api/client.go

@@ -334,14 +334,18 @@ func (c *Client) mergeSetCookies(resp *http.Response) {
 // logRequest prints a one-line summary of an HTTP request to stderr if
 // Verbose is on. The query string is stripped because it carries the
 // GraphQL `variables` blob which can include user-controlled IDs that
-// are uninteresting noise. The Cookie header is never logged regardless
-// of verbosity — the only place we ever read it is applyHeaders.
+// are uninteresting noise — unless X_CLI_FULL_URL is set, in which
+// case we print the whole thing for debugging rotation drift. The
+// Cookie header is never logged regardless of verbosity — the only
+// place we ever read it is applyHeaders.
 func (c *Client) logRequest(method, rawURL string, status int, dur time.Duration, err error) {
 	if !c.Verbose {
 		return
 	}
-	if i := strings.IndexByte(rawURL, '?'); i > 0 {
-		rawURL = rawURL[:i] + "?…"
+	if os.Getenv("X_CLI_FULL_URL") == "" {
+		if i := strings.IndexByte(rawURL, '?'); i > 0 {
+			rawURL = rawURL[:i] + "?…"
+		}
 	}
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "» %s %s → ERROR (%s): %v\n", method, rawURL, dur.Round(time.Millisecond), err)
@@ -398,7 +402,16 @@ func (c *Client) GraphQL(ctx context.Context, name string, variables map[string]
 	if err != nil {
 		return err
 	}
-	featsJSON, err := json.Marshal(c.endpoints.Features)
+	// Per-op features override the global blob when present. Each
+	// GraphQL operation expects a specific feature set; using one
+	// global blob with the union of all keys causes the gateway to
+	// 404 the request (it treats unknown-for-this-op features as a
+	// schema mismatch).
+	featuresMap := c.endpoints.Features
+	if ep.Features != nil {
+		featuresMap = ep.Features
+	}
+	featsJSON, err := json.Marshal(featuresMap)
 	if err != nil {
 		return err
 	}

api/endpoints.go

@@ -28,6 +28,15 @@ type GraphQLEndpoint struct {
 	Kind          string  `yaml:"kind"`
 	RPS           float64 `yaml:"rps"`
 	Burst         int     `yaml:"burst"`
+
+	// Features is the per-operation features blob. When non-nil it
+	// REPLACES the global EndpointMap.Features for this op. Different
+	// X GraphQL operations expect different feature sets — passing a
+	// "union of everything" blob causes the gateway to 404 the
+	// request (it treats unknown features as a schema mismatch).
+	// Capture the per-op set from the live web client and put it
+	// here when an op stops working after a rotation.
+	Features map[string]bool `yaml:"features,omitempty"`
 }
 
 type RESTEndpoint struct {

api/relationships.go

@@ -148,13 +148,23 @@ type PageOptions struct {
 }
 
 // Followers scrapes the followers of `screenName`. Requires authentication.
+//
+// Variables match the modern web client (April 2026 capture):
+// `withGrokTranslatedBio: true` is required — without it x.com returns
+// 404 with an empty body (the gateway treats the missing var as an
+// unknown query).
 func (c *Client) Followers(ctx context.Context, screenName string, opts PageOptions) ([]*UserSummary, error) {
 	uid, err := c.resolveUserID(ctx, screenName)
 	if err != nil {
 		return nil, err
 	}
 	return c.scrapeUserList(ctx, "Followers",
-		map[string]any{"userId": uid, "count": 20, "includePromotedContent": false},
+		map[string]any{
+			"userId":                 uid,
+			"count":                  20,
+			"includePromotedContent": false,
+			"withGrokTranslatedBio":  true,
+		},
 		opts,
 		"data", "user", "result", "timeline", "timeline", "instructions")
 }
@@ -166,7 +176,12 @@ func (c *Client) Following(ctx context.Context, screenName string, opts PageOpti
 		return nil, err
 	}
 	return c.scrapeUserList(ctx, "Following",
-		map[string]any{"userId": uid, "count": 20, "includePromotedContent": false},
+		map[string]any{
+			"userId":                 uid,
+			"count":                  20,
+			"includePromotedContent": false,
+			"withGrokTranslatedBio":  true,
+		},
 		opts,
 		"data", "user", "result", "timeline", "timeline", "instructions")
 }

api/search.go

@@ -56,11 +56,14 @@ func (c *Client) SearchPosts(ctx context.Context, query string, opts SearchOptio
 	out := make([]*Tweet, 0, limit)
 
 	for len(out) < limit {
+		// withGrokTranslatedBio is required by the modern gateway.
+		// Sending it as `false` keeps the response shape compact.
 		vars := map[string]any{
-			"rawQuery":    rawQuery,
-			"count":       20,
-			"querySource": "typed_query",
-			"product":     product,
+			"rawQuery":              rawQuery,
+			"count":                 20,
+			"querySource":           "typed_query",
+			"product":               product,
+			"withGrokTranslatedBio": false,
 		}
 		if cursor != "" {
 			vars["cursor"] = cursor
@@ -107,10 +110,11 @@ func (c *Client) SearchUsers(ctx context.Context, query string, opts SearchOptio
 
 	for len(out) < limit {
 		vars := map[string]any{
-			"rawQuery":    query,
-			"count":       20,
-			"querySource": "typed_query",
-			"product":     "People",
+			"rawQuery":              query,
+			"count":                 20,
+			"querySource":           "typed_query",
+			"product":               "People",
+			"withGrokTranslatedBio": false,
 		}
 		if cursor != "" {
 			vars["cursor"] = cursor

api/tweets.go

@@ -433,10 +433,14 @@ func (c *Client) UserTweets(ctx context.Context, screenName string, opts Timelin
 	if err != nil {
 		return nil, err
 	}
+	// `includePromotedContent: true` matches the live web client.
+	// Sending false used to work but no longer does — the gateway
+	// returns a different schema variant. UserTweets in particular is
+	// strict about this flag.
 	return c.scrapeUserTimeline(ctx, "UserTweets", map[string]any{
 		"userId":                                 userID,
 		"count":                                  20,
-		"includePromotedContent":                 false,
+		"includePromotedContent":                 true,
 		"withQuickPromoteEligibilityTweetFields": true,
 		"withVoice":                              true,
 	}, opts)

cmd/auth.go

@@ -145,7 +145,11 @@ func runAuthImport(cmd *cobra.Command, _ []string) error {
 	}
 
 	cookies := api.ParseCookieString(raw)
-	if err := api.RequireAuthCookies(cookies); err != nil {
+	// In the browser path, ct0 is optional — chromebrowser fetches a
+	// fresh one from x.com via Set-Cookie on the first navigation.
+	// In the http path, ct0 is mandatory because the http transport
+	// has no way to mint one.
+	if err := api.RequireAuthCookiesFor(cookies, useHTTP); err != nil {
 		return err
 	}
 

cmd/doctor.go

@@ -102,11 +102,12 @@ func runDoctor(cmd *cobra.Command, _ []string) error {
 
 func verifyLiveSession(ctx context.Context, eps *api.EndpointMap, sess *store.Session) error {
 	client := api.New(api.Options{
-		Endpoints: eps,
-		Throttle:  api.NewThrottle(api.Defaults{}),
-		Session:   api.Session{Cookies: sess.Cookies},
+		Endpoints:  eps,
+		Throttle:   api.NewThrottle(api.Defaults{}),
+		Session:    api.Session{Cookies: sess.Cookies},
+		UseBrowser: !useHTTP,
 	})
-	ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
+	ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
 	defer cancel()
 	_, err := client.VerifyCredentials(ctx)
 	return err

endpoints.yaml

@@ -25,9 +25,22 @@ bases:
 # twikit and twitter-scraper. Rotate if X revokes it.
 bearer: "AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"
 
-# GraphQL features blob captured from the live web client. Must match
-# what the gateway expects at the time of the bundle. If the server
-# rejects a query with a "feature error", append the missing key here.
+# Default GraphQL features blob — the "timeline" set captured from
+# Followers/Following/SearchTimeline/UserTweets/BlueVerifiedFollowers
+# requests in the live web client. Profile-page ops (UserByScreenName,
+# UserByRestId) use their own override under the per-op features key
+# because they need a different set (see below).
+#
+# Profile-only keys NOT included here:
+#   hidden_profile_subscriptions_enabled
+#   subscriptions_verification_info_is_identity_verified_enabled
+#   subscriptions_verification_info_verified_since_enabled
+#   highlights_tweets_tab_ui_enabled
+#   responsive_web_twitter_article_notes_tab_enabled
+#   subscriptions_feature_can_gift_premium
+#
+# Adding them here breaks Followers/SearchTimeline (the gateway returns
+# 404 with empty body when an op receives features it doesn't expect).
 features:
   rweb_video_screen_enabled: false
   profile_label_improvements_pcf_label_in_post_enabled: true
@@ -65,12 +78,6 @@ features:
   responsive_web_grok_imagine_annotation_enabled: true
   responsive_web_grok_community_note_auto_translation_is_enabled: true
   responsive_web_enhance_cards_enabled: false
-  hidden_profile_subscriptions_enabled: true
-  subscriptions_verification_info_is_identity_verified_enabled: true
-  subscriptions_verification_info_verified_since_enabled: true
-  highlights_tweets_tab_ui_enabled: true
-  responsive_web_twitter_article_notes_tab_enabled: true
-  subscriptions_feature_can_gift_premium: true
 
 # GraphQL queries and mutations.
 #
@@ -85,6 +92,24 @@ graphql:
     kind: read
     rps: 0.5
     burst: 2
+    # Profile-specific features blob. Different from the global
+    # "timeline" set: includes hidden_profile_subscriptions and
+    # subscriptions_verification_info_* which the timeline ops
+    # reject as unknown.
+    features:
+      hidden_profile_subscriptions_enabled: true
+      profile_label_improvements_pcf_label_in_post_enabled: true
+      responsive_web_profile_redirect_enabled: false
+      rweb_tipjar_consumption_enabled: false
+      verified_phone_label_enabled: false
+      subscriptions_verification_info_is_identity_verified_enabled: true
+      subscriptions_verification_info_verified_since_enabled: true
+      highlights_tweets_tab_ui_enabled: true
+      responsive_web_twitter_article_notes_tab_enabled: true
+      subscriptions_feature_can_gift_premium: true
+      creator_subscriptions_tweet_preview_api_enabled: true
+      responsive_web_graphql_skip_user_profile_image_extensions_enabled: false
+      responsive_web_graphql_timeline_navigation_enabled: true
   UserByRestId:
     queryId: VQfQ9wwYdk6j_u2O4vt64Q
     operationName: UserByRestId
@@ -128,7 +153,7 @@ graphql:
     rps: 0.3
     burst: 2
   Followers:
-    queryId: gC_lyAxZOptAMLCJX5UhWw
+    queryId: "-WcGoRt8IQuPm-l1ymgy6g"
     operationName: Followers
     kind: read
     rps: 0.2