feat: v0.2 — port XActions scraper patterns, ship real commands

v0.1 was scaffolding + transport. v0.2 makes the commands real by
porting the production-grade subset of XActions' HTTP scraper
(`reference/XActions/src/scrapers/twitter/http/*`) into a typed Go
domain layer with the same stable rules:

api/extract.go
  Shared map-walking helpers (getMap, getString, getInt with string→int
  for X's view counts, getBool, getSlice, walkPath, walkPathSlice,
  walkPathMap, copyMap). Single source of truth for the parser layer.

api/tweets.go — ParseTweet projection
  - Recognizes Tweet | TweetWithVisibilityResults (unwrap one level) |
    TweetTombstone (typename → non-nil marker).
  - Recursive quote and retweet projection with a hard maxQuoteDepth=5
    cap so a malicious or pathological response cannot drive the parser
    into stack exhaustion.
  - Picks the highest-bitrate video/mp4 variant for video URLs.
  - parseTimelineInstructions dispatches on TimelineAddEntries /
    TimelineAddToModule / TimelinePinEntry, with cursor-bottom-/
    cursor-top-/tweet-/user- entry-ID prefix typing.
  - GetThread reconstructs a conversation from TweetDetail, filters to
    the focal tweet's author by default (self-thread), sorts
    chronologically.
  - resolveUserID is cached per Client via sync.Map so paginated reads
    and the monitor poll loop only pay the UserByScreenName roundtrip
    once per session.

api/relationships.go — Followers / Following / Likers / Retweeters
  - ParseUserSummary projects user_results.result, drops UserUnavailable,
    bumps avatar from _normal to _400x400.
  - Generic scrapeUserList paginator with username-keyed Map dedup,
    progress callback, multi-page cursor advance.
  - Endpoint-specific instructions paths:
      Followers/Following → data.user.result.timeline.timeline.instructions
      Retweeters          → data.retweeters_timeline.timeline.instructions
      Favoriters          → data.favoriters_timeline.timeline.instructions

api/search.go — SearchPosts / SearchUsers
  - Wraps SearchTimeline with the product flag (Latest|Top|People|
    Photos|Videos).
  - BuildAdvancedQuery composes from:, to:, since:, until:, min_faves:,
    min_retweets:, min_replies:, lang:, filter:, -filter: into the raw
    query string.
  - Dedicated parseSearchUserInstructions for product=People shape.

api/actions.go — Follow / Unfollow + classifyMutationErrors
  - REST form POST to friendshipsCreate / friendshipsDestroy with
    user_id + include_profile_interstitial_type=1 + skip_status=true,
    matching what real browsers send.
  - classifyMutationErrors inspects the response `errors[]` envelope
    and treats idempotent failures ("already followed", etc.) as
    success, so a re-run after a partial batch does not log false
    failures. Maps "rate limit" / "spam protection" to RateLimitError,
    "cannot find specified user" to NotFoundError, suspended to
    APIError.

api/media.go — DownloadTweetMedia
  - Streams every media asset on a tweet to disk via the same
    http.Client (so any future TLS impersonation round-tripper applies).
  - Atomic write via temp + rename so a crash mid-download cannot
    corrupt a previously valid file.
  - applyImageQuality appends the X CDN size hint (?name=large by
    default; small|medium|large|orig).

cmd/* — real commands replacing v0.1 stubs (cmd/stubs.go deleted)
  cmd/tweets.go      tweets list <user> [-n --replies], tweets get <id>
  cmd/relationships.go followers, following (shared paginated runner)
  cmd/search.go      search posts (--product --since --until --from
                     --to --lang --filter --exclude --min-likes
                     --min-retweets), search users
  cmd/thread.go      thread unroll [--all-authors]
  cmd/media.go       media download <id|url> [-o --quality]
  cmd/monitor.go     monitor account [-i --once] poll loop with new-
                     tweet streaming and follower delta; interval is
                     clamped to a minimum of 15s
  cmd/grow.go        grow follow-engagers <tweet>, grow
                     follow-by-keyword <query>; dry-run by default,
                     --apply required to mutate; refuses to mutate
                     from a cloud ASN unless --i-know-its-a-cloud-ip
                     is also passed (now wired to cmd/doctor.go's
                     EgressIsCloud — was a stub in the first draft);
                     --max --min-followers; mergeUserCandidates dedups,
                     sorts into log-buckets by follower count, then
                     shuffles within each bucket so the action pattern
                     is not deterministically biased toward the same
                     whales every run.

cmd/doctor.go
  Exports EgressIsCloud(ctx) for grow's mutation gate.

internal/cmdutil/render.go
  PrintJSON, TabPrinter, HumanCount (1.2k / 3.4M), RelTime, TruncateRunes
  (rune-safe so multi-byte text doesn't get corrupted), SingleLine.

Tests
  api/tweets_test.go        rich UserTweets fixture covering Pinned +
                            wrapped + tombstone + quote + video bitrate;
                            full UserTweets pipeline against a fake
                            server with two pages; ParseTweet depth-cap
                            regression.
  api/relationships_test.go full Followers pipeline with cross-page
                            dedup + UserUnavailable filter.
  api/search_test.go        BuildAdvancedQuery cases,
                            parseSearchUserInstructions.
  api/actions_test.go       FollowUser form body shape,
                            classifyMutationErrors dispatch
                            (idempotent / rate-limit / not-found /
                            unknown).
  api/media_test.go         image+video download against a fake server,
                            applyImageQuality, extFromURL.
  api/profile_test.go       getInt now accepts string-encoded ints.

Coverage: api/ 70.8%, internal/store/ 62.8%. Race-clean under
go test -race -count=1 ./...

Reviewed by an internal Linus-style pass; the four mandatory fixes
(asnIsCloud wired to real doctor egress check, ParseTweet recursion
depth cap, dead sentinel + joinArgs reimpl removed, resolveUserID
cached per Client) are all in this commit.

Docs
  skills/x-cli/SKILL.md       updated to match real flag names
  docs/comparison-xactions.md §6 marked features Implemented with file
                              references; new bullet for media chunked
                              uploader = Drop (out of scope for v0.1).

Author: lroolle

Makefile

@@ -1,5 +1,5 @@
 BINARY      := x
-PKG         := github.com/lroolle/x-cli
+PKG         := github.com/thevibeworks/x-cli
 VERSION     := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
 LDFLAGS     := -s -w -X $(PKG)/internal/version.Version=$(VERSION)
 GOFLAGS     := -trimpath

api/actions.go

@@ -0,0 +1,133 @@
+package api
+
+// Mutating actions: follow / unfollow. Mirrors the GraphQL/REST mutation
+// helpers in XActions' src/scrapers/twitter/http/engagement.js, with one
+// important addition: x-cli inspects the response body for X's "errors"
+// envelope and treats idempotent failures (already following, etc.) as
+// success. Without that, a retry after a partial success looks like a
+// failure and the caller can't tell the difference.
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strings"
+)
+
+// FollowUser follows a user by their numeric `rest_id`. Routes through
+// the REST `friendshipsCreate` endpoint, which is what real browsers use
+// for follow actions. Throttle-aware via Client.REST.
+func (c *Client) FollowUser(ctx context.Context, userID string) error {
+	if userID == "" {
+		return &APIError{Endpoint: "friendshipsCreate", Status: 0, Body: "empty user id"}
+	}
+	form := url.Values{}
+	form.Set("user_id", userID)
+	form.Set("include_profile_interstitial_type", "1")
+	form.Set("skip_status", "true")
+	return c.restMutationCheckErrors(ctx, "friendshipsCreate", form)
+}
+
+// UnfollowUser unfollows a user by their numeric `rest_id`.
+func (c *Client) UnfollowUser(ctx context.Context, userID string) error {
+	if userID == "" {
+		return &APIError{Endpoint: "friendshipsDestroy", Status: 0, Body: "empty user id"}
+	}
+	form := url.Values{}
+	form.Set("user_id", userID)
+	form.Set("include_profile_interstitial_type", "1")
+	form.Set("skip_status", "true")
+	return c.restMutationCheckErrors(ctx, "friendshipsDestroy", form)
+}
+
+// FollowByUsername resolves a screen name to a user ID and follows them.
+func (c *Client) FollowByUsername(ctx context.Context, screenName string) error {
+	uid, err := c.resolveUserID(ctx, strings.TrimPrefix(screenName, "@"))
+	if err != nil {
+		return err
+	}
+	return c.FollowUser(ctx, uid)
+}
+
+// restMutationCheckErrors wraps Client.REST and inspects the decoded body
+// for X's `errors[]` envelope. The envelope dispatch is:
+//
+//   - "already following" / "you have already" / variants → silent success
+//   - "rate limit" / "to protect our users from spam"     → RateLimitError
+//   - "cannot find specified user" / "user not found"     → NotFoundError
+//   - "suspended"                                         → APIError (final)
+//   - anything else                                       → APIError
+//
+// Throttle accounting and exponential backoff are handled inside Client.REST.
+// This wrapper only handles the message-level dispatch.
+func (c *Client) restMutationCheckErrors(ctx context.Context, endpointName string, form url.Values) error {
+	var body map[string]any
+	if err := c.REST(ctx, endpointName, form, &body); err != nil {
+		return err
+	}
+	return classifyMutationErrors(endpointName, body)
+}
+
+// classifyMutationErrors dispatches the `errors[]` envelope. Exported as
+// an internal helper for use by tests and future GraphQL mutation paths.
+func classifyMutationErrors(endpointName string, body map[string]any) error {
+	errs := getSlice(body, "errors")
+	if len(errs) == 0 {
+		return nil
+	}
+	for _, e := range errs {
+		em, ok := e.(map[string]any)
+		if !ok {
+			continue
+		}
+		msg := strings.ToLower(getString(em, "message"))
+
+		// Idempotent — already in the desired state.
+		if strings.Contains(msg, "already favorited") ||
+			strings.Contains(msg, "already retweeted") ||
+			strings.Contains(msg, "already bookmarked") ||
+			strings.Contains(msg, "you have already") ||
+			strings.Contains(msg, "already followed") ||
+			strings.Contains(msg, "already following") ||
+			strings.Contains(msg, "not found in list of retweets") {
+			return nil
+		}
+
+		// Rate limit (server-side message variant — Throttle.Observe handles
+		// HTTP 429 separately).
+		if strings.Contains(msg, "rate limit") ||
+			strings.Contains(msg, "to protect our users from spam") ||
+			strings.Contains(msg, "too many requests") {
+			return &RateLimitError{Endpoint: endpointName}
+		}
+
+		// Not found.
+		if strings.Contains(msg, "cannot find specified user") ||
+			strings.Contains(msg, "user not found") ||
+			strings.Contains(msg, "user has been suspended") {
+			return &NotFoundError{Endpoint: endpointName}
+		}
+
+		// Suspended (the actor — i.e. our own session).
+		if strings.Contains(msg, "your account is suspended") ||
+			strings.Contains(msg, "this account is suspended") {
+			return &APIError{Endpoint: endpointName, Status: 0, Body: getString(em, "message")}
+		}
+	}
+
+	// Unrecognized error envelope — surface the raw messages.
+	parts := make([]string, 0, len(errs))
+	for _, e := range errs {
+		if em, ok := e.(map[string]any); ok {
+			if msg := getString(em, "message"); msg != "" {
+				parts = append(parts, msg)
+			}
+		}
+	}
+	if len(parts) == 0 {
+		raw, _ := json.Marshal(body)
+		return &APIError{Endpoint: endpointName, Status: 0, Body: string(raw)}
+	}
+	return &APIError{Endpoint: endpointName, Status: 0, Body: fmt.Sprintf("graphql errors: %s", strings.Join(parts, "; "))}
+}

api/actions_test.go

@@ -0,0 +1,163 @@
+package api
+
+import (
+	"context"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestFollowUserSendsCorrectFormBody(t *testing.T) {
+	var captured map[string]string
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ := io.ReadAll(r.Body)
+		captured = map[string]string{
+			"path":   r.URL.Path,
+			"method": r.Method,
+			"body":   string(body),
+			"ctype":  r.Header.Get("Content-Type"),
+		}
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(`{"id": 1, "name": "test"}`))
+	}))
+	defer srv.Close()
+
+	eps := &EndpointMap{
+		Bases:  Bases{REST: srv.URL, GraphQL: srv.URL},
+		Bearer: "B",
+		REST: map[string]RESTEndpoint{
+			"friendshipsCreate": {
+				Path:     "/1.1/friendships/create.json",
+				Method:   "POST",
+				Kind:     "mutation",
+				MinGap:   10 * 1000 * 1000, // 10ms
+				MaxGap:   10 * 1000 * 1000,
+				DailyCap: 10,
+			},
+		},
+	}
+	c := New(Options{
+		Endpoints: eps,
+		Throttle:  NewThrottle(Defaults{}),
+		Session:   Session{Cookies: map[string]string{"auth_token": "x", "ct0": "y"}},
+	})
+
+	if err := c.FollowUser(context.Background(), "12345"); err != nil {
+		t.Fatal(err)
+	}
+
+	if captured["path"] != "/1.1/friendships/create.json" {
+		t.Errorf("path = %q", captured["path"])
+	}
+	if captured["method"] != "POST" {
+		t.Errorf("method = %q", captured["method"])
+	}
+	if !strings.HasPrefix(captured["ctype"], "application/x-www-form-urlencoded") {
+		t.Errorf("content-type = %q", captured["ctype"])
+	}
+	for _, want := range []string{"user_id=12345", "skip_status=true", "include_profile_interstitial_type=1"} {
+		if !strings.Contains(captured["body"], want) {
+			t.Errorf("body %q missing %q", captured["body"], want)
+		}
+	}
+}
+
+func TestFollowUserEmptyIDReturnsError(t *testing.T) {
+	c := &Client{}
+	if err := c.FollowUser(context.Background(), ""); err == nil {
+		t.Error("expected error for empty user id")
+	}
+}
+
+func TestClassifyMutationErrors(t *testing.T) {
+	cases := []struct {
+		name    string
+		body    map[string]any
+		wantErr error
+	}{
+		{
+			name:    "no errors",
+			body:    map[string]any{},
+			wantErr: nil,
+		},
+		{
+			name: "already following",
+			body: map[string]any{
+				"errors": []any{
+					map[string]any{"message": "You have already followed this user."},
+				},
+			},
+			wantErr: nil, // idempotent
+		},
+		{
+			name: "rate limited",
+			body: map[string]any{
+				"errors": []any{
+					map[string]any{"message": "Rate limit exceeded"},
+				},
+			},
+			wantErr: &RateLimitError{},
+		},
+		{
+			name: "user not found",
+			body: map[string]any{
+				"errors": []any{
+					map[string]any{"message": "Cannot find specified user"},
+				},
+			},
+			wantErr: &NotFoundError{},
+		},
+		{
+			name: "spam protection",
+			body: map[string]any{
+				"errors": []any{
+					map[string]any{"message": "To protect our users from spam"},
+				},
+			},
+			wantErr: &RateLimitError{},
+		},
+		{
+			name: "unknown error",
+			body: map[string]any{
+				"errors": []any{
+					map[string]any{"message": "Some unrecognised error"},
+				},
+			},
+			wantErr: &APIError{},
+		},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			err := classifyMutationErrors("test", tc.body)
+			if tc.wantErr == nil {
+				if err != nil {
+					t.Errorf("want nil, got %T: %v", err, err)
+				}
+				return
+			}
+			if err == nil {
+				t.Fatalf("want %T, got nil", tc.wantErr)
+			}
+			switch tc.wantErr.(type) {
+			case *RateLimitError:
+				var target *RateLimitError
+				if !errors.As(err, &target) {
+					t.Errorf("want *RateLimitError, got %T: %v", err, err)
+				}
+			case *NotFoundError:
+				var target *NotFoundError
+				if !errors.As(err, &target) {
+					t.Errorf("want *NotFoundError, got %T: %v", err, err)
+				}
+			case *APIError:
+				var target *APIError
+				if !errors.As(err, &target) {
+					t.Errorf("want *APIError, got %T: %v", err, err)
+				}
+			}
+		})
+	}
+}

api/client.go

@@ -35,6 +35,8 @@ type Client struct {
 	sessionMu sync.RWMutex
 	session   Session
 
+	userIDCache sync.Map // screen_name(lowercased) → rest_id
+
 	userAgent    string
 	retryBackoff time.Duration // base unit for exponential backoff; overridable in tests
 }