diff --git a/.agents/commands/line-capture.md b/.agents/commands/line-capture.md deleted file mode 100644 index b0ecd69c..00000000 --- a/.agents/commands/line-capture.md +++ /dev/null @@ -1,38 +0,0 @@ -Launch Chrome with network capture enabled for LINE Chrome Extension analysis. - -## What to do - -1. **Close any running Chrome instances first** (Chrome only supports one instance per profile with remote debugging). Warn the user if Chrome is running. - -2. Run the capture script: - ``` - bash scripts/line-chrome-capture.sh /tmp/line-chrome-extension-capture - ``` - This launches Chrome with: - - Remote debugging on port 9222 - - Full network logging to `/tmp/line-chrome-extension-capture/line-net-log.json` - - A persistent capture-only Chrome profile at - `~/.cache/line-chrome-extension-capture/chrome-debug-profile` - - Auto-loaded LINE and Codex extensions staged from the user's normal Chrome - profile into `~/.cache/line-chrome-extension-capture/extensions` - - A LINE extension version preflight: - - repo version from `pkg/line/client.go` `ExtensionVersion` - - installed/staged Chrome extension version from `manifest.json` - - a warning if those versions differ - - First-run Chrome prompts pre-skipped: - - "Make Google Chrome the default browser" stays unchecked - - "Automatically send usage statistics and crash reports" stays unchecked - - Chrome/Sync sign-in is skipped; use "Stay Signed Out" if it appears - - If the LINE extension is missing and the user asked for setup, relaunch with - `LINE_CAPTURE_AUTO_LOAD_EXTENSIONS=0`, open the official Web Store listing - for `ophjlpahpchlmihnnnihgmmeilfjmjjc`, verify LINE / LY Corporation, click - Add to Chrome, and accept the standard "Add extension" confirmation - -3. Tell the user: - - Open the LINE Chrome Extension (browser toolbar icon) - - Log in manually if the capture profile is not already logged in - - Perform whatever flow they want to analyze - - When done, quit Chrome completely (Cmd+Q) - -4. Once Chrome is closed, the net-log file is ready. Continue with the - line-analyze workflow. diff --git a/.agents/commands/line-analyze.md b/.agents/skills/line-analyze/SKILL.md similarity index 51% rename from .agents/commands/line-analyze.md rename to .agents/skills/line-analyze/SKILL.md index 98554177..14226037 100644 --- a/.agents/commands/line-analyze.md +++ b/.agents/skills/line-analyze/SKILL.md @@ -1,32 +1,47 @@ -Parse and analyze a LINE Chrome Extension network capture. +--- +name: line-analyze +description: Parses and redacts a LINE Chrome Extension network capture (CDP capture or Chrome net-log) and compares it with the beeper/line bridge code. Use after a line-capture run, or when given a LINE net-log or CDP capture file to analyze. +--- + +# LINE Analyze + +Turns a raw LINE Chrome Extension capture into redacted, readable findings and +compares them with the bridge. Run the commands below from the repository root. ## Input -Use the provided path to a net-log JSON file. If no path is provided, default to +Use the provided path to a capture file. Prefer the CDP capture +(`line-cdp-capture.json`): it includes SSE messages and request bodies. The +Chrome net-log is a backup. If no path is provided, default to +`/tmp/line-chrome-extension-capture/line-cdp-capture.json`, then `/tmp/line-chrome-extension-capture/line-net-log.json`. ## What to do -1. Run the parser: +1. Run the parser. This extracts LINE-related HTTP requests and writes a + `-parsed.json` file: + + ```bash + python3 .agents/skills/line-analyze/scripts/parse-line-traffic.py [] ``` - python3 scripts/parse-line-traffic.py + +2. Redact it before reading or quoting anything. This removes tokens, MIDs, + `picturePath` and raw body bytes: + + ```bash + python3 .agents/skills/line-analyze/scripts/redact-line-capture.py ``` - This extracts LINE-related HTTP requests from the Chrome net-log and writes a `-parsed.json` file. -2. Read the parsed output JSON file. + Read the raw files only locally, and only when the redacted output loses + information you need. Quote or document only redacted snippets. -3. Analyze the traffic and present findings organized by category: +3. Read the redacted output and present findings by category: - **Thrift RPC** (`/api/talk/thrift/`): The main LINE API. Requests use LINE's Thrift-over-HTTP protocol. The `x-lhm` header indicates the Thrift method name. Compare against methods in `pkg/line/methods.go`. - - **Long-poll / SSE** (`/api/talk/long-polling/`, `/api/operation/receive`): Real-time event delivery. LF1 is used during login verification, JQ for login polling, and SSE for ongoing message/operation streaming. Compare against `pkg/line/sse.go` and `pkg/line/client.go`. - - **Auth** (`/api/auth/`): Token refresh and login flows. Compare against login methods in `pkg/line/client.go` (loginV2, confirmE2EELogin, tokenRefresh). - - **Media/OBS** (`obs.line-apps.com`): Media upload/download. Check `x-obs-params` header for operation details, `x-obs-oid` in responses. Compare against `pkg/line/client.go` (UploadOBS, DownloadOBS). - - **Stickers/CDN** (`stickershop`, `scdn`): Sticker image downloads. - - **Secondary API** (`legy-jp.line-apps.com`): Link previews and page info. Compare against `pkg/line/client.go` (GetPageInfo). 4. For each interesting request, explain: @@ -34,9 +49,16 @@ Use the provided path to a net-log JSON file. If no path is provided, default to - The request/response structure - Any headers that carry protocol-specific meaning -5. Highlight any **unimplemented** endpoints or behaviors that differ from what the bridge currently does. Cross-reference with `pkg/line/methods.go` and `pkg/connector/`. +5. Highlight any **unimplemented** endpoints or behaviors that differ from what + the bridge currently does. Cross-reference with `pkg/line/methods.go` and + `pkg/connector/`. -6. After presenting the analysis, ask the user if they want to clean up the capture files (`/tmp/line-chrome-extension-capture/`). Warn that the raw net-log and parsed output contain sensitive data (auth tokens in `X-Line-Access` headers, message IDs, etc.) and should not be left on disk or committed to the repository. If the user agrees, delete the entire capture directory. +6. After presenting the analysis, ask the user if they want to clean up the + capture files (`/tmp/line-chrome-extension-capture/`). Warn that the raw + capture and parsed output contain sensitive data (auth tokens in + `X-Line-Access` headers, message IDs, etc.) and must not be left on disk or + committed to a repository. If the user agrees, delete the entire capture + directory. ## LINE protocol context @@ -44,5 +66,5 @@ Use the provided path to a net-log JSON file. If no path is provided, default to - Thrift RPC uses HTTP POST to `line-chrome-gw.line-apps.com/api/talk/thrift/Talk//`. - Auth token is in `x-line-access` header. - E2EE messages have `chunks` field containing encrypted content (base64 segments). -- Operations have numeric `type` codes — see `pkg/connector/sync.go` and `pkg/connector/handle_message.go` for the operation type switch. +- Operations have numeric `type` codes. See `pkg/connector/sync.go` and `pkg/connector/handle_message.go` for the operation type switch. - Body content in the parsed output may be base64-encoded binary (Thrift serialization). Note the method name from headers rather than trying to decode the binary body. diff --git a/scripts/parse-line-traffic.py b/.agents/skills/line-analyze/scripts/parse-line-traffic.py similarity index 90% rename from scripts/parse-line-traffic.py rename to .agents/skills/line-analyze/scripts/parse-line-traffic.py index dd21a688..bf79df4e 100755 --- a/scripts/parse-line-traffic.py +++ b/.agents/skills/line-analyze/scripts/parse-line-traffic.py @@ -2,7 +2,7 @@ """Parse LINE network captures (CDP or net-log format). Usage: - python3 scripts/parse-line-traffic.py [output.json] + python3 .agents/skills/line-analyze/scripts/parse-line-traffic.py [output.json] Supports two input formats: - CDP capture (from line-cdp-capture.py): has SSE message bodies, Thrift response bodies @@ -231,14 +231,17 @@ def parse_cdp_capture(data: dict) -> dict: entry["status"] = req["status"] # Extract interesting request headers - headers = req.get("headers", {}) + # requestWillBeSentExtraInfo carries the headers Chrome really sent + # (Cookie, Origin, Sec-*, X-Line-*), which requestWillBeSent omits. + headers = {**req.get("headers", {}), **req.get("request_extra_headers", {})} interesting_req = {} for k, v in headers.items(): if k.lower() in ( "content-type", "x-line-access", "x-line-application", "x-lhm", "x-lpv", "x-lsr", "x-obs-params", - "x-line-channeltoken", "accept", - ): + "x-line-channeltoken", "accept", "origin", "user-agent", + "x-line-chrome-version", "x-line-mid", + ) or k.lower().startswith("x-line-") or k.lower().startswith("x-ls"): interesting_req[k] = v if interesting_req: entry["request_headers"] = interesting_req @@ -270,7 +273,19 @@ def parse_cdp_capture(data: dict) -> dict: # Include request body if available if "request_body" in req: - entry["request_body_text"] = req["request_body"] + body = req["request_body"] + if isinstance(body, dict): + # Raw captures keep the bytes in base64; the parsed output only + # keeps the text form and a size + sha256 for binary bodies. + if body.get("error"): + entry["request_body_error"] = body["error"] + else: + entry["request_body_size"] = body.get("size") + entry["request_body_sha256"] = body.get("sha256") + if body.get("text") is not None: + entry["request_body_text"] = body["text"] + else: + entry["request_body_text"] = body entry["timestamp"] = req.get("timestamp", "") line_requests.append(entry) diff --git a/scripts/redact-line-capture.py b/.agents/skills/line-analyze/scripts/redact-line-capture.py similarity index 87% rename from scripts/redact-line-capture.py rename to .agents/skills/line-analyze/scripts/redact-line-capture.py index 8c20f481..fdea3fce 100755 --- a/scripts/redact-line-capture.py +++ b/.agents/skills/line-analyze/scripts/redact-line-capture.py @@ -2,7 +2,7 @@ """Redact sensitive values from LINE capture JSON files. Usage: - python3 scripts/redact-line-capture.py input.json [output.json] + python3 .agents/skills/line-analyze/scripts/redact-line-capture.py input.json [output.json] The default output path is input-redacted.json. This helper preserves JSON structure for protocol analysis while removing credentials, cookies, tokens, @@ -39,6 +39,16 @@ "refresh_token", "session", "certificate", + # Identifiers that point at a person, group or uploaded object. + "mid", + "chatmid", + "contactmid", + "picturepath", + "picturestatus", + "oid", + "x-obs-oid", + # Raw request-body bytes from the CDP recorder (uploads, Thrift payloads). + "base64", } SENSITIVE_KEY_FRAGMENTS = ( @@ -53,7 +63,8 @@ EMAIL_RE = re.compile(r"(?i)[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}") JWT_RE = re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}") -MID_RE = re.compile(r"\b([ucr][0-9a-f]{20,})\b", re.IGNORECASE) +# Not \b: "_" counts as a word character there, which hid MIDs after "_". +MID_RE = re.compile(r"(? -m pip + install --user` for Python packages, using the Python the preflight + reported). Show the user the missing items with the exact commands you plan + to run, ask them to confirm, then install and re-run the preflight until it + exits 0. Never install without that confirmation. The launcher refuses to + start while dependencies are missing. + +2. **Close any running Chrome instances first.** Chrome only supports one + instance per profile with remote debugging. Warn the user if Chrome is + running. + +3. Run the capture script from the checkout: + + ```bash + bash .agents/skills/line-capture/scripts/line-chrome-capture.sh /tmp/line-chrome-extension-capture + ``` + + This launches Chrome with: + - Remote debugging on port 9222 + - Full network logging to `/tmp/line-chrome-extension-capture/line-net-log.json` + - A persistent capture-only Chrome profile at + `~/.cache/line-chrome-extension-capture/chrome-debug-profile` + - Auto-loaded LINE and Codex extensions staged from the user's normal Chrome + profile into `~/.cache/line-chrome-extension-capture/extensions` + - A LINE extension version preflight: + - repo version from `pkg/line/client.go` `ExtensionVersion` + - installed/staged Chrome extension version from `manifest.json` + - a warning if those versions differ + - First-run Chrome prompts pre-skipped: + - "Make Google Chrome the default browser" stays unchecked + - "Automatically send usage statistics and crash reports" stays unchecked + - Chrome/Sync sign-in is skipped; use "Stay Signed Out" if it appears + - If the LINE extension is missing and the user asked for setup, relaunch + with `LINE_CAPTURE_AUTO_LOAD_EXTENSIONS=0`, open the official Web Store + listing for `ophjlpahpchlmihnnnihgmmeilfjmjjc`, verify LINE / LY + Corporation, click Add to Chrome, and accept the standard "Add extension" + confirmation + +4. Tell the user: + - Chrome opens on `about:blank`. Open LINE from the extension's toolbar icon + (puzzle-piece menu if it is not pinned), not by opening the extension's + `index.html` as a tab: that behaves differently from the real popup. + `LINE_CAPTURE_START_URL` overrides the start URL. + - Log in manually if the capture profile is not already logged in + - Perform whatever flow they want to analyze + - Do not open DevTools on the extension: LINE has `debugger;` traps that + pause it, and the recorder does not need DevTools + - Windows the extension opens (Settings, group popups) can start frozen + and blank, with LINE greyed out as "Debugger paused in another tab". The + recorder releases them automatically. If one still freezes, quit Chrome + and run the capture again. + - When done, quit Chrome completely (all windows, not only the tab) + +5. Once Chrome is closed, the capture files are ready. Continue with the + `line-analyze` skill. + +Raw captures are secrets (tokens, cookies, MIDs, message bodies). Never commit +them. See the `line-implement` skill for the full security rules. diff --git a/.agents/skills/line-capture/scripts/line-capture-preflight.sh b/.agents/skills/line-capture/scripts/line-capture-preflight.sh new file mode 100755 index 00000000..ba7aa33c --- /dev/null +++ b/.agents/skills/line-capture/scripts/line-capture-preflight.sh @@ -0,0 +1,112 @@ +#!/bin/bash +# Check that everything the LINE Chrome Extension capture needs is installed. +# Works on macOS, Linux and Windows (Git Bash / MSYS2). +# +# Usage: +# .agents/skills/line-capture/scripts/line-capture-preflight.sh # human/agent-readable report +# eval "$(.agents/skills/line-capture/scripts/line-capture-preflight.sh --print-env)" # sets CHROME and the PYTHON array +# +# The report prints the detected OS, one OK line per dependency found and one +# MISSING line per dependency that is not. It deliberately does not say how to +# install anything: the agent running it picks the install method that fits +# the OS and the package managers available, and asks the user first. +# Exits 1 when anything required is missing. +# +# Override detection with CHROME=/path/to/chrome or PYTHON=/path/to/python +# (one executable; spaces in the path are fine). + +set -uo pipefail + +case "$(uname -s)" in + Darwin) OS=macos ;; + Linux) + if grep -qi microsoft /proc/version 2>/dev/null; then OS=wsl; else OS=linux; fi ;; + MINGW* | MSYS* | CYGWIN*) OS=windows ;; + *) OS=unknown ;; +esac + +find_chrome() { + if [ -n "${CHROME:-}" ]; then + [ -x "$CHROME" ] && echo "$CHROME" + return + fi + local candidates=() + case "$OS" in + macos) + candidates=("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" + "$HOME/Applications/Google Chrome.app/Contents/MacOS/Google Chrome") ;; + windows) + candidates=("${PROGRAMFILES:-/c/Program Files}/Google/Chrome/Application/chrome.exe" + "${PROGRAMFILES_X86:-/c/Program Files (x86)}/Google/Chrome/Application/chrome.exe" + "${LOCALAPPDATA:-$HOME/AppData/Local}/Google/Chrome/Application/chrome.exe") ;; + *) + local name + for name in google-chrome google-chrome-stable chromium chromium-browser; do + command -v "$name" >/dev/null 2>&1 && { command -v "$name"; return; } + done ;; + esac + local path + for path in "${candidates[@]}"; do + [ -x "$path" ] && { echo "$path"; return; } + done +} + +# Sets the PYTHON_FOUND array: the executable first, then optional arguments +# ("py -3"). An array keeps a PYTHON override with spaces in its path in one piece. +find_python() { + PYTHON_FOUND=() + local candidate + if [ -n "${PYTHON:-}" ]; then + "$PYTHON" -c "import sys; sys.exit(sys.version_info < (3, 8))" 2>/dev/null && PYTHON_FOUND=("$PYTHON") + return + fi + # "python3" on Windows can be the Microsoft Store stub, so the version check + # also filters out commands that exist but don't run. + for candidate in "python3" "python" "py -3"; do + # shellcheck disable=SC2206 # intentional split of "py -3" + local cmd=($candidate) + if "${cmd[@]}" -c "import sys; sys.exit(sys.version_info < (3, 8))" 2>/dev/null; then + PYTHON_FOUND=("${cmd[@]}") + return + fi + done +} + +CHROME_FOUND="$(find_chrome)" +find_python + +if [ "${1:-}" = "--print-env" ]; then + printf 'CHROME=%q\nPYTHON=(' "$CHROME_FOUND" + [ ${#PYTHON_FOUND[@]} -gt 0 ] && printf '%q ' "${PYTHON_FOUND[@]}" + printf ')\n' + exit 0 +fi + +missing=0 +ok() { echo "OK $1"; } +need() { echo "MISSING $1"; missing=1; } + +echo "OS $OS ($(uname -sm))" +if [ "$OS" = wsl ]; then + echo "NOTE WSL detected: use a Linux Chrome or Chromium inside WSL. Windows chrome.exe cannot use WSL paths or reach the recorder, so run from Git Bash on Windows instead." +fi + +if [ -n "$CHROME_FOUND" ]; then + ok "Google Chrome: $CHROME_FOUND" +else + need "Google Chrome (or Chromium on Linux); set CHROME=/path/to/chrome if it is installed somewhere else" +fi + +if [ ${#PYTHON_FOUND[@]} -gt 0 ]; then + ok "Python 3.8+: ${PYTHON_FOUND[*]} ($("${PYTHON_FOUND[@]}" --version 2>&1))" + # line-cdp-capture.py records request bodies and SSE events through CDP. + if "${PYTHON_FOUND[@]}" -c "import websocket" >/dev/null 2>&1; then + ok "Python package websocket-client" + else + need "Python package websocket-client (pip name: websocket-client), for ${PYTHON_FOUND[*]}" + fi +else + need "Python 3.8+" +fi + +exit $missing diff --git a/scripts/line-cdp-capture.py b/.agents/skills/line-capture/scripts/line-cdp-capture.py similarity index 74% rename from scripts/line-cdp-capture.py rename to .agents/skills/line-capture/scripts/line-cdp-capture.py index 142befe9..c76b41e0 100755 --- a/scripts/line-cdp-capture.py +++ b/.agents/skills/line-capture/scripts/line-cdp-capture.py @@ -4,15 +4,18 @@ Connects to Chrome's remote debugging port and captures: - SSE EventSource messages (the critical piece net-log misses) - Thrift RPC request/response bodies -- All LINE-domain request headers +- All LINE-domain request headers, including the ones Chrome adds itself + (Network.requestWillBeSentExtraInfo) +- Request bodies of every LINE request, binary ones (image uploads) included + (stored as size + sha256, and the bytes in base64 in this raw capture) Usage: # First, launch Chrome with --remote-debugging-port=9222 # Then: - python3 scripts/line-cdp-capture.py [output.json] + python3 .agents/skills/line-capture/scripts/line-cdp-capture.py [output.json] # Or use the wrapper: - bash scripts/line-chrome-capture.sh [output_dir] + bash .agents/skills/line-capture/scripts/line-chrome-capture.sh [output_dir] """ import json @@ -21,6 +24,7 @@ import signal import time import base64 +import hashlib import threading from collections import defaultdict from datetime import datetime, timezone @@ -69,7 +73,10 @@ def __init__(self, output_path: str): self.thrift_responses = {} # requestId -> response body self.request_bodies = {} # requestId -> request body self.pending_body_requests = {} # msg_id -> (requestId, session_id) + self.pending_post_requests = {} # msg_id -> requestId + self.extra_headers = {} # requestId -> headers from requestWillBeSentExtraInfo self.sessions = {} # sessionId -> targetInfo + self.target_sessions = {} # targetId -> sessionId self.running = True @@ -143,6 +150,11 @@ def send_to_session(self, session_id: str, method: str, params: dict = None) -> self.ws.send(json.dumps(msg)) return mid + def release_target(self, session_id: str, target_type: str): + """Let a page-like target run if it is held waiting for a debugger.""" + if target_type in ("page", "iframe", "other"): + self.send_to_session(session_id, "Runtime.runIfWaitingForDebugger") + def handle_event(self, msg: dict): method = msg.get("method", "") params = msg.get("params", {}) @@ -162,18 +174,39 @@ def handle_event(self, msg: dict): "maxTotalBufferSize": 100 * 1024 * 1024, "maxResourceBufferSize": 10 * 1024 * 1024, }) + # Windows opened by the extension (the per-group popup.html, the + # Settings window) can arrive held at "waiting for debugger" even + # with waitForDebuggerOnStart off, and Chrome does not always set + # waitingForDebugger on the event. Left alone they stay blank and + # grey out the extension with "Debugger paused in another tab". + # runIfWaitingForDebugger is a no-op on a running target, so send + # it to every page-like target whatever the flag says. + self.release_target(sid, ttype) # Track session -> target mapping self.sessions[sid] = target_info + self.target_sessions[target_info.get("targetId", "")] = sid + return + + # A window can start waiting after attach (it attaches with an empty + # URL and navigates), so release again whenever its info changes. + if method == "Target.targetInfoChanged": + info = params.get("targetInfo", {}) + sid = self.target_sessions.get(info.get("targetId", "")) + if sid: + self.release_target(sid, info.get("type", "")) return if method == "Target.detachedFromTarget": sid = params.get("sessionId", "") target = self.sessions.pop(sid, {}) + self.target_sessions.pop(target.get("targetId", ""), None) print(f" Detached from [{target.get('type', '?')}] {target.get('title', '?')}", file=sys.stderr) return if method == "Network.requestWillBeSent": self._on_request(params, session_id) + elif method == "Network.requestWillBeSentExtraInfo": + self._on_request_extra_info(params) elif method == "Network.responseReceived": self._on_response(params, session_id) elif method == "Network.loadingFinished": @@ -186,6 +219,27 @@ def handle_event(self, msg: dict): def handle_response(self, msg: dict): mid = msg.get("id") + if mid in self.pending_post_requests: + request_id = self.pending_post_requests.pop(mid) + result = msg.get("result", {}) + post_data = result.get("postData") + if post_data is not None: + # Newer Chrome flags binary bodies with base64Encoded; older + # ones return text only, which is lossy for binary data. + if result.get("base64Encoded"): + raw = base64.b64decode(post_data) + else: + raw = post_data.encode("utf-8", errors="surrogateescape") + self.request_bodies[request_id] = { + "size": len(raw), + "sha256": hashlib.sha256(raw).hexdigest(), + "base64": base64.b64encode(raw).decode("ascii"), + "text": raw.decode("utf-8") if self._is_text(raw) else None, + } + else: + error = msg.get("error", {}).get("message", "unavailable") + self.request_bodies[request_id] = {"error": error} + return if mid in self.pending_body_requests: request_id, session_id = self.pending_body_requests.pop(mid) result = msg.get("result", {}) @@ -216,16 +270,33 @@ def _on_request(self, params: dict, session_id: str = ""): "session_id": session_id, } - # Capture POST body for Thrift calls - if request.get("hasPostData") and "/thrift/" in url: - post_data = request.get("postData", "") - if post_data: - self.request_bodies[request_id] = post_data + # Request bodies: the inline postData is missing for large or binary + # bodies (e.g. obs/uploadProfile), so always ask for it explicitly. + if request.get("hasPostData"): + target = session_id + if target: + mid = self.send_to_session(target, "Network.getRequestPostData", {"requestId": request_id}) + else: + mid = self.send("Network.getRequestPostData", {"requestId": request_id}) + self.pending_post_requests[mid] = request_id count = len(self.requests) if count % 10 == 1: print(f" [{count} LINE requests captured]", file=sys.stderr) + @staticmethod + def _is_text(raw: bytes) -> bool: + try: + raw.decode("utf-8") + except UnicodeDecodeError: + return False + return b"\x00" not in raw + + def _on_request_extra_info(self, params: dict): + """Headers as sent on the wire (Cookie, Origin, Sec-*, X-Line-* ...).""" + request_id = params.get("requestId") + self.extra_headers.setdefault(request_id, {}).update(params.get("headers", {})) + def _on_response(self, params: dict, session_id: str = ""): request_id = params.get("requestId") if request_id not in self.requests: @@ -295,6 +366,8 @@ def save(self): entry.pop("session_id", None) # internal, don't export if rid in self.thrift_responses: entry["response_body"] = self.thrift_responses[rid] + if rid in self.extra_headers: + entry["request_extra_headers"] = self.extra_headers[rid] if rid in self.request_bodies: entry["request_body"] = self.request_bodies[rid] requests_list.append(entry) diff --git a/scripts/line-chrome-capture.sh b/.agents/skills/line-capture/scripts/line-chrome-capture.sh similarity index 70% rename from scripts/line-chrome-capture.sh rename to .agents/skills/line-capture/scripts/line-chrome-capture.sh index 101967ed..627dac8e 100755 --- a/scripts/line-chrome-capture.sh +++ b/.agents/skills/line-capture/scripts/line-chrome-capture.sh @@ -2,7 +2,9 @@ # Launch Chrome with CDP-based network capture for LINE extension analysis. # # Usage: -# ./scripts/line-chrome-capture.sh [output_dir] +# bash .agents/skills/line-capture/scripts/line-chrome-capture.sh [output_dir] +# +# The script reads the bridge's ExtensionVersion from pkg/line/client.go. # # This starts Chrome with remote debugging and runs a CDP listener that # captures all LINE traffic including SSE EventSource message bodies. @@ -29,13 +31,63 @@ # This is useful when installing LINE from the Chrome Web Store into the # persistent capture profile. # -# When done, close Chrome (Cmd+Q). The capture files will be at: +# The browser opens on about:blank; open LINE from the extension's toolbar icon +# (LINE_CAPTURE_START_URL overrides the start URL). When done, quit Chrome +# completely. The capture files will be at: # /line-cdp-capture.json — CDP capture (SSE + request bodies) # /line-net-log.json — Chrome net-log (low-level backup) set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# .agents/skills/line-capture/scripts -> repository root +REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)" + +# Without its dependencies the CDP listener dies silently and only the net-log +# is saved, which lacks request bodies and SSE events. +if ! "$SCRIPT_DIR/line-capture-preflight.sh" >/dev/null; then + "$SCRIPT_DIR/line-capture-preflight.sh" + echo "" + echo "==> Install the missing dependencies above, then run this again." + exit 1 +fi +# Sets CHROME and the PYTHON array (it can be several words, like "py -3"). +eval "$("$SCRIPT_DIR/line-capture-preflight.sh" --print-env)" + +# Windows Chrome cannot use WSL paths or reach the recorder inside WSL. +if grep -qi microsoft /proc/version 2>/dev/null && [[ "$CHROME" == *.exe ]]; then + echo "Windows chrome.exe does not work from WSL. Use a Linux Chrome or Chromium in WSL, or run this from Git Bash on Windows." >&2 + exit 1 +fi + +warn() { echo " WARNING: $*" >&2; } + +# Chrome on Windows needs native paths; Git Bash paths only convert implicitly +# for some arguments. +native_path() { + if command -v cygpath >/dev/null 2>&1; then + cygpath -w "$1" + else + echo "$1" + fi +} + +# Directories that can hold the user's normal Chrome profiles, per OS. +normal_chrome_dirs() { + case "$(uname -s)" in + Darwin) + echo "$HOME/Library/Application Support/Google/Chrome" ;; + MINGW* | MSYS* | CYGWIN*) + local local_appdata="${LOCALAPPDATA:-$HOME/AppData/Local}" + command -v cygpath >/dev/null 2>&1 && local_appdata="$(cygpath -u "$local_appdata")" + echo "$local_appdata/Google/Chrome/User Data" ;; + *) + echo "$HOME/.config/google-chrome" + echo "$HOME/.config/chromium" ;; + esac +} + OUTPUT_DIR="${1:-/tmp/line-chrome-extension-capture}" mkdir -p "$OUTPUT_DIR" NETLOG_FILE="$OUTPUT_DIR/line-net-log.json" @@ -56,7 +108,7 @@ seed_chrome_capture_profile() { # below keep the choices explicit and readable for future maintainers. touch "$profile_dir/First Run" - python3 - "$profile_dir" <<'PY' + "${PYTHON[@]}" - "$profile_dir" <<'PY' import json import sys from pathlib import Path @@ -139,10 +191,20 @@ PY find_latest_extension_dir() { local extension_id="$1" - local base="$HOME/Library/Application Support/Google/Chrome/Default/Extensions/$extension_id" - if [ -d "$base" ]; then - find "$base" -mindepth 1 -maxdepth 1 -type d | sort -V | tail -n 1 - fi + local chrome_dir base found + while IFS= read -r chrome_dir; do + base="$chrome_dir/Default/Extensions/$extension_id" + [ -d "$base" ] || continue + # Chrome's folders can be unreadable (macOS privacy protection answers + # "Operation not permitted"); skip those instead of aborting. + found="$(find "$base" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -V | tail -n 1)" + if [ -n "$found" ]; then + # First usable match only: Chrome and Chromium can both have it. + echo "$found" + return 0 + fi + warn "cannot read $base, skipping" + done < <(normal_chrome_dirs) } find_latest_profile_extension_dir() { @@ -150,14 +212,14 @@ find_latest_profile_extension_dir() { local extension_id="$2" local base="$profile_dir/Default/Extensions/$extension_id" if [ -d "$base" ]; then - find "$base" -mindepth 1 -maxdepth 1 -type d | sort -V | tail -n 1 + find "$base" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -V | tail -n 1 || true fi } read_manifest_version() { local extension_dir="$1" if [ -n "${extension_dir:-}" ] && [ -f "$extension_dir/manifest.json" ]; then - python3 - "$extension_dir/manifest.json" <<'PY' + "${PYTHON[@]}" - "$extension_dir/manifest.json" <<'PY' import json import sys from pathlib import Path @@ -171,9 +233,9 @@ PY } find_repo_extension_version() { - local client_file="$SCRIPT_DIR/../pkg/line/client.go" + local client_file="$REPO_ROOT/pkg/line/client.go" if [ -f "$client_file" ]; then - python3 - "$client_file" <<'PY' + "${PYTHON[@]}" - "$client_file" <<'PY' import re import sys from pathlib import Path @@ -193,7 +255,7 @@ stage_extension_dir() { fi local version - version="$(python3 - "$source_dir/manifest.json" <<'PY' + version="$("${PYTHON[@]}" - "$source_dir/manifest.json" <<'PY' import json import sys from pathlib import Path @@ -211,13 +273,21 @@ PY # Chrome Web Store extension directories contain _metadata/, which Chrome # rejects when loading as unpacked. Stage a clean copy for capture runs. + # An unreadable source must not abort the capture; skip with a warning. if command -v rsync >/dev/null 2>&1; then - rsync -a --delete --exclude '_metadata' "$source_dir/" "$dest/" + rsync -a --delete --exclude '_metadata' "$source_dir/" "$dest/" 2>/dev/null || { + warn "could not copy $source_dir, skipping this extension" + return 0 + } else local tmp="$dest.tmp" rm -rf "$tmp" mkdir -p "$tmp" - (cd "$source_dir" && tar --exclude './_metadata' -cf - .) | (cd "$tmp" && tar -xf -) + if ! { (cd "$source_dir" && tar --exclude './_metadata' -cf - .) | (cd "$tmp" && tar -xf -); } 2>/dev/null; then + rm -rf "$tmp" + warn "could not copy $source_dir, skipping this extension" + return 0 + fi rm -rf "$dest" mv "$tmp" "$dest" fi @@ -250,13 +320,6 @@ if [ -n "${CODEX_EXTENSION_DIR:-}" ] && [ -f "$CODEX_EXTENSION_DIR/manifest.json EXTENSION_DIRS+=("$CODEX_EXTENSION_DIR") fi -CHROME="${CHROME:-/Applications/Google Chrome.app/Contents/MacOS/Google Chrome}" -if [ ! -x "$CHROME" ]; then - echo "Chrome not found at $CHROME" - echo "Set CHROME env var to your Chrome binary path." - exit 1 -fi - FIRST_RUN=false if [ ! -d "$DEBUG_PROFILE" ]; then FIRST_RUN=true @@ -298,8 +361,9 @@ if [ "$FIRST_RUN" = true ]; then echo " The profile persists across capture dirs — you only need to do this once." echo "" fi -echo " 1. Open the LINE Chrome Extension and do your flow" -echo " 2. When done, close Chrome (Cmd+Q)" +echo " 1. Click the LINE extension icon in the toolbar (puzzle-piece menu if not pinned) and do your flow" +echo " Do not open DevTools on the extension: LINE has debugger traps and the recorder does not need it." +echo " 2. When done, quit Chrome completely (all windows, not just the tab)" echo "" # Chrome 146+ requires --user-data-dir for remote debugging. @@ -310,14 +374,18 @@ CHROME_ARGS=( --disable-features=SignInPromo,SigninInterception,ChromeSigninIntercept,DiceWebSigninInterception --remote-debugging-port=9222 --remote-allow-origins="*" - --user-data-dir="$DEBUG_PROFILE" - --log-net-log="$NETLOG_FILE" + --user-data-dir="$(native_path "$DEBUG_PROFILE")" + --log-net-log="$(native_path "$NETLOG_FILE")" --net-log-capture-mode=IncludeSensitive ) if [ ${#EXTENSION_DIRS[@]} -gt 0 ]; then + NATIVE_EXTENSION_DIRS=() + for ext_dir in "${EXTENSION_DIRS[@]}"; do + NATIVE_EXTENSION_DIRS+=("$(native_path "$ext_dir")") + done IFS=, - EXTENSION_ARG="${EXTENSION_DIRS[*]}" + EXTENSION_ARG="${NATIVE_EXTENSION_DIRS[*]}" unset IFS if [ -z "${LINE_PROFILE_EXTENSION_DIR:-}" ]; then CHROME_ARGS+=(--disable-extensions-except="$EXTENSION_ARG") @@ -325,22 +393,15 @@ if [ ${#EXTENSION_DIRS[@]} -gt 0 ]; then CHROME_ARGS+=(--load-extension="$EXTENSION_ARG") fi -START_URL="${LINE_CAPTURE_START_URL:-}" -if [ -z "$START_URL" ] && [ -n "${LINE_PROFILE_EXTENSION_DIR:-}" ] && [ -f "$LINE_PROFILE_EXTENSION_DIR/manifest.json" ]; then - START_URL="chrome-extension://ophjlpahpchlmihnnnihgmmeilfjmjjc/index.html" -fi -if [ -z "$START_URL" ] && [ -n "${LINE_EXTENSION_DIR:-}" ] && [ -f "$LINE_EXTENSION_DIR/manifest.json" ]; then - START_URL="chrome-extension://ophjlpahpchlmihnnnihgmmeilfjmjjc/index.html" -fi -if [ -z "$START_URL" ]; then - START_URL="about:blank" -fi +# about:blank on purpose: opening the extension's index.html as a tab behaves +# differently from the real popup, so the user opens LINE from the toolbar icon. +START_URL="${LINE_CAPTURE_START_URL:-about:blank}" "$CHROME" "${CHROME_ARGS[@]}" "$START_URL" 2>/dev/null & CHROME_PID=$! # Start the CDP capture (it waits for Chrome to be ready) -python3 "$SCRIPT_DIR/line-cdp-capture.py" "$CDP_FILE" & +"${PYTHON[@]}" "$SCRIPT_DIR/line-cdp-capture.py" "$CDP_FILE" & CDP_PID=$! # Wait for Chrome to exit diff --git a/.agents/skills/line-implement/REFERENCE.md b/.agents/skills/line-implement/REFERENCE.md index 649075ea..e6827ec3 100644 --- a/.agents/skills/line-implement/REFERENCE.md +++ b/.agents/skills/line-implement/REFERENCE.md @@ -5,8 +5,8 @@ Start with: - `AGENTS.md` / `CLAUDE.md` -- `.agents/commands/line-capture.md` -- `.agents/commands/line-analyze.md` +- `.agents/skills/line-capture/SKILL.md` +- `.agents/skills/line-analyze/SKILL.md` - `.agents/commands/line-implement.md` - `docs/protocol/README.md` - `docs/protocol/endpoints/README.md` @@ -61,7 +61,7 @@ IDs, MIDs, contacts, message bodies, media object IDs, QR/login artifacts, and E2EE metadata. - Use `/tmp/line-chrome-extension-capture/-` with `umask 077`. -- Prefer the persistent dedicated Chrome profile created by `scripts/line-chrome-capture.sh` +- Prefer the persistent dedicated Chrome profile created by `.agents/skills/line-capture/scripts/line-chrome-capture.sh` at `~/.cache/line-chrome-extension-capture/chrome-debug-profile`. - Ask before using the user's normal Chrome profile. - Ask before stopping a running bridge or invalidating an active LINE session. @@ -88,7 +88,7 @@ Use CDP capture first because it includes SSE event bodies: ```bash umask 077 CAPTURE_DIR="/tmp/line-chrome-extension-capture/$(date +%Y%m%d-%H%M%S)-feature-slug" -bash scripts/line-chrome-capture.sh "$CAPTURE_DIR" +bash .agents/skills/line-capture/scripts/line-chrome-capture.sh "$CAPTURE_DIR" ``` The launcher stages clean unpacked copies of the LINE and Codex Chrome extension @@ -112,15 +112,15 @@ sign-in screen, choose "Stay Signed Out" and continue to LINE login only. When Chrome closes: ```bash -python3 scripts/parse-line-traffic.py "$CAPTURE_DIR/line-cdp-capture.json" "$CAPTURE_DIR/line-cdp-capture-parsed.json" -python3 scripts/redact-line-capture.py "$CAPTURE_DIR/line-cdp-capture-parsed.json" +python3 .agents/skills/line-analyze/scripts/parse-line-traffic.py "$CAPTURE_DIR/line-cdp-capture.json" "$CAPTURE_DIR/line-cdp-capture-parsed.json" +python3 .agents/skills/line-analyze/scripts/redact-line-capture.py "$CAPTURE_DIR/line-cdp-capture-parsed.json" ``` Use the net-log only as a backup: ```bash -python3 scripts/parse-line-traffic.py "$CAPTURE_DIR/line-net-log.json" "$CAPTURE_DIR/line-net-log-parsed.json" -python3 scripts/redact-line-capture.py "$CAPTURE_DIR/line-net-log-parsed.json" +python3 .agents/skills/line-analyze/scripts/parse-line-traffic.py "$CAPTURE_DIR/line-net-log.json" "$CAPTURE_DIR/line-net-log-parsed.json" +python3 .agents/skills/line-analyze/scripts/redact-line-capture.py "$CAPTURE_DIR/line-net-log-parsed.json" ``` ## Analysis Checklist diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 00000000..2b7a412b --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file