Scout — 21 candidates

[github-actions]

Candidates (21)

• "It's Hard to Eval" Is a Product Smell — eval-philosophy · article
  Source: hamel-husain | Proposed slug: eval-smell-product-design
  Blurb: Difficulty evaluating AI outputs is a product design failure, not an inherent property. Three case studies (data agent, PE curriculum builder, workers-comp report) show how designing for user verification — provenance, progressive disclosure, diffs, cited findings — makes outputs checkable and evals tractable.
  Rationale: Strong, substantive piece from a practitioner with 3 years focused on evals. The core argument — that "hard to eval" is a product smell, not a technical constraint — is original and actionable. Goes well beyond philosophy into concrete before/after interface designs with reproducible principles (provenance, progressive disclosure, smaller checkable units). Aligns squarely with eval-philosophy as foundational thinking on how to structure evaluation loops upstream at the product design stage. Hamel Husain is a credible voice in this space with multiple related works cited. No substantially similar resource in the guide covers this product-design-first angle on evalability.

• Patterns for Building Cybersecurity Evals — eval-frameworks · article
  Source: eugene-yan | Proposed slug: cybersecurity-evals-patterns
  Blurb: Four-component pattern for cybersecurity evals: sandboxed target, difficulty-modulating inputs, tools, and a grader. Concrete design decisions for evaluating agents on offensive/defensive security tasks.
  Rationale: Eugene Yan is a credible practitioner author. The piece covers reproducible eval harness design for cybersecurity tasks — a niche but important intersection of eval-frameworks and security. It's not a generic listicle; it describes a concrete structural pattern (sandboxed target + inputs + tools + grader) with practical implications for anyone building security-domain agents. Cross-cuts both eval-frameworks and security sections; eval-frameworks is the stronger fit given the focus is on harness construction rather than threat modelling.

• Computer-Use and TOCTOU: What You Click Is Not What You Get! — security · article
  Source: embrace-the-red | Proposed slug: toctou-agent-computer-use
  Blurb: TOCTOU (time-of-check/time-of-use) race condition exploited against computer-use agents: what the agent visually confirms differs from what it actually clicks. Reproduces Jun Kokatsu's ChatGPT Operator vulnerability and generalizes the attack pattern to agentic UI interaction.
  Rationale: Covers an agent-specific attack surface (TOCTOU race conditions in computer-use agents) that is distinct from classic prompt injection. The research is reproducible — it explicitly reproduces a known CVE/advisory against ChatGPT Operator and generalizes the technique. Fits squarely in the security section's mandate for agent-specific vulnerability taxonomies and tool misuse. Embrace the Red is a well-regarded practitioner source for agentic security research.

• How we built saga rollbacks for Cloudflare Workflows — durable-execution · article
  Source: cloudflare-ai-agents | Proposed slug: cloudflare-workflows-saga-rollbacks
  Blurb: Saga-pattern compensation in a durable execution engine: each step.do() takes an optional compensating action, letting Cloudflare Workflows automatically unwind partial progress on failure. Covers the design trade-offs, execution model, and implementation details.
  Rationale: Substantive engineering post from Cloudflare on a meaningful new capability in their durable execution runtime. Saga-pattern rollbacks are a non-trivial design problem for agentic workflows (partial-progress recovery is a core concern), and the post covers architecture decisions and implementation — not just a feature announcement. Directly relevant to the durable-execution section.

• Build your own vulnerability harness — coding-agent-infra · article
  Source: cloudflare-ai-agents | Proposed slug: cloudflare-vulnerability-harness
  Blurb: Multi-stage vulnerability discovery harness with automated triage loop: state control design, adversarial review to suppress false positives, and strategies for routing around LLM context limits.
  Rationale: Substantive technical architecture post from Cloudflare engineering covering harness design, multi-stage orchestration, state management, adversarial review loops, and context-window handling — all directly relevant to coding-agent-infra. Reproducible patterns (state controls, adversarial reviewer, context routing) make it actionable, not just marketing. Fits the "harness design, hook systems, tool allowlists, and operational patterns" mandate of that section exactly.

• Improving Agents is a Data Mining Problem — eval-philosophy · article
  Source: langchain-blog | Proposed slug: improving-agents-data-mining
  Blurb: Agent improvement reframed as trace mining: harvest production runs to surface failure modes, fine-tune cheap judge models from frontier LLM labels, then hill-climb with evals — a concrete flywheel from telemetry to performance gain.
  Rationale: Substantive vendor post with reproducible techniques — the data-mining framing (mine traces → label with frontier LLM → distill cheap judge → eval gate) is a concrete, reusable architecture pattern. Directly relevant to eval-philosophy and the production-trace-back-into-eval loop. Not a press release or listicle. No substantially similar resource already present in the guide.

• Tuning the harness, not the model: a Nemotron 3 Ultra playbook — coding-agent-infra · article
  Source: langchain-blog | Proposed slug: tuning-harness-not-model-nemotron
  Blurb: Matched Opus 4.8's best agent run at ~8x lower cost by tuning only scaffolding — tool allowlists, prompt structure, and harness config — around Nemotron 3 Ultra, leaving weights untouched.
  Rationale: Directly addresses harness design and operational patterns for coding agents — the core topic of the coding-agent-infra section. The central claim (closing a performance gap vs. a frontier model purely through scaffolding changes at 8x lower cost) is a reproducible, concrete technique with clear infrastructure implications. Vendor blog origin is acceptable here because the post contains specific architectural decisions rather than product marketing.

• How to Use RLMs in Deep Agents — multi-agent-frameworks · article
  Source: langchain-blog | Proposed slug: rlms-deep-agents-langchain
  Blurb: Recursive language models (RLMs) as a fix for context rot: agents write code that fans out work to subagents over context chunks via grep/map/reduce patterns. Implemented in Deep Agents via dynamic subagent dispatch and a lightweight code interpreter; benchmarked on OOLONG long-context reasoning.
  Rationale: Substantive vendor post with reproducible architecture decisions and benchmark evidence. Introduces a concrete pattern (RLMs / programmatic subagent fan-out) for a real production failure mode (context rot / long-context degradation), rather than being a marketing piece. OOLONG benchmark grounding makes claims verifiable. Relevant to multi-agent orchestration design and context-window management strategy.

• How Deep Agents Run Untrusted Code Without a Sandbox — sandboxing · article
  Source: langchain-blog | Proposed slug: deep-agents-wasm-quickjs-untrusted-code
  Blurb: In-process isolation via WASM + QuickJS as a lightweight alternative to full VM sandboxes — covers least-privilege capability gating and snapshot-based durable pauses for agent code execution.
  Rationale: Substantive technical content on a specific, under-documented sandboxing approach (WASM + QuickJS in-process isolation) with direct infrastructure implications for code-executing agents. Covers capability least-privilege and snapshot-based pausing — reproducible architectural decisions rather than marketing. Vendor blog but technically grounded. Fits sandboxing section well; no substantially similar resource covering this WASM/QuickJS angle appears to exist.

• Benchmarking Single Agent Performance — eval-frameworks · article
  Source: langchain-blog | Proposed slug: langchain-react-agent-benchmarking
  Blurb: Empirical study of how instruction count and tool-set size degrade ReAct agent performance across frontier models (claude-3.5-sonnet, gpt-4o, o1, o3-mini) — concrete signal for capacity-planning a single-agent harness.
  Rationale: Substantive benchmark with reproducible methodology across multiple frontier models. Directly relevant to practitioners designing single-agent systems — the scaling behaviour of instruction count and tool count is an under-documented failure mode. Vendor blog post but contains concrete empirical results, not marketing. Fits eval-frameworks as a practical benchmark study.

• Evaluating Large Language Models With OpenEvals — eval-frameworks · article
  Source: langchain-blog | Proposed slug: langchain-openevals-agenteval
  Blurb: Pre-built evaluator library (OpenEvals + AgentEvals) covering LLM-as-judge, structured-output correctness, and agent trajectory scoring — usable standalone or wired into LangSmith.
  Rationale: Covers a concrete, production-oriented eval framework with evaluators purpose-built for agentic trajectories — directly relevant to eval-frameworks. LangChain vendor content but technically substantive: describes specific evaluator types, interfaces, and composition patterns rather than pure marketing. Trajectory eval is a meaningful agent-specific capability not well-served by most generic eval libraries.

• Designing Efficient Verifiers for Legal Agents — eval-philosophy · article
  Source: langchain-blog | Proposed slug: designing-efficient-verifiers-legal-agents
  Blurb: Harvey + LangChain Labs study on making LLM-as-judge verifiers cheaper and more reliable for legal-agent evaluation and post-training signal. Covers verifier calibration, cost-accuracy tradeoffs, and lessons applicable beyond the legal domain.
  Rationale: Vendor blog post from LangChain, but co-authored with Harvey (a production legal AI shop) and grounded in a real applied study on verifier design — a concrete evaluation infrastructure problem. Covers reproducible techniques (verifier calibration, cost/reliability tradeoffs) with practical implications for anyone building LLM-as-judge eval pipelines. Fits eval-philosophy as foundational thinking on how to build reliable, cost-efficient evaluation loops. Substantive enough to clear the vendor-blog bar.

• Context Management for Deep Agents — memory-systems · article
  Source: langchain-blog | Proposed slug: context-management-deep-agents
  Blurb: How Deep Agents SDK tackles context rot in long-running tasks: offloading, summarization, and filesystem abstraction as first-class context management primitives.
  Rationale: Covers concrete, reproducible techniques (offloading, summarization, filesystem abstraction) for a real engineering problem — context rot in long-horizon agents. Vendor blog but contains architecture decisions and SDK-level design patterns, not marketing. Directly relevant to memory-systems section. Summary is thin so confidence is moderate, but the topic and framing suggest substantive technical content worth including.

• How we build evals for Deep Agents — eval-philosophy · article
  Source: langchain-blog | Proposed slug: langchain-evals-deep-agents
  Blurb: Practical eval construction for deep/long-horizon agents: data curation strategy, behavior-targeted metrics, and testing patterns grounded in real agent failure modes.
  Rationale: Vendor blog from LangChain, but covers reproducible eval construction techniques specific to deep/long-horizon agents — data curation, metric selection, and behavioral testing strategies. This is substantive methodology content, not marketing. Fits well in eval-philosophy as a practitioner's first-person account of building agent evals. No substantially similar resource on deep/long-horizon agent eval pipelines currently in the guide.

• Building a 100x Cheaper Trace Judge with Fireworks — eval-philosophy · article
  Source: langchain-blog | Proposed slug: langchain-fireworks-trace-judge
  Blurb: Fine-tune a small open model on error signals mined from production traces to match frontier-model judge performance at ~1% of the cost. Covers the distillation pipeline, signal extraction from real agent traces, and calibration against GPT-4-class judges.
  Rationale: Substantive vendor post with reproducible technique: using production trace data to distill a cheap, fine-tuned judge model. Directly relevant to eval infrastructure for agentic systems. The cost/quality trade-off methodology is practically useful and not covered by existing resources on LLM-as-judge patterns.

• Iterating Towards LLM Reliability with Evaluation Driven Development — eval-philosophy · article
  Source: langchain-blog | Proposed slug: iterating-towards-llm-reliability-edd
  Blurb: Dosu's production account of evaluation-driven development: building offline eval suites first, using production traces to seed regression datasets, and closing the loop between monitoring and iteration.
  Rationale: Vendor blog post but passes the bar: it documents reproducible engineering practices (eval-before-ship discipline, trace-to-dataset pipelines, production monitoring feedback loops) from a real production system (Dosu). The core methodology — evaluation-driven development as a workflow — is substantive and transferable beyond LangSmith. Fits eval-philosophy as a practitioner account of why and how to structure evals around reliability rather than vibes.

• Aligning LLM-as-a-Judge with Human Preferences — eval-philosophy · article
  Source: langchain-blog | Proposed slug: aligning-llm-as-a-judge-human-preferences
  Blurb: Self-improving LLM-as-a-Judge evaluators via few-shot learning: how human-preference signals can be folded back into the evaluator to close the gap between automated scores and human judgment.
  Rationale: Vendor blog, but contains reproducible technique (few-shot calibration of LLM judges against human preference labels) with direct infrastructure implications for eval pipelines. The self-improving evaluator pattern is substantive and goes beyond marketing. Fits eval-philosophy as foundational thinking on closing the vibes-vs-automated-score gap; could also sit in eval-frameworks but the conceptual argument earns philosophy placement.

• Building LangGraph: Designing an Agent Runtime from First Principles — multi-agent-frameworks · article
  Source: langchain-blog | Proposed slug: building-langgraph-runtime
  Blurb: First-principles walkthrough of LangGraph's design: why graph-based execution, how control flow and durability were architected, and the tradeoffs made to support production-scale agents.
  Rationale: Vendor blog, but this is a first-party architectural deep-dive from the LangGraph authors explaining design decisions — control flow model, durability primitives, execution model — rather than a marketing piece. That makes it substantive and non-duplicative of generic LangGraph usage tutorials. Relevant to multi-agent-frameworks as it explains why the runtime is designed the way it is, which is directly useful for engineers evaluating or building on top of it. Conditionally includeable pending confirmation the post contains genuine architectural depth (the summary is thin, but "from first principles" framing and LangChain's track record of technical posts supports inclusion).

• How to turn Claude Code into a domain specific coding agent — coding-agent-infra · article
  Source: langchain-blog | Proposed slug: claude-code-domain-specific-agent
  Blurb: Context engineering techniques for specializing Claude Code to domain-specific libraries — CLAUDE.md, tool allowlists, example injection — with eval results comparing approaches.
  Rationale: Substantive vendor post with reproducible techniques: compares concrete context engineering strategies (CLAUDE.md, tool configuration, few-shot examples) for domain-specialization of a coding agent harness, backed by evaluation results. Directly relevant to coding-agent-infra harness design and context engineering. Not a marketing piece or listicle — contains architecture decisions and measurable outcomes.

• Evaluating Deep Agents: Our Learnings — eval-philosophy · article
  Source: langchain-blog | Proposed slug: evaluating-deep-agents-langchain
  Blurb: Five evaluation patterns for long-horizon agents: bespoke unit tests, single-step validation, full-turn checks, multi-turn simulations, and environment setup — grounded in production experience.
  Rationale: Covers reproducible evaluation patterns specific to long-horizon ("deep") agents, including multi-turn simulation and environment setup — topics underserved in generic LLM eval literature. Vendor origin is acceptable here because the content is technique-focused and actionable, not a product announcement. Fits eval-philosophy as foundational thinking on structuring agent eval loops, though it could also sit in eval-frameworks; the emphasis on conceptual patterns tips it toward philosophy.

• Fault Tolerance in LangGraph: Retries, Timeouts and Error Handlers — durable-execution · article
  Source: langchain-blog | Proposed slug: langgraph-fault-tolerance
  Blurb: Three fault-tolerance primitives inside LangGraph — RetryPolicy (backoff), TimeoutPolicy (wall-clock and idle caps), and error_handler (post-retry cleanup) — and how they compose. Covers SAGA pattern for multi-step workflows with real-world side effects.
  Rationale: Vendor blog, but contains reproducible, concrete architecture decisions — specific named primitives, composition patterns, and the SAGA pattern applied to agentic workflows. Directly relevant to the durable-execution section's focus on fault-tolerance and workflow resilience. Substantive enough to clear the vendor-blog bar.