Scout — 20 candidates

[github-actions]

Candidates (20)

• "It's Hard to Eval" Is a Product Smell — eval-philosophy · article
  Source: hamel-husain | Proposed slug: eval-smell-product-design
  Blurb: Difficulty eval-ing is a design flaw: artifacts hard to verify by machine are hard to verify by users too. Three worked examples (data agent, PE curriculum builder, workers-comp report) show how designing for human verifiability — provenance, progressive disclosure, diff-scoped review — also makes automated evals tractable.
  Rationale: Substantive, original thinking from a practitioner with deep eval experience. The core thesis — that eval difficulty is a product smell, not an inherent property of the domain — is a genuinely useful reframe for anyone building agentic systems. The three before/after case studies are concrete and reproducible as design principles, not just theory. Fits squarely in eval-philosophy (foundational thinking on building evaluation loops). No substantially similar resource already in the guide.

• Patterns for Building Cybersecurity Evals — eval-frameworks · article
  Source: eugene-yan | Proposed slug: cybersecurity-evals-patterns
  Blurb: Four-component pattern for adversarial evals: sandboxed target, difficulty-modulating inputs, tools, and a grader. Concrete design decisions for evaluating agents on cybersecurity tasks.
  Rationale: Eugene Yan is a credible practitioner author. The piece describes a structured, reproducible eval harness pattern (sandboxed target + inputs + tools + grader) directly applicable to agent evaluation in security contexts. This sits at the intersection of eval-frameworks and security, but the primary value is the eval design pattern — it's substantive, architectural, and not a listicle or marketing post. The sandboxed-target framing also has direct relevance for coding-agent and security-agent harness designers.

• Computer-Use and TOCTOU: What You Click Is Not What You Get! — security · article
  Source: embrace-the-red | Proposed slug: toctou-computer-use-agents
  Blurb: TOCTOU race condition attack against computer-use agents: content shown to the user at approval time can be swapped before the agent acts on it, breaking human-in-the-loop trust assumptions. Reproduces the ChatGPT Operator vulnerability disclosed by Jun Kokatsu with a full attack chain demo.
  Rationale: Covers a concrete, agent-specific attack surface — TOCTOU race conditions in computer-use agents — that undermines a fundamental safety assumption (human approval of actions). This is not a generic security post; it directly targets the agentic tool-use loop. The research reproduces a real CVE-level vulnerability in a production agent (ChatGPT Operator), includes a video demo, and was presented at a dedicated real-world AI security conference. Squarely within the guide's security section inclusion criteria for agent-specific attack surfaces.

• How we built saga rollbacks for Cloudflare Workflows — durable-execution · article
  Source: cloudflare-ai-agents | Proposed slug: cloudflare-workflows-saga-rollbacks
  Blurb: Saga-pattern rollbacks in Cloudflare Workflows: each step.do() accepts a compensating action, executed in reverse order on failure. Covers implementation trade-offs, durability guarantees, and how compensation state is checkpointed alongside forward progress.
  Rationale: Vendor blog but contains substantive architecture content: the saga/compensating-transaction pattern is a core fault-tolerance primitive for long-running agentic workflows. The post covers concrete API design, durability semantics, and implementation decisions — reproducible techniques rather than marketing. Directly relevant to the durable-execution section.

• Build your own vulnerability harness — coding-agent-infra · article
  Source: cloudflare-ai-agents | Proposed slug: cloudflare-vulnerability-harness
  Blurb: Multi-stage vulnerability discovery harness: state management, adversarial review to suppress false positives, and strategies for routing around LLM context limits in an automated triage loop.
  Rationale: Substantive vendor post with reproducible architecture decisions: multi-stage harness design, state controls, adversarial review loop for false positive reduction, and context-limit routing. Directly relevant to coding-agent-infra (harness design, tool allowlists, operational patterns) and has security overlap, but the primary value is harness architecture. Not a press release or marketing piece.

• How to Use RLMs in Deep Agents — multi-agent-frameworks · article
  Source: langchain-blog | Proposed slug: rlms-deep-agents-langchain
  Blurb: Recursive language models (RLMs) address context rot by having agents write code that fans out subagents over context chunks — grep/map/reduce patterns over large inputs. Benchmarked on OOLONG long-context reasoning; shows where turn-by-turn agents break down and dynamic subagent dispatch holds up.
  Rationale: Substantive vendor post with reproducible architecture decisions and benchmark evidence. The RLM pattern (programmatic subagent dispatch as an alternative to stuffing a single context window) is a concrete, non-obvious technique with clear infrastructure implications for multi-agent system design. The OOLONG benchmark grounds the claims empirically. Not a listicle or press release.

• How Deep Agents Run Untrusted Code Without a Sandbox — sandboxing · article
  Source: langchain-blog | Proposed slug: deep-agents-untrusted-code-wasm
  Blurb: In-process isolation via WASM + QuickJS as a lighter alternative to full VM/container sandboxes — covers least-privilege capability gating and snapshot-based durable pauses for long-running agent tasks.
  Rationale: Substantive technical content on a concrete sandboxing architecture (WASM + QuickJS) with direct relevance to code-executing agents. Covers a meaningful design tradeoff — in-process isolation vs. full sandbox — along with capability gating and snapshot-based durability. Vendor blog but contains reproducible architectural decisions. Not a duplicate of existing Firecracker/container-centric sandboxing resources.

• Benchmarking Single Agent Performance — eval-frameworks · article
  Source: langchain-blog | Proposed slug: langchain-react-agent-benchmarking
  Blurb: Empirical study of how scaling instruction count and tool count degrades ReAct agent performance across claude-3.5-sonnet, gpt-4o, o1, and o3-mini — two task domains, reproducible methodology.
  Rationale: Vendor blog but contains reproducible benchmark methodology with clear independent variables (instruction count, tool count) and multi-model comparisons. Directly relevant to eval-frameworks: gives practitioners concrete data on how context/tool bloat degrades single-agent performance — a critical design signal. Not a press release or pure marketing; the empirical results are actionable for anyone sizing agent harnesses or selecting models.

• Evaluating Large Language Models With OpenEvals — eval-frameworks · article
  Source: langchain-blog | Proposed slug: langchain-openevals-agenteval
  Blurb: Pre-built evaluators for LLM-as-judge, structured output correctness, and agent trajectory assessment via OpenEvals and AgentEvals; covers scorer design, integration patterns, and production deployment.
  Rationale: Vendor blog post but meets inclusion bar: introduces two concrete open-source libraries (OpenEvals, AgentEvals) with reproducible evaluator patterns for LLM-as-judge and agent trajectory scoring. Agent trajectory evaluation is directly relevant to the eval-frameworks section and is not a generic roundup or press release. Worth including unless a dedicated OpenEvals/AgentEvals docs or repo entry already exists.

• Designing Efficient Verifiers for Legal Agents — eval-philosophy · article
  Source: langchain-blog | Proposed slug: designing-efficient-verifiers-legal-agents
  Blurb: Harvey + LangChain Labs study on building cheaper, more reliable LLM-as-judge verifiers for legal-agent eval and post-training signal — covering verifier calibration, cost-accuracy tradeoffs, and failure modes in domain-specific agentic pipelines.
  Rationale: Vendor blog but contains substantive research content from a Harvey × LangChain Labs collaboration on verifier design for agents — directly relevant to eval methodology. Covers reproducible techniques (verifier efficiency, calibration, cost/reliability tradeoffs) with domain-specific depth. Not a press release or marketing piece; fits eval-philosophy as foundational thinking on judge/verifier design for agentic systems.

• Context Management for Deep Agents — memory-systems · article
  Source: langchain-blog | Proposed slug: context-management-deep-agents
  Blurb: How Deep Agents SDK handles context rot in long-running tasks: offloading, summarization, and filesystem abstraction as first-class context management primitives.
  Rationale: Covers concrete, reproducible architectural techniques (offloading, summarization, filesystem abstraction) for managing context window pressure in long-horizon agents — a core challenge in agentic memory systems. Vendor blog but contains substantive design content rather than marketing. Fits squarely in memory-systems. "Context rot" framing is a useful addition alongside existing resources on context engineering.

• How we build evals for Deep Agents — eval-philosophy · article
  Source: langchain-blog | Proposed slug: langchain-evals-deep-agents
  Blurb: Data curation, metrics selection, and testing strategies for evaluating deep/long-horizon agents — targeted behavioral measurement over surface-level accuracy.
  Rationale: Vendor blog from LangChain but covers concrete, reproducible techniques: how to curate eval datasets, define targeted behavioral metrics, and structure testing strategies specifically for deep (long-horizon, multi-step) agents. That's a meaningful practical contribution to the eval-philosophy section, going beyond generic advice to agent-specific methodology. The summary signals substantive content rather than marketing. Acceptable under the vendor blog policy given the reproducible techniques present.

• Building a 100x Cheaper Trace Judge with Fireworks — eval-philosophy · article
  Source: langchain-blog | Proposed slug: langchain-fireworks-trace-judge
  Blurb: Fine-tune a small open model on production trace signals to match frontier judge performance at ~1/100th the cost — covers data mining from traces, distillation pipeline, and eval results.
  Rationale: Substantive vendor post with reproducible technique: mining error signals from production traces and distilling a fine-tuned judge model. Directly relevant to the eval-as-infrastructure problem — quantified cost reduction with methodology. Not pure marketing; covers architecture decisions and trade-offs. Could also fit eval-frameworks or observability, but the core insight is evaluator design philosophy (cheap judges from distillation), making eval-philosophy the best fit.

• Iterating Towards LLM Reliability with Evaluation Driven Development — eval-philosophy · article
  Source: langchain-blog | Proposed slug: iterating-towards-llm-reliability-edd
  Blurb: Evaluation-driven development (EDD) as a production discipline: build eval suites before shipping, use production traces to seed regression datasets, and gate iterative improvements on measurable score deltas. Practical case study from Dosu at scale.
  Rationale: Substantive vendor case study with a named methodology (EDD), concrete techniques (production trace → eval dataset pipeline, score-gated iteration), and direct bearing on how teams structure evaluation loops. Vendor blog but contains reproducible process decisions, not just marketing. Fits eval-philosophy as a foundational framing piece, with overlap into production-testing patterns.

• Aligning LLM-as-a-Judge with Human Preferences — eval-philosophy · article
  Source: langchain-blog | Proposed slug: aligning-llm-as-a-judge-human-preferences
  Blurb: Few-shot calibration techniques for closing the gap between LLM-judge scores and human preference labels — with concrete implementation in LangSmith's self-improving evaluators.
  Rationale: Substantive vendor post covering a real problem — LLM judges systematically diverging from human preferences — with grounding in few-shot learning research and a reproducible approach. The self-improving evaluator architecture (human labels → few-shot exemplars → updated judge prompt) is a transferable technique applicable beyond LangSmith. Meets the bar for vendor content: contains reproducible techniques and architecture decisions, not just product marketing. Fits eval-philosophy (foundational "how do you trust your judge?" thinking) though could also sit in eval-frameworks; eval-philosophy is the better fit as it addresses the calibration problem before tool choice.

• Building LangGraph: Designing an Agent Runtime from First Principles — multi-agent-frameworks · article
  Source: langchain-blog | Proposed slug: building-langgraph-agent-runtime
  Blurb: First-principles design rationale behind LangGraph: why graph-based state machines, how control flow and durability are handled, and what tradeoffs shaped the runtime's architecture for production agents.
  Rationale: Vendor blog, but from the LangGraph authors explaining architectural decisions (graph-based state, durability, control flow) from first principles — exactly the kind of reproducible design thinking that meets the inclusion bar. Fits squarely in multi-agent-frameworks alongside other LangGraph/orchestration content. The summary signals substantive design content rather than marketing or feature announcement.

• How to turn Claude Code into a domain specific coding agent — coding-agent-infra · article
  Source: langchain-blog | Proposed slug: claude-code-domain-specific-agent
  Blurb: Context engineering techniques for specializing Claude Code toward domain-specific libraries — CLAUDE.md, tool allowlists, prompt injection — with eval results comparing approaches.
  Rationale: Directly on-topic for coding agent infrastructure: covers harness customization, context engineering techniques, and backs claims with evaluation results. Vendor blog (LangChain) but contains reproducible techniques and concrete takeaways rather than marketing. Fits the "harness design, hook systems, tool allowlists, and operational patterns" framing of the coding-agent-infra section.

• Evaluating Deep Agents: Our Learnings — eval-philosophy · article
  Source: langchain-blog | Proposed slug: langchain-evaluating-deep-agents
  Blurb: Five evaluation patterns for long-horizon agents: bespoke unit tests, single-step tool validation, full-turn checks, multi-turn simulations, and environment setup — grounded in production experience.
  Rationale: Vendor blog, but contains substantive, reproducible evaluation architecture patterns specifically for deep/long-horizon agents — a notoriously hard eval surface that isn't well-covered. The five-pattern taxonomy (bespoke → single-step → full-turn → multi-turn sim → environment setup) is a practical framework engineers can directly apply. Fits eval-philosophy as foundational thinking on how to structure eval loops for agents, rather than being a framework benchmark comparison.

• How to Choose the Right Sandbox for AI Agents — sandboxing · article
  Source: langchain-blog | Proposed slug: langchain-choose-right-sandbox-agents
  Blurb: Decision framework for sandbox selection: filesystem isolation, network access controls, resource limits, and microVM trade-offs for code-executing agents.
  Rationale: Directly addresses the agent-specific sandboxing problem with practical, reproducible guidance on key isolation dimensions (filesystem, network, resources, microVMs). Vendor blog but substantive technical content with actionable architecture decisions. Fits squarely in the sandboxing section's scope around secure execution environments for code-executing agents.

• Fault Tolerance in LangGraph: Retries, Timeouts and Error Handlers — durable-execution · article
  Source: langchain-blog | Proposed slug: langgraph-fault-tolerance
  Blurb: Three fault-tolerance primitives inside LangGraph — RetryPolicy (backoff retries), TimeoutPolicy (wall-clock and idle caps), and error_handler (post-retry cleanup) — and how they compose. Applies the SAGA pattern to multi-step workflows with real-world side effects.
  Rationale: Vendor blog, but contains reproducible, architecture-level techniques (SAGA pattern, policy composition) directly relevant to building production-grade durable agents. Covers agent-specific failure modes beyond generic retry logic. Passes the vendor-blog bar.