Search results

video_classification: 4/10 tools are pass stubs, paper claims 25 tools but repo has 10

Status: Open.

#9 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 23, 2026

know_your_business: CSV labels for 'awaiting information' contradict SOP escalation rules

Status: Open.

#8 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 23, 2026

email_intent: unresolved git merge conflicts in sop.txt, tools.py, and test_set_with_outputs.csv

Status: Open.

#7 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 23, 2026

video_annotation: 20 of 26 tool implementations are pass-only stubs returning None

Status: Open.

#6 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 23, 2026

traffic_spoofing_detection: 39 Medium-risk tasks labeled "Warning Issued" instead of "Temporary Suspension"

Status: Open.

#5 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 22, 2026

referral_abuse_detection_v1: CSV labels follow closure-priority rule that contradicts SOP scoring

Status: Open.

#4 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 22, 2026

warehouse_package_inspection: tools use mock arithmetic instead of CSV lookups

Status: Open.

#3 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 22, 2026

Bug: content_flagging benchmark is non-deterministic and internally inconsistent

Status: Open.

#2 In amazon-science/SOP-Bench;

· vivekhaldar opened on Mar 22, 2026