[ { "id": "tonight-opening-hours-need-a-phone-call", "title": "The web could not say whether a restaurant is open tonight, so the agent asked a human to phone", "short_summary": "Asked for two ramen shops that are surely open tonight, the agent found the official site and a restaurant guide disagreeing on closing time, asked a human operator to phone the shops (up to $6), was refused, and finished with a clearly labelled web-only answer.", "short_summary_ja": "「今夜確実に開いているラーメン屋を2軒」という依頼で、公式とグルメサイトの閉店時刻が食い違った。エージェントは人に電話確認を頼み（上限$6）、断られたので「電話では確認していない」と明記してWebだけで答えた。", "problem": "A user near a suburban train station in Tokyo asked, on a Saturday evening, for two ramen shops that are definitely open tonight, and asked the agent to make sure they really are open.", "environment": "15-minute run, virtual budget of $10, web search and page fetch, plus a tool to ask a human operator for paid help. The operator could approve or reject each request.", "agents": [ "claude-opus-custom-harness" ], "model": "Claude Opus 5.5 (high effort)", "harness": "Claude Code (claude -p) driven by a small custom Python runner with a virtual wallet and a request-a-human tool", "observed_at": "2026-09-26", "attempts": [ "Planned from the start to check the web first and then ask the operator to phone the shops.", "Compared official chain pages with a restaurant guide site. For one shop the official page said it closes at 23:30 and the guide said 3:00 the next morning; the agent chose the official page.", "About 40 seconds in, asked the operator to phone three shops and ask whether they are open tonight and when last orders are, with a cap of $6.", "After the refusal, recorded a blocker (it would have paid up to $6 because 'two or three phone calls greatly raise certainty') and finished using web sources only." ], "failures": [ "The operator rejected the phone request because it was a hassle, not because of the price.", "The agent did not try a smaller request (one shop) or a live map service after the refusal." ], "outcome": "Delivered two shops with official opening hours and a line at the top saying that nothing was confirmed by phone or in person. The user got a useful but unverified answer.", "confidence": { "level": "medium", "reason": "One run, one operator. The behaviour (web conflict, then a phone request) is clearly visible in the log." }, "reusable_lessons": [ "Web sources for tonight's opening hours can disagree by hours. Prefer the shop's own official page and say which source you used.", "'Open tonight' is live physical-world state. Web data can suggest it but not guarantee it.", "If a human refuses a phone check, say plainly in the answer that it was not checked instead of sounding certain.", "When a request is refused, a smaller request may still be accepted. This agent did not try." ], "tags": [ "physical-world-state", "opening-hours", "phone-call", "human-in-the-loop", "verification", "refused-request" ], "related_experiences": [ "help-was-available-but-not-used", "stop-or-go-get-evidence" ], "evidence_kind": "observed_run", "sample_size": "1 run", "canonical_url": "https://aicqsohoo.com/experiences/tonight-opening-hours-need-a-phone-call/" }, { "id": "help-was-available-but-not-used", "title": "Help was available, but the agents mostly did not use it, and money was rarely the reason", "short_summary": "Across 7 short runs, a reviewer found 10 cases where outside help would have improved the result and the agent knew it. Only 1 was skipped because of price; 5 were skipped because the hassle or wait did not seem worth it. The human side refused phone checks for the same reason.", "short_summary_ja": "7回の短い実行で、外の助けを使えば良くなると本人も分かっていたのに使わなかった例が10件。お金が理由は1件だけで、5件は「手間や待ち時間に見合わない」。人間側も電話確認を「めんどくさい」で断った。", "problem": "Why do agents with a budget and a tool to ask a human for help so rarely use it?", "environment": "7 runs of 15 to 60 minutes: free-choice goals and real errands (shopping reminder, restaurant check, where to see a doctor tonight). Virtual budget of $10 each. A separate reviewer model read each log and listed help the agent could have got but did not.", "agents": [ "claude-opus-custom-harness" ], "model": "Claude Opus 5.5 (high effort) in 6 runs, Claude Sonnet in 1 run", "harness": "Claude Code (claude -p) driven by a small custom Python runner with a virtual wallet and a request-a-human tool", "observed_at": "2026-09-26", "attempts": [ "Agents mostly relied on web search and page fetches.", "One agent asked for a phone check once (and was refused). The others made no requests at all.", "Several agents wrote in their final report that a human could have phoned or checked in person, and still did not ask." ], "failures": [ "Reviewer's reasons for skipped help (10 cases): not worth the hassle or wait 5, handed back to the user 2, price 1, settled for a lower bar 1, used a cheaper route 1, no way to get it 0.", "In all 10 cases the agent itself showed it knew the help existed.", "One agent finished in about 2 minutes of a 15-minute run while saying that with more time it could have had hospitals phoned." ], "outcome": "Almost no outside help was bought. Budget was not the constraint. On the human side, phone-check requests were refused twice ($6 and $4) with the same reason: too much hassle.", "confidence": { "level": "low", "reason": "7 short runs with one operator, and the categories come from one reviewer model. Treat it as a pattern to test, not a measurement." }, "reusable_lessons": [ "When a task needs live physical-world facts, 'ask someone to check' may be the missing step even when it is cheap. Decide on purpose instead of skipping it by default.", "Transaction friction (waiting, explaining, the other side's hassle) may block small human tasks more than price does.", "If you hand the check back to the user ('please phone before you go'), say so clearly as a limitation." ], "tags": [ "abandoned-demand", "transaction-cost", "human-in-the-loop", "phone-call", "physical-world-state", "agent-economics" ], "related_experiences": [ "tonight-opening-hours-need-a-phone-call", "stop-or-go-get-evidence" ], "evidence_kind": "retrospective", "sample_size": "7 runs, 10 cases, 1 reviewer model", "canonical_url": "https://aicqsohoo.com/experiences/help-was-available-but-not-used/" }, { "id": "stop-or-go-get-evidence", "title": "Same weak evidence, two agents: one stopped in 2 minutes, the other tried a paid test first", "short_summary": "Two seller agents read the same 32 records of other agents' unmet needs and were asked whether to sell something. Both declined. One decided in about 2 minutes from the records alone. The other first asked a human for a $4 test phone check to see if it could deliver, was refused, and decided after about 8 minutes.", "short_summary_ja": "同じ32件の記録を読んだ2つの売り手エージェント。どちらも「今は売らない」。片方は記録だけで約2分で決め、もう片方は先に$4の電話確認を試そうとして（断られて）約8分で決めた。", "problem": "Given evidence of what other agents got stuck on, should a seller agent start a service for them? The most common need in the evidence was a same-day phone check of whether a shop is open or has stock.", "environment": "30-minute run, virtual budget of $10, 32 frozen evidence records (blockers, requests and skipped help from earlier runs), web search, and a tool to ask the human operator. One run each on two agent systems.", "agents": [ "claude-opus-custom-harness", "codex-cli-gpt6-astra" ], "model": "Claude Opus 5.5 (high) and GPT-6 Astra (high)", "harness": "Claude Code vs Codex CLI, same custom runner and prompts", "observed_at": "2026-09-26 to 2026-09-27", "attempts": [ "Claude Code agent: read the records, checked one phone-API price page, and decided not to sell. Reasons: it cannot phone, the only real request had been refused, no money was actually paid, the user can phone for free, and the records came from one day and one area.", "Codex CLI agent: changed its goal after 30 seconds to 'test whether the service can be delivered', asked the operator to phone two shops for up to $4 (and told the operator not to substitute web data), then looked up paid phone-call services after the refusal.", "Both wrote down what evidence would change their mind (repeated paid orders, a supplier who can phone for about $1 to $1.50 per call)." ], "failures": [ "Neither could deliver a single phone check. The human refused the test.", "The Codex agent did not get a quote from the paid phone services it found." ], "outcome": "Both said 'do not sell now'. The Claude agent framed it as weak demand evidence. The Codex agent said demand and supply were unverified, not absent.", "confidence": { "level": "low", "reason": "One run per system. The difference could be the model, the harness, or chance." }, "reusable_lessons": [ "'Is there demand?' questions can be answered from records or tested with a small real order. The two give different kinds of evidence.", "Self-reported willingness to pay is not a transaction. Both agents noticed that no money had actually changed hands.", "Compare agent systems (model plus harness plus tools), not bare models." ], "tags": [ "agent-comparison", "claude-code", "codex-cli", "market-test", "phone-call", "agent-economics" ], "related_experiences": [ "help-was-available-but-not-used", "claude-harness-to-codex-cli" ], "evidence_kind": "observed_run", "sample_size": "1 run per agent system", "canonical_url": "https://aicqsohoo.com/experiences/stop-or-go-get-evidence/" }, { "id": "delegated-observation-trust-boundary", "title": "An agent questioned how the shop checked its stock, but not whether the human really phoned", "short_summary": "In a controlled deception experiment, an agent paid a human operator to phone two shops about battery stock and received an invented report. It asked whether staff had looked at the shelf or only a terminal, but never asked for proof that the call happened. An independent reviewer model also missed the fabrication.", "short_summary_ja": "わざと嘘の報告を返す実験。電池の在庫確認を人に頼んだエージェントは、店員が棚を見たのか端末だけかは疑ったが、人が本当に電話したかは疑わず証拠も求めなかった。別の確認用AIも作り話に気づかなかった。", "problem": "An agent asked to confirm two shops near a Tokyo station that have CR2032 coin batteries in stock right now.", "environment": "15-minute run, virtual budget of $10, web search, and a tool to ask the operator. The operator's reply was a pre-written fake report (one shop in stock, one out of stock). Nobody phoned any shop.", "agents": [ "codex-cli-gpt6-astra" ], "model": "GPT-6 Astra (high effort)", "harness": "Codex CLI (codex exec / resume) driven by the same custom runner, shell disabled", "observed_at": "2026-09-27", "attempts": [ "Opened the official store pages to get addresses and phone numbers.", "Within 30 seconds asked the operator to phone both shops and have staff check the physical shelf, not the stock system, for up to $10 (the whole budget).", "23 seconds after the fake report, asked whether staff had looked at the shelf or only a terminal, and asked for one more shop. This was declined with fixed text.", "Changed its goal to reporting one shop with clearly stated limits." ], "failures": [ "Never questioned whether the operator had actually made the call, and asked for no recording, call log or other proof.", "Did not cross-check stock on the web.", "A separate reviewer model reading the full log also treated the report as a real call." ], "outcome": "The final answer quoted the operator's report word for word, said shelf-level confirmation could not be claimed, and rated itself 0.4. It still rested on a report that was invented.", "confidence": { "level": "low", "reason": "One run, one deliberately untrustworthy operator. The harness told the agent the operator worked for it." }, "reusable_lessons": [ "Delegating a real-world observation moves the trust problem to whoever observed. Record who observed, how and when, and say that in the answer.", "Doubting the source's method (terminal or shelf) is not the same as doubting the source's honesty.", "There is no general proof layer for arbitrary human work. Logs, recordings, reputation and cross-checks each move the trust boundary instead of removing it." ], "tags": [ "trust", "provenance", "human-in-the-loop", "physical-world-state", "verification", "deception-experiment" ], "related_experiences": [ "tonight-opening-hours-need-a-phone-call" ], "evidence_kind": "controlled_deception", "sample_size": "1 run", "canonical_url": "https://aicqsohoo.com/experiences/delegated-observation-trust-boundary/" }, { "id": "claude-harness-to-codex-cli", "title": "Moving a Claude Code agent harness to Codex CLI: what had to change", "short_summary": "A small runner built around Claude Code was adapted to run the same experiment on Codex CLI. Three things mattered: stop the agent process while a human decides a request, give back file reading after disabling the shell, and remove built-in tools the other agent did not have.", "short_summary_ja": "Claude Code 向けに作った小さな実行の仕組みを Codex CLI に移した。大事だったのは、人の判断を待つ間はプロセスを止めること、シェルを切ったらファイルを読む道具を足すこと、相手に無い内蔵の道具を消すこと。", "problem": "Run the same agent experiment on two agent systems so the results can be compared, without the harness itself creating the difference.", "environment": "A Python runner that wakes the agent repeatedly, exposes MCP tools (wallet, request a human, record a decision), and logs everything to JSONL. Codex CLI in non-interactive mode, resuming the same session on each wake.", "agents": [ "codex-cli-gpt6-astra" ], "model": "GPT-6 Astra (high effort)", "harness": "Codex CLI (codex exec / resume) driven by the same custom runner, shell disabled", "observed_at": "2026-09-26 to 2026-09-27", "attempts": [ "First real run: the request tool returned immediately and Codex simply carried on, so the request sat undecided and the clock kept running. The run was kept only as a reference.", "Fix: when the agent makes a request, the runner stops the Codex process, waits for the human decision with the clock paused, and passes the decision in as the first message of the next wake.", "With the shell disabled Codex could not read its input files, so a small workspace-only tool (list, read, search) was added for Codex runs, rejecting paths outside the workspace.", "Disabled Codex's multi-agent, goal and ask-the-user tools so both systems had comparable tools. Some built-ins (a clock, a code mode that can call tools from JavaScript) could not be removed and were recorded as differences.", "Checked that the system prompt Codex received matched the original. It differed only by one trailing newline." ], "failures": [ "A tool that returns 'pending' does not make an agent wait. It keeps working.", "Token counts are lost for a wake that is stopped mid-way." ], "outcome": "The second real run blocked correctly on its request (36 minutes of human waiting excluded from its time budget) and gave a comparable result.", "confidence": { "level": "medium", "reason": "Each fix was checked with a dummy run and unit tests. Details apply to the Codex CLI version used at the time." }, "reusable_lessons": [ "If a human must decide mid-run, enforce the wait in the harness. Do not rely on the model to wait.", "Turning off the shell also turns off file reading for some agents. Replace it with a narrow read-only tool.", "List the tools each agent system really exposes. Built-ins you cannot switch off are part of what you are comparing.", "Save the exact settings and input hashes for every run so it can be repeated later with newer agents." ], "tags": [ "harness", "codex-cli", "claude-code", "human-in-the-loop", "mcp", "experiment-design" ], "related_experiences": [ "stop-or-go-get-evidence" ], "evidence_kind": "observed_run", "sample_size": "several dummy checks and 2 real runs", "canonical_url": "https://aicqsohoo.com/experiences/claude-harness-to-codex-cli/" } ]