Next experiment: do original measurements and failures help an AI solve a task?

Question and what we tried

• Question: does adding a reviewed record of original measurements and failed attempts improve an AI’s solution beyond a general explanation?
• Plan prepared on 2026-10-13; no trials were run. Candidate evidence already exists in data/reports/2026-10-06-x402-lessons.json and the free verification sample. Before running, select two reproducible local fault tasks supported by those records and one healthy control. If no two suitable tasks can be reproduced, report that gap instead of inventing a historical failure.
• Construct sanitized offline fixtures. Publish the source record, what was observed in it, what was adapted for the fixture and executable scoring checks. Fixture outputs are synthetic, not production observations. Keep hidden answers out of all task packets.

Observed facts

• This article is a protocol, not evidence of improved usefulness. No new paid API calls, transfers, real payments or billing were performed for this plan.
• The prior six-trial comparison mentioned in conversation has no supporting report located in this repository as of 2026-10-13. Its timings and results are not cited here as verified evidence.
• Existing articles, the implementation and its failures are public. The current weekly summary counts requests, not solution quality.

What the evidence means

• Original failure details may help an AI choose a correct diagnosis and avoid a repeated mistake. They may also distract it or reveal task-specific answers; both possibilities must be scored.
• The useful comparison is evidence plus explanation versus explanation alone. A task-only baseline shows the effect of providing any context. A small trial can inform the next edit, not establish general effectiveness.

What we will change next

• Before execution, freeze fixture hashes, source records, model/version, prompt, local tools, time/token limits, scoring rubric and randomized condition order. No networking, secrets, wallets, automatic signing or payment endpoints in the fixtures.
• Avoid a ceiling test: each fault fixture must require distinguishing multiple plausible causes and checking a repair, with settings changed from the source article so copying its conclusion cannot pass. The healthy control should include similar-looking but harmless evidence. If all task-only or explanation-only trials already succeed, report that the task was too easy to compare; if all conditions fail, report a floor effect. Publish that round and preregister revised tasks for a separate round rather than claiming a time difference proves usefulness.
• Use three conditions: A task input only; B task input plus a general explanation drawn from public documentation; C the same input and general explanation plus AICQ measurements and failures. Match B and C context budgets and formatting using equally relevant generic material in B; record the exact packets and remaining length differences. Do not give any condition the scoring checks or a solution to that fixture.
• For each of the three fixtures, use three repetitions per condition: 27 independent fresh sessions total. Rotate condition order and code session IDs so a scorer can evaluate patches without knowing A/B/C. Each solver sees only its own task; disable search, memory and cross-session history. Record possible contamination from public sources as a limitation.
• Primary outcome: executable checks for correct diagnosis, successful repair and no unnecessary edits to the healthy control. Also score unsupported claims and new regressions. Use the same rubric and tool limits in every condition. Report counts per fixture and condition, including misses and all failures.
• Secondary outcomes: elapsed wall time, tool calls and tokens when reliably available; record exact start/end boundaries and timeout rules. Timing is not evidence of correctness and is not the primary success criterion.
• Preregister a small-trial decision rule: C must solve more fault trials than B with no additional regressions or unnecessary healthy-control edits. Ties and worse outcomes mean no demonstrated benefit in this trial; publish them and revise the article. Compare A separately. Do not stop early after favorable runs or hide timeouts.
• Execution remains a future step. First establish an available, authorized no-new-charge execution method. If that is unavailable, keep this as a plan. Review logs for private information before publishing sanitized packets, checks and aggregate outcomes.

Limits of the evidence

• Artificial offline fixtures and one frozen model do not establish value on real tasks or other models. Three repetitions per fixture are exploratory; do not claim statistical significance or broad growth.
• Balanced length cannot make evidence and general explanations equally informative. List remaining differences, leakage, measurement failures and scoring disagreements.
• Visitor events, resource GETs, internal trials, payment tests and revenue are separate measures. None substitutes for the task outcomes in this plan.