What we sell, and what it actually rests on.

The register

1. "If someone has to re-read behind the software, you have gained nothing."

   Homepage · the post « Re-reading behind the AI »

   Measured

   Vaccaro, Almaatouq & Malone, "When combinations of humans and AI are useful", Nature Human Behaviour 2024;8:2293–2303, pre-registered: 5,126 articles screened, 74 studies, 106 experiments. On average the human–AI pair does worse than the better of the two alone (g = −0.23, 95% CI −0.39 to −0.07). The authors write themselves that over 95% of the systems measured had a human make the final decision after the AI's advice, and that better results might have been obtained by designing in advance processes in which the AI does only the parts where it is clearly better. Qazi et al., NEJM AI 2025: 44 physicians trained twenty hours in AI, accuracy from 84.9% to 73.3% when the advice is tampered with. Skitka et al., 1999: the bias persists in participants told the aid is fallible.

   Where it stopsThe split designed in advance was tested only 3 times out of 106 (g = 0.22, p = 0.494): it is the path the authors point to, not a path demonstrated. Automation bias is measured on decision aids from before language models, mostly in the lab and in clinical settings; carrying it over to these models is a strong analogy, not a measurement. And the opposite literature exists, algorithm aversion: the direction depends on the task and on what the person has seen the machine miss.

   Open the primary →

2. "In three weeks, we count it on your last twenty files, with the AI you already have."

   Homepage · the measure

   Measured, once

   The arrangement the measure prepares, the tool handles alone what it can read and hands back the rest, is the only one whose effect has been measured in a randomised trial against the real standard of care: MASAI, The Lancet 2026;407:505–514, 105,934 women, breast-cancer screening in Sweden. Sensitivity 80.5% against 73.8%, identical specificity (98.5%), reading workload down 44%, interval cancers non-inferior. Funded by the Swedish Cancer Society, not by the vendor. Leibig et al., Lancet Digital Health 2022, on 1,193,197 mammograms, had found the same: the AI alone is worse than the average radiologist, but the AI that handles the 63% of cases it can hold and hands back the other 37% beats the radiologist on both axes. What those two studies share with the measure: before letting the tool decide alone, someone counted on which cases it is right.

   Where it stopsAll the evidence comes from medical imaging; we looked for a measured equivalent in SMEs, on quotes or reports, and there is none. And twenty files are a small sample: fifteen right out of twenty gives 75%, but the plausible interval runs from about 51% to 91%. So the measure gives an order of magnitude and the questions on which the tool fails, not a rate to the decimal. That is the reason for the clause: if the result does not let you decide, it is not due.

   Open the primary →

3. "A model that handles one file well does not handle a hundred in a row."

   Service page

   Measured

   τ-bench (Yao et al., arXiv:2406.12045), verbatim: "even state-of-the-art function calling agents (like gpt-4o) succeed on < 50% of the tasks, and are quite inconsistent (pass^8 < 25% in retail)." CRMArena-Pro (Salesforce Research, arXiv:2505.18878): leading agents reach ~58% single-turn success and drop to 35% in multi-turn settings, across nineteen expert-validated tasks. METR (arXiv:2503.14499) finds the 80%-reliability time horizon is "roughly 5x shorter" than the 50% one — every "AI can now do X-hour tasks" headline quotes the coin-flip threshold.

   Where it stopsτ-bench runs on simulated users and a synthetic database, with 2024 models. And the counterweight belongs here: on GDPval (arXiv:2510.04374, OpenAI's own benchmark), 47.6% of Claude Opus 4.1 deliverables beat or matched professionals averaging 14 years of experience — on the gold subset. One-shot quality is the demo. Repetition against state is production. Both are true.

   Open the primary →

4. "Nobody is replaced. The system applies their rules."

   Profile · Experience entry

   Measured

   Goldberg, "Man versus model of man", Psychological Bulletin 73(6), 1970, pp. 422–432. Least-squares models were built of each of 29 clinical psychologists, on their own judgements of 861 MMPI profiles. Built on all 861 cases, 86% of the models predicted the actual diagnoses better than the clinician they were derived from (p < .001, sign test); for 28 of the 29 judges (97%) the model was at least as valid. The paper states the mechanism plainly: the clinician "has his days" — boredom, fatigue, distraction — and the model removes that noise, not the knowledge.

   Where it stopsThe advantage shrinks with the sample the model is derived from: 86% on all cases, 79% on a seventh, 72% on a tenth, 62% on 33 cases. And the honest limit: modelling did NOT improve on the composite judgement of all 29 clinicians pooled. Against several experts pooled, the edge disappears. 1970s clinical judgement is not job pricing.

   Open the primary →

5. "Your competitors can get the same models you do. Not what your firm knows how to do."

   Homepage

   Measured

   Brynjolfsson, Li & Raymond, "Generative AI at Work", NBER w31161 (published in the Quarterly Journal of Economics 140(2), 2025). 5,179 customer-support agents, 3 million chats, staggered rollout, difference-in-differences on the firm's own administrative logs. Access to the assistant raised issues resolved per hour by 14% on average — 34% for novice and low-skilled workers, with minimal impact on the experienced and highly skilled. The tool was trained on the firm's own corpus, up-weighting the chats of top performers.

   Where it stopsThe authors call the knowledge-diffusion mechanism "suggestive" themselves, and the design is quasi-experimental, not a randomised trial. We read the free NBER working paper (5,179 agents, 14%); the paywalled published version reports 5,172 and 15% — we cite what we opened. 89% of the agents worked outside the United States, mostly in the Philippines. And there is no generic-AI control arm: see the last entry.

   Open the primary →

6. "What it costs you today, estimated from your own figures."

   Homepage · frequently asked questions

   Argues against us

   METR (arXiv:2507.09089): a randomised controlled trial on 16 experienced open-source developers completing 246 real issues in repositories they had worked on for about five years. They forecast AI would cut their time by 24%. Afterwards they estimated it had cut it by 20%. Measured, it increased completion time by 19%. Economists had predicted −39%, ML experts −38%. Humlum & Vestergaard (NBER w33777): 25,000 workers across 7,000 workplaces in 11 exposed occupations in Denmark, survey responses linked to administrative registers — "precise null effects on earnings and recorded hours… ruling out effects larger than 2%" two years on. Adopters report saving about 3% of their work hours, and 85% of them reallocate that time to other tasks.

   Where it stopsMETR is 16 developers, all expert maintainers, and its authors say explicitly this is not evidence that AI fails to speed up most developers. Generalise the perception gap, never the +19%. Humlum measures earnings and hours, not the quality of the work. What both establish is narrow and enough: self-report gets the sign wrong, so a number agreed before the work starts is not a formality.

   Open the primary →

7. "The system is yours: it evolves without me, with no subscription to my presence."

   Company page · profile

   Measured, once

   Soloway, Bachant & Jensen (the last two at Digital Equipment Corporation), AAAI-87, pp. 824–829, published while XCON was in production. Verbatim: "Over 7 years, XCON has grown to about 6,200 rules, of which approximately 50% change every year. While the performance of XCON is satisfactory, it is increasingly becoming more difficult to change." It started at about 700 rules, and the paper notes that "the problems of continually updating such a large system do not grow linearly". The system worked. Keeping it alive was the problem.

   Where it stopsOne industrial case, 1987, a rule-based system rather than a model. The mechanism transfers — custody and maintenance, not knowledge quality, is what kills these systems — the figures do not. The study we wanted beside it, Gill's 1995 survey of early expert systems in MIS Quarterly, is paywalled everywhere we looked. We did not read it, so it is not here.

   Open the primary →

8. "Ours gives you its error rate, measured, and a sample drawn at random to check it."

   Homepage

   Argues against us

   Kwan et al., BMJ 2020;370:m3216 — 108 studies, 122 trials, 1,203,053 patients and 10,790 providers. Computerised decision support raised the share of patients receiving the desired care by 5.8 percentage points (95% CI 4.0 to 7.6). But in the 30 trials that reported clinical endpoints, the median improvement was 0.3% (IQR −0.7% to +1.9%). Behaviour moves reliably. Outcomes much less so. Roshanov et al., BMJ 2013;346:f657 — meta-regression of 162 randomised trials: systems evaluated by their own developers report an odds ratio of 4.35 (95% CI 1.66 to 11.44), a result the authors describe as robust across methods and internal validation.

   Where it stopsThat last number is about us. We build the system and we would be the ones reporting on it. There is no rhetorical answer to it — only a structural one: the baseline is agreed by you before we start, and measured against what the process costs today, which is the one arrangement a developer cannot quietly grade. Both papers are clinical; carrying them across to job pricing is reasoning, not measurement. And heterogeneity is high (I²=76%) — the top quartile of results ran from 10% to 62%.

   Open the primary →

9. "For French SMEs of 50 to 500 people."

   Homepage · service page

   Measured, once

   Aldasoro, Gambacorta, Pal, Revoltella, Weiss & Wolski, "AI adoption, productivity and employment: evidence from European firms", EIB Economics Working Paper 2026/02. The European Investment Bank's annual survey matched to company accounts, more than 12,000 non-financial firms. On average, AI adoption raises labour productivity by 4%. By size (Table 5): 5 to 9 employees, −0.059, not significant; 10 to 49, −0.041, not significant; 50 to 249, +0.040, significant at the 10% level; 250 and above, +0.079, significant at the 1% level. Fifty employees is the smallest size at which an effect shows.

   Where it stopsThe study is European and gives no result for France alone. It counts "big data analytics and AI" as AI, which is wider than generative AI. Productivity is turnover per employee as declared in the survey. The 50-to-249 effect clears only the 10% level, the loosest of the usual thresholds. The upper half of our range, 250 to 500, sits inside a "250 and above" group the study does not split. We had set the threshold of 50 before reading this study.

   Open the primary →

10. "In six weeks, we write with your teams what software can take on alone in this work, what stays with the people who know, and the plan to build it."

    Homepage

    Argues against us

    Calvino & Fontanelli, "AI users are not all alike: the characteristics of French firms buying and developing AI", CESifo Working Paper 11466 (published in Research Policy, 2026). INSEE's ICT survey on AI use in 2018, about 9,000 French firms of 10 employees or more. The authors separate firms that buy their AI from an external provider — "off-the-shelf or customised" — from firms that develop it in-house. For buyers (9.83% of firms), the link with productivity is explained by selection: they were already more productive. For developers (3.11%), it "remains positive and significant beyond selection". A system we build for you is, in this classification, a bought one.

    Where it stopsThe buyers group pools off-the-shelf and customised systems and the study never separates them, so it says nothing about customised work on its own. The data are from 2018 and cover predictive AI, before generative AI. The authors' explanation — building in-house lets a firm tailor the tool to its needs, integrate it into its organisation and adapt it on feedback — is an interpretation they do not test. Building with the people who do the work answers the first two. The third depends on who, in your firm, will change a rule after we have left; in the data, what sets developers apart is having ICT specialists on the payroll. We read the working paper — the published version is paywalled.

    Open the primary →

11. "An assistant grounded in your know-how beats a generic model on your work."

    The load-bearing claim of the whole offer

    Untested

    Nothing tests it directly. No randomised field experiment compares an internally-grounded assistant against a generic model on the same task; two research passes went looking and found none. What exists in that space is vendor marketing with no method and no control. The closest indirect evidence is a pair: Brynjolfsson's +34% for novices with a firm-tuned corpus, set against Otis et al. — 640 Kenyan entrepreneurs, randomised, given a generic GPT-4 business mentor over WhatsApp — where low performers did about 8% worse.

    Where it stopsDifferent populations, different tasks, different designs: this is not a head-to-head, and we will not present it as one. Otis reports a null average effect under a pre-registered plan; the gain for high performers ("just over 15%") is not a significant average effect, only a quantile effect at the median. What Otis establishes solidly is the harm to low performers, not the benefit to anyone. And we read the working paper — the Management Science version is paywalled.

    Open the primary →

What we could not read

The rule for this page is that nothing enters it whose primary we have not opened. Three sources we wanted are behind paywalls with no repository copy, so they support nothing here:

1. Garg et al., JAMA 293(10), 2005 — the landmark review of decision support. Closed everywhere. Its findings are echoed by Kwan (2020) and Roshanov (2013), which we did read, so the entry above stands without it.
2. Gill, MIS Quarterly 19(1), 1995 — the five-year follow-up on early expert systems. Closed. Its abstract matches what we argue, which is exactly why we will not cite it on an abstract.
3. The published Quarterly Journal of Economics version of Brynjolfsson, Li & Raymond. We read the free NBER working paper instead, and cite its numbers — 5,179 agents and 14%, not the published 5,172 and 15%.