Why “Just Ask an LLM” Falls Short for Regulatory Coverage
This report was originally posted on 17th August, 2026. Further regulatory developments may have occurred after publication. To keep up-to-date with the latest compliance news, sign up to our newsletter.
AUTHORED BY STEPHEN BUGGY, ENGINEERING MANAGER, ADHERENT
Why “Just Ask an LLM” Falls Short for Regulatory Coverage
Asking a general-purpose LLM which regulations apply to a product gets you a fast, confident answer – but confidence isn’t the same as correctness, and in compliance the cost of a missed rule shows up as recalls, fines, and blocked shipments.
To find out exactly how wide that gap is, Adherent ran a like-for-like benchmark: the same plain product description, given once, to ChatGPT, Claude, and Gemini, with no structured profiling or coaching. Every regulation each model named was checked against Adherent’s own list using official identifiers – EU CELEX numbers, national statutory instrument numbers – across five product/market combinations spanning the EU, individual member states, and the US.
The starkest result: for a single Bluetooth speaker sold across the EU, Adherent identified 352 applicable regulations. The best-performing model found 20 – leaving roughly 300 requirements no business would have known to look for.
What’s Inside?
- Full benchmark results across five product/market combinations – from a Bluetooth speaker in the EU to a Wi-Fi robot vacuum in the US
- Why the gap is largest in the EU, where implementing laws and technical amendments do the real work, not the parent directive
- Why the US gap sits at state level – privacy, product stewardship, and automated decision-making rules that federal-only models miss
- Two accuracy risks: models citing a repealed directive, and one model failing to answer the same market five times in a row
- How Adherent validates applicability decisions against an expert-curated dataset built over nearly 25 years – and why that traceability matters under audit
