What an automated accessibility scan can and can’t find
Automated accessibility scanners, including the free one on this site, are fast and cheap. In a minute they can list hundreds of concrete problems with the exact element to fix. But they can’t tell you that your site is accessible, and anyone who says otherwise is overselling. The W3C puts it plainly: evaluation tools “can not determine accessibility, they can only assist in doing so” (W3C, Selecting Web Accessibility Evaluation Tools).
This guide explains where the line is, so you can use a scan for what it’s good at and cover the rest yourself.
How an automated scan works
Our scanner opens your pages in a real headless Chrome browser and runs axe-core, the open-source rules engine from Deque Systems, against the rendered page. axe-core has rules for WCAG 2.0, 2.1 and 2.2 at Levels A, AA and AAA, plus best practices. Each rule checks something a computer can verify from the page’s code and computed styles: does this image have an alt attribute, does this text have enough contrast against its background, does this button have a name.
Results come in three kinds: violations (definitely failing), passes, and incomplete items where the tool couldn’t be sure and a person needs to look, for example text over a background image. axe-core’s maintainers say it is designed to return zero false positives, bugs notwithstanding, which is why uncertain cases go to “incomplete” rather than being reported as errors.
How much do automated tools find?
It depends on how you count, and the two best-known studies measure different things:
- In 2017 the UK Government Digital Service built a test page with 143 deliberate barriers and ran 10 tools against it. The best single tool found 41% of the barriers (counting prompts for manual inspection); one popular tool found only 17%; and 29% of barriers were missed by every tool (GDS accessibility blog).
- In 2021 Deque analyzed issues from over 2,000 real audits covering about 13,000 pages and found that its automated testing covered 57% of the issues by volume (Deque). Deque makes axe-core, so read it as a vendor study; its point is that the common issues on real sites happen to be the automatable ones.
Either way, a large share of barriers need a person to find. That’s why we say on every page that automated tools detect only a portion of accessibility issues.
What a scan finds reliably
| Issue | Why it matters in a store | WCAG 2.2 |
|---|---|---|
| Images with no alt attribute | Product photos are silent or read as file names | 1.1.1 (A) |
| Text contrast on solid backgrounds | Gray prices and sale badges are unreadable for many | 1.4.3 (AA) |
| Form fields without labels | Search, newsletter and quantity fields can’t be identified | 1.3.1, 4.1.2 (A) |
| Buttons and links with no name | Icon-only cart, close and menu buttons are announced as just “button” | 4.1.2, 2.4.4 (A) |
| Missing page language or title | Wrong pronunciation; tabs and history can’t be told apart | 3.1.1, 2.4.2 (A) |
| Invalid or misused ARIA | Custom widgets announce the wrong thing | 4.1.2 (A) |
| Zoom disabled in the viewport tag | Phone users can’t pinch to enlarge | 1.4.4 (AA) |
| Small tap targets | Quantity buttons and swatches are hard to hit | 2.5.8 (AA) |
What needs a person
- Whether alt text is right. A tool can see that alt text exists, not that “IMG_2231” or “product” is useless. (How to write it.)
- Keyboard use from start to finish. Can you open the menu, choose a size, add to cart and reach checkout using only the keyboard? Is the focus order logical? Can you close the pop-up? Tools check pieces of this, not the journey.
- What happens after interaction. A scan sees the page as loaded. Menus, cart drawers, quick-view modals and error states that appear later may never be tested.
- Screen reader experience. Is “Added to cart” announced? Do headings make sense read out? Does a price read as a price?
- Meaning and clarity. Link text that’s technically present but vague, confusing error messages, instructions that rely on shape or position.
- Media. Whether captions exist in a video player and are accurate, and whether audio descriptions are needed.
- Text on images and gradients, which tools usually flag for manual review.
- Pages the scanner can’t reach: password-protected stores, customer accounts, and hosted checkouts.
A 30-minute manual check you can do yourself
- Keyboard only (10 min): unplug the mouse. Buy something up to the payment step using Tab, Shift+Tab, Enter, Space, arrows and Escape. Note anywhere you get stuck or can’t see focus.
- Zoom (5 min): zoom to 200% and 400% on the home, product and cart pages. Nothing should be cut off or overlap.
- Screen reader (10 min): VoiceOver is built into Mac (Command+F5) and iPhone; NVDA is free for Windows; TalkBack is on Android. Listen to a product page: the title, price, images, variant options and the add-to-cart button.
- Read your alt text and link text (5 min) in the scan report and ask whether each would make sense out of context.
The W3C’s Easy Checks is a good free next step.
How to use a scan well
Use the scan to clear the long list of mechanical problems quickly, then spend your human time on the journeys that matter most: finding a product, choosing options, adding to cart, and getting help. Scan again after theme changes and new apps. And be careful with the result: the DOJ’s web guidance notes that “a ‘clean’ report does not necessarily mean everything is accessible,” and a report is not legal advice or a certification.
Frequently asked questions
If my scan shows zero issues, is my site accessible?
Not necessarily. A clean automated result means none of the automatically testable rules failed on the pages that were scanned. Keyboard use, the accuracy of alt text, and how the site works with a screen reader still need a person to check.
Can an automated scan produce false positives?
Occasionally. The W3C notes that evaluation tools can produce false or misleading results. axe-core is designed to avoid false positives and reports uncertain cases separately for manual review, but it is still worth checking a result before changing code.
How often should I scan my store?
After any theme change, new app, or big batch of new products, and otherwise every few months. Accessibility problems are often introduced by new content and third-party apps rather than by the original theme.
Sources
- W3C WAI, Selecting Web Accessibility Evaluation Tools, Easy Checks, and WCAG 2.2.
- US Department of Justice, Guidance on Web Accessibility and the ADA.
- UK Government Digital Service, What we found when we tested tools on the world’s least-accessible webpage (2017).
- Deque Systems, Automated testing study identifies 57 percent of digital accessibility issues (2021); axe-core on GitHub.