Too often, I see companies touting their high accessibility scores. They use tools like aXe Core, Lighthouse, Access Engine, or other free automated tools to derive them. But this only tells a part of the story, and it doesn’t even tell half of it. Let’s explore what is needed, why, and how we go about continuing the testing.
Important Disclosure
I work at Level Access. We have our own toolset that we use internally. While I work there, I try to remain agnostic in talking about tools and techniques.
Tools
Like any project, we need a few tools to help us do the job we’ll need.
- Computing Device – This can be a Windows, Mac, iPhone, Android device, tablet, or even Linux. We can’t test websites without accessing them.
- Screen reader – This will be device dependent. The Apple products have VoiceOver. Android has Talkback. For Windows, there is Narrator built in, JAWS, and NVDA. I recommend NVDA as it doesn’t lie and is free open source software. I also caution on relying on Narrator. It’s initial use was just to get someone to the point where they can download a “real” screen reader. Microsoft has continued to update it. It is acting more like a screen reader now. However, there’s still work to be done.
- Browser – Unless you are working with VoiceOver, the recommended browser to use is Chrome. It has the largest market share and is at the tech edge of production browsers.
- Spreadsheet – to log findings. Include columns for at least:
- Finding description
- Finding recommendation
- Page found on (URL)
- Guideline
- Screenshot (I like to use ScreenCast for storing these images.)
- Steps to reproduce
- WCAG’s Quickref for the level I am testing against. I test to 2.2 A and AA as my standard.
- Mouse
- Keyboard (Doing a mobile audit? Use a Bluetooth keyboard without the screen reader.)
- Automated tooling – It does help to run automatics at the beginning of an audit. It will reduce the time you spend documenting issues.
- Contrast checker
- Scope document
- Objective of the audit
Setup
We’ve gathered our tools and we are ready to begin testing…or are we? The last two items I mention in the Tools section are a scope and objective. We need to know the objective of the audit. Is it to update their backlog with any accessibility issues? Is it because they need a VPAT? I ask these questions because it influences how I scope.
Scoping is an art to itself. I’ll give a brief overview here. The first part of scoping is understanding the objective and the primary flows.
Look at each primary flow. Are there shared experiences? Yes, we probably only need to test one. Do they use the same header or footer? That can be one unit to test as it is global. Identify patterns used the site and capture them. If pages use templates, say a product details page, capture one of each to test. A thorough audit will likely involve more than 10 pages or units to test. Most of my clients come in around 15-20.
If there is a need for a VPAT, we will add a few more pages. We want to make sure we capture a sample of everything. In the first audit style we may not look at the About Us, Contact, Terms & Conditions, etc. But with a VPAT to be authored, I would be reviewing these as well.
Setup your spreadsheet. I like to make pivot tables and drop downs for key sections. Like which guideline is involved, what disabilities are affected, anything where you will be repeating things from a limited collection.
Step 1 – Mouse
If your vision requires a screen reader, use it instead of the mouse.
Time to start testing. You have your scope. The key here is getting to know the site layout and functionality. Go through every single page in your scope with the mouse and only the mouse. Figure out what items are active controls and how they respond to the mouse.
Step 2 – Automatics
Now that I have familiarized myself with site, I run my automatic testing tools. Depending on the tool you use and how you have it configured, you’ll have a few ways to do this. Some tools look at one page and report findings back in the Chrome Developer tools. If they allow you to export them to a spreadsheet, collect that. You have one for each page tested to compile later. If it doesn’t export, you will need to manually copy them to your spreadsheet.
You may want to roll your own testing bot that can push content to a database. There are several free and open source tooling that will let you run a headless browser with Node.JS to make calls to pages and return the tested output. I’m now playing around with this for a different project.
Automatic tools at best cover 40% of all WCAG criteria. If a company says their AI enhanced tool does better, you may hit 50% coverage. This is because AI doesn’t understand intent, purpose, or how people think into account.
Review every automatic to make sure there are no false positives. Every company out there is saying they get fewer or no false positives. I have yet to find a tool that can honestly say “zero false positives.”
Step 2 – Keyboard
If your vision requires a screen reader, skip this step.
Keyboard testing is based off our mouse testing. We now want to navigate the site like we did in step one without the use of the mouse. We want to ensure that only active controls receive focus. We also check that controls can be used by the keyboard fully in their design. Log anything that doesn’t work properly with keyboard only. Log anything that gets focus, yet is a static element. Caveat – if the link is an in-page link and the target is static a tabindex of -1 is allowable. Repeat this until the page is completely reviewed.
Now you may want to move onto another screen, and that’s fine. Many folks like to go through a project with one tool, then the next. I test my projects by page or screen. So I go through all the steps on a screen, then mark the screen done. You do this as it best works for you.
Step 3 – Screen reader
Pick your screen reader. Remember, these are often device dependent. Repeat Step 2 with the screen reader on. Now, we are determining if the active controls have proper accessible names. Do the names include the visual text? Will they work with the screen reader?
Advice
A control might work with the screen reader off but not with it on. You could be in the wrong mode of the screen reader. There is virtual browse mode. Here the screen reader intercepts all keystrokes as a way to provide faster navigating. This uses Tab to access active controls and arrows to read by line.
There is also a forms mode where the screen reader doesn’t intercept keystrokes. Usually this is invoked by moving into a form field. If it is not, use the screen reader’s hot key plus the space bar will usually change this. In NVDA, this is Insert/Caps Lock + Space bar. Try that widget now. It should work, and if it doesn’t, log it.
Step 4 – Color contrast
Pick your picker! Until you are capable of computing contrast ratios with your own eyes, you will need a color contrast testing tool. There are dozens out there. My favorite is Level Access’s Accessible Color Picker. Before that it was Colour Contrast Analyzer.
Which one matters less than if it is giving you accurate ratios.
Advice
There are multiple contrast ratio methods currently being discussed among the Accessibility Guidelines Working Group (authors of WCAG). We realize there are many color combinations that fail the WCAG standards. However, these combinations are perfectly viable for colorblind and low vision users. One of the calculation methods is called APCA. There is a strong probability it will be used in WCAG 3. However, it is not the standard now, so use a color contrast tool based on the WCAG 2.x algorithm.
Step 5 – Reading level
You need to know the audience of the site or product you are testing. For instance, the targeted audience of this blog is people interested or involved in accessibility work. I have some basic posts, and some more advanced. My language is always meant to be as simple as possible. However, there are specific things I expect others in the domain to understand. One of these is the term WCAG.
Why do we need to know the audience? Because we need to check if the writing on the site is hitting the right audience.
For instance, providing a legal interpretation of a document. If the audience is other lawyers, you use one style. You include as much jargon and as many shortcuts as possible, because you expect them to know the domain.
If the audience is the general layperson, we need to write as if speaking to 13-year-olds. Our language should be clear and simple.
Over a decade ago, I wrote about this for the A11yProject.com. The post is still up and accurate. I’ll relist the resources for testing before the next step.
AI has many great features. One is its ability to identify when text should be simpler. I use it all the time to help clarify my writing. You may be able to use AI-based tools for this. It depends on your access to the admin portion or source code.
Resources:
Step 6 – WCAG SC
When it seems that I have completed my testing, I bring out the WCAG Quickref again. One by one, I read each guideline’s purpose and understanding. For each one, I’ll think through if I might have missed this in my testing. I check each page in scope and log anything I forgot to log earlier.
Follow Up
Now what do you do? Well, this depends on your contract. Good things to do with your client are review the list. Pull out 5-10 of the most important issues. Explain why you pulled them, their impact to users, and how to fix them. Leave them with a priority list of fixes. You could complete a VPAT and issue an ACR. You can work with their team on remediation efforts.
You don’t want to leave the client hanging. Don’t just log 100 or more bugs and walk away. Because next year when they update the product, they’ll need more testing. When they roll out a new product, they will need more testing. Help the client improve through training, fixing, or guiding them.
Don’t do this for free either. When I was independent, I would charge for the evaluation based on scope, not hours. But I’d also put in a recurring monthly support charge. This included 5 – 10 hours monthly paid up front. It is a retainer I take regardless if they use that support. If the time used in a month exceeds the retainer, that gets added to the next invoice. But now we’re entering business practices and that is a different blog post.
Happy Global Accessibility Awareness Day (GAAD)! Have questions or want to discuss this, hit me up on LinkedIn and BlueSky.
A little bonus – Level Access has a lot going on today if you’d like to check out training.
