Automating accessibility testing

Accessibility on the web has always been an afterthought, with a great number of websites having no compliance with any standards and adhering to no laws. But in this day and age that shouldn’t have to be the case, when the tools to improve this are readily available.

The state of the tools #

Admittedly, the tooling for accessibility-testing is nowhere near the tooling for writing code or writing generally. It never has been and never will be, because accessibility is a specialist-area that often gets overlooked or only catered to by large Content Management Systems (CMS) that have developers who care and know how to do it.

Developers and designers shouldn’t have to lock themselves into arcane CMS’ to maintain accessibility though, nor should it be the domain of the CMS to deliver this. Every necessary part of making a website accessible lies within the front-end – the HTML, CSS, and JS – and as such almost all of the necessary work lies within the theming.

Rather than a never-ending list of tools to use, here’s a shortlist of the best available right now, covering most of what you need:

• Lighthouse: Easy-to-run accessibility audits right in Chrome’s DevTools – Firefox has an equivalent Accessibility Inspector
• tota11y: An easy tool to add to any project for testing HTML semantics, ARIA landmarks, and more
• GTmetrix: Accessibility testing as a service, with free tracking of PageSpeed and YSlow score, load times, and page size, over time
• a11yresources: The never-ending list of reference resources

This covers the basics of what you need to get started with accessibility testing, but we’re after something quite more involved. Let’s talk standards.

Accessibility standards #

Every piece of pertinent and relevant information about standards are defined in the W3C’s Web Content Accessibility Guidelines (WCAG). It’s quite a hefty read though, so I don’t recommend actually reading all of it unless you’re particularly interested in the standards themselves. Most other standards I’ve seen are basically subsets of the WCAG, and achieving any of their three success criteria – A, AA, AAA – should satisfy other requirements.

For an overview, read Mozilla Developer Network’s (MDN) “Understanding the Web Content Accessibility Guidelines”, “What is accessibility?”, and especially the section on legal requirements. Most countries have specific laws^[1] that require accessibility for websites to prevent discrimination of services, and you should take heed of this – both large and small businesses have been sued for lack of compliance.

We’re going to be testing against these standards because they allow us broad compliance with legal requirements, but they also cover a lot of ground for varying disabilities. Mind you, there are no binary levels of disabilities here – though perhaps legally – every visual, audible or physical disability comes in graded varieties, and our objective is to make websites as easy to access for everyone as we can.

Down the rabbit hole #

Building the perfect beast requires a mindset that considers structure and goals at the outset, and compliance and testing at the end.

HTML #

HTML is in many ways the easiest thing to make accessible, because it’s very standardized, even with HTML5. What’s more difficult though, is that you quickly realize that even popular HTML/CSS frameworks like Bootstrap, Materialize, Foundation, and web component libraries like Polymer, vary greatly within themselves in the quality of the code delivered.

However, rather than reinventing the wheel, or spend days on picking something “right for your needs”, pick something you’re familiar with or that looks promising. Their worth will prove in testing, and I’ve yet to find a framework that is not bloated in several ways. Accept this fault as part of their nature, and focus on whether their structure is difficult to follow or not.

What’s more important is the semantics of the HTML-structure. This is where HTML5^[2] comes in as important, especially with the Web Accessibility Initiative - Accessible Rich Internet Applications (WAI-ARIA). Rather than superfluously declare every element using a variety of attributes, classes, and pseudo-solutions, minimize the complexity of both structure and element-declarations.

A <nav>-element conveys the role="navigation"-attribute well to screen-readers, but you’ll want to the whole structure of the website to be clear about what goes where and does what. A little bit of extra thought here will pay dividends later on.

CSS #

A quick sanity-check regarding CSS is in order: Frameworks and libraries will deliver some sensible defaults, but they are almost always so basic that you end up customizing it. It is greatly beneficial if they utilize custom properties^[3], or at least some variables that can be tweaked before compilation.

Higher on the list of priorities should be well-structured files and folders that clearly separate structure from style. You want to deliver stylesheets that complement the elegant HTML-structure you have, and minimally yield sufficient contrast to be readable and understandable to your audience. The aforementioned WCAG-standards ensures this, and modern CSS-tooling like PostCSS can automate this by telling you when the contrast is insufficient between elements.

Modern CSS-modules such as Grid will work well to make navigating by keyboard accessible using the tabindex-attribute^[4]. As is the Flexible Box module^[5], which works great within or wrapping a grid, and both modules reduce the amount of CSS-code needed even for complex layouts.

It is also important, and greatly simplified with the above two modules, to ensure that your website works across devices. MDN has a good article that gives an overview of responsive design.

JS #

The old rule of JS-development for the web was that everything should be accessible even without JS enabled. Given how commonplace JS has become for websites, it’s hard to argue that this should be the governing design principle anymore. Rather, consider how scripts impact the performance, responsiveness, and accessibility of your website.

Whatever functionality that the scripts you are using offer, should be available in a minimal state even if parts of the script fails or is otherwise impaired from executing. This is the principle of graceful degradation, closely related to the principle of progressive enhancement: Assume that most visitors can utilize modern technology in the browser, but fall back to simpler versions when necessary, and polyfill in what older browsers may be missing.

As we’ll see, doing this automatically is actually easier than you’d think. But be aware that many libraries and scripts are inaccessible by default; in general, always, unless the author explicitly states that it has been tested and built for accessibility.

A practical implementation #

I’ll use the Scholar theme for Grav CMS as a practical example. In brief:

Scholar is an academic-focused theme, for publishing papers, articles, books, documentation, your blog, and even your resumé, with Grav.

HTML #

More to the point, it’s written to be modular, extendable, and adaptable. The basic HTML-structure is this:

<html lang="en">
  <head>
    ...
  </head>
  <body>
    <header role="banner">
      <nav
        class="links"
        role="navigation"
        aria-label="Links"
        aria-checked="false"
      >
        ...
      </nav>
      <nav class="menu" role="navigation" aria-label="Menu">...</nav>
    </header>
    <main>
      <section>
        <article>...</article>
      </section>
      <aside></aside>
    </main>
    <footer>...</footer>
  </body>
</html>

This solves the following:

1. The structure is declarative, asserting semantic tags that assistive technologies can recognize without additional markup^[6]
2. There is a balance between the necessary amount of high-level elements and nested elements, allowing common layouts and styles
3. Keyboard-navigation is elegantly handled, when the order of importance is directly available in the Document Object Model (DOM)
4. Broad landmarks such as the content of <main> or <section> can be easily replaced with templating, to adapt to more specific layout-variants

CSS #

The stylesheets are compiled from a neatly organized set of folders and files:

/src/css/
| grid.pcss
| normalize.pcss
| print.pcss
| responsive.pcss
| root.pcss
| styles.base.pcss
| theme.pcss
+---book
| book.pcss
+---components
| | cv.pcss
| | drawer.pcss
| | header.pcss
| | metadata.pcss
| | related.pcss
| | search.pcss
| | selectr.pcss
| | sequence.pcss
| \---formbase
| control.pcss
| defaults.pcss
| input.pcss
| main.pcss
| select.pcss
+---docs
| article.pcss
| docs.pcss
| footer.pcss
| sidebar.pcss
| toolbar.pcss
+---page
| page.pcss
\---styles
berlin.pcss
burgundy.pcss
haleakala.pcss
hawa-mahal.pcss
illawong.pcss
izamal.pcss
kaliningrad.pcss
kuzbass.pcss
london.pcss
longyearbyen.pcss
metal.pcss
miami.pcss
norilsk.pcss
prague.pcss
shizuoka.pcss
spitsbergen.pcss
tuscany.pcss

Each folder holds specifics for a component the theme delivers, that is somewhat different but still generally compatible with the basic layout. All files have the extension .pcss to make it easier for a code-editor to acknowledge them as PostCSS-files.

• theme.pcss: Imports all high-level files, and sets some basics that override browser-resets provided by normalize.css
• root.pcss: Declares a few CSS custom properties that provides font stacks
• grid.pcss: Simple helper-classes for the Grid module, useful for when it’s not desirable to declare this directly on common elements
• responsive.pcss: Adjustments for devices, based on Bootstrap’s breakpoints and the 320 and Up principle
• styles.base.pcss: A file which applies each set of custom properties in /styles to create a separate stylesheet for each distinct Style

With this, the layout of the website is separate from its style, and it has minimal, neutral styling even when a Style has not been applied. The separation of components into modules also makes it easier for developers to utilize source maps.

JS #

Following the guidelines set out above, the JS is comparatively small and largely additive.

• theme.js: Loads all high-level files
• accessibility.js: Helper-functions to enhance accessibility, using native/pure JS to create event listeners
• search.js: Initializes the FlexSearch search engine
• mobile.js: Minimal mutation observer to make opening navigation on mobile devices as smooth as possible
• tinyDrawer.js: A lightweight drawer-menu, for internal page-navigation across devices
• utilities.js: Utility-functions

Whilst FlexSearch does require JS to be used, it is a very accurate, flexible, and lightweight alternative to a back-end solution to searching. As such the balance was struck between usability even without a server, and degradation. Other scripts strike the same balance, opting for simple, lightweight, modern solutions to common user interface necessities rather than complex, heavy solutions.

This is to say, that there is no reason to presuppose that accessibility means you should cater to every outdated piece of software or hardware still in existence. Some assistive equipment are legacy-compatible in that regard, but many follow modern standards and works with any device. We’ll also compensate for some of this in testing.

Testing #

Rather than checking off items on a checklist, we want to automate testing as much as possible. There are of course limits here, where testing is only as good as the tests you write. But because we have a quite exhaustive list of standards to follow, we can cover a huge base of tests at once.

There is a sizable handful of testing tools and frameworks, ranging from very involved to easy to get running. To test accessibility, we need something that can run a browser to look at how things actually render – rather than just test code. I like Cypress because it’s fast, easy, and tests and captures the DOM across browsers.

Considering that we’re testing the front-end, we should already have a manifest of dependencies for our tooling. For me, that mostly means using the Node Package Manager (commonly “npm”) to record specific versions for each required package. This is quite clear from the dependencies and devDependencies sections of any manifest (package.json) – where the latter typically describes the tools rather than assets.

Setting up #

Cypress is a Node-package, installable from npm install cypress --save-dev and directly downloadable. At the time of writing, version 5.3.0 is the newest version. Installing Cypress is more involved than most scripts, and it will have to unpack binaries.

We’ll need two additions to Cypress; axe for compliance-testing and a script to simulate the tab-key commonly used for navigating websites via the keyboard. Run npm install cypress-axe cypress-plugin-tab --save-dev. Now we run npx cypress open to open Cypress in its electron browser.

Then we’ll need to configure Cypress, for example with a cypress.json-file. This file will co-exist along with package.json in the root of our workspace. It’ll look like this:

{
  "baseUrl": "http://localhost:8000/",
  "numTestsKeptInMemory": 1000,
  "watchForFileChanges": false,
  "env": {
    "context": {
      "exclude": [["#query"], ["label[for=\"query\"]"], [".search-button"]]
    },
    "config": {
      "runOnly": ["wcag2a", "section508", "best-practice"],
      "rules": {
        "landmark-complementary-is-top-level": { "enabled": false },
        "landmark-no-duplicate-banner": { "enabled": false },
        "landmark-banner-is-top-level": { "enabled": false }
      }
    },
    "routes": {
      "index": "/",
      "article": "/article",
      "blog": "/blog",
      "post": "/blog/classic-modern-architecture",
      "book": "/book",
      "cv": "/cv",
      "docs": "/docs/features"
    },
    "dynamicRoutes": {
      "data": "/data",
      "embed": "/embed",
      "search": "/search",
      "print": "/print"
    },
    "landmarks": {
      "index": {
        "body": "body",
        "header": "header[role=\"banner\"]",
        "main-article": "main section article",
        "sidebar-header": "aside header",
        "sidebar-article": "aside section article"
      },
      "article": {
        "body": "body"
      },
      "blog": {
        "body": "body"
      },
      "post": {
        "body": "body"
      },
      "book": {
        "body": "body"
      },
      "cv": {
        "body": "body"
      },
      "docs": {
        "body": "body",
        "header": "header[role=\"banner\"]",
        "sidebar": "aside.sidebar",
        "main-article": "main section article"
      }
    },
    "breakpoints": {
      "xs": 320,
      "sm": 576,
      "md": 768,
      "lg": 992,
      "xl": 1200,
      "xxl": 1600
    },
    "activeElements": {
      "index": ["header[role=\"banner\"] .menu .search-button", "a"],
      "article": ["a"],
      "blog": ["a"],
      "cv": ["a"],
      "docs": ["a"]
    },
    "tabElements": "button, a[href], input, select, textarea, [tabindex]:not([tabindex=\"-1\"])"
  }
}

First of note is "baseUrl": "http://localhost:8000/", which tells Cypress what domain it should be testing. In this example, Grav will be running on localhost with the port 8000. "numTestsKeptInMemory": 1000 increases how many test results are remembered, and "watchForFileChanges": false disables retesting when files change.

Second, the env.context includes some elements not to test. The standard configuration to test all elements is generally sane, but sometimes there are elements which are customized to fit in the DOM that are less standards-compliant, but still accessible. These are passed along with env.config to Cypress-Axe which are used in the rests.

Third, the tests are defined in the /cypress/integration/accessibility-folder using their standard setup. Rather than write out every test, parts of env is looped over:

• env.routes defines which URLs to test
• env.dynamicRoutes tests special routes appended to the normal routes
• env.landmarks assigns common, high-level elements to capture visually
• env.activeElements specifies which elements to test contrast for
• env.tabElements are used to select elements to tab through, emulating a keyboard
• env.breakpoints defines what responsive breakpoints to capture at

For example, in cypress/integration/accessibility/aria.spec.js:

/// <reference types="Cypress" />

for (const [index, route] of Object.entries(Cypress.env("routes"))) {
  describe(`ARIA at ${index} (${route})`, () => {
    before(function () {
      cy.visit(route);
      cy.injectAxe();
    });
    it("Has no violations", function () {
      cy.checkA11y(Cypress.env("context"), Cypress.env("config"));
    });
  });
}

This is fairly simple: Loop through the routes, and report back the state of page. With the configuration above, rules that are tagged with “wcag2a”, “section508”, or “best-practice” from Axe’s rulesets will be ran.

Things to test #

In addition to testing the HTML for violations to the ARIA-rules, we’ll want to cover some basics:

• Contrast: What WCAG-level the current colors satisfy, if any. If any contrast is below AA, it’s certainly hard to read for large groups of people, especially with the varieties of screens
• Routes: Tests that dynamically generated URLs are reachable. This would also be a way to test that assets can be found by the server
• Search: Tests the search-field on every page
• Tab: Simulates keyboard-navigation by pressing the tabulator-key and shows the behavior on the page

Below is an example of how the tabbing-behavior manifests, so we can see in Cypress what the actual order of elements is. 3 and 5 are actually obscured, because 3 and 4 happen at the same place in the window – writing what to search for automatically renders results – and 5 is a button to hide the results, which becomes invisible when the results are.

Testing #

Back in Cypress’ window, we just click the test to run it. Cypress will automatically get started, and render each URL as we have told it to do. The test, as described, is pretty plain: It expects the URL to have no ARIA-violations. The test results will be in the left menu, which Cypress populates as tests run.

The error we’re told about here is a11y error! aria-allowed-attr. When we click the line, it highlights the element in question. Perhaps somewhat counterintuitively, the Cypress-Axe plugin hides a lot of information in the test-window’s console. When clicking them, a tooltip will notify you of this. The easiest way to get more information, is to right-click the error and choose “Inspect element”, then click the console-tab.

Expanding the nodes-property, we see that failureSummary explains ‘Fix any of the following: ARIA attribute is not allowed: aria-checked=“false”’. Cypress will also link to a relevant resource to explain it, in this case DequeUniversity.com/rules/axe/3.4/aria-allowed-attr. The offending attribute is aria-checked, which, whilst disallowed for a nav-element, is used to keep this element accessible on mobile devices. There are more compliant, and quirky, ways to solve this particular problem, but as it stands it’s an effective middle-ground for opening a navigation-menu that doesn’t interfere with screen readers, tabulated navigation, or requires special cross-device compatibility.

As discussed there are no perfect solutions here, that do not require either polyfilling or quirky solutions with CSS-hacks. In essence, it’s better to break a rule in the rulesets we’re applying rather than create a heavyset fix for a problem we solved robustly with a modern solution. Still, we’d like to turn this thinking back to the testing we’re doing, and so it’s necessary to exclude it. That is, exclude this particular error for this particular element, not the rule altogether.

We’ll add nav[aria-checked] to env.context.exclude in cypress.json, so it now looks like

"exclude": [
  ["#query"],
  ["label[for=\"query\"]"],
  [".search-button"],
  ["nav[aria-checked]"]
]

This ignores this element when it has this attribute.

Linting on the move – Tooling for Front-End Development #

Finally, there are a few best practices for development that helps you avoid errors much earlier on. In a sentence: DO lint and format code as you type, DO NOT write massive, monolithic stylesheets and scripts. This follows along with the previous argument of using semantic HTML. Well-structured, initially decoupled, optimized assets makes your life much easier, and lets you develop rapidly with a great deal fewer faults.

I am partial to VSCode as an editor for almost all types of code, because compared to all others I’ve come across, it has the best, most stable features, and the best ecosystem of extensions. A lot could be mentioned here, but briefly:

• Prettier: The best formatter and linter bar none, supporting many languages
• PostCSS Syntax: Syntax-highlighting for PostCSS, which adds some features on-top of regular CSS
• Native JS support: Syntax highlighting, IntelliSense, debugging, formatting, code navigation, and refactoring out-of-the-box
• Native HTML support: Syntax highlighting, IntelliSense, and Emmet
• Native CSS support: Syntax highlighting, IntelliSense, Emmet, linting, and formatting

The aforementioned PostCSS has support for plugins, including preset environments. One such plugin is Stylelint, which marks errors and enforces conventions in your stylesheets, and another is the PresetEnv, which calculates what polyfills are needed for browsers to use the styles. The latter will use Browserlist to by default target the most common browsers by usage and age. The image below shows a snapshot of the types of errors that PostCSS will report on with Stylelint, using rulesets very much like WCAG. This helps you catch problems even before testing.

The same, elegant system is available for JS with the Parcel Bundler. It too will target browsers with Browserlist, and can gather up all your scripts, namespaced for use so they don’t conflict with other scripts or packages. You won’t need to worry about rulesets because JS is more loose in this regard, but it will immediately let you know if there are errors in your code that prevents compilation. Cutting-edge JS doesn’t actually need compilation, but Parcel is handy for when you need to support a wider range of legacy browsers or contain your script as a package.

Both PostCSS and Parcel can output with source maps for development, which makes it easy for your browser’s DevTools to pinpoint exactly what file declares the CSS or JS your are inspecting. And with VSCode as your editor, you’ll speed up and improve your development immensely.

Conclusion #

To ensure that a theme works as expected in regards to accessibility, all errors from tests must be resolved satisfactorily. But this requires that your tests encompass all areas of accessibility, or at least a minimum basis to cover most disabilities like a lack of sight, touch, or hearing to a sufficient degree. We’ve looked at standards implementing the broadest set of requirements for accessibility, and how to test various areas in a way that could be used for any front-end.

As said, the implemented solutions are more elegant, robust, and modern than building out the DOM and using old-school tricks to make the theme behave as expected. Using a test-methodology that can be automated makes it a great deal easier to catch deficiencies in the structure and style that would otherwise go unnoticed. You’ll also make great strides in the quality of your code by using modern techniques and applying modern standards.