slicing and dicing accessibility violations with exception groups

exception groups are a new feature, motivated by async applications, to raise multiple unrelated exceptions. this is the ideal representation of bulk accessibility audits. capturing accessibility and inaccessibility in formal testing environments prevents regressions and organizes the entropy of bulk inaccessibilities.

    from nbconvert_a11y import pytest_axe
    assert pytest_axe.get_npm_directory("axe-core"), "axe isn't installed"

use the new async axe machinery to generate accessibility test results.

    async with pytest_axe.AsyncAxe(
        url="https://nbviewer.org/github/jupyterlab/jupyterlab-demo/blob/master/notebooks/Lorenz.ipynb"
    ) as axe:
            await axe.run()

    known_failures = axe.exc["serious-color-contrast-enhanced"], axe.exc["serious-color-contrast"], axe.exc["a fixed violation"]

we discover accessibility violations when we run axe. this audit is a snapshot in time. code changes and accessibility focused efforts may eliminate known accessibility violations. known inaccessibilities are expected_failures while remediated violations are contained in unexpected_passes.

    expected_failures, failures = axe.results.exception().split(known_failures)
    assert all((expected_failures, failures))
    print(
        "expected failures", expected_failures,
        "undocumented accessibility violations", failures, sep="\n"
    )

expected failures
8 accessibility violations (2 sub-exceptions)
undocumented accessibility violations
8 accessibility violations (6 sub-exceptions)

unexpected_passes are interesting scenarios require maintenance to the accessibility tests to update and document improvements.

    (unexpected_passes := tuple(set(map(type, expected_failures.exceptions)).symmetric_difference(known_failures)))

(nbconvert_a11y.pytest_axe.a fixed violation,)

these sets of exceptions and types provide different perspectives on the state of accessibility. i try to capture some of these ideas in the four states of knowing we can on a test:

i know it is	i know it is not
i don't know it is	i don't know it is not

these states map to different collections and sets of failures.

	it is	it is not
i know	pass	expected failure
i don't know	unexpected pass	failure

there is value in this multi dimensional testing system that verifies accessibility and admits to inaccessibilities. this approach will prove useful in retrofitting inaccessible systems.