Why Silktide Index scoring changed in 2023

Summary

The previous Silktide Index was using our testing engine from over two years ago (late 2020). A lot has changed, and with WCAG 2.2 being added to our tests, we decided it was time to switch over to our latest engine.

This means more consistency and more accurate results. For some websites, this can also mean large changes in scores. Here we drill down into the gritty details of what is different.

New WCAG 2.2 checks

Silktide have added 9 new checks for WCAG 2.2, of which 4 are fully automated and count towards your Silktide Index scores. These are:

• Ensure controls clearly indicate when they are selected (WCAG AA 2.4.11)
• Ensure focus is not partly obscured (WCAG AAA 2.4.13)
• Ensure focus is not fully obscured (WCAG AA 2.4.12)
• Ensure interactive components are far enough apart (WCAG AA 2.5.8)

The impact of WCAG 2.2 on your accessibility score is limited to around 5%, so most websites will experience only a slight adjustment here.

First-time visits

The older Silktide Index always attempted to dismiss common cookie banners and popups. To test for WCAG 2.2, Silktide needs to experience websites before closing any popups.

We accomplish this by testing an extra ‘page’, which we call a “First-time visit”. Here Silktide tests the homepage of the website before anything is clicked or closed.

You may see this as an extra copy of the homepage. We continue to check every other page on the website as before, closing any popups or cookie banners that we can.

Weighting for PDFs

Previously, the Silktide Index calculated a “PDF Score”, which contributed to about 15% of your overall accessibility score.

This sounds simple, but the results could be counter-intuitive, for example:

• Website A has one PDF, and that PDF was not accessible, resulting in a PDF score of 0%. The whole website would lose 15% of its score.
• Website B has 100 PDFs, and 50 of them are not accessible, resulting in a PDF score of 50%. The whole website would lose 7.5% of its score (50% of 15%).
• So a website with more problems could score higher.

In our new version, Silktide weights the PDF Score by the number of PDFs found. More precisely, Silktide gives a PDF 30% of the weight of an HTML page. This results in much fairer scoring.

Newer technologies

In the last two years, we’ve updated our checks to support many new browser technologies, that were either introduced recently, or previously impossible to test:

• Psuedo-elements can be used to add text to a page (e.g. “open in new tab” to links), as well as for background colors (e.g. for color contrast). Silktide now detects and handles these automatically.
• Styled component frameworks (such as Styled-Components, Emotion, and Themed Styles) can be used to add CSS to a page in a way that doesn’t normally appear in the source code. Silktide can now detect and handle these automatically.
• JavaScript event handlers can be used to attach complex behavior to parts of a page; Silktide can now detect and analyze many of these automatically.

This list is not exhaustive; there have been hundreds of minor changes made over two years.

Testing different pages

Because the Silktide Index only tests a sample of your pages, it has to choose which ones to test. This can affect scores if it chooses to test different pages.

The algorithm we use to decide which pages are tested has changed. In our internal tests, this resulted in more appropriate pages being tested, but the selection of different webpages and PDFs can significantly impact your scores.

New WCAG 2.1 check: Aim for large interactive controls

Silktide can now automate testing of one WCAG 2.1 Success Criterion (2.5.5): “Aim for large interactive controls”. This check was not technically feasible for us to implement in the older Index but became possible due to improvements we made for WCAG 2.2.

This is a Level AAA check that is extremely difficult for almost any website to fully satisfy, and it has a low weight.

Cookie banner sensitivity

Many banners and popups can prevent a keyboard-only user from seeing what they have selected with their keyboard. Two of the new WCAG 2.2 checks look for this (2.4.12 and 2.4.13).

Because of this, Silktide scores can now be affected by our ability to dismiss a cookie banner successfully. If a cookie banner cannot be dismissed, it is possible for Silktide to experience each page (via the keyboard) only through the banner or popup.

This is like if you are pressing tab repeatedly, and find yourself trapped inside a popup; Silktide can never escape to experience what is behind it. This only affects our keyboard accessibility checks.

We have trained Silktide to recognize most banners and popups, and we continue to improve this, but given the infinite variety of popups, no approach can eliminate them all perfectly. If you notice a cookie banner or popup appearing in the screenshots for one of your websites, let us know, and we’ll be sure to remove it wherever possible.

Order based on frequency and severity

Previously Silktide ordered issues by their severity – for example, a completely broken mobile page would always be more important than say a duplicate ID tag.

Now Silktide also considers the frequency of these issues, when deciding what order to show them in. For example, this might mean that a particularly bad issue, which occurs on every page, can appear higher than a bigger issue with a single page.

This has no impact on scoring, but it does affect what issues we might show you in the Silktide Index, as we only show you the top 10.