Flaky visual regression tests. Not a problem.
Flakiness is one of the most common issues engineers face with visual regression testing. In this article, we will explore ways to reduce flakiness, improve the stability of your visual tests and decrease the maintenance time the engineers spend.
One of the cornerstones of proper visual regression testing is tooling. There are many ways of ensuring your UI is rendered in a stable way across CI/CD runs but in the end, it's the flexibility of the VRT(Visual Regression Testing) tools that reduce the noise to the maximum extent.
In this guide, we will focus on setting up the right configuration for different flakiness cases and eliminate them one by one using the flexible and extendable config of Lost Pixel - open source visual regression testing tool.
What is Lost Pixel Platform 🔎
Lost Pixel Platform is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of web pages and applications. It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to quickly identify and fix visual bugs. It offers straight integration with GitHub & allows for easy collaboration between team members.
Lost Pixel provides great flexibility when it comes to setting up your tests. We will go through 5 most common flakiness scenarios and fix them with various methods provided by Lost Pixel Platform.
Before we start
The Lost Pixel open-source engine already packs numerous methods to reduce flakiness and ensure smooth visual test execution. Namely, you get the following internal methods out of the box by simply using the Lost Pixel base configuration.
- Disabling animation - we try to disable animations on the web browser when running your visual tests, as animation usually does not complete with > 1-millisecond precision that would ensure consistent results.
- Requests handling - when doing visual tests, you can run them on isolated components and full-blown apps with network functionality. Lost Pixel ensures that all requests on the given page are executed before the snapshot is taken to show the state you intend to have.
- Antialiasing - non-deterministic rendering is a common problem that causes flakiness in comparing fonts, color hues, and other pixel entities. Lost Pixel uses modern diffing engines that use the latest research on comparing images that already consider all potential problems.
Although the above approaches to reducing flakiness in visual regression tests prove to help a lot, there is far more to it because we are discussing various tools people use to render their frontends. Sometimes there might be derailments, and this guide focuses on fixing them.
Non-deterministic rendering
Even using a scientific approach to identifying false-positive regressions sometimes fails. Enter thresholds - configuration object to fine-tune the sensitivity of diffing engine. You can define the thresholds on the mode level(storybook, ladle, pages, custom) or on the individual test level making it even more granular. Furthermore, you can decide if you want sensitivity to work on an absolute or relative level by choosing the amount of % difference or pixel amount, which is tolerable and won't be considered a regression.
import { CustomProjectConfig } from 'lost-pixel';
export const config: CustomProjectConfig = {
storybookShots: {
storybookUrl: './storybook-static',
threshold: 100,
},
pageShots: {
pages: [
{ path: '/layouts/books/fiction', name: 'fiction-books', threshold: 0.1 },
{ path: '/layouts/books/biography', name: 'biography-pages' },
],
baseUrl: 'http://172.17.0.1:3000',
},
lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
apiKey: process.env.LOST_PIXEL_API_KEY,
};
Live elements on the page
Sometimes your page contains elements like videos, lazy-loaded images, or other non-css animated elements that you still don't want to remove from the flow of the page, but you want to test everything else. In this case, masking is one of the easier methods to achieve this.
You can easily target elements with basic html attributes and mask them - this will ensure that whenever the element is rendered, it will be hidden behind the box of solid color, ensuring they are displayed the same way across the test runs.
import { CustomProjectConfig } from 'lost-pixel';
export const config: CustomProjectConfig = {
pageShots: {
pages: [
{ path: '/layouts/books/fiction', name: 'fiction-books', threshold: 0.1 },
{ path: '/layouts/books/biography', name: 'biography-pages',
mask: [{selector: 'video'}]
},
],
baseUrl: 'http://172.17.0.1:3000',
mask: [{ selector: 'code' }, { selector: 'h2' }],
},
lostPixelProjectId: 'YOUR_PROJECT_ID',
apiKey: process.env.LOST_PIXEL_API_KEY,
};
Elements that don't belong to visual testing
Sometimes in your visual tests, you might have elements that don't provide immediate value and can be excluded from the general flow of the page.
In this case, we can use the configure browser functionality of Lost Pixel and prepare the browser the way we wish before taking the shot. In this case, we can exclude the elements from DOM and do it in the centralized place that belongs to visual tests.
module.exports = {
pageShots: {
pagesJsonUrl: 'http://172.17.0.1:9000/lost-pixel.json',
baseUrl: 'http://172.17.0.1:9000',
},
lostPixelProjectId: 'YOUR_PROJECT_ID',
apiKey: process.env.LOST_PIXEL_API_KEY,
beforeScreenshot: async (page) => {
await page.addStyleTag({
content: `iframe {
visibility: hidden;
}
/* do not show underline animation */
#toc-holder a {
background-size: 0 !important;
background-image: none !important;
}
/* skip image display within section */
section img {
visibility: hidden;
}
/* hide cookie banner */
#onetrust-consent-sdk {
display: none;
}`,
})
},
}
Elements that take time to prepare
Sometimes you have an element on the page that needs to participate in visual regression testing flow, but it takes time to prepare itself. In case it's not about animation and network requests but some other functional aspect of your app - you can fine-tune various wait parameters of lostpixel.config.js|ts to ensure you test the UI in the right time of its lifecycle.
module.exports = {
pageShots: {
pagesJsonUrl: 'http://172.17.0.1:9000/lost-pixel.json',
baseUrl: 'http://172.17.0.1:9000',
},
lostPixelProjectId: 'YOUR_PROJECT_ID',
apiKey: process.env.LOST_PIXEL_API_KEY,
waitBeforeScreenshot: 5000,
waitForFirstRequest: 2000,
waitForLastRequest: 10000,
}
Some tests are better not run
In extreme cases, you have a very hard test to fix, but you still want it to lay around for documentation purposes, as in the case of testing Storybook or Ladle. In such a scenario, you could still leave your story be but disable all Lost Pixel runs on it:
export const WithPref = () => (
<Stack>
<Button prefix={<CircularProgress />}>Button with prefix</Button>
<ButtonMinimal prefix={<IconAccess />}>
Button with Icon
</ButtonMinimal>
</Stack>
)
WithPref.parameters = {
lostpixel: { disable: true },
}
Summary
We have gone through the most common ways of battling flakiness in visual testing and explored how the Lost Pixel Platform can help your tests to be stable and robust! Be aware that extremely flaky tests are not doing good to your overall confidence hence if you can't fix them either with utilities from Lost Pixel or on your end with the code - it's almost always better to get rid of them!
If you need help, I am always happy to chat on Visual Regression Testing Discord!
About Dima Ivashchuk
Hey, I'm - Dima the co-founder of Lost Pixel. I like modern frontends, building stuff on the internet, and educating others. I am committed to building the best open-source visual regression testing platform!