## Introduction 👀

Are you looking for the best [visual regression](https://stackonfire.com/visual-regression-testing-101) testing tools to ensure your website or application looks and functions as intended? Look no further than our **Ultimate Guide to Visual Regression Testing**. In this article, we'll explore the top five tools you should try to streamline your testing process and catch any design or functionality issues before they impact your users. We will explore both free and paid options and go through the following tools and their respective use cases:

1. Playwright snapshots (Open Source)
    
2. Loki (Open Source)
    
3. Chromatic (SaaS)
    
4. Percy (Saas)
    
5. Lost Pixel (Open Souce + SaaS)
    

## Playwright snapshots 🎭

[Playwright](https://playwright.dev/) is an open-source Node.js library for automating web browsers. Microsoft developed it and designed it to work with all major browsers, including Chrome, Firefox, and Safari. Playwright allows developers to write automated tests that interact with web pages, simulate user actions, and capture screenshots or videos of the browser window.

**Playwright** is a tool not explicitly designed for visual testing but offered utilities to streamline the whole visual testing process. Anytime in your E2E test run, you can `await expect(page).toHaveScreenshot();` Ensure that your page looks exactly like you expect it to look at any given point in time.

Playwright offers various utilities to reduce the tests' flakiness and ensures you can configure the thresholds for maximum tolerable pixel differences in your tests. It comes in handy when your app relies on non-deterministic rendering and can have minor visual differences not distinguishable to the human eye and thus not relevant as negative cases.

Being an **open-source tool,** Playwright relies on your running the testing & baseline update process on your premises - either on CI or locally on a developer machine. This is both pros & cons, as you are missing a user interface to inspect the results of the visual tests, but at the same time, you are in complete control of your difficulties and don't need to use a third-party service to manage your visual tests. Being able to render practically anything, Playwright is a great tool to test your marketing pages, Storybook components or anything that can be caused in the browser. However, this would almost always require additional setup and tinkering.

## Loki.js 👁️

[Loki.js](https://github.com/oblador/loki) is an open-source visual regression testing tool for your Storybook components. It offers a low-maintenance workflow to test your Storybook components across different devices (React native supported). Although as robust as Playwright for the Storybook test, Loki.js is limited in power when trying your application pages or running a visual regression analysis after interacting with your front end. To achieve that, you would need to do some extra setup and probably add everything you want to test in Storybook.

Loki.js integrates perfectly with any of the CI providers, which makes it super easy to incorporate into your existing testing pipeline. It possesses the same limitation as the Playwright here, though, as you don't have an integrated user interface to work with the test results, and it usually boils down to comparing the images locally with a tool like [Kaleidoscope](https://kaleidoscope.app/). Although this workflow is robust and is used by many companies, the main struggle comes from blocking the developer machines to run the tests locally and producing the new snapshots, thus making it very costly to maintain. This also hinders collaboration between designers/engineers as the test results can be scattered across different user machines/ ci runs.

## Chromatic 🌈

[Chromatic](https://www.chromatic.com/) is a SaaS visual regression testing tool designed to work with [Storybook](https://storybook.js.org/), a popular open-source tool for building UI components. Chromatic integrates with your existing Git workflow to automatically test and review UI changes in your components, ensuring they look and function as intended.

As we are delving more into cloud solutions, Chromatic is a clear step up in most aspects of the **visual regression testing flow.** It is easy to set up and integrates well with your Git flow, meaning you can be sure you don't accidentally release breaking visuals to your customers. Finally, it offers a UI that fosters collaboration and makes it very easy to review the test results.

The main limitation here is that being a Storybook tool, Chromatic excels at providing utilities for working with Storybook but does not make it easy to run visual tests on your pages, automated tests and other use cases where they span beyond Storybook.

## Percy 🦔

[Percy](https://percy.io/) is a SaaS visual regression testing tool that allows you to automate your visual testing process. It integrates with your existing testing framework and captures screenshots of your web application or website to detect visual changes.

Percy offers a user-friendly interface to manage your visual tests and provides a collaborative environment for designers and developers to work together on UI changes. It also provides features like version control, commenting, and collaboration tools to streamline your testing process.

Being one of the oldest tools on the market, Percy covers a lot of use cases. Therefore, he can easily integrate both into your marketing pages, Storybook, and other visual testing use cases. In addition, it is owned by [BrowserStack](https://www.browserstack.com/), so it logically offers to test the magnitude of different operating systems, devices and viewports. Percy being held by a large testing firm spawns some cons, though - it is one of the most expensive tools on the market. In addition, it is usually bulkier in setup & maintenance compared to other tools mentioned in this article.

## Lost Pixel 🖼️

Lost Pixel is a combination of both open-source and SaaS offerings. It uses a [lost-pixel](https://github.com/lost-pixel/lost-pixel) open-source engine that resembles Loki.js in the easiness of setup and maintenance but covers all of the use cases for visual testing(SStorybook, application pages, E2E test) referred to as **Holistic Visual Regression Testing** and offered [Lost Pixel Platform](https://lost-pixel.com/), which integrates into your existing Git workflow and enables collaboration through UI for reviewing the test results.

It's seldom a case that you want to ***just*** unit test your components, Lost Pixel becomes a single source of truth for your visual regression testing strategy. The **Holistic Visual Regression Testing** concept allows you to combine the results of your visual tests into a single run and achieve even more confidence when developing your UIs. For example, imagine a CI run that runs your Storybook tests, then covers a dedicated set of marketing pages that use those components, finishes with interaction tests, and snapshots the results into a single place. Luckily [it is an easy setup with Lost Pixel](https://stackonfire.com/holistic-visual-testing).

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)

A potential drawback of **Lost Pixel & Lost Pixel Platform** is its native integration with GitHub only(no support of GitLab or other git providers).

## Summary 📊

All five tools have their pros and cons. Playwright and Loki are open-source tools that offer utilities to streamline the visual testing process, but they have limitations. Chromatic and Percy are SaaS tools that are easy to set up and provide user-friendly interfaces, but they are limited to testing Storybook components and can be expensive.

Lost Pixel combines open-source and SaaS offerings that provide a **Holistic Visual Regression Testing** approach. It integrates with Git workflow to test E2E, marketing pages, and Storybook components, making it the best option for engineers striving to achieve the best results with little maintenance effort.

The Ultimate Guide to Visual Regression Testing: 5 Tools to Try

## Introduction 📚

Visual regression testing is a crucial aspect of software development that ensures the user interface remains consistent across different versions of an application. However, if you are new to this field, the terminology and jargon used in visual regression testing may be overwhelming. That's where this visual regression testing glossary comes in handy! In this article, we'll define and explain the most commonly used terms in visual regression testing so you can quickly become a pro.

## Visual Regression Testing 👁️

Visual regression testing(commonly referred to as **VRT, Visual testing or Visual snapshot testing**) is a technique used to compare screenshots of an application's user interface to detect any visual changes that may have occurred between different versions of the application. This technique ensures that the user interface remains consistent after the code changes. Usually, **VRT** runs as the last part of your CI/CD pipeline closing the whole test suit and ensuring that your Frontend looks precisely like it should.

## Shots 📸

In visual regression testing, a shot(also referred to as a **visual snapshot**) refers to a screenshot of an application's UI. Shots are used to compare the user interface between different versions of the application to detect any visual changes that may have occurred. Shot is a generally accepted unit of measure for visual regression testing SaaS.

## Baseline images 🖼️ 🖼️

Baseline images(a **source of truth or baseline snapshots**) are reference screenshots of an application's user interface captured during the initial testing phase. These images are used as a reference point for future tests to detect any visual changes that may have occurred between different versions of the application.

Usually, baseline images are manually approved by people responsible for maintaining the **VRT.** However, on the first run of your chosen testing suite, you must approve all your images - those images will now become baselines.

## Visual regression testing flow 🖼️ 🔄

[Visual regression testing flow](https://lost-pixel.com/blog/post/visual-regression-testing-101) refers to capturing baseline images, running tests to detect any visual changes, comparing the results to the baseline images, and updating the baseline images in cases with intended changes. This process is repeated for each code change to ensure that the user interface remains consistent.

![image](https://cdn.sanity.io/images/809vdkvp/production/84afb76aba78719a09bd897ddd95ff81d7fe3011-450x377.png)

## Visual difference ⛰️ ↔️🏔️

Visual difference refers to the changes between two shots of an application's user interface. These changes can be detected using the visual regression testing techniques mentioned above. ***Although not all*** visual differences are **Visual Bugs**, sometimes the changes are intended, and a baseline update is required.

## Visual bug 🖼️ 🪲

A visual bug is a defect in an application's user interface detected and caught by **VRT.** Here are a couple of good examples of visual bugs:

* Text overlapping or not aligning correctly with other elements on the page
    
* Images not displaying correctly or appearing distorted
    

Visual bugs are the exact reason we are running **VRT.** We aim to catch those bugs early in our release pipeline and ensure that end customers never face a breaking UI.

## Flaky visual tests ❄️

The term "**flaky"** refers to tests that produce inconsistent results. For example, in visual regression testing, flaky visual tests are tests that sometimes pass and sometimes fail, even though the application's UI has not changed. Various factors, such as network issues, browser inconsistencies, or timing issues, can cause flaky visual tests. Therefore, it is essential to identify and fix flaky visual tests to ensure that your visual regression testing results are consistent and reliable.

Luckily modern **VRT** tools provide a range of tools and practices to battle flaky tests and make the **VRT** easily one of the most valuable parts of testing on CI/CD.

Visual regression testing glossary

## Introduction to Holistic Visual Testing 🖼️

As front-end development continues to evolve, it's more important than ever to have a robust testing strategy in place to ensure that your applications look and function as expected. One of the most effective ways to do this is through visual regression testing, which allows you to catch and fix visual bugs before they reach production.

**Holistic visual regression testing** is an approach that aims to test the entire application, including both layout and interactions, as a whole.

In this example, we will set up a complex visual testing suite for our Next.js application. Here is the list of tools we will use:

* **Storybook**
    
* **Playwright**
    
* **GitHub actions**
    
* **Lost Pixel Platform**
    

As the final result, we want to make sure that the following parts of our applications are looking the way they are intended to look:

* Individual components - **visual unit tests**
    
* Pages - **visual integration tests**
    
* Results of playwright/cypress tests - **visual e2e tests**
    

Lost Pixel makes it easy to compose all of this in a single place to make the management of your visual testing a breeze. Here is the final version of the code for the whole integration:

Here is the final version of the [code on GitHub](https://github.com/lost-pixel/lost-pixel-next-holistic-visual-tests)

## Setting up Next application 🔼

We will clone the [app-playground](https://github.com/vercel/app-playground.git) repo from Next.js, it has rudimentary parts of the actual application, and that's precisely what we need.

```bash
git clone https://github.com/vercel/app-playground.git

pnpm install

pnpm dev
```

![image](https://cdn.sanity.io/images/809vdkvp/production/066def66917e0662bae053bf76bd4fa759d167c5-1896x1194.png)

## Visual unit tests 🖼️

We will use Storybook in combination with Lost Pixel to make sure our app components are looking the way we expect them to look.

Let's install Storybook:

```bash
npx storybook@next init && pnpm install postcss-loader style-loader @storybook/addon-postcss
```

Inside `.storybook/main.js` we need to make Storybook & Tailwind.css play well with each other :D

```javascript
const path = require('path');
module.exports = {
  stories: ['../stories/**/*.mdx', '../stories/**/*.stories.@(js|jsx|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    {
      name: '@storybook/addon-postcss',
      options: {
        cssLoaderOptions: {
          // When you have splitted your css over multiple files
          // and use @import('./other-styles.css')
          importLoaders: 1,
        },
        postcssLoaderOptions: {
          // When using postCSS 8
          implementation: require('postcss'),
        },
      },
    },
  ],
  framework: {
    name: '@storybook/nextjs',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
};
```

Inside `.storybook/preview.js` we need to import tailwind styles on top of the file:

```javascript
import '../styles/globals.css';

export const parameters = {
  actions: { argTypesRegex: '^on[A-Z].*' },
  controls: {
    matchers: {
      color: /(background|color)$/i,
      date: /Date$/,
    },
  },
};
```

Inside `stories/Button.stories.tsx` let's write our first story that we will use in tests:

```javascript
import type { Meta, StoryObj } from '@storybook/react';
import React from 'react';

import Button from 'ui/Button';

// More on how to set up stories at: https://storybook.js.org/docs/7.0/react/writing-stories/introduction
const meta: Meta<typeof Button> = {
  title: 'Example/Button',
  component: Button,
};

export default meta;
type Story = StoryObj<typeof Button>;

export const Default: Story = {
  render: () => <Button kind="default">Default</Button>,
};

export const Error: Story = {
  render: () => <Button kind="error">Error</Button>,
};
```

Now we run our storybook to see our components displayed in an isolated manner:

```bash
npm run storybook
```

![image](https://cdn.sanity.io/images/809vdkvp/production/9c7106ebbc5756aac595d56fde22d7777624567d-1024x450.png)

Our stories will serve as the basis for the unit visual tests. Let's integrate this first step with Lost Pixel Platform to see how it will work there.

## Lost Pixel Platform ⚙️

You can use the [quickstart documentation](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform) from [lost-pixel.com](http://lost-pixel.com) to set up the platform to work with your repositories via installing the official Lost Pixel GitHub app. In this guide, we will focus on setting up the right configuration, assuming you have done the platform setup already!

You can use Lost Pixel Onboarding to easily set up the project by copying & pasting the respective files.

In our `lostpixel.config.ts` at the root of our project, let's define the tests for Storybook

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },

  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

In our `.github/workflows/vrt.yml` at the root of our project let's define our CI/CD job:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

With this simple setup, we are already covering the visual unit tests 🙌🏼

![image](https://cdn.sanity.io/images/809vdkvp/production/d0cb270199226fe61560ed0c9ae61ce57b24d9a6-1856x2050.png?)

Let's make it really holistic now!

## Visual integration tests 🖼️ ↔️ 🖼️

Sometimes we care not only about how our components will be used in isolation but also about how they will look on the page when they are combined with a plethora of other components & styles. Let's explore how we can easily expand our initial setup to monitor our pages continuously!

Let's extend our `lostpixel.config.ts` to include our page shots.

```typescript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

We also need to extend `.github/workflows/vrt.yml` to build & run our Next.js app so we can access pages in Lost Pixel:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

Now we are talking 🚀 Our visual tests are covering not only **unit** parts with individual components but also **the integration** part where we are making the screenshots for the whole pages.

![image](https://cdn.sanity.io/images/809vdkvp/production/14cd5947e14ffbc2af96c9b11168133fca0e5714-2300x2038.png)

Let's make it an ultimate setup with a magic flavour of the feature of Lost Pixel called custom shots 🪄✨✨✨

## Visual E2E tests 🖼️ ⚙️ 🖼️ ⚙️ 🖼️

What if I told you that on any given CI run, you can create the screenshots using your logic & they will be available immediately on Lost Pixel Platform for the visual regression testing? Enter custom shots 📦

We will be using Playwright to interact with the page & make sure that after some set of interactions, we are always looking at the same UI.

```bash
npm install --save-dev @playwright/test
```

Let's also extend our `package.json` with `test:e2e` command:

```bash
...
"scripts": {
  "dev": "next dev",
  "build": "next build",
  "start": "next start",
  "test:e2e": "playwright test",
}
```

Now we will create our E2E test that will not do much, but will give you a good understanding of how you could integrate Lost Pixel into any test like this!

In `e2e/example.spec.ts` :

```typescript
import { test } from '@playwright/test';

test('lost-pixel e2e', async ({ page }) => {
  // Perform some action on the page, like clicking the button.
  // 📒 http://172.17.0.1:3000 is already fit for GitHub action runtime
  await page.goto('http://172.17.0.1:3000/context');
  await page.click('data-test-id=context-click-counter');
  await page.click('data-test-id=context-click-counter');
  // Make a shot for Lost Pixel to test
  // 📒 Path is arbitrary, but I advise you to store all of the custom shots in a single folder, e.g. lost-pixel. 
  await page.screenshot({ path: 'lost-pixel/a.png', fullPage: true });
});
```

Our `ContextClickCounter` component gets a new `data-test-id`, so the code above works:

```javascript
      <button
        // Here is the important part
        data-test-id="context-click-counter"
        onClick={() => setCount(count + 1)}
        className="rounded-lg bg-gray-700 px-3 py-1 text-sm font-medium tabular-nums text-gray-100 hover:bg-gray-500 hover:text-white"
      >
        {count} Clicks
      </button>
```

Now we are ready to integrate our E2E with Lost Pixel! Let's extend our `lostpixel.config.ts` to include our custom shots.

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  customShots: {
    // 📒 currentShotsPath should be matching the one above from playwright test
    currentShotsPath: './lost-pixel',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

And finally, we extend our GitHub Action declaration file by running Playwright tests:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Playwright tests
        run: npx playwright install --with-deps && npm run test:e2e

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

After Lost Pixel Platform runs this time, you should see something similar to this:

![image](https://cdn.sanity.io/images/809vdkvp/production/f36f66c4976292959369bc045746e71539648066-1856x854.png)

After interacting with our page & clicking the button two times, we made a screenshot of a page to test visually. Sometimes it's very useful because apart from asserting the presence or absence of some elements on the page that E2E tests traditionally rely on, the layout could be broken after a particular interaction and here visual regression tests will catch this for us.

## Summary 🔼

We have built a holistic visual testing framework over an existing Next.js application using the tools suitable for every task: **Storybook** for visual unit tests, **Pages** for visual integration tests & **Playwright** for visual E2E tests. We have use **GitHub Actions** to make sure our setup runs continuously on every push. Finally, we have used the **Lost Pixel Platform** to manage our visual tests in one central place & notify us whenever visual regression is found.

![image](https://cdn.sanity.io/images/809vdkvp/production/61df865a0b5598939a29b186556504e42024be10-1696x1082.png)

If you liked the read and enjoyed setting up your visual tests, you might as well like our [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m). There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

Holistic visual regression testing

## Introduction ✨

Visual regression testing is a critical aspect of modern web development that helps ensure that code changes don't break the intended design and functionality of a website or application.

In this blog post, we'll discuss five compelling reasons why every development team should invest in writing visual regression tests. From improved efficiency and reduced time to market to enhanced user experience and increased collaboration, the benefits of visual regression testing are many. Keep reading to learn more about how visual regression testing can help your team produce better code and ship it more frequently to customers without fear of regressions.

## #1 Efficiency 👟

Visual regression tests automate the process of verifying UI changes, reducing the time and effort required to manually test each change. There are multiple stages of how advanced your setup could be but think of it this way - after each code change, you could deploy your application to a preview environment & ask your team to go through all of the pages, click around, resize the browser & see if nothing is broken visually. **OR** you could automate that saving hours & hours weekly for all of the stakeholders

## #2 Reduced time to market ⏰

With visual regression tests in place, development teams can quickly identify and fix UI issues, leading to faster time to market and a more efficient development process. This point flows nicely from the previous one and is impacting one of the most notable metrics all teams strive to improve on. If your processes are lean & automated I bet you can ship fast and turn your paddle boat 🛶 into a speed boat 🚤.

## #3 Enhanced User Experience ✨

Would you rather have your UI explode in your user's face or ship a pixel-perfect application that is pleasing to look at & work with? I guess an answer is clear here - broken layout, mixed-up design tokens or even incorrectly rendered tooltip could all cost the business money, sometimes substantial amounts. With visual regression testing, we make sure that our users are protected(at least to some degree) from this and we have a safety net for all the problems just before they hit the end consumer of our software.

## #4 Increased Collaboration 🤝🏼

Visual regression tests provide a clear, concise and standardized way to communicate UI changes between development teams, designers and stakeholders, fostering collaboration and reducing miscommunication. If you are using a SaaS solution for visual regression testing you can enjoy the streamlined process that resembles the following:

**Designer** 🎨: hey, *Developer*, it seems like we have a problem with our Button, the success style for it is somehow not applied. Our visual tests are failing!  
**Developer** 💻: oh wow, indeed it's somewhat of an edge case I missed. Pushed the fix to the `fix/button-styles` branch  
**Designer** 🎨 *(after some time)*: nice, the fix seems to be working, looked at the results of the visual tests and the buttons look great again!

Not a lot of back & forth, easy access to results for validation and many other collaborative benefits!

## #5 Improved Confidence 💪🏼

This could easily trump all of the above reasons! Visual tests increase the confidence of development teams in their work by providing a comprehensive and consistent way to validate UI changes. Teams can trust that their work meets the desired standards and that no critical issues have been introduced with any code changes.

One of the main goals of any software testing is just to make sure we, humans, are not doing things we are not good at(checking lots of tiny details of something in hundreds of permutations). Doing this over and over again & failing hurts our confidence of course, and we start to ship less frequently just because we are scared to break stuff. Tests to the rescue! Now we are all confident and shipping new releases 10 times per day 🚀

## Summary 🔼

Visual tests are just another medium to help us produce better code & ship it more frequently to the customers without fear of regressions. Embrace them and enjoy 😊

If you liked the read and enjoyed setting up your visual tests you might as well like our [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m). There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

I am also a co-founder of a Visual Regression Testing SaaS called [Lost Pixel](https://www.lost-pixel.com) and I would be excited to set up visual testing for your company and deliver you from all the traps that I've seen people stepping in when implementing their Visual Tests. Consulting & setup is totally free 💸

Feel free to reach out to me [on twitter](https://www.twitter.com/divdev_) or via email `dimitri@lost-pixel.com`

5 reasons to write visual regression tests

## Foreword 👋🏼

Hey, my name is Dima! I am a frontend-engineer who loves to solve problems. One day at my contracting work we faced serious developer experience issues with running our Storybook.js visual regression tests. At the time we had a custom solution that was working mostly fine but required a lot of manual steps & was mostly blocking our engineers for multiple hours each week.

Having evaluated external solutions we quickly understood that for our monthly amounts of tests, the price could get ridiculously high in no time. Remember how I said I love to solve problems? Together with my friend Chris, we decided to give this problem a shot, and that's how **Lost Pixel** & **Lost Pixel Platform** was born. I welcome you to embark on a short retrospective with me to see why & how we did it.

***Spoiler alert***: we cut the costs significantly, made engineers happy & built it in an open-source fashion 🚀

## Motivation 🎯

There are multiple tools out on the market both open-source & commercial that do the job of visual regression testing well. Why even build a new one you may ask? The answer lay in plain sight for us:

> There was nothing out there being **a)** open-source **b)** able to run on customers' premises **c)** would have a modern approval flow as a cloud service **AND** in virtue of previously mentioned facts still, *be affordable*

From the very start, we were building Lost Pixel on the foundation of it being an open-source technology that could be used in a **self-hosted** manner **for free** or in Lost Pixel Platform as a managed service *but still based on our open-source engine.*

This gives us much more room for growth as we are planning to cater to the needs of both OSS & commercial clients in the future with the mission of making visual regression testing **convenient**, **comprehensive** & **affordable**.

## Differentiating factor 🧞‍♂️

While being similar to most of the existing OSS & SaaS solutions on the market we decided to focus on a couple of premises and make a difference there:

1. Focus on **GitHub** as the main vendor of where your CI/CD runs & make the process seamless by offering **Github Action packed** with everything you need to setup up visual tests on the platform in a couple of lines of code.
    
2. Have an accent on **Developer Experience** by providing strongly typed Lost Pixel configuration, extensive logs for easier debugging and a user interface designed to be used collaboratively with your team
    
3. Offer a **holistic solution** to visual tests. Visual tests are not limited to Storybook, you can run them on your marketing pages, and components or E2E tests like Cypress or Playwright. Our action supports all of them and makes it easier to compose them so you have all of your visual tests gathered in one place - **Lost Pixel Platform.**
    
4. We natively support monorepos, everybody loves monorepos, right? 😀
    

## Building the solution ⚒️

We started with nothing but the premise that we need to have a tool that would be able to handle hundreds of thousands of screenshots monthly, be easily hostable with any CI/CD process & have a user interface for collaboration. As with all other great things, Lost Pixel started with a Slack message 😀

![image](https://cdn.sanity.io/images/809vdkvp/production/3c34e703e7a451198a198a7e572e73e880a26a39-950x308.png)


In less than a couple of weeks we already had a working solution, and in less than 3 months it was approved for internal usage in large scale-up:

![image](https://cdn.sanity.io/images/809vdkvp/production/8018ff65fb980097da6dd09044b65ee4ecd6849c-912x990.png)

We started testing the whole solution including the platform behind closed doors to see its viability and soon enough we realized that making it open-source will benefit much more people. So we did that!

![image](https://cdn.sanity.io/images/809vdkvp/production/6190843870941a17573b7f22d3b4e52162be6d01-1844x1238.png)

The open-source launch was quite successful and it further validated our idea & got us more closed-beta customers to closely cooperate with & build a solution they would be true advocates of.

We took the Q4 of 2022 to polish the Platform, improve its performance & usability and did a soft launch to open-beta in the middle of January 2023 🎊

![image](https://cdn.sanity.io/images/809vdkvp/production/6fa1a320d1d2043dce709aa5a4367d191d0de686-3484x1000.png)

At this time we are highly persuaded that we have one of the most solid Visual Testing Platforms on the market with a high potential to become a go-to solution when it comes to visual regression tests. We have battle-tested the platform on ridiculously large storybooks & pages and achieved above-benchmark performance.

We have a huge list of things that we want to ship, so stay tuned - Lost Pixel Platform will become even better(faster? 🤔) in the future!

## Learnings 🎓

* our beta customers were/are our unfair competitive advantage - just being in constant communication with them increased our probability of success drastically
    
* sometimes passion in some topic & constant stream of feedback are the only things you need to build meaningful things
    
* technical cofounder + technical cofounder = ✨
    
* focusing on making a limited group of customers successful &gt; going for a big-bang approach to capture the whole market(I think it's reasonable when starting out, might revise in the future)
    

## Closing thoughts 🧠

It's wild to imagine what we were able to achieve in the past 6 months and I am being super excited for the future of Lost Pixel, both Open Source & Lost Pixel Platform commercial solutions. I wanted to write this post to inspire you a little bit and maybe give some insights on things that helped us to succeed in the ship/improve/release cycle.

I'd love to hear your thoughts on [https://lost-pixel.com](https://lost-pixel.com) and [lost-pixel open-source engine](https://github.com/lost-pixel/lost-pixel) and if you have any questions or just want to chat about random topics I highly encourage you to join our [Lost Pixel discord](https://discord.gg/WqVjk49g9m) or just drop me a [DM on Twitter](https://twitter.com/divdev_).

Cheers and have fun building stuff!

Visual regression testing that makes sense

## Visual regression testing 101

### Introduction ✍🏼

Hey! My name is Dima, I am from Ukraine and I am keen on building products, designing beautiful UI/UX and testing automation. I am so passionate about visual regression tests that I've built an open-source tool called [Lost Pixel](https://lost-pixel.com) and guess what? It is a holistic visual regression testing framework!

I am excited to give you an introduction to the world of visual tests and walk you through the use cases in which they are most beneficial for your design & development flow. All of the below information is from years of me working with Visual Regression Tests digested into a short read.

### What 🖼️

**Visual regression testing** (a.k.a *vis reg tests, visual tests, UI snapshot tests, visual diff tests*) is an approach to automated QA. In this approach, we try to capture unintended changes to UI resulting from code modification. Visual tests can be applied across the whole frontend hierarchy starting from the atomic component, continuing with UI patterns utilising the components and ending with a page accommodating multiple UI patterns to present the user with a finished frontend. 

The visual regression test consists of three essential moving parts:

1. Screenshot of the element we want to test before the code change (a.k.a *baseline, intended version*)
2. Screenshot of the same element after the code change (a.k.a *compared image*)
3. Generated difference image in case the change in source code resulted in a different image(automatically generated by comparison engines like [pixel-match](https://github.com/mapbox/pixelmatch) or [odiff](https://github.com/dmtrKovalenko/odiff)) 


![image](https://cdn.sanity.io/images/809vdkvp/production/2e79892440f17379adbc2bcf974eb7b2c3a0caa9-940x788.png?w=450)

There are three possible outcome scenarios when doing a visual regression test:

1. **No difference** - tests pass as expected
2. **Intended difference** - test fails but the change is expected. In this case we need to update the baseline in our testing suite which will result in a no-difference run after we re-trigger the test run.
3. **Unintended difference** - test fails and change is unexpected. That's the reason we are doing visual tests in the first place - to easily catch such outcomes.

![image](https://cdn.sanity.io/images/809vdkvp/production/80975f2fe5d066c3aa0ad1fefc9873d44f42050b-940x788.png?w=450)

We have successfully covered the process of visual regression testing and now are ready to look into the most frequent use cases.

### How 🔀

If we talk tool-agnostic this is how Visual Regression Testing workflow look on the CI/CD pipeline.

![image](https://cdn.sanity.io/images/809vdkvp/production/9442aab8e86a1d9b4a9d20a80ffe20d83830c314-2151x1572.png?w=450)


### Use cases 🤔
Visual regression tests stand in between unit and integration tests on the testing trophy and are mostly used in the following products:

1. Design systems 
2. Component libraries
3. Application interfaces
4. Marketing & documentation pages

There are two main applications utilised in companies of different sizes and spheres. **Catching visual bugs** & **cross-functional teams collaboration**.


### Catching visual Bugs 🪲

You might already have some e2e tests in your application done with Cypress or Playwright but in most cases, they are not enough. Focusing purely on functional testing we always forget that sometimes totally functional elements of UI can look totally broken visually. Thus most frequent use case is to catch Visual Bugs with the ultimate goal being **shipping code more confidently**

#### Case studies 🔍

- Engineer is making amendments to the signature of the `<Box/>` component which is the primitive utilised in 100s of other design system components. By sheer number, it's hard to verify manually that none of the dependent components broke visually.

**❌ Problem:** The dependent components were too tightly coupled to `<Box/>` component and the changes rendered `<Datepicker/>` and `<FormControl/>` look totally broken for the end users.

**✅ Solution:** Visual tests picked the problem, automatically reported it and gave a bird-eye overview of places where it happens.

-  Product designer gives the engineer the high fidelity prototype in which an already existing component is slightly modified to accommodate to new design vision. The changes are looking good and the engineer ships the ticket. 

**❌ Problem**: Engineer misses a couple of usages of the modified component where it's used in the non-traditional layout with end users experiencing UI which looks broken

**✅ Solution:** Visual tests identified layout shifts, automatically reported it and gave a common ground for the product designer and engineer to solve the problem.

In both cases, Visual Regression Testing would be an automated way of making sure that change in one place would not cascade with visual bugs into other places.

### Collaborating in cross-functional teams 🤝🏼
With the usage of Visual Regression Tests, you start getting another tangible benefit which stems from the fact that you always **have a snapshot of the item** that you are testing. This opens multiple opportunities to collaborate:

1. Frontend Engineers can between themselves verify the correctness of the final result of their work
2. Engineers and Designers can sync in a way faster manner on the iterations done on the UI using the snapshots as common ground for discussion

#### Case study 🔍

- Engineer is working on the UX improvement of CTA-button prepared by the product designer. The Design System with high fidelity prototypes is a little bit out of date from the platform

**❌ Problem:** It's hard for an engineer to manually QA all the occurrences of the CTA button and even if possible would take lots of engineering hours. Happened that one of the missed cases is CTA button is used within different contexts making its label confusing for the users. Without  common ground, the communication between the two becomes a constant back and forth of sending screenshots of UI from one to another

**✅ Solution:** With help of Visual tests as the shared communication medium the scope of the problem is isolated giving a clear path for both collaborators to continue. The discussion happens by attaching concrete results of Visual Regression Tests.
