[Adverity](https://adverity.com) is the integrated data platform for connecting, managing, and using your data at scale. As Adverity scales, ensuring high-quality and comprehensive good-looking UI becomes vital. Visual Regression Testing is one of the fastest and easiest methods of testing aimed at ensuring a consistent & error-free UI. The Adverity team faced a unique set of challenges when it came to Visual Tests:

* Running million of visual tests monthly, requiring a fast and efficient tool for CI/CD pipelines ***that is affordable***
    
* Managing a large monorepo codebase, necessitating native monorepo support
    
* Catering to diverse visual testing needs, including testing the design systems, full pages with all the components integrated together, and end-to-end test results coming from Cypress & Playwright
    
* Collaborating within a large team responsible for managing and maintaining tests
    

## Lost Pixel Platform: A Scalable Solution for Visual Regression Testing

The [Lost Pixel Platform](https://lost-pixel.com) is a cutting-edge visual regression testing solution tailored to address the specific challenges faced by Adverity and other large organizations that need to scale their visual tests and not break the bank. It offers a range of features that cater to Adverity's unique requirements, ensuring streamlined processes and robust testing.

At this stage, Adverity is running **~1.5 million visual tests per month** on Lost Pixel Platform - let's explore what features of the platform the team finds appealing and how they leverage those to achieve their goals.

![](https://cdn.sanity.io/images/809vdkvp/production/6b7618da5686d2d3cb7f7cf6c3aacb07a0d67854-2232x1512.png)

### Efficient CI/CD Pipeline Integration

Lost Pixel Platform's fast and automated visual regression testing allows Adverity to run millions of tests monthly within their CI/CD pipelines. This efficiency enables the Adverity team to maintain high-quality standards across their product without compromising on speed.

### Native Monorepo Support

The platform provides native support for monorepos, making it an ideal choice for Adverity's extensive codebase. This feature allows the team to manage and maintain their tests efficiently, ensuring consistency and organization. Adverity leverages nx to ensure there is no minute wasted on unnecessary builds on the CI and Lost Pixel offers useful hooks to adapt to this on the platform side.

![](https://cdn.sanity.io/images/809vdkvp/production/c287ec286488d7596b63b3fdb89750e512c50a76-1828x1030.png)

### Flexible Visual Testing Options

Lost Pixel Platform accommodates diverse visual testing needs, from design systems with Storybook stories to full-page and end-to-end Cypress test results. This flexibility ensures that Adverity can comprehensively test all aspects of its product, catching visual regressions early in the development process.

### Enhanced Collaboration Capabilities

The Lost Pixel Platform offers collaboration tools such as comments, change rejection and integration with Slack and GitHub notifications. These features facilitate seamless communication and collaboration within Adverity's large development team, ensuring that all front-end developers can manage and maintain tests effectively.

## Lost Pixel + Adverity: A Partnership for Success

Adverity's adoption of the Lost Pixel Platform has led to substantial benefits and increased confidence in the visual part of Adverity offering across all boards of products:

* **Greater Efficiency**: Lost Pixel's fast and automated testing allows Adverity to run millions of tests monthly, maintaining high-quality standards across their product without sacrificing speed.
    
* **Improved Monorepo Management**: Native monorepo support streamlines the management and maintenance of Adverity's extensive codebase, ensuring consistency and organization of frontend components
    
* **Holistic visual testing**: The platform's flexible testing options cater to diverse visual testing needs, enabling Adverity to catch visual regressions early in the development process by following [Holistic Visual Regression Testing](https://lost-pixel.com/blog/post/holistic-visual-regression-testing) approach.
    
* **Seamless Team Collaboration**: Enhanced collaboration features allow Adverity's frontend developers to efficiently manage and maintain tests, fostering a cohesive and effective development environment.
    

By leveraging the Lost Pixel Platform, Adverity has effectively addressed its unique testing challenges, ensuring a consistent and high-quality user experience across its product.

How Lost Pixel Platform helps Adverity to run 1m visual tests per month

 ## Understanding Visual Regression Testing

As the world of coding continues to evolve, so must our testing strategies. As a result, a modern trend known as [visual regression testing](https://en.wikipedia.org/wiki/Regression_testing#Visual_regression_testing) is gaining traction. Visual regression testing is essentially a `spot the difference` game for your app, ensuring that new updates haven't led to unwanted visual changes for your end-users.

## Visual Regression Testing with Playwright

This guide will introduce you to a tool called [Playwright](https://playwright.dev/) for effective visual regression testing. Playwright can capture a screenshot of your website and then compare it with an older, approved version to detect any visual discrepancies.

Playwright is a free tool from Microsoft for visual regression testing of web apps. It can write scripts in various languages and run tests on different browsers, making it a versatile tool for modern web testing. In addition, playwright tests can run on Windows, Linux, and macOS, proving its adaptability across different platforms.

Imagine automated agents that understand your app's most critical user flows and can execute those flows without a hitch - filling inputs, clicking buttons, reacting to changes on the page, and much more. Playwright is a highly capable tool that makes writing such automation effortless.

### Who might find this article useful

1. **Software Developers**: Especially front-end developers or full-stack developers tasked with maintaining the visual consistency of applications and websites.

2. **Quality Assurance Engineers**: Professionals responsible for testing software to ensure it's bug-free and visually consistent.

3. **Product Managers**: Those who need to understand the testing process and may be deciding on the tools and processes used in their teams.

4. **DevOps Engineers**: They are often responsible for setting up [continuous integration (CI)](https://en.wikipedia.org/wiki/Continuous_integration) environments that might run these visual regression tests.

5. **UI/UX Designers**: They might be interested in understanding how their designs are being tested for visual consistency.

This guide will go over each step in detail and is also suitable for beginners!

### What This Guide Is About

This guide is a hands-on tutorial on using Playwright for visual regression testing in your continuous integration (CI) workflow. First, we'll dive into a real-world example to demonstrate how you can use Playwright to capture and compare screenshots of your web app, spotting any visual changes caused by updates to your code.

We'll also guide you on integrating these visual tests into your CI pipeline with [GitHub Actions](https://github.com/features/actions). This way, your tests run automatically every time you update your code, helping to maintain your app's visual consistency.

We will cover several approaches using the free native Playwright option and a cloud visual regression testing solution like [Lost Pixel](https://lost-pixel.com/) and compare their differences and use cases.

### Setting up native Playwright visual regression testing 

We are using [the official Playwright example as the starter code](https://github.com/microsoft/playwright/tree/main/examples/todomvc)

Install the dependencies:

`npm install && npx playwright inst
all`

Run tests locally to see them passing:

`npm run test`

See the report of the Playwright execution:

`npx playwright show-report`

Let's add couple of visual tests to the Playwright execution:

```javascript
test.describe("New Todo", () => {
  test("should allow me to add todo items", async ({ page }) => {
    // create a new todo locator
    const newTodo = page.getByPlaceholder("What needs to be done?");
    // Create 1st todo.
    await newTodo.fill(TODO_ITEMS[0]);
    await newTodo.press("Enter");

    // Make sure the list only has one todo item.
    await expect(page.getByTestId("todo-title")).toHaveText([TODO_ITEMS[0]]);

    // Create 2nd todo.
    await newTodo.fill(TODO_ITEMS[1]);
    await newTodo.press("Enter");

    // Make sure the list now has two todo items.
    await expect(page.getByTestId("todo-title")).toHaveText([
      TODO_ITEMS[0],
      TODO_ITEMS[1],
    ]);

    await checkNumberOfTodosInLocalStorage(page, 2);

    //  🚀🚀🚀 We are adding visual tests here 🚀🚀🚀
    await expect(page).toHaveScreenshot("new-todo.png");
  });
```
After running the tests(in this case, I've added a couple more) we should start seeing our snapshots appear in the file system
![](https://cdn.sanity.io/images/809vdkvp/production/4a58681db6e93fb3a7c4b7ffa9cd541d8e33a091-1850x1082.png)

Moreover, we should see the actual images corresponding to our test cases.

> Note how cool it is that we can make visual assertions numerous times in one test run, thus making sure that our application looks a certain way after meaningful user interactions

![](https://cdn.sanity.io/images/809vdkvp/production/210dea6b79fbfeaecc2ddb71f937e56d610d6dda-472x436.png)

In visual testing, we always operate with the following concepts: baselines, differences, and regressions. You can read more about them in this glossary. Let's assume that we have changed our application and want to ensure that the baselines are up to date with our tests.

```javascript
// Let's rename couple of the items in our TODO list
const TODO_ITEMS = [
  "buy some Cheddar cheese", 
  "feed the dog",
  "book a doctors appointment",
];
```

After running the tests, we should see an excellent report from Playwright showing us that we've got something new, and we did not expect to see this.

![](https://cdn.sanity.io/images/809vdkvp/production/670b1465df3f466f700acb417db97e229e1bab54-1544x1174.png)

If this was a visual regression - we would need to fix our application code, in this case we made this change deliberately so we can run Playwright with the following params to update the screenshots:

`npx playwright test --update-snapshots`

This code will update our snapshots stored as local files and ensure that our tests pass this time!

### GitHub Actions

Let's extend our setup to run the Playwright test suite on each and every commit. For this purpose let's create the following folders:

`.github/workflows`

Inside this folder, let's create `vrt.yaml` - where VRT is Visual Regression Testing, you can name it in anyhow; I just prefer this convention! The contents of the file should look like this:

```yaml
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      # Checkout and setup
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Install dependencies
        run: npm install
      - name: Install Playwright Browsers
        run: npx playwright install --with-deps chromium
      # Run Playwright tests
      - name: Run Playwright tests
        run: npx playwright test
      # Upload the test result artifacts
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: tests
          path: tests/
          retention-days: 30
```

Note the artifacts upload step - on your first run, you will most probably miss the baselines for the images due to them being taken on a different OS(Linux defaults on GitHub). You can download the `/tests` artifacts folder and replace your local one with it - after pushing new baselines, the run should pass.

You could use this method to update the baselines whenever the changes are intended, or you can go with a more elaborate way of creating a new GitHub action that could partially automate this process for you. I will not be covering it in this guide as it requires more than beginner knowledge in how GitHub actions work

Congratulations you now have a fully functional visual regression testing CI/CD setup that you could use on your pages!

If you want a more production-ready automated setup with a big boost of developer experience, continue reading because we will also cover how to migrate this setup to Lost Pixel Platform(Visual Regression Testing cloud).

## What is Lost Pixel Platform 🔎

[Lost Pixel Platform](https://lost-pixel.com) is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of [web pages and applications](https://lost-pixel.com/blog/post/holistic-visual-regression-testing). It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to quickly identify and fix visual bugs. In addition, it offers straight integration with GitHub & allows for easy collaboration between team members.

Lost Pixel is **free for open source** & offers a **generous free plan** for small projects!

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)

## Setting up Lost Pixel

You can use the [quickstart documentation](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform) from [lost-pixel.com](https://lost-pixel.com) to set up the platform to work with your repositories via installing the official Lost Pixel GitHub app. In this guide, we will focus on setting up the right configuration, assuming you have done the platform setup already!

You can use Lost Pixel Onboarding to quickly set up the project by copying & pasting the respective files.

In our `lostpixel.config.ts` that we need to create at the root of our project, let's define the custom tests(how to use Lost Pixel with Playwright or similar tools). This will make Lost Pixel aware that screenshotting process is done on your side, and it only needs to take care of comparing images & offering them on the platform

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
   customShots: {
    // 📒 currentShotsPath should be matching the one above from playwright test
    currentShotsPath: './lost-pixel',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

In our `.github/workflows/vrt.yml` at the root of our project let's define our CI/CD job with Lost Pixel. Note that you you need to use a `LOST_PIXEL_API_KEY` that you previosly set up on the Lost Pixel Platform.

```yaml
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      # Checkout and setup
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Install dependencies
        run: npm install
      - name: Install Playwright Browsers
        run: npx playwright install --with-deps chromium
      # Run Playwright tests
      - name: Run Playwright tests
        run: npx playwright test

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

Now let's replace our Playwright visual tests with just making screenshots:

Before: 

`await expect(page).toHaveScreenshot("new-todo.png")`

After:

`await page.screenshot({ path: 'lost-pixel/a.png', fullPage: true })`


With this simple setup, we are done with visual regression tests on Lost Pixel Platform 🙌🏼 Our first shot is arleady on the Platform

![](https://cdn.sanity.io/images/809vdkvp/production/25969909e68b0a383e5262abdf4cbe44da19b96c-1882x892.png)


## Summary

We have gone through the main aspects of why Playwright is an excellent tool for doing visual regression testing and covered two ways of managing your tests - fully open-source and with a bit of manual work & using Lost Pixel Platform that offers a battle-tested visual regression testing workflow.

If you need help, I am always happy to chat on [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m)!

Complete guide on Playwright visual regression testing

### What is Ladle

According to the founder, Ladle is a tool for developing and testing your React components in an isolated and faster environment than most real-world applications. It supports [Component Story Format](https://github.com/ComponentDriven/csf) – a concept widely popular thanks to [Storybook](https://storybook.js.org/). Ladle has been developed as a drop-in replacement for Storybook – it should already work with your existing stories.

### Why replacing Storybook

Storybook is an amazing tool with a thriving community of engineers. Storybook's vast ecosystem and feature richness bring their challenges and create a need for lighter/faster replacement. The main pain points when using Storybook are the following, [according to the creator of Ladle](https://ladle.dev/blog/introducing-ladle#storybook-%EF%B8%8F):

* **production build time** - often, it's the slowest part of the CI
    
* **dev mode start-up time** - not always faster than the related prod app
    
* **bundle output** - hosting storage + slow initial time to interactive
    
* **maintenance** - we repackaged Storybook, its dependencies, and configuration to provide a seamless setup for our developers; however, the addon versioning and monorepo setup make maintenance difficult
    
* **testing** - for our automated visual testing, we need to crawl and parse stories in a separate process since Storybook doesn't export a static list of stories (and other metadata)
    

The previous points are hard to disagree with as many engineers when asked about their experience using Storybook on a large-scale project, complain about its cumbersomeness and maintenance overhead.

### Visual tests in Ladle

As a perfect component playground, Ladle is frequently used for development & documentation purposes. This means that your Ladle instance holds your application's most important building blocks at any given time, starting from design system primitives and ending with pattern components. This makes Ladle a perfect tool for [visual regression testing](https://lost-pixel.com/blog/post/visual-regression-testing-101).

There are numerous ways of running the visual tests, you can start simple and [use low-level tools like playwright](https://ladle.dev/blog/visual-snapshots):

```javascript
import { test, expect } from "@playwright/test";
// we can't create tests asynchronously, thus using the sync-fetch lib
import fetch from "sync-fetch";

// URL where Ladle is served
const url = "http://127.0.0.1:61000";

// fetch Ladle's meta file
// https://ladle.dev/docs/meta
const stories = fetch(`${url}/meta.json`).json().stories;

// iterate through stories
Object.keys(stories).forEach((storyKey) => {
  // create a test for each story
  test(`${storyKey} - compare snapshots`, async ({ page }) => {
    // skip stories with `meta.skip` set to true
    test.skip(stories[storyKey].meta.skip, "meta.skip is true");
    // navigate to the story
    await page.goto(`${url}/?story=${storyKey}&mode=preview`);
    // stories are code-splitted, wait for them to be loaded
    await page.waitForSelector("[data-storyloaded]");
    // take a screenshot and compare it with the baseline
    await expect(page).toHaveScreenshot(`${storyKey}.png`);
  });
});
```

This works well for simpler projects with little maintenance and fewer collaborators. Whenever the need comes to set up visual tests on larger projects, it pays off to set up a more robust cloud solution.

Compared to simple Playwright visual regression testing, cloud platforms offer numerous benefits:

* easier maintenance flow with lots of utilities to compare images
    
* direct CI integration with zero-effort setup
    
* faster collaboration flow that allows easier sharing and working together on failing tests
    
* no overhead of keeping all of the images stored in the repository(becomes a real problem with lots of collaborators & commits)
    

### Lost Pixel Platform

When creating Lost Pixel some half a year ago, we decided to bet on Ladle. This decision seems great in retrospect, as we see a great need for a "lighter/faster/better to maintain" Storybook. Lost Pixel offers native ladle support, meaning you could start testing with a simple pointer to the running instance.

To run Lost Pixel Platform on your CI, you must first [set up a GitHub installation](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform#set-up-the-lost-pixel-platform).

After that, create lostpixel.config.ts:

`lostpixel.config.ts`

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  ladleShots: {
  // IP should be localhost when running locally & 172.17.0.1 when running in GitHub action(CI example)
    baseUrl: 'http://172.17.0.1:61000',
  },

  lostPixelProjectId: "xxxx",
  process.env.LOST_PIXEL_API_KEY,
};
```

And, of course, a CI run declaration - here, we are using GitHub actions as it is one of the simplest ways of running your CI/CD pipelines on GitHub

`visual-tests.yml`

```yaml
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
   steps:
     - name: Checkout
       uses: actions/checkout@v3

     - name: Setup Node
       uses: actions/setup-node@v3
       with:
         node-version: 18.x
         cache: "npm"

     - name: Install dependencies
       run: npm install

     - name: Build ladle
       run: npm run build

     - name: Serve ladle
       run: npm run serve &

     - name: Lost Pixel
       uses: lost-pixel/lost-pixel@v3.0.3
```

After pushing some new commits, you will see a new Lost Pixel Check on your GitHub repo & and also have a nice new build on Lost Pixel Platform:

![image](https://cdn.sanity.io/images/809vdkvp/production/e0134ff51f86ebe62c0e09b8dd60a45d04f97fc2-2092x1386.png)

### Summary

We have gone through the main aspects of why Ladle is a great alternative for Storybook and also covered how to do effortless visual regression tests with Ladle in Playwright & Lost Pixel Platform.

If you need help, I am always happy to chat on [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m)!

Visual regression testing with Ladle, so far the best Storybook alternative

## What is Storybook 🖼️

Storybook is an **open-source** tool for developing UI components in isolation. It allows developers to build and test components in a controlled environment, separate from the main application. It also provides a way to document and showcase the components, making it easier for other team members and stakeholders to understand and use them. Storybook can be used with various front-end frameworks like React, Vue, and Angular.

## **What is Lost Pixel Platform 🔎**

[Lost Pixel Platform](https://lost-pixel.com) is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of [web pages and applications](https://lost-pixel.com/blog/post/holistic-visual-regression-testing). It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to quickly identify and fix visual bugs. It offers straight integration with GitHub & allows for easy collaboration between team members.

Lost Pixel is **free for open source** & offers a **generous free plan** for small projects!

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)


## Let's integrate ⚒️

In this blog post, we will explore how to integrate **Storybook** with **Lost Pixel Platform**. We will run the process on **GitHub** with **GitHub Actions.** By combining these powerful tools, we can streamline the process of testing and validating our stories' visual appearance and ensure we capture all unintended changes that might occur from day-to-day development. Let's get started!

The final code for this tutorial [is available here](https://github.com/lost-pixel/lost-pixel-next-storybook)

### Installing storybook 🔄

Storybook is not usually used on its own, so we will use the starter of Storybook packaged with the Next.js application. This tutorial focuses on a fresh project, but you will have little problems using the Lost Pixel Platform with your existing storybook projects!

```bash
npx create-next-app@latest

npx storybook@next init
```

```bash
npm run storybook
```

This should result in the Storybook opening on your localhost:

![image](https://cdn.sanity.io/images/809vdkvp/production/a7393464db0f4bc9f14c329e291301ff750d2531-1874x816.png)

Congrats, if you are following this from the ground up, you have your **Storybook** ready for **Lost Pixel Platform** integration.

### Integrating Lost PIxel Platform 🔄

Before setting up our CI/CD tests, we need to ensure that the repository is all set & all of our previous work is pushed to **GitHub!**

![image](https://cdn.sanity.io/images/809vdkvp/production/0157f91d456b2b4bf8a7b9ca8b447b5ef47afd28-1870x1092.png)

To start working with Lost Pixel, visit [https://app.lost-pixel.com](https://app.lost-pixel.com) and sign in! You will be presented with a call to action to set up your GitHub repositories to work with lost-pixel action.

I already have something installed there, so UI might look different for you, but you will be anyways guided to GitHub if you click this(or the link in your respective UI):

![image](https://cdn.sanity.io/images/809vdkvp/production/107a21eedb865d98b2f599a3baa63d294d4cf252-2356x714.png)

![image](https://cdn.sanity.io/images/809vdkvp/production/d804c56006a5b860b5b3f215c683e8e105caa2a3-2382x1960.png)



Add your repository to the installation & save it! Congratulations, you are almost there, and now we will finally add a couple of code chunks!

### Lost Pixel configuration ⚙️

At this stage you need to follow onboarding instructions and copy paste couple of code blocks into your project:

![image](https://cdn.sanity.io/images/809vdkvp/production/f426c6d961d7edcdb53632b076b9ae9afadb439e-2358x1724.png)

Copy-paste the contents of the first block into your `lostpixel.config.ts`. This will do three important things:

1. Tell lost-pixel that we are operating on the level of storybook
    
2. Identify the project on the platform with its id
    
3. Provide a way to verify the run with `LOST_PIXEL_API_KEY`
    

Lost Pixel configuration end here & we need to set up GitHub Action to run the job continuously, so we are moving to copy-paste the code chunk #2 from the onboarding. For this, at the root of your project, create `.github/workflows` folder and paste the chunk from onboarding into `.github/workflows/vrt.yml`

![image](https://cdn.sanity.io/images/809vdkvp/production/e5b45190a4309fa6e685b295de86ee1b49c38e8d-2094x1730.png)

The last step before we push the code and enjoy our first Lost Pixel run is to put `LOST_PIXEL_API_KEY` into GitHub secrets. For this, let's paste the key from the third step of the onboarding into GitHub Actions secrets:

![image](https://cdn.sanity.io/images/809vdkvp/production/031b1b84966489ee167a8f417d465fec551e3999-2060x1232.png)


After pushing the code, you should see our shots on the platform

![image](https://cdn.sanity.io/images/809vdkvp/production/9c112d7330ada72ddc832063a17bf8938c5646e0-1668x1292.png)

## Summary 

That's it for the whole integration, you can see how in a couple of simple steps, we managed to set up our visual regression testing on [Lost Pixel Platform](https://lost-pixel.com). 

If you liked the read and enjoyed setting up your visual tests, you might like our [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m). There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

Storybook visual regression testing with Lost Pixel Platform

## Flaky visual regression tests. Not a problem.

Flakiness is one of the most common issues engineers face with [visual regression testing](https://lost-pixel.com/blog/post/visual-regression-testing-101). In this article, we will explore ways to reduce flakiness, improve the stability of your visual tests and decrease the maintenance time the engineers spend.

One of the cornerstones of proper visual regression testing is tooling. There are many ways of ensuring your UI is rendered in a stable way across CI/CD runs but in the end, it's the flexibility of the VRT(Visual Regression Testing) tools that reduce the noise to the maximum extent. 

In this guide, we will focus on setting up the right configuration for different flakiness cases and eliminate them one by one using the flexible and extendable config of Lost Pixel - open source visual regression testing tool.

## **What is Lost Pixel Platform 🔎**

[Lost Pixel Platform](https://lost-pixel.com) is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of web pages and applications. It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to quickly identify and fix visual bugs. It offers straight integration with GitHub & allows for easy collaboration between team members.

Lost Pixel provides great flexibility when it comes to setting up your tests. We will go through **5 most common** flakiness scenarios and fix them with various methods provided by Lost Pixel Platform.

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)

## Before we start
The Lost Pixel open-source engine already packs numerous methods to reduce flakiness and ensure smooth visual test execution. Namely, you get the following internal methods out of the box by simply using the Lost Pixel base configuration.

- **Disabling animation** - we try to disable animations on the web browser when running your visual tests, as animation usually does not complete with > 1-millisecond precision that would ensure consistent results.
- **Requests handling** -  when doing visual tests, you can run them on isolated components and full-blown apps with network functionality. Lost Pixel ensures that all requests on the given page are executed before the snapshot is taken to show the state you intend to have.
- **Antialiasing** - non-deterministic rendering is a common problem that causes flakiness in comparing fonts, color hues, and other pixel entities. Lost Pixel uses modern diffing engines that use the latest research on comparing images that already consider all potential problems.

Although the above approaches to reducing flakiness in visual regression tests prove to help a lot, there is far more to it because we are discussing various tools people use to render their frontends. Sometimes there might be derailments, and this guide focuses on fixing them.

### Non-deterministic rendering

Even using a scientific approach to identifying false-positive regressions sometimes fails. Enter thresholds - configuration object to fine-tune the sensitivity of diffing engine. You can define the thresholds on the mode level(storybook, ladle, pages, custom) or on the individual test level making it even more granular. Furthermore, you can decide if you want sensitivity to work on an absolute or relative level by choosing the amount of % difference or pixel amount, which is tolerable and won't be considered a regression.

```
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
    threshold: 100,

  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books', threshold: 0.1 },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

```

### Live elements on the page 
Sometimes your page contains elements like videos, lazy-loaded images, or other non-css animated elements that you still don't want to remove from the flow of the page, but you want to test everything else. In this case, masking is one of the easier methods to achieve this. 

You can easily target elements with basic html attributes and mask them - this will ensure that whenever the element is rendered, it will be hidden behind the box of solid color, ensuring they are displayed the same way across the test runs.

```
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {

  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books', threshold: 0.1 },
      { path: '/layouts/books/biography', name: 'biography-pages',     
      mask: [{selector: 'video'}]
     
 },
    ],
    baseUrl: 'http://172.17.0.1:3000',
    mask: [{ selector: 'code' }, { selector: 'h2' }],
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};

```


### Elements that don't belong to visual testing

Sometimes in your visual tests, you might have elements that don't provide immediate value and can be excluded from the general flow of the page.

In this case, we can use the configure browser functionality of Lost Pixel and prepare the browser the way we wish before taking the shot. In this case, we can exclude the elements from DOM and do it in the centralized place that belongs to visual tests.


```
module.exports = {
  pageShots: {
    pagesJsonUrl: 'http://172.17.0.1:9000/lost-pixel.json',
    baseUrl: 'http://172.17.0.1:9000',
  },
  lostPixelProjectId: 'clb5ek3mm1772001qqg7yban38',
  apiKey: process.env.LOST_PIXEL_API_KEY,
  beforeScreenshot: async (page) => {
    await page.addStyleTag({
      content: `iframe {
          visibility: hidden;
        }
        /* do not show underline animation */
        #toc-holder  a {
          background-size: 0 !important;
          background-image: none !important;
        }
        /* skip image display within section */
        section img {
          visibility: hidden;
        }
        /* hide cookie banner */
        #onetrust-consent-sdk {
          display: none;
        }`,
    })
  },
}
```

### Elements that take time to prepare

Sometimes you have an element on the page that needs to participate in visual regression testing flow, but it takes time to prepare itself. In case it's not about animation and network requests but some other functional aspect of your app - you can fine-tune various wait parameters of `lostpixel.config.js|ts` to ensure you test the UI in the right time of its lifecycle.

```
module.exports = {
  pageShots: {
    pagesJsonUrl: 'http://172.17.0.1:9000/lost-pixel.json',
    baseUrl: 'http://172.17.0.1:9000',
  },
  lostPixelProjectId: 'clb5ek3mm1772001qqg7yban38',
  apiKey: process.env.LOST_PIXEL_API_KEY,
  waitBeforeScreenshot: 5000,
  waitForFirstRequest: 2000,
  waitForLastRequest: 10000,
}
```

### Some tests are better not run

In extreme cases, you have a very hard test to fix, but you still want it to lay around for documentation purposes, as in the case of testing Storybook or Ladle. In such a scenario, you could still leave your story be but disable all Lost Pixel runs on it:

```
export const WithPref = () => (
    <Stack>
        <Button prefix={<CircularProgress />}>Button with prefix</Button>
        <ButtonMinimal prefix={<IconAccess />}>
            Button with Icon
        </ButtonMinimal>
    </Stack>
)

WithPref.parameters = {
    lostpixel: { disable: true },
}
```


### Summary

We have gone through the most common ways of battling flakiness in visual testing and explored how the Lost Pixel Platform can help your tests to be stable and robust! Be aware that extremely flaky tests are not doing good to your overall confidence hence if you can't fix them either with utilities from Lost Pixel or on your end with the code - it's almost always better to get rid of them!

If you need help, I am always happy to chat on [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m)!


Handling flaky visual regression tests with Lost Pixel Platform

## Why testing PDFs 🗃️

Visual regression testing for PDFs is vital in today's digital world, ensuring document accuracy and consistency. Organizations can detect unintended visual differences by implementing PDF visual regression testing, maintaining quality, and enhancing user experience. This approach to PDF testing not only safeguards against errors but also strengthens client trust and preserves an organization's reputation. In this tutorial, we will learn how to test your PDFs generated by the **Lost Pixel Platform**, where you can test **7,000** PDFs each month **for free**.

The code for the result of this tutorial [can be found here](https://github.com/lost-pixel/lost-pixel-pdf-tests).

## **What is Lost Pixel Platform 🔎**

[Lost Pixel Platform](https://lost-pixel.com) is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of web pages and applications. It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to identify and fix visual bugs quickly. It offers straight integration with GitHub & allows for easy collaboration between team members.

We will be utilizing the **Custom Shots** mode of Lost Pixel Platform to test our PDFs, let's jump straight into that.

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)

## Create PDF

We will be using `react-pdf/renderer` which has a lot of nice utilities to create our template & allow PDF generation:

```typescript
import React from "react";
import ReactPDF, {
  Page,
  Text,
  View,
  Document,
  StyleSheet,
} from "@react-pdf/renderer";

type TemplateData = {
  companyName: string;
  companyPhone: string;
  companyEmail: string;
  receiptNumber: string;
  datePaid: string;
  paymentMethod: string;
  amount: string;
};

interface PDFProps {
  data: TemplateData;
}

const styles = StyleSheet.create({
  page: {
    backgroundColor: "white",
  },
  section: {
    margin: 10,
    padding: 10,
  },
  heading: {
    fontSize: 24,
    fontWeight: 600,
    color: "salmon",
    marginBottom: 8,
  },
  statement: {
    fontSize: 20,
    color: "#131925",
    lineHeight: 1.4,
    marginBottom: 4,
  },
  divider: {
    width: "100%",
    height: 1,
    backgroundColor: "#999999",
    margin: "24px 0 24px 0",
  },
  paragraph: {
    fontSize: 12,
    color: "#212935",
    lineHeight: 1.67,
  },
  columnParent: {
    flexDirection: "row",
    justifyContent: "space-between",
  },
  columnStart: {
    flex: 1,
  },
  columnEnd: {
    flex: 1,
    alignItems: "flex-end",
  },
});

const PDF = ({ data }: PDFProps) => {
  return (
    <Document>
      <Page size="A4" style={styles.page}>
        <View style={styles.section}>
          <View style={styles.columnParent}>
            <View style={styles.columnStart}>
              <Text style={styles.heading}>{data.companyName}</Text>
              <Text style={styles.paragraph}>{data.companyPhone}</Text>
              <Text style={styles.paragraph}>{data.companyEmail}</Text>
            </View>
            <View style={styles.columnEnd}>
              <Text style={styles.heading}>Receipt</Text>
              <Text style={styles.paragraph}>
                Receipt number: {data.receiptNumber}
              </Text>
              <Text style={styles.paragraph}>Date paid: {data.datePaid}</Text>
              <Text style={styles.paragraph}>
                Payment method: {data.paymentMethod}
              </Text>
            </View>
          </View>
          <View style={styles.divider}></View>
          <View>
            <Text
              style={styles.statement}
            >{`${data.amount} paid on ${data.datePaid}`}</Text>
            <Text style={styles.paragraph}>Thank you for your business!</Text>
          </View>
        </View>
      </Page>
    </Document>
  );
};

export default async (data: TemplateData) => {
  return await ReactPDF.renderToStream(<PDF {...{ data }} />);
};
```

Our server to create actual PDF files will look like this(you can find the detailed explanation of the individual code chunks in the **comments**)

```typescript
import express from "express";
import createTemplate from "./create-template";
import fs from "fs";
import { Readable } from "stream";
import { fromPath } from "pdf2pic";
import { stableDataExamples } from "./data";

const app = express();
app.use(express.json());
const port = 3000;

// use react pdf template & create a real PDF out of it

const savePDFToFile = async (pdfStream, fileName) => {
  const writeStream = fs.createWriteStream(fileName);
  const readable = new Readable().wrap(pdfStream);
  readable.pipe(writeStream);

  return new Promise((resolve, reject) => {
    writeStream.on("finish", resolve);
    writeStream.on("error", reject);
  });
};

// to test our PDFs for visual regressions we will need to get an image representation of them. This function will be responsible for converting PDF into PNG

const savePDFToPNG = async (pdfPath, pngFileName) => {
  try {
    const converter = fromPath(pdfPath, {
      density: 100, // output pixels per inch
      saveFilename: pngFileName.split(".")[0], // output file name
      savePath: "./lost-pixel", // output file location
      format: "png", // output file format
      //This accounts to A4 size
      width: 2480,
      height: 3508,
    });

    const data = await converter(1);
    console.log(`Saved PNG: ./lost-pixel/${pngFileName}`);
  } catch (error) {
    console.error("Error converting PDF to PNG:", error);
  }
};

// to simplify the PDF example and give a proper attention to viusal tests we will limit the setup to the following. Whenever we execute npm run test command we will run the following code that will generate PDFs & respective PNGs

const server = app.listen(port, async () => {
  console.log(`The sample PDF app is running on port ${port}.`);

  for (let i = 0; i < stableDataExamples.length; i++) {
    const data = stableDataExamples[i];
    console.log(`Generating PDF with stable data example ${i + 1}: `, data);

    const pdfStream = await createTemplate(data);
    const pdfFileName = `stable-data-example-${i + 1}-export.pdf`;
    await savePDFToFile(pdfStream, pdfFileName);
    console.log(
      `Saved PDF with stable data example ${i + 1} to the file system.`
    );

    await savePDFToPNG(pdfFileName, `stable-data-example-${i + 1}-export.png`);

    server.close();
  }
});
```

## **Lost Pixel setup ⚙️**

### Lost Pixel Config

Use the [**quickstart documentation**](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform) from [**lost-pixel.com**](https://lost-pixel.com) to set up the platform to work with your repositories via installing the official **Lost Pixel GitHub app**. We will go straight ahead into the setup for the tests!

`lostpixel.config.js` setup is super minimal, we need to create this file at the root of the repository with the following contents:

```javascript
module.exports = {
  customShots: {
    currentShotsPath: "./lost-pixel",
  },
  lostPixelProjectId: "clgqfqifx0is5k80ef9a72utq",
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

This will ensure that we are using Lost Pixel in custom mode and will point it to the right project on the Lost Pixel Platform via its `id` that you should have already created in the previous step!

---

### GitHub action

You could you the CI/CD pipeline of your liking, we will be using GitHub Actions in this tutorial. Let's create `.github/workflows/vrt.yml` at the root of our project with the following contents:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Install ImageMagick
        run: sudo apt-get install -y imagemagick graphicsmagick

      - name: Create lost-pixel folder
        run: mkdir lost-pixel

      - name: Run test script
        run: npm run test

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.4.0
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

This will ensure that on every commit, we are running our CI flow consisting of installing dependencies and image utils that we need to convert PDF into PNG. After that, we create the lost-pixel directory where we will expect our custom shots and create the shots by running the test script: `"test": "tsc --build && node build/index.js"` . Finally, we run the Lost Pixel tests!

---

### Lost Pixel Platform 🖼️

After committing the code and pushing it to the remote, we should in almost no time see the results of the test run on the platform:

![image](https://cdn.sanity.io/images/809vdkvp/production/633dac4912c1d838761672e3f7f7144f73b1bd61-1746x2062.png)

Now you can easily monitor your PDFs for visual integrity and ensure they all look exactly like they should!

### Summary

We have implemented a Visual Regression Testing workflow for our PDFs, but it does not end here. While this tutorial is quite simplistic and focuses on the core implementation of Visual Regression Testing on CI, you can extend your visual tests with many flexible options to make them less flaky and more robust!

If you need help, I am always happy to chat on [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m)!

PDF visual regression testing with react-pdf and Lost Pixel Platform

### Why you might need visual tests in monorepo 📦

Monorepos have become increasingly popular in software development due to their ability to centralize and streamline code management. Ensuring consistency and preventing visual regressions becomes crucial with multiple interconnected components or applications in a single repository. Here are a few reasons why visual regression testing is essential in monorepos:

1. **Multiple Frontends**: In a monorepo with several front-end applications, visual regression testing can help detect unintended design changes and ensure a consistent user experience across all frontends.
    
2. **Design Systems**: Monorepos often include design systems that provide reusable UI components for multiple applications. Visual regression testing ensures that updates to the design system do not cause unexpected visual discrepancies in the consuming applications.
    
3. **Component Libraries**: Monorepos may contain component libraries shared by different projects. Visual regression testing guarantees that changes to these shared components do not introduce visual bugs in the dependent projects.
    
4. **Collaboration**: As monorepos facilitate collaboration among multiple teams working on different parts of the project, visual regression testing helps maintain design consistency and prevents visual issues introduced by cross-team contributions.
    

In summary, visual regression testing is critical in maintaining the integrity of a monorepo's design and user experience. By incorporating it into your development workflow, you can minimize visual inconsistencies and improve the overall quality of your projects.

> Our end goal is to have a continuous integration flow with GitHub actions that notifies us whenever there is a visual regression in our monorepo

Let's learn how you can add visual tests to your monorepo with **Turbo & Lost Pixel.**

**The finalized code is accessible:** [here](https://github.com/lost-pixel/monorepo-example)

### What is Turborepo 🖼️

[Turborepo](https://turbo.build/repo) is a high-performance monorepo build system that aims to optimize and simplify the management of codebases. By enabling efficient caching, parallelization, and incremental building, it provides a robust solution for developers to maintain and scale their projects seamlessly. In this blog post, we will explore why there is a need for Visual Regression Testing in monorepos & how we can easily add them using the Lost Pixel Platform.

### What is Lost Pixel Platform 🔎

[Lost Pixel Platform](https://lost-pixel.com) is a cloud-based visual regression testing platform that enables developers to test and validate the visual appearance of web pages and applications. It automates the testing process, helping to ensure that the visual integrity of the application is maintained. The platform captures screenshots of web pages and compares them to previous versions, allowing developers to identify and fix visual bugs quickly. It offers straight integration with **GitHub** & allows for easy collaboration between team members.

[Lost Pixel is free for open source](https://github.com/lost-pixel/lost-pixel) & offers a generous free plan for small projects!

![image](https://cdn.sanity.io/images/809vdkvp/production/43df0c7c3b10ffaf5d71b88900e1b7ee02bab03a-3484x1000.png)

### Setup ⚙️

**Turborepo**

We will be using the [basic Turborepo example](https://github.com/vercel/turbo/tree/main/examples/basic) - it contains two Next.js web apps that we will use to demonstrate the Visual Regression Testing workflow.

Pull the repo and copy the contents of the basic into a separate repo, we will be using [GitHub actions](https://github.com/features/actions) so you will need to create a repo on GitHub as well to follow this tutorial in the best way! Your repo contents should look similar to this:

![image](https://cdn.sanity.io/images/809vdkvp/production/a4bb100e894670a9cfa779ec094d0f2899a16dfd-304x460.png)

Running `pnpm run dev` at the root of monorepo should launch two browser windows with the simple demo apps!

---

**Lost Pixel Platform**

Follow the [documentation of Lost Pixel to install the app on your GitHub repository](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform#set-up-lost-pixel-platform).

> ❗Note: when you enable the monorepo mode, which we are going through in this section you will have **2 distinct project IDs** that we will use in the last integration step.

After following a simple setup on GitHub & Lost Pixel Platform you should see your repo in the repositories list.

![image](https://cdn.sanity.io/images/809vdkvp/production/7b36fe062ae3f729320478ce6d4d2fa75ddf810b-1688x968.png?)

In the repository view, you now need to switch to monorepo mode, and then you can rename your project to something more convenient:

![image](https://cdn.sanity.io/images/809vdkvp/production/be8ea3429d72bf769b37be50aa4e4b81bdeed714-1614x830.png)

Finally, create a new project inside of this repository:

![image](https://cdn.sanity.io/images/809vdkvp/production/e091b37a5abb5b63e9cc23827f1ebdac8861c50d-826x430.png?w=450)

You can now easily switch between the projects using the project dropdown shown above and are ready to do the final for integrating visual regression tests into your monorepo - setting up the GitHub action to run visual tests on every new commit to your repository!

---

**GitHub Action**

Let's look at `.github/workflows/vrt.yml`. This file will instruct GitHub action runners to run our Visual Tests in monorepo in parallel by utilizing the Matrix jobs. If boiled down to one sentence, it will: install dependencies for each part of monorepo, run both web apps & execute `lost-pixel` on both of them(we will still need to define what pages we need to test, and we will do this straight up next).

> Don't forget to add your: `LOST_PIXEL_API_KEY` to [GitHub repository secrets](https://docs.github.com/en/rest/actions/secrets?apiVersion=2022-11-28). You will see your `LOST_PIXEL_API_KEY` in the onboarding UI inside the repository on the Lost Pixel Platform. You can also get a secret by visiting the [workspace settings](https://app.lost-pixel.com/app/settings)

```yaml
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  lost_pixel:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        config:
          - {
              package: "apps/web",
              name: "Lost Pixel for Web",
              command: "pnpm run dev",
            }
          - {
              package: "apps/docs",
              name: "Lost Pixel for Docs",
              command: "pnpm run dev",
            }
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install pnpm
        uses: pnpm/action-setup@v2
        with:
          version: 7.17.1

      - name: Install dependencies
        run: pnpm install

      - name: Cache Build
        uses: actions/cache@v2
        with:
          path: |
            apps/web/.next
            apps/docs/.next
          key: ${{ runner.os }}-build-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-build-${{ github.sha }}

      - name: Start App
        run: cd ${{ matrix.config.package }} && ${{ matrix.config.command }} &
        env:
          CI: true

      - name: ${{ matrix.config.name }}
        uses: lost-pixel/lost-pixel@v3.3.0
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
          LOST_PIXEL_CONFIG_DIR: ${{ matrix.config.package }}
  finalize:
    needs: [lost_pixel]
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Lost Pixel Finalize
        uses: lost-pixel/lost-pixel@v3.3.0
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
          LOST_PIXEL_CONFIG_DIR: apps/web
        with:
          FINALIZE: true
```

---

**Lost Pixel config**

The last step of the integration is to instruct Lost Pixel in each individual monorepo project what tests we are expecting to create `lostpixel.config.ts` with the following contents:

```typescript
import { CustomProjectConfig } from "lost-pixel";

export const config: CustomProjectConfig = {
  pageShots: {
    pages: [
      { path: "/", name: "web-home" },
    ],
    baseUrl: "http://172.17.0.1:3000",
  },
  lostPixelProjectId: "your-web-project-id-from-lost-pixel-platform",
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

In `apps/docs` create `lostpixel.config.ts` with the following contents:

```typescript
import { CustomProjectConfig } from "lost-pixel";

export const config: CustomProjectConfig = {
  pageShots: {
    pages: [
      { path: "/", name: "docs-home" },
    ],
    baseUrl: "http://172.17.0.1:3000",
  },
  lostPixelProjectId: "your-docs-project-id-from-lost-pixel-platform",
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

This will instruct Lost Pixel to test the base route of your applications running in GitHub action. Note that you need to paste the correct `lostPixelProjectId` from respective projects on the Lost Pixel Platform for this to function!

We aim at this as our result - our visual tests are on the platform now, and you can look at them by switching the projects in the tab!

![image](https://cdn.sanity.io/images/809vdkvp/production/d5f200ffe68f048ce107b201cfe49c9b043a0a4a-1796x1054.png)

Furthermore, we have a full monorepo setup, and our commit status checks are indicating if our code has visual regressions:

![image](https://cdn.sanity.io/images/809vdkvp/production/492c7fc627048a94453040716e23c60171a64ebe-874x476.png)

### Summary

We have implemented a Visual Regression Testing workflow for our turborepo, but it does not end here. You can extend your Lost Pixel runs with a [holistic approach to visual regression testing](https://lost-pixel.com/blog/post/holistic-visual-regression-testing) by running visual tests not only on your pages but also on storybooks and E2E tests, you could also extend your mono repo with a Design System & set up [Storybook component visual tests](https://stackonfire.com/storybook-visual-regression-testing) there still fitting it into your monorepos setup.

If you need help, I am always happy to chat on [Visual Regression Testing discord](https://discord.gg/WqVjk49g9m)!

Monorepo Visual Regression Testing. Why & How. (Turborepo example)

[Prisma](https://www.prisma.io/) is the next-generation Node.js and TypeScript ORM that streamlines database access, schema migrations, and code generation, enabling developers to build data-driven applications with greater efficiency and ease. Prisma's comprehensive documentation excels in providing extensive coverage and clear guidance, allowing developers to harness the full potential of the tool and get successful in writing their type-safe database queries in minutes

## Documentation team challenges

As Prisma continues to grow in popularity to become the most downloaded ORM, maintaining high-quality and comprehensive documentation is paramount. [The documentation](https://www.prisma.io/docs) serves as a crucial resource for developers seeking to harness the full potential of both Prisma ORM & Prisma Data Platform. However, the Prisma documentation team faces a variety of challenges while working to ensure that their extensive guides and resources remain accurate, visually appealing, and consistent. Those challenges include:

![image](https://cdn.sanity.io/images/809vdkvp/production/3125a13b44c40004ce6c9657163076643ef10396-3260x2010.png)

### Time-Consuming Manual Checks

Manually verifying the content and appearance of numerous documentation pages is a labor-intensive process. With constant updates and a large number of pages, manual checks become a significant time sink for the team.

### Ensuring Visual Appeal and Consistency

The documentation serves as the first point of contact for many users, making it crucial to maintain a visually appealing and consistent appearance. This helps to reinforce Prisma's image as a reliable and professional tool.

### Managing Contributions from Multiple Sources

Both internal team members and external contributors provide content updates, requiring a robust system to ensure that changes maintain the desired quality and look. Coordinating and reviewing these contributions can be a complex and time-consuming task.

### Efficient and Reliable Testing for Long Pages

With numerous extensive pages in the documentation, the team must have an efficient and dependable solution that enables them to review and identify changes effectively. This allows them to iterate quickly and ensure that all pages meet the desired standard of quality.

## Lost Pixel Platform: A Solution for Visual Regression Testing

The [Lost Pixel Platform](https://lost-pixel.com
) is a powerful solution for visual regression testing, addressing the challenges faced by Prisma's documentation team. By leveraging the DX first approach of Lost Pixel, the team can streamline their processes and enhance the quality of their documentation.

![image](https://cdn.sanity.io/images/809vdkvp/production/a6429ed1332265037d8f9d8a10f7a27b3b52bbff-2128x1150.png)

### Fast, Automated Testing

Lost Pixel Platform offers fast and automated visual regression testing, significantly reducing the time spent on manual checks. This enables the Prisma documentation team to focus on creating and updating content, ensuring that their API references, guides and other content pieces remain relevant and up-to-date.

### Intuitive Review flow

The platform provides intuitive review features, allowing the team to easily navigate and identify changes in their documentation that could potentially be regressions. This simplifies the process of reviewing lengthy pages and ensures that all visual aspects are consistent and appealing.

### Seamless Integration with Collaboration Tools

Lost Pixel Platform integrates seamlessly with popular collaboration tools and has first-class GitHub support, making it easier for the Prisma documentation team to coordinate and manage contributions from both internal and external sources. Being tightly integrated into the CI pipelines Lost Pixel allows to catch visual regressions long before the PR gets merged and hits customers. This streamlined workflow leads to improved efficiency and better control over the quality and consistency of the documentation.

### Simplifying Content Changes and Previews

Lost Pixel Platform makes it easy to compare the updated version of a page with its previous state, improving the process of reviewing content changes. This eliminates the need to manually visit each updated section, saving time and reducing the likelihood of overlooking crucial differences between versions.

## Lost Pixel + Prisma partnership

The Prisma team saw the potential of the Lost Pixel Platform early on and decided to become one of its first adopters. This partnership has been beneficial for both parties, as it has helped shape the development of the platform to better suit the needs of Prisma's documentation team. Prisma saw some immediate benefits from adopting Visual Regression Testing workflow into their daily routine:

1. **Improved Efficiency**: The Lost Pixel Platform's automated visual regression testing has significantly reduced the time spent by the team on manual checks. This has allowed the Prisma documentation team to focus on creating and updating valuable content for developers.
    
2. **Enhanced Quality and Consistency**: Lost Pixel's intuitive review flow and seamless integration with GitHub have facilitated better coordination and management of contributions. This has led to more consistent and visually appealing documentation that reflects Prisma's commitment to professionalism and reliability.
    
3. **Rapid Detection of Regressions**: The platform's integration into Prisma's CI pipelines catches visual regressions before they reach users, ensuring that the documentation remains accurate and up-to-date at all times. This way Prisma spends way less time fixing visual bugs that already went live thus increasing user satisfaction with their docs.
    
4. **Streamlined Content Review**: Lost Pixel Platform simplifies the process of reviewing content changes, making it easy for the team to spot crucial differences between versions. This eliminates the need for manual checks, saving time and minimizing the risk of overlooking essential updates.

Prisma Docs: Visual regression testing case study

## Introduction 👀

Are you looking for the best [visual regression](https://stackonfire.com/visual-regression-testing-101) testing tools to ensure your website or application looks and functions as intended? Look no further than our **Ultimate Guide to Visual Regression Testing**. In this article, we'll explore the top five tools you should try to streamline your testing process and catch any design or functionality issues before they impact your users. We will explore both free and paid options and go through the following tools and their respective use cases:

1. Playwright snapshots (Open Source)
    
2. Loki (Open Source)
    
3. Chromatic (SaaS)
    
4. Percy (Saas)
    
5. Lost Pixel (Open Souce + SaaS)
    

## Playwright snapshots 🎭

[Playwright](https://playwright.dev/) is an open-source Node.js library for automating web browsers. Microsoft developed it and designed it to work with all major browsers, including Chrome, Firefox, and Safari. Playwright allows developers to write automated tests that interact with web pages, simulate user actions, and capture screenshots or videos of the browser window.

**Playwright** is a tool not explicitly designed for visual testing but offered utilities to streamline the whole visual testing process. Anytime in your E2E test run, you can `await expect(page).toHaveScreenshot();` Ensure that your page looks exactly like you expect it to look at any given point in time.

Playwright offers various utilities to reduce the tests' flakiness and ensures you can configure the thresholds for maximum tolerable pixel differences in your tests. It comes in handy when your app relies on non-deterministic rendering and can have minor visual differences not distinguishable to the human eye and thus not relevant as negative cases.

Being an **open-source tool,** Playwright relies on your running the testing & baseline update process on your premises - either on CI or locally on a developer machine. This is both pros & cons, as you are missing a user interface to inspect the results of the visual tests, but at the same time, you are in complete control of your difficulties and don't need to use a third-party service to manage your visual tests. Being able to render practically anything, Playwright is a great tool to test your marketing pages, Storybook components or anything that can be caused in the browser. However, this would almost always require additional setup and tinkering.

## Loki.js 👁️

[Loki.js](https://github.com/oblador/loki) is an open-source visual regression testing tool for your Storybook components. It offers a low-maintenance workflow to test your Storybook components across different devices (React native supported). Although as robust as Playwright for the Storybook test, Loki.js is limited in power when trying your application pages or running a visual regression analysis after interacting with your front end. To achieve that, you would need to do some extra setup and probably add everything you want to test in Storybook.

Loki.js integrates perfectly with any of the CI providers, which makes it super easy to incorporate into your existing testing pipeline. It possesses the same limitation as the Playwright here, though, as you don't have an integrated user interface to work with the test results, and it usually boils down to comparing the images locally with a tool like [Kaleidoscope](https://kaleidoscope.app/). Although this workflow is robust and is used by many companies, the main struggle comes from blocking the developer machines to run the tests locally and producing the new snapshots, thus making it very costly to maintain. This also hinders collaboration between designers/engineers as the test results can be scattered across different user machines/ ci runs.

## Chromatic 🌈

[Chromatic](https://www.chromatic.com/) is a SaaS visual regression testing tool designed to work with [Storybook](https://storybook.js.org/), a popular open-source tool for building UI components. Chromatic integrates with your existing Git workflow to automatically test and review UI changes in your components, ensuring they look and function as intended.

As we are delving more into cloud solutions, Chromatic is a clear step up in most aspects of the **visual regression testing flow.** It is easy to set up and integrates well with your Git flow, meaning you can be sure you don't accidentally release breaking visuals to your customers. Finally, it offers a UI that fosters collaboration and makes it very easy to review the test results.

The main limitation here is that being a Storybook tool, Chromatic excels at providing utilities for working with Storybook but does not make it easy to run visual tests on your pages, automated tests and other use cases where they span beyond Storybook.

## Percy 🦔

[Percy](https://percy.io/) is a SaaS visual regression testing tool that allows you to automate your visual testing process. It integrates with your existing testing framework and captures screenshots of your web application or website to detect visual changes.

Percy offers a user-friendly interface to manage your visual tests and provides a collaborative environment for designers and developers to work together on UI changes. It also provides features like version control, commenting, and collaboration tools to streamline your testing process.

Being one of the oldest tools on the market, Percy covers a lot of use cases. Therefore, he can easily integrate both into your marketing pages, Storybook, and other visual testing use cases. In addition, it is owned by [BrowserStack](https://www.browserstack.com/), so it logically offers to test the magnitude of different operating systems, devices and viewports. Percy being held by a large testing firm spawns some cons, though - it is one of the most expensive tools on the market. In addition, it is usually bulkier in setup & maintenance compared to other tools mentioned in this article.

## Lost Pixel 🖼️

Lost Pixel is a combination of both open-source and SaaS offerings. It uses a [lost-pixel](https://github.com/lost-pixel/lost-pixel) open-source engine that resembles Loki.js in the easiness of setup and maintenance but covers all of the use cases for visual testing(SStorybook, application pages, E2E test) referred to as **Holistic Visual Regression Testing** and offered [Lost Pixel Platform](https://lost-pixel.com/), which integrates into your existing Git workflow and enables collaboration through UI for reviewing the test results.

It's seldom a case that you want to ***just*** unit test your components, Lost Pixel becomes a single source of truth for your visual regression testing strategy. The **Holistic Visual Regression Testing** concept allows you to combine the results of your visual tests into a single run and achieve even more confidence when developing your UIs. For example, imagine a CI run that runs your Storybook tests, then covers a dedicated set of marketing pages that use those components, finishes with interaction tests, and snapshots the results into a single place. Luckily [it is an easy setup with Lost Pixel](https://stackonfire.com/holistic-visual-testing).

![image](https://cdn.sanity.io/images/809vdkvp/production/232017c893977dfe441ba6572ef471713725cbdf-3176x1786.png)

A potential drawback of **Lost Pixel & Lost Pixel Platform** is its native integration with GitHub only(no support of GitLab or other git providers).

## Summary 📊

All five tools have their pros and cons. Playwright and Loki are open-source tools that offer utilities to streamline the visual testing process, but they have limitations. Chromatic and Percy are SaaS tools that are easy to set up and provide user-friendly interfaces, but they are limited to testing Storybook components and can be expensive.

Lost Pixel combines open-source and SaaS offerings that provide a **Holistic Visual Regression Testing** approach. It integrates with Git workflow to test E2E, marketing pages, and Storybook components, making it the best option for engineers striving to achieve the best results with little maintenance effort.

The Ultimate Guide to Visual Regression Testing: 5 Tools to Try

## Introduction 📚

Visual regression testing is a crucial aspect of software development that ensures the user interface remains consistent across different versions of an application. However, if you are new to this field, the terminology and jargon used in visual regression testing may be overwhelming. That's where this visual regression testing glossary comes in handy! In this article, we'll define and explain the most commonly used terms in visual regression testing so you can quickly become a pro.

## Visual Regression Testing 👁️

Visual regression testing(commonly referred to as **VRT, Visual testing or Visual snapshot testing**) is a technique used to compare screenshots of an application's user interface to detect any visual changes that may have occurred between different versions of the application. This technique ensures that the user interface remains consistent after the code changes. Usually, **VRT** runs as the last part of your CI/CD pipeline closing the whole test suit and ensuring that your Frontend looks precisely like it should.

## Shots 📸

In visual regression testing, a shot(also referred to as a **visual snapshot**) refers to a screenshot of an application's UI. Shots are used to compare the user interface between different versions of the application to detect any visual changes that may have occurred. Shot is a generally accepted unit of measure for visual regression testing SaaS.

## Baseline images 🖼️ 🖼️

Baseline images(a **source of truth or baseline snapshots**) are reference screenshots of an application's user interface captured during the initial testing phase. These images are used as a reference point for future tests to detect any visual changes that may have occurred between different versions of the application.

Usually, baseline images are manually approved by people responsible for maintaining the **VRT.** However, on the first run of your chosen testing suite, you must approve all your images - those images will now become baselines.

## Visual regression testing flow 🖼️ 🔄

[Visual regression testing flow](https://lost-pixel.com/blog/post/visual-regression-testing-101) refers to capturing baseline images, running tests to detect any visual changes, comparing the results to the baseline images, and updating the baseline images in cases with intended changes. This process is repeated for each code change to ensure that the user interface remains consistent.

![image](https://cdn.sanity.io/images/809vdkvp/production/84afb76aba78719a09bd897ddd95ff81d7fe3011-450x377.png)

## Visual difference ⛰️ ↔️🏔️

Visual difference refers to the changes between two shots of an application's user interface. These changes can be detected using the visual regression testing techniques mentioned above. ***Although not all*** visual differences are **Visual Bugs**, sometimes the changes are intended, and a baseline update is required.

## Visual bug 🖼️ 🪲

A visual bug is a defect in an application's user interface detected and caught by **VRT.** Here are a couple of good examples of visual bugs:

* Text overlapping or not aligning correctly with other elements on the page
    
* Images not displaying correctly or appearing distorted
    

Visual bugs are the exact reason we are running **VRT.** We aim to catch those bugs early in our release pipeline and ensure that end customers never face a breaking UI.

## Flaky visual tests ❄️

The term "**flaky"** refers to tests that produce inconsistent results. For example, in visual regression testing, flaky visual tests are tests that sometimes pass and sometimes fail, even though the application's UI has not changed. Various factors, such as network issues, browser inconsistencies, or timing issues, can cause flaky visual tests. Therefore, it is essential to identify and fix flaky visual tests to ensure that your visual regression testing results are consistent and reliable.

Luckily modern **VRT** tools provide a range of tools and practices to battle flaky tests and make the **VRT** easily one of the most valuable parts of testing on CI/CD.

Visual regression testing glossary

## Introduction to Holistic Visual Testing 🖼️

As front-end development continues to evolve, it's more important than ever to have a robust testing strategy in place to ensure that your applications look and function as expected. One of the most effective ways to do this is through visual regression testing, which allows you to catch and fix visual bugs before they reach production.

**Holistic visual regression testing** is an approach that aims to test the entire application, including both layout and interactions, as a whole.

In this example, we will set up a complex visual testing suite for our Next.js application. Here is the list of tools we will use:

* **Storybook**
    
* **Playwright**
    
* **GitHub actions**
    
* **Lost Pixel Platform**
    

As the final result, we want to make sure that the following parts of our applications are looking the way they are intended to look:

* Individual components - **visual unit tests**
    
* Pages - **visual integration tests**
    
* Results of playwright/cypress tests - **visual e2e tests**
    

Lost Pixel makes it easy to compose all of this in a single place to make the management of your visual testing a breeze. Here is the final version of the code for the whole integration:

Here is the final version of the [code on GitHub](https://github.com/lost-pixel/lost-pixel-next-holistic-visual-tests)

## Setting up Next application 🔼

We will clone the [app-playground](https://github.com/vercel/app-playground.git) repo from Next.js, it has rudimentary parts of the actual application, and that's precisely what we need.

```bash
git clone https://github.com/vercel/app-playground.git

pnpm install

pnpm dev
```

![image](https://cdn.sanity.io/images/809vdkvp/production/066def66917e0662bae053bf76bd4fa759d167c5-1896x1194.png)

## Visual unit tests 🖼️

We will use Storybook in combination with Lost Pixel to make sure our app components are looking the way we expect them to look.

Let's install Storybook:

```bash
npx storybook@next init && pnpm install postcss-loader style-loader @storybook/addon-postcss
```

Inside `.storybook/main.js` we need to make Storybook & Tailwind.css play well with each other :D

```javascript
const path = require('path');
module.exports = {
  stories: ['../stories/**/*.mdx', '../stories/**/*.stories.@(js|jsx|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    {
      name: '@storybook/addon-postcss',
      options: {
        cssLoaderOptions: {
          // When you have splitted your css over multiple files
          // and use @import('./other-styles.css')
          importLoaders: 1,
        },
        postcssLoaderOptions: {
          // When using postCSS 8
          implementation: require('postcss'),
        },
      },
    },
  ],
  framework: {
    name: '@storybook/nextjs',
    options: {},
  },
  docs: {
    autodocs: 'tag',
  },
};
```

Inside `.storybook/preview.js` we need to import tailwind styles on top of the file:

```javascript
import '../styles/globals.css';

export const parameters = {
  actions: { argTypesRegex: '^on[A-Z].*' },
  controls: {
    matchers: {
      color: /(background|color)$/i,
      date: /Date$/,
    },
  },
};
```

Inside `stories/Button.stories.tsx` let's write our first story that we will use in tests:

```javascript
import type { Meta, StoryObj } from '@storybook/react';
import React from 'react';

import Button from 'ui/Button';

// More on how to set up stories at: https://storybook.js.org/docs/7.0/react/writing-stories/introduction
const meta: Meta<typeof Button> = {
  title: 'Example/Button',
  component: Button,
};

export default meta;
type Story = StoryObj<typeof Button>;

export const Default: Story = {
  render: () => <Button kind="default">Default</Button>,
};

export const Error: Story = {
  render: () => <Button kind="error">Error</Button>,
};
```

Now we run our storybook to see our components displayed in an isolated manner:

```bash
npm run storybook
```

![image](https://cdn.sanity.io/images/809vdkvp/production/9c7106ebbc5756aac595d56fde22d7777624567d-1024x450.png)

Our stories will serve as the basis for the unit visual tests. Let's integrate this first step with Lost Pixel Platform to see how it will work there.

## Lost Pixel Platform ⚙️

You can use the [quickstart documentation](https://docs.lost-pixel.com/user-docs/setup/lost-pixel-platform) from [lost-pixel.com](https://lost-pixel.com) to set up the platform to work with your repositories via installing the official Lost Pixel GitHub app. In this guide, we will focus on setting up the right configuration, assuming you have done the platform setup already!

You can use Lost Pixel Onboarding to easily set up the project by copying & pasting the respective files.

In our `lostpixel.config.ts` at the root of our project, let's define the tests for Storybook

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },

  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

In our `.github/workflows/vrt.yml` at the root of our project let's define our CI/CD job:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

With this simple setup, we are already covering the visual unit tests 🙌🏼

![image](https://cdn.sanity.io/images/809vdkvp/production/d0cb270199226fe61560ed0c9ae61ce57b24d9a6-1856x2050.png?)

Let's make it really holistic now!

## Visual integration tests 🖼️ ↔️ 🖼️

Sometimes we care not only about how our components will be used in isolation but also about how they will look on the page when they are combined with a plethora of other components & styles. Let's explore how we can easily expand our initial setup to monitor our pages continuously!

Let's extend our `lostpixel.config.ts` to include our page shots.

```typescript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

We also need to extend `.github/workflows/vrt.yml` to build & run our Next.js app so we can access pages in Lost Pixel:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

Now we are talking 🚀 Our visual tests are covering not only **unit** parts with individual components but also **the integration** part where we are making the screenshots for the whole pages.

![image](https://cdn.sanity.io/images/809vdkvp/production/14cd5947e14ffbc2af96c9b11168133fca0e5714-2300x2038.png)

Let's make it an ultimate setup with a magic flavour of the feature of Lost Pixel called custom shots 🪄✨✨✨

## Visual E2E tests 🖼️ ⚙️ 🖼️ ⚙️ 🖼️

What if I told you that on any given CI run, you can create the screenshots using your logic & they will be available immediately on Lost Pixel Platform for the visual regression testing? Enter custom shots 📦

We will be using Playwright to interact with the page & make sure that after some set of interactions, we are always looking at the same UI.

```bash
npm install --save-dev @playwright/test
```

Let's also extend our `package.json` with `test:e2e` command:

```bash
...
"scripts": {
  "dev": "next dev",
  "build": "next build",
  "start": "next start",
  "test:e2e": "playwright test",
}
```

Now we will create our E2E test that will not do much, but will give you a good understanding of how you could integrate Lost Pixel into any test like this!

In `e2e/example.spec.ts` :

```typescript
import { test } from '@playwright/test';

test('lost-pixel e2e', async ({ page }) => {
  // Perform some action on the page, like clicking the button.
  // 📒 http://172.17.0.1:3000 is already fit for GitHub action runtime
  await page.goto('http://172.17.0.1:3000/context');
  await page.click('data-test-id=context-click-counter');
  await page.click('data-test-id=context-click-counter');
  // Make a shot for Lost Pixel to test
  // 📒 Path is arbitrary, but I advise you to store all of the custom shots in a single folder, e.g. lost-pixel. 
  await page.screenshot({ path: 'lost-pixel/a.png', fullPage: true });
});
```

Our `ContextClickCounter` component gets a new `data-test-id`, so the code above works:

```javascript
      <button
        // Here is the important part
        data-test-id="context-click-counter"
        onClick={() => setCount(count + 1)}
        className="rounded-lg bg-gray-700 px-3 py-1 text-sm font-medium tabular-nums text-gray-100 hover:bg-gray-500 hover:text-white"
      >
        {count} Clicks
      </button>
```

Now we are ready to integrate our E2E with Lost Pixel! Let's extend our `lostpixel.config.ts` to include our custom shots.

```javascript
import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  storybookShots: {
    storybookUrl: './storybook-static',
  },
  pageShots: {
    pages: [
      { path: '/layouts/books/fiction', name: 'fiction-books' },
      { path: '/layouts/books/biography', name: 'biography-pages' },
    ],
    baseUrl: 'http://172.17.0.1:3000',
  },
  customShots: {
    // 📒 currentShotsPath should be matching the one above from playwright test
    currentShotsPath: './lost-pixel',
  },
  lostPixelProjectId: 'clde9r3rh00v3m50vlq8y0k78',
  apiKey: process.env.LOST_PIXEL_API_KEY,
};
```

And finally, we extend our GitHub Action declaration file by running Playwright tests:

```yaml
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    name: Lost Pixel

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Install dependencies
        run: npm install --legacy-peer-deps

      - name: Build Storybook
        run: npm run build-storybook

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Playwright tests
        run: npx playwright install --with-deps && npm run test:e2e

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v3.0.4
        env:
          LOST_PIXEL_API_KEY: ${{ secrets.LOST_PIXEL_API_KEY }}
```

After Lost Pixel Platform runs this time, you should see something similar to this:

![image](https://cdn.sanity.io/images/809vdkvp/production/f36f66c4976292959369bc045746e71539648066-1856x854.png)

After interacting with our page & clicking the button two times, we made a screenshot of a page to test visually. Sometimes it's very useful because apart from asserting the presence or absence of some elements on the page that E2E tests traditionally rely on, the layout could be broken after a particular interaction and here visual regression tests will catch this for us.

## Summary 🔼

We have built a holistic visual testing framework over an existing Next.js application using the tools suitable for every task: **Storybook** for visual unit tests, **Pages** for visual integration tests & **Playwright** for visual E2E tests. We have use **GitHub Actions** to make sure our setup runs continuously on every push. Finally, we have used the **Lost Pixel Platform** to manage our visual tests in one central place & notify us whenever visual regression is found.

![image](https://cdn.sanity.io/images/809vdkvp/production/61df865a0b5598939a29b186556504e42024be10-1696x1082.png)

If you liked the read and enjoyed setting up your visual tests, you might as well like our [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m). There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

Holistic visual regression testing

## Introduction ✨

Visual regression testing is a critical aspect of modern web development that helps ensure that code changes don't break the intended design and functionality of a website or application.

In this blog post, we'll discuss five compelling reasons why every development team should invest in writing visual regression tests. From improved efficiency and reduced time to market to enhanced user experience and increased collaboration, the benefits of visual regression testing are many. Keep reading to learn more about how visual regression testing can help your team produce better code and ship it more frequently to customers without fear of regressions.

## #1 Efficiency 👟

Visual regression tests automate the process of verifying UI changes, reducing the time and effort required to manually test each change. There are multiple stages of how advanced your setup could be but think of it this way - after each code change, you could deploy your application to a preview environment & ask your team to go through all of the pages, click around, resize the browser & see if nothing is broken visually. **OR** you could automate that saving hours & hours weekly for all of the stakeholders

## #2 Reduced time to market ⏰

With visual regression tests in place, development teams can quickly identify and fix UI issues, leading to faster time to market and a more efficient development process. This point flows nicely from the previous one and is impacting one of the most notable metrics all teams strive to improve on. If your processes are lean & automated I bet you can ship fast and turn your paddle boat 🛶 into a speed boat 🚤.

## #3 Enhanced User Experience ✨

Would you rather have your UI explode in your user's face or ship a pixel-perfect application that is pleasing to look at & work with? I guess an answer is clear here - broken layout, mixed-up design tokens or even incorrectly rendered tooltip could all cost the business money, sometimes substantial amounts. With visual regression testing, we make sure that our users are protected(at least to some degree) from this and we have a safety net for all the problems just before they hit the end consumer of our software.

## #4 Increased Collaboration 🤝🏼

Visual regression tests provide a clear, concise and standardized way to communicate UI changes between development teams, designers and stakeholders, fostering collaboration and reducing miscommunication. If you are using a SaaS solution for visual regression testing you can enjoy the streamlined process that resembles the following:

**Designer** 🎨: hey, *Developer*, it seems like we have a problem with our Button, the success style for it is somehow not applied. Our visual tests are failing!  
**Developer** 💻: oh wow, indeed it's somewhat of an edge case I missed. Pushed the fix to the `fix/button-styles` branch  
**Designer** 🎨 *(after some time)*: nice, the fix seems to be working, looked at the results of the visual tests and the buttons look great again!

Not a lot of back & forth, easy access to results for validation and many other collaborative benefits!

## #5 Improved Confidence 💪🏼

This could easily trump all of the above reasons! Visual tests increase the confidence of development teams in their work by providing a comprehensive and consistent way to validate UI changes. Teams can trust that their work meets the desired standards and that no critical issues have been introduced with any code changes.

One of the main goals of any software testing is just to make sure we, humans, are not doing things we are not good at(checking lots of tiny details of something in hundreds of permutations). Doing this over and over again & failing hurts our confidence of course, and we start to ship less frequently just because we are scared to break stuff. Tests to the rescue! Now we are all confident and shipping new releases 10 times per day 🚀

## Summary 🔼

Visual tests are just another medium to help us produce better code & ship it more frequently to the customers without fear of regressions. Embrace them and enjoy 😊

If you liked the read and enjoyed setting up your visual tests you might as well like our [Visual Regression Testing Discord](https://discord.gg/WqVjk49g9m). There you can find help with any issues, chat about your setup & discuss the good practices for visual tests.

I am also a co-founder of a Visual Regression Testing SaaS called [Lost Pixel](https://www.lost-pixel.com) and I would be excited to set up visual testing for your company and deliver you from all the traps that I've seen people stepping in when implementing their Visual Tests. Consulting & setup is totally free 💸

Feel free to reach out to me [on twitter](https://www.twitter.com/divdev_) or via email `dimitri@lost-pixel.com`

5 reasons to write visual regression tests

## Foreword 👋🏼

Hey, my name is Dima! I am a frontend-engineer who loves to solve problems. One day at my contracting work we faced serious developer experience issues with running our Storybook.js visual regression tests. At the time we had a custom solution that was working mostly fine but required a lot of manual steps & was mostly blocking our engineers for multiple hours each week.

Having evaluated external solutions we quickly understood that for our monthly amounts of tests, the price could get ridiculously high in no time. Remember how I said I love to solve problems? Together with my friend Chris, we decided to give this problem a shot, and that's how **Lost Pixel** & **Lost Pixel Platform** was born. I welcome you to embark on a short retrospective with me to see why & how we did it.

***Spoiler alert***: we cut the costs significantly, made engineers happy & built it in an open-source fashion 🚀

## Motivation 🎯

There are multiple tools out on the market both open-source & commercial that do the job of visual regression testing well. Why even build a new one you may ask? The answer lay in plain sight for us:

> There was nothing out there being **a)** open-source **b)** able to run on customers' premises **c)** would have a modern approval flow as a cloud service **AND** in virtue of previously mentioned facts still, *be affordable*

From the very start, we were building Lost Pixel on the foundation of it being an open-source technology that could be used in a **self-hosted** manner **for free** or in Lost Pixel Platform as a managed service *but still based on our open-source engine.*

This gives us much more room for growth as we are planning to cater to the needs of both OSS & commercial clients in the future with the mission of making visual regression testing **convenient**, **comprehensive** & **affordable**.

## Differentiating factor 🧞‍♂️

While being similar to most of the existing OSS & SaaS solutions on the market we decided to focus on a couple of premises and make a difference there:

1. Focus on **GitHub** as the main vendor of where your CI/CD runs & make the process seamless by offering **Github Action packed** with everything you need to setup up visual tests on the platform in a couple of lines of code.
    
2. Have an accent on **Developer Experience** by providing strongly typed Lost Pixel configuration, extensive logs for easier debugging and a user interface designed to be used collaboratively with your team
    
3. Offer a **holistic solution** to visual tests. Visual tests are not limited to Storybook, you can run them on your marketing pages, and components or E2E tests like Cypress or Playwright. Our action supports all of them and makes it easier to compose them so you have all of your visual tests gathered in one place - **Lost Pixel Platform.**
    
4. We natively support monorepos, everybody loves monorepos, right? 😀
    

## Building the solution ⚒️

We started with nothing but the premise that we need to have a tool that would be able to handle hundreds of thousands of screenshots monthly, be easily hostable with any CI/CD process & have a user interface for collaboration. As with all other great things, Lost Pixel started with a Slack message 😀

![image](https://cdn.sanity.io/images/809vdkvp/production/3c34e703e7a451198a198a7e572e73e880a26a39-950x308.png)


In less than a couple of weeks we already had a working solution, and in less than 3 months it was approved for internal usage in large scale-up:

![image](https://cdn.sanity.io/images/809vdkvp/production/8018ff65fb980097da6dd09044b65ee4ecd6849c-912x990.png)

We started testing the whole solution including the platform behind closed doors to see its viability and soon enough we realized that making it open-source will benefit much more people. So we did that!

![image](https://cdn.sanity.io/images/809vdkvp/production/6190843870941a17573b7f22d3b4e52162be6d01-1844x1238.png)

The open-source launch was quite successful and it further validated our idea & got us more closed-beta customers to closely cooperate with & build a solution they would be true advocates of.

We took the Q4 of 2022 to polish the Platform, improve its performance & usability and did a soft launch to open-beta in the middle of January 2023 🎊

![image](https://cdn.sanity.io/images/809vdkvp/production/6fa1a320d1d2043dce709aa5a4367d191d0de686-3484x1000.png)

At this time we are highly persuaded that we have one of the most solid Visual Testing Platforms on the market with a high potential to become a go-to solution when it comes to visual regression tests. We have battle-tested the platform on ridiculously large storybooks & pages and achieved above-benchmark performance.

We have a huge list of things that we want to ship, so stay tuned - Lost Pixel Platform will become even better(faster? 🤔) in the future!

## Learnings 🎓

* our beta customers were/are our unfair competitive advantage - just being in constant communication with them increased our probability of success drastically
    
* sometimes passion in some topic & constant stream of feedback are the only things you need to build meaningful things
    
* technical cofounder + technical cofounder = ✨
    
* focusing on making a limited group of customers successful &gt; going for a big-bang approach to capture the whole market(I think it's reasonable when starting out, might revise in the future)
    

## Closing thoughts 🧠

It's wild to imagine what we were able to achieve in the past 6 months and I am being super excited for the future of Lost Pixel, both Open Source & Lost Pixel Platform commercial solutions. I wanted to write this post to inspire you a little bit and maybe give some insights on things that helped us to succeed in the ship/improve/release cycle.

I'd love to hear your thoughts on [https://lost-pixel.com](https://lost-pixel.com) and [lost-pixel open-source engine](https://github.com/lost-pixel/lost-pixel) and if you have any questions or just want to chat about random topics I highly encourage you to join our [Lost Pixel discord](https://discord.gg/WqVjk49g9m) or just drop me a [DM on Twitter](https://twitter.com/divdev_).

Cheers and have fun building stuff!

Visual regression testing that makes sense

## Visual regression testing 101

### Introduction ✍🏼

Hey! My name is Dima, I am from Ukraine and I am keen on building products, designing beautiful UI/UX and testing automation. I am so passionate about visual regression tests that I've built an open-source tool called [Lost Pixel](https://lost-pixel.com) and guess what? It is a holistic visual regression testing framework!

I am excited to give you an introduction to the world of visual tests and walk you through the use cases in which they are most beneficial for your design & development flow. All of the below information is from years of me working with Visual Regression Tests digested into a short read.

### What 🖼️

**Visual regression testing** (a.k.a *vis reg tests, visual tests, UI snapshot tests, visual diff tests*) is an approach to automated QA. In this approach, we try to capture unintended changes to UI resulting from code modification. Visual tests can be applied across the whole frontend hierarchy starting from the atomic component, continuing with UI patterns utilising the components and ending with a page accommodating multiple UI patterns to present the user with a finished frontend. 

The visual regression test consists of three essential moving parts:

1. Screenshot of the element we want to test before the code change (a.k.a *baseline, intended version*)
2. Screenshot of the same element after the code change (a.k.a *compared image*)
3. Generated difference image in case the change in source code resulted in a different image(automatically generated by comparison engines like [pixel-match](https://github.com/mapbox/pixelmatch) or [odiff](https://github.com/dmtrKovalenko/odiff)) 


![image](https://cdn.sanity.io/images/809vdkvp/production/2e79892440f17379adbc2bcf974eb7b2c3a0caa9-940x788.png?w=450)

There are three possible outcome scenarios when doing a visual regression test:

1. **No difference** - tests pass as expected
2. **Intended difference** - test fails but the change is expected. In this case we need to update the baseline in our testing suite which will result in a no-difference run after we re-trigger the test run.
3. **Unintended difference** - test fails and change is unexpected. That's the reason we are doing visual tests in the first place - to easily catch such outcomes.

![image](https://cdn.sanity.io/images/809vdkvp/production/80975f2fe5d066c3aa0ad1fefc9873d44f42050b-940x788.png?w=450)

We have successfully covered the process of visual regression testing and now are ready to look into the most frequent use cases.

### How 🔀

If we talk tool-agnostic this is how Visual Regression Testing workflow look on the CI/CD pipeline.

![image](https://cdn.sanity.io/images/809vdkvp/production/9442aab8e86a1d9b4a9d20a80ffe20d83830c314-2151x1572.png?w=450)


### Use cases 🤔
Visual regression tests stand in between unit and integration tests on the testing trophy and are mostly used in the following products:

1. Design systems 
2. Component libraries
3. Application interfaces
4. Marketing & documentation pages

There are two main applications utilised in companies of different sizes and spheres. **Catching visual bugs** & **cross-functional teams collaboration**.


### Catching visual Bugs 🪲

You might already have some e2e tests in your application done with Cypress or Playwright but in most cases, they are not enough. Focusing purely on functional testing we always forget that sometimes totally functional elements of UI can look totally broken visually. Thus most frequent use case is to catch Visual Bugs with the ultimate goal being **shipping code more confidently**

#### Case studies 🔍

- Engineer is making amendments to the signature of the `<Box/>` component which is the primitive utilised in 100s of other design system components. By sheer number, it's hard to verify manually that none of the dependent components broke visually.

**❌ Problem:** The dependent components were too tightly coupled to `<Box/>` component and the changes rendered `<Datepicker/>` and `<FormControl/>` look totally broken for the end users.

**✅ Solution:** Visual tests picked the problem, automatically reported it and gave a bird-eye overview of places where it happens.

-  Product designer gives the engineer the high fidelity prototype in which an already existing component is slightly modified to accommodate to new design vision. The changes are looking good and the engineer ships the ticket. 

**❌ Problem**: Engineer misses a couple of usages of the modified component where it's used in the non-traditional layout with end users experiencing UI which looks broken

**✅ Solution:** Visual tests identified layout shifts, automatically reported it and gave a common ground for the product designer and engineer to solve the problem.

In both cases, Visual Regression Testing would be an automated way of making sure that change in one place would not cascade with visual bugs into other places.

### Collaborating in cross-functional teams 🤝🏼
With the usage of Visual Regression Tests, you start getting another tangible benefit which stems from the fact that you always **have a snapshot of the item** that you are testing. This opens multiple opportunities to collaborate:

1. Frontend Engineers can between themselves verify the correctness of the final result of their work
2. Engineers and Designers can sync in a way faster manner on the iterations done on the UI using the snapshots as common ground for discussion

#### Case study 🔍

- Engineer is working on the UX improvement of CTA-button prepared by the product designer. The Design System with high fidelity prototypes is a little bit out of date from the platform

**❌ Problem:** It's hard for an engineer to manually QA all the occurrences of the CTA button and even if possible would take lots of engineering hours. Happened that one of the missed cases is CTA button is used within different contexts making its label confusing for the users. Without  common ground, the communication between the two becomes a constant back and forth of sending screenshots of UI from one to another

**✅ Solution:** With help of Visual tests as the shared communication medium the scope of the problem is isolated giving a clear path for both collaborators to continue. The discussion happens by attaching concrete results of Visual Regression Tests.
