Visual regression testing that makes sense

Foreword 👋🏼

Hey, my name is Dima! I am a frontend-engineer who loves to solve problems. One day at my contracting work we faced serious developer experience issues with running our Storybook.js visual regression tests. At the time we had a custom solution that was working mostly fine but required a lot of manual steps & was mostly blocking our engineers for multiple hours each week.

Having evaluated external solutions we quickly understood that for our monthly amounts of tests, the price could get ridiculously high in no time. Remember how I said I love to solve problems? Together with my friend Chris, we decided to give this problem a shot, and that's how Lost Pixel & Lost Pixel Platform was born. I welcome you to embark on a short retrospective with me to see why & how we did it.

Spoiler alert: we cut the costs significantly, made engineers happy & built it in an open-source fashion 🚀

Motivation 🎯

There are multiple tools out on the market both open-source & commercial that do the job of visual regression testing well. Why even build a new one you may ask? The answer lay in plain sight for us:

There was nothing out there being a) open-source b) able to run on customers' premises c) would have a modern approval flow as a cloud service AND in virtue of previously mentioned facts still, be affordable

From the very start, we were building Lost Pixel on the foundation of it being an open-source technology that could be used in a self-hosted manner for free or in Lost Pixel Platform as a managed service but still based on our open-source engine.

This gives us much more room for growth as we are planning to cater to the needs of both OSS & commercial clients in the future with the mission of making visual regression testing convenient, comprehensive & affordable.

Differentiating factor 🧞‍♂️

While being similar to most of the existing OSS & SaaS solutions on the market we decided to focus on a couple of premises and make a difference there:

1. Focus on GitHub as the main vendor of where your CI/CD runs & make the process seamless by offering Github Action packed with everything you need to setup up visual tests on the platform in a couple of lines of code.

2. Have an accent on Developer Experience by providing strongly typed Lost Pixel configuration, extensive logs for easier debugging and a user interface designed to be used collaboratively with your team

3. Offer a holistic solution to visual tests. Visual tests are not limited to Storybook, you can run them on your marketing pages, and components or E2E tests like Cypress or Playwright. Our action supports all of them and makes it easier to compose them so you have all of your visual tests gathered in one place - Lost Pixel Platform.

4. We natively support monorepos, everybody loves monorepos, right? 😀

Building the solution ⚒️

We started with nothing but the premise that we need to have a tool that would be able to handle hundreds of thousands of screenshots monthly, be easily hostable with any CI/CD process & have a user interface for collaboration. As with all other great things, Lost Pixel started with a Slack message 😀

In less than a couple of weeks we already had a working solution, and in less than 3 months it was approved for internal usage in large scale-up:

We started testing the whole solution including the platform behind closed doors to see its viability and soon enough we realized that making it open-source will benefit much more people. So we did that!

The open-source launch was quite successful and it further validated our idea & got us more closed-beta customers to closely cooperate with & build a solution they would be true advocates of.

We took the Q4 of 2022 to polish the Platform, improve its performance & usability and did a soft launch to open-beta in the middle of January 2023 🎊

At this time we are highly persuaded that we have one of the most solid Visual Testing Platforms on the market with a high potential to become a go-to solution when it comes to visual regression tests. We have battle-tested the platform on ridiculously large storybooks & pages and achieved above-benchmark performance.

We have a huge list of things that we want to ship, so stay tuned - Lost Pixel Platform will become even better(faster? 🤔) in the future!

Learnings 🎓

• our beta customers were/are our unfair competitive advantage - just being in constant communication with them increased our probability of success drastically

• sometimes passion in some topic & constant stream of feedback are the only things you need to build meaningful things

• technical cofounder + technical cofounder = ✨

• focusing on making a limited group of customers successful > going for a big-bang approach to capture the whole market(I think it's reasonable when starting out, might revise in the future)

Closing thoughts 🧠

It's wild to imagine what we were able to achieve in the past 6 months and I am being super excited for the future of Lost Pixel, both Open Source & Lost Pixel Platform commercial solutions. I wanted to write this post to inspire you a little bit and maybe give some insights on things that helped us to succeed in the ship/improve/release cycle.

I'd love to hear your thoughts on https://lost-pixel.com and lost-pixel open-source engine and if you have any questions or just want to chat about random topics I highly encourage you to join our Lost Pixel discord or just drop me a DM on Twitter.

Cheers and have fun building stuff!