Ran into a situation last week that got me thinking about how most teams handle visual changes in production.
We pushed a CSS update that was supposed to fix button alignment on one page. Looked fine in staging. Looked fine in the PR preview. Got merged, deployed, and broke the card layout on three other pages because of a shared utility class.
Caught it about four hours later when a customer mentioned it in a support ticket. Not a great look.
What we tried
After that incident we talked about adding visual regression testing to CI. Looked at a few options:
-
Percy — solid, but the pricing gets steep once you have more than a handful of pages and multiple PRs a day
-
Playwright screenshots in CI — free, but you need to maintain baseline images and the diffs are noisy. Every font rendering difference across runners triggers a false positive
-
Manual spot checks after deploy — what we were doing. Obviously not sufficient
We ended up going with Playwright screenshot comparisons but with a pretty generous diff threshold (0.3%) to cut down on false positives. It catches big layout shifts but ignores sub-pixel rendering differences. Good enough for now, but not ideal.
The production monitoring gap
CI-based visual testing only covers what happens before deploy. It doesn't catch things that break after — CDN caching serving old assets, third-party scripts injecting unexpected elements, A/B test variants rendering wrong.
Some teams run synthetic monitoring that takes periodic screenshots of production pages and compares them. That covers the post-deploy gap but it's a separate system to set up and maintain.
Curious what others do
How are you handling visual verification in production? Specifically:
-
Do you run any visual checks post-deploy, or only in CI?
-
If you use screenshot comparison, how do you deal with dynamic content (dates, user-specific data, ads)?
-
Anyone running continuous visual monitoring on production pages on a schedule?
Feels like there should be a simpler answer than "maintain 200 baseline images and pray the CI runner has the same font rendering as last time."
Last time I worked on a large, long running web project, a person would manually check nearly everything. He kept a checklist of what to check, which covered nearly every page, modal, and dialog on staging before we deployed. We did try the snapshot thing for a little bit, but it caused more problems than it solved.
At my last job all testing was manual, it was so painful. Like we had a c# application and there was literally a requirement of "system must run on c# framework 4.8.1" so test gave instructions to go into registry and the key to check and the tester had to write the value seen, it was then scanned in as a pdf and reviewed by 6 people who all had to check that the value matched what was in the test case.
Lol, yeah. We also had to do stuff like that for auditing and compliance because the application interacted with the company's accounting system.
This was medical device testing so I get it but oh my go so annoying
Error handling verification was go into visual studio, set break point, run in debug, run till break point hit, unplug network cable, confirm application stopped running with appropriate error code