Ran into a situation last week that got me thinking about how most teams handle visual changes in production.
We pushed a CSS update that was supposed to fix button alignment on one page. Looked fine in staging. Looked fine in the PR preview. Got merged, deployed, and broke the card layout on three other pages because of a shared utility class.
Caught it about four hours later when a customer mentioned it in a support ticket. Not a great look.
What we tried
After that incident we talked about adding visual regression testing to CI. Looked at a few options:
-
Percy — solid, but the pricing gets steep once you have more than a handful of pages and multiple PRs a day
-
Playwright screenshots in CI — free, but you need to maintain baseline images and the diffs are noisy. Every font rendering difference across runners triggers a false positive
-
Manual spot checks after deploy — what we were doing. Obviously not sufficient
We ended up going with Playwright screenshot comparisons but with a pretty generous diff threshold (0.3%) to cut down on false positives. It catches big layout shifts but ignores sub-pixel rendering differences. Good enough for now, but not ideal.
The production monitoring gap
CI-based visual testing only covers what happens before deploy. It doesn't catch things that break after — CDN caching serving old assets, third-party scripts injecting unexpected elements, A/B test variants rendering wrong.
Some teams run synthetic monitoring that takes periodic screenshots of production pages and compares them. That covers the post-deploy gap but it's a separate system to set up and maintain.
Curious what others do
How are you handling visual verification in production? Specifically:
-
Do you run any visual checks post-deploy, or only in CI?
-
If you use screenshot comparison, how do you deal with dynamic content (dates, user-specific data, ads)?
-
Anyone running continuous visual monitoring on production pages on a schedule?
Feels like there should be a simpler answer than "maintain 200 baseline images and pray the CI runner has the same font rendering as last time."
I can say a little bit on tooling: We’ve experimented with Chromatic. We’re on a mobile app though, so the idea with Chromatic is to render via react-native-web, which has its own issues, so the ROI is such that we haven’t taken it beyond POC. Seems nice though, and probably more in line with your domain.
But for org strategy, I take it you’re on a much bigger team than I ever have been. (10 people max for me) So I can’t speak to personal experience there, but what I will say is that being able to control your entire env top to bottom is crucial for pretty much anything where you want high fidelity testing.
Like, being able to set up an env that allows you to play with stuff like externally-sourced ads and CDN caches in a space that’s private and temporary. No small feat, but that’s pretty much what you have to do if you want to catch those Rube Goldberg style failures before prod.
It depends how much it’s worth it to you, to have a test env that’s 99.999% realistic. As with anything, the first two 9s are the cheapest.