flutter

Passing a threshold for screenshot tests in Flutter

wafrat

Mar 14, 2023 • 1 min read

Since this was much harder than expected, I figured I should document it for future reference, and also in order to refine the solution as I get used to it.

The regular way to screenshot tests is to write:

await expectLater(find.byType(MaterialApp),
  matchesGoldenFile('goldens/render_$description.png'));

However this does not let you set a threshold. So if you run the screenshot tests on different platforms, let's say MacOS from your laptop, and Linux on GitHub actions, it might work on one platform and still fail on the other by a tiny margin:

  ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
  The following assertion was thrown while running async test code:
  Golden "goldens/clipped_path_kanji_10_percent.png": Pixel test failed, 0.03% diff detected.
  Failure feedback can be found at
  /home/runner/work/svg_drawing_animation/svg_drawing_animation/test/failures
  
  When the exception was thrown, this was the stack:
  #0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:101:7)

Screenshot test failure on GitHub actions.

After browsing tickets and PRs on GitHub, I ended up finding a solution that involves making your own comparison function at https://github.com/Piinks/cocoon/blob/f427f28acf6933ab693b44f9d74f52a4906405f2/app_flutter/test/utils/golden.dart.

The function additionally prepends the goldens prefix, so usage becomes:

await expectGoldenMatches(
  find.byType(MaterialApp), 'render_$description.png');

You can find the tweaked file and its usage here:

Sign up for more like this.