rr can be used to record and replay Firefox and Firefox tests on Linux. See https://github.com/mozilla/rr/wiki/Recording-Firefox. If you have rr installed and have a Linux Debug build of Firefox handy, recording a mochitest is as simple as:
./mach mochitest --debugger=rr ...
For example, to record a single mochitest:
./mach mochitest testing/mochitest/tests/Harness_sanity/test_sanitySimpletest.html \ --keep-open=false --debugger=rr
Even better, use –run-until-failure to repeat the mochitest until an intermittent failure occurs:
./mach mochitest testing/mochitest/tests/Harness_sanity/test_sanitySimpletest.html \ --keep-open=false --run-until-failure --debugger=rr
To replay and debug the most recent recording:
Similar techniques can be applied to reftests, xpcshell tests, etc.
For a fun and simple experiment, you can update a test to fail randomly, maybe based on Math.random(). Run the test in a loop or with –run-until-failure to reproduce your failure, then replay: Your “random” failure should occur at exactly the same point in execution on replay.
In recent weeks, I have run many mochitests on my laptop in rr, hoping to improve my understanding of how well rr can record and replay intermittent test failures.
rr has some, but only a little, effect on test run-time. I can normally run mochitest-1 via mach on my laptop in about 17 minutes; with rr, that increases to about 22 minutes (130% of normal). That’s consistent with :roc’s observations at http://robert.ocallahan.org/2015/11/even-more-rr-replay-performance.html.
I observed no difference in test results, when running on my laptop: the same tests passed and failed with or without rr, and intermittent failures occurred with approximately the same frequency with or without rr. (This may not be universal; others have noted differences: https://mail.mozilla.org/pipermail/rr-dev/2015-December/000310.html.)
So my experience with rr has been very encouraging: If I can reproduce an intermittent test failure on my laptop, I can record it with rr, then debug it at my leisure and benefit from rr “extras” like reverse execution. This seems great!
I still have a concern about the practical application of rr to recording intermittent failures reported on treeherder…I’ll try to write a follow-up post on that soon.