Finding flaky tests in JavaScript applications using stress and test suite reordering

Flaky tests are an issue when software tests fail non-deterministically on the same version of code. Research has been conducted on flaky tests in Java, C/C++, and Python, but little research has been conducted in JavaScript even though it is a popular programming language. Thus, we conducted an empirical study of flaky tests found from 58 JavaScript GitHub repositories that used Jest as their testing framework. We developed a technique called StressSequence to detect flaky tests in JavaScript applications. StressSequence stresses the machine using Stress-Ng and a random sequencer for reordering test suites in Jest. StressSequence then reruns the tests multiple times under these conditions, detecting a flaky test when it observes the test both pass and fail. We evaluated StressSequence on our subjects, comparing the detected flaky tests against a baseline technique of simply rerunning tests without stress or reordering, Reruns. The subjects were run 10 times with StressSequence and 10 times each with Reruns as a comparison. We found 71 flaky tests from both StressSequence and Reruns. StressSequence found 56 and Reruns found 41 with 26 overlapping. Of the 71 flaky tests, 30 were from timeouts and 30 came from one application due to test dependencies. Of the remaining eleven flaky tests the majority of them are of the Async Wait and UI categories. Based on the manual inspection the random sequencer did not help reveal any flaky tests. Running with stress, however, did impact the flake rate of certain flaky tests and found flaky tests not found using Reruns.