GSoC 2020 Week 12 Recap: Testing the Tests

Hello! We've reached the end (almost). The twelve weeks of coding are complete. Later this week I'll make a big post with a video summarizing the entire summer. For now, here's one last weekly recap.

Update from last week

I spent this week working on tests. The idea is to automatically warn future developers if they make a change that introduces a bug in something that was previously working. MuseScore has lots of tests in place for existing features, and I added some more tests for the new features I've implemented.

Visual tests/vtests

The simplest tests are the ones that load a file, export it to PNG, and throw up a warning if it looks different than a reference image. For these I just had to add a few strategically chosen examples. So, for MeasureRepeats, the vtests will check the three subtypes, in a somewhat crowded context (to see that they are positioned correctly even when the measures have different widths), and also check that the correct numbers are applied to a sequence of one-measure repeats when that style setting is enabled. For MMRests, one will check a couple of the H-bar style settings (in Bravura), and others will check the display of old-style multimeasure rests in each font.

Script tests

These scripts replay a sequence of user actions and compare the result against a reference file. Not all actions can be recorded—a notable gap is user responses to dialog boxes (Cancel/OK)—but enough were available for me to create scripts for most of the special handling I described in my Week 7 recap. Interestingly, these tests are not currently available for their prescribed function (i.e., they are not run automatically to warn developers if they make a change that introduces a bug in something that was previously working), having been disabled while some of the behavior is in flux due to MuseScore 4 development. I also had to write some parts of the script that weren't recorded automatically, because for whatever reason that was disabled too. But, when script tests are re-enabled, these ones will run.

?/mtests

I spent some time figuring out how run mtests (I don't know what the "m" is for) on my computer, and ultimately found that I could, but the output only told me whether or not a test passed, and I didn't get the detailed information about what the failures were. So, I had to "test the tests"—commit and push to the pull request in order to get the mtests to run on the server—many times as I worked on them. The end result is a handful of tests for MeasureRepeats with linked parts and undo/redo, as well as one for MIDI and a few for MusicXML import/export that I added earlier (these were easy, as rather than writing tests I only needed to add test files to be checked by existing test functions).

Existing tests

Naturally, my MeasureRepeat-related changes introduced a number of problems with things that were previously working. This week I fixed those things which the current tests alerted me to. I also had to update many of the existing reference files, as since even the one-measure repeat is now stored differently the newly created file would necessarily be different than the reference.

Along the way, I discovered many small bugs in MeasureRepeat behavior and made many other small improvements that occurred to me, to help make them as robust and functional as they can possibly be.

Looking forward to this week

There's nothing left to do! :-) I've even removed "[WIP]" from the title of PR #6365. Of course, some things may occur to me and I'll sneak them in there. It's also possible that, if either or both of the other projects is merged first, I'll need to rebase mine because there will be conflicts (MuseScore team: merge mine first! :-)). Other than that—watch for that big Work Product post.

Thanks for reading along these past twelve weeks!