Automated tests

This is a short overview. For a more technical explanation, see the mtest README.

Before any pull request is merged, it is required that it pass all of the automated tests which are set up in the mtest folder. The tests automatically perform actions which would be performed by a user, and create a new score, which is compared against the expected output.

There are two places you can find mtests: in the mtest directory (a subdirectory of the musescore root directory), or in the build.*/mtest directory.

In the root mtest directory

Tests are arranged into different directories. For example, copy and paste tests are in mtest/libmscore/copypaste. Each test has a .mscx file without a -ref, which is the score that tests are performed on, and a counterpart .mscx file with a -ref. The -ref file is the reference file: once the non-ref file has had the tests performed on it, it's compared with the reference file. If the files are exactly the same, the tests pass. But you can't run tests in the root mtest directory.

In the build.*/mtest directory

Here, you can run tests, once you've built release, debug etc.

Before you run any tests, make sure that your default style in MuseScore is the same as the MuseScore Default. As of version 3.1 there is a Reset to MuseScore Default button in the Format menu Style... dialog. Click that button prior to running ctest or mtest or any specific test executable. If you have your own styles you wish to preserve, save them in a style file that you can restore later.

To run the entire suite of tests, first make the tests in the build.*/mtest directory. To run them all, run the mtest executable file that is created, or just run ctest.

To run just a single suite of tests, enter the relevant directory, e.g. build.*/mtest/libmscore/copypaste/. Then, make the tests, and run the relevant executable file that is created, e.g. tst_copypaste.

After running tests, depending on which tests you ran, loads of [testname].mscx files will be created. These are the files that are compared against the reference files in the root mtest directory. If there is a difference between them, the mtest will fail.

Changing Tests

If you have made a change that makes the output of a test different, you need to update the reference file(s). You can do this by running updateReference in the root mtest directory for whichever test you want to update, although this sometimes may not work. Or, you can manually copy accross the changed file(s), for example:

cp build.debug/mtest/libmscore/copypaste/copypaste23.mscx mtest/libmscore/copypaste/copypaste23-ref.mscx

Adding new tests

To add a new test, edit the relevant tst_*.cpp file in the root mtest directory. It should be easy to figure out how these work.
If you add a new test that diffs files, please create an updateReference script to copy the reference files when any future changes are made.
If you have added a new file to an existing test, and it uses a " -ref" reference file for diff, please update the updateReference script in the test directory, to assist with future maintenance.