Test script facility: a new approach to testing and reporting bugs

Hello everyone.

We are pleased to announce the new testing facility which allows recording, playing and automated running tests on Travis. The new facility is based on self-developed scripts engine which keeps user actions in human-readable format. The new engine makes creating tests easier for both developers and testers. Moreover, they extend the number of scenarios which could be caught, recorded and automated in Travis.

Quick tour

We recorded a video to show you a quick tour of the new testing facility: https://youtu.be/Ua35MB0qnRc.

How to add a new test to the repository (developers)

1. Open script recorder tool (Tools → Script Recorder)
2. Change current directory to /mtest/testscript/scripts
3. Open a file from testscript/scripts/init folder (if you need to use a file which is not there yet, just put it there)
4. Enter a for the script (don’t use extension, e.g. “quarter_note_input_failed”)
5. Do necessary operations in the score (see “Limitations of the script engine” section)
6. Press “Stop recording”
7. .script and .mscx files should appear in scripts/ directory. They are required for the automated testing facility so add them to Git repository (as well as init file if you added a new one).

How to use scripts when reporting bugs (testers)

1. Open script recorder tool (Tools → Script Recorder)
2. Open the file which is affected by the issue
3. Choose any folder where you want to write the script, enter a name.
4. Press “Record”
5. Do the operations to reproduce the issue (see “Limitations of the script engine” section)
6. Press “Stop recording” (not applicable if a crash is an issue. Script will be written in this case by the way)
7. Attach the archive with the original file and the “.script” file which appeared in the chosen folder to the issue description. If you want to demonstrate the final state of the score you can attach the corresponding “.mscx” file too (which can be found alongside with “.script” file in case there was no crash during script recording)

Limitations of the script engine

Currently, we support the following actions in the scripts:

• keyboard shortcuts
• menu actions which do not open additional dialogs

We are planning to improve the scripting facility to support adding palette elements to the elements, and capture properties changed via Inspector panel. Text editing is not recorded, too.

The new facility works along with mtests and vtests frameworks, but significantly simplifies writing tests.

We need your feedback

• Use test scripts each time you report issues.
• Use tests in Pull Requests to keep fixed behavior.

Please, let us know about all the inconsistencies you faced while using the tool. Follow this thread to be notified about the improvements we are introducing to the facility.