automatically build & upload debug builds of AppImage to bintray
I'd like to configure the Makefile+Recipe+travis+bintray scripts to build and upload a debug version of musescore whenever a commit is made to the official MuseScore repo. Currently, those scrips only upload "release" builds. But there will inevitably be bugs, and it dramatically helps out debugging when all the qDebug() strings are printed to console and for producing stack traces when musescore crash.
I would suggest that commits to the numbered version branches (i.e. 2.0.3) produce release builds, since those branches are being prepared for release, while commits to the musescore master branch produce debug builds, since master will likely include more new features that need to be tested hence might produce bugs.
What do people think?
I think I know what changes to make:
- Makefile portable target will need to be broken into a "make portable debug" and "make portable release" (which would be the same as "make portable" unspecified). Would need to use "build.debug" instead of "build.release" for build directory name.
- Recipe would need to accept a paremeter for triggering debug build, or would have to make a seperate Recipedebug. Instead of make "$@" portable would be make "$@" portable debug. Might want to also have the CentOS image install and use the debug versions of libraries.
- build/travis/job2_AppImage/build.sh would need to use whatever debug command is decided when calling the Recipe
- .travis.yml would have a parralel build to make the debug version.
Comments
I think it could be extremely useful also to have debug builds for Windows, since there is no console output for "release" Windows build (except maybe if they are run inside a debugger).
I'm not sure, but maybe also Linux release build do print debug output when run with "-d" command from a terminal.
I sometimes provided debug Windows build to identify difficult-to-be-reproduced bugs, but it would be much easier, if possible, to have automatic builds and point users experiencing difficult-to-be-identified bugs to them, and then use custom debug builds only in the case a "usual" debug build is not enough (to pin down the Windows 10 font bug I had to add more comments to the code to identify where exactly the crash was happening).
For Windows debug build, I usually attach a batch file which saves the console output to a log file that can then be attached to a forum/bug report thread (so that non-expert users do not have to deal with a command prompt), it reads:
set QML_IMPORT_TRACE=1start /B ./bin/MuseScore.exe -d -F > ./log_file.txt 2>&1
However, some crashes or bugs happen ONLY in release builds, so I think the debug builds should not be the only one exposed to people, even alpha/beta testers.
Moreover, the debug builds are much larger than release build and would occupy a lot of space on the server.
In reply to I think it could be extremely by ABL
@ABL, nightlies for windows need some love that I can't give them right now. Would you be willing to do that? If yes, drop me an email or pass by IRC so we can discuss what needs to be done.
@ericfont I agree. I would put the debug builds in another repository. Feel free to make a PR.
In reply to I think it could be extremely by ABL
>> "I'm not sure, but maybe also Linux release build do print debug output when run with "-d" command from a terminal."
You're right! I was mistaken.
An additional idea: in the debug build for Linux, if they are performed with a sufficiently recent gcc version, we could possibly also enable address-sanitizer (it simply requires setting a couple of flags during compilation): in the case of crashes it displays a backtrace and other useful information.
https://en.wikipedia.org/wiki/AddressSanitizer
In reply to An additional idea: in the by ABL
Unfortunately, the AppImages are built on a very old version of linux to ensure maximal compatibility (CentOS 6.7 Linux 2.6.32-573.el6), so I don't think AddressSanitizer would work with those AppImage built in that VM. Although could conceivably have a seperate bintray AppImage package made on recent linux that includes AddressSanitizer...
CORRECTION: Actually those builds use gcc 4.8.2, and according to https://gcc.gnu.org/gcc-4.8/changes.html that gcc version should support AddressSanitizer...I'll look into this when making debug builds...
In reply to Unfortunately, the AppImages by ericfontainejazz
For gcc 4.8, if I remember correctly, in order to obtain the file name and lines instead of hexadecimal address values a script should be used, see:
http://wiki.blender.org/index.php/Dev:Doc/Tools/Debugging/GCC_Address_S…
in particular about asan_symbolize.py
I'm not entirely sure how having debug AppImages would be useful given that you'd need to have a working copy of the code anyway if you are planning to actually fix the bug. Once you've compiled the code once future compilations don't take nearly as long (unless you do "
make clean" first), so it's not as though that's much of a burden. If compilation is slow on your machine it is annoying, but it's not like you have to sit and watch it happen. But the point is you would still have to do it to make changes.Also, I'm wary about taking advantage of the free resources provided by Bintray and Travis. It seems a bit unfair to have so many builds going on, all producing essentially the same thing. If you can point out a good, genuine use case then I wouldn't be opposed to this.
There are ways to manually trigger Travis builds, so perhaps we could make a button somewhere to manually request a debug AppImage build when you need one. (We could combine it with the button to build Pull Requests.) I'm thinking we put this button on a nightlies page on the MuseScore website, and when someone clicks the button it sends a message to Travis to trigger a build (having first checked that there isn't already a build of that particular commit). This would prevent resources being wasted.
In reply to I'm not entirely sure how by shoogle
There are a bunch of qDebug statements in the code which print output to the screen, but are only included if compiled with the CMAKE DEBUG variable. This output can be very useful when a non-developer user is experiencing a bug. Especially when there is a crash involved, and if user hasn't been able to communicate the *exact* reproduction steps, then asking the user to run the latest debug build from the terminal and report back the console output can provide a lot of info. And with the suggestion to add the Address Saniter enabled for these debug builds, I think that can be extremely useful because will quickly identify issue as a memory corruption bug (which I find occur quite a bit for musescore crashes!).
Regarding testing other contributor's PR's: my most powerful computer is from May 2010, and it does take me unecessary time to checkout someone's branch and compile. I would very much appreciate builds of PRs, and I can agree to your suggestion about manually triggering Travis builds. Are you going to implement that, or should I (try)?
In reply to There are a bunch of qDebug by ericfontainejazz
To me, providing debug builds for Linux is just a small step towards the real goal.
Most of our users are on windows, some are on Mac too... so a debug build on Linux would not really help a lot. The real goal is a complete cross platform crash reporting system.... For every nightly, we keep the symbols, we distribute a release build without them (it's smaller), and if a crash happens, the crash log is sent to a server and the server uses the symbols to make a proper, readable crash report. We archive these crash reports. Symbol generation could be done in Travis. I can provide the necessary servers/storage to host the system. Of course, we would do the same for release versions!
I started to investigate a long time ago and I proposed this as a gsoc project this year and I think most of previous years. See https://musescore.org/en/developers-handbook/google-summer-code/ideas-2…
The problem are:
1/ We need time and resources to investigate and create such an infrastructure.
2/ It's somehow a less sexy project compared to lyric hyphenation or chord playback (or mobile app development in my case) And it's a huge project compared to just adding a couple of flags to enable Linux debug build on travis
3/ We would need eyeballs to go through the crash reports and cluster them, investigate them, file bug reports. But that would be an happy problem if the system is in place.
In reply to To me, providing debug builds by [DELETED] 5
would OSLOSU be suitable place to host a Socorro server? Apparently they provide VM's http://osuosl.org/services/hosting/details
In reply to would OSLOSU be suitable by ericfontainejazz
Yes sure. We probably have some work to do ourself on MuseScore before going to them but I'm sure they would be happy to help. If not we can find another place. Really, hosting the server is not the most daunting task in the whole setup.
In reply to Yes sure. We probably have by [DELETED] 5
Right. I mentioned because I saw they provide VMs.
I never really know how complicated these things are until actually try them out. I suppose I can just try it out setting up this server in my home network on a VM, and see if I can get it to receive a crash from a test program, and then try musescore.
I take it that you haven't got any GSoC students applying thus year interested in this. I have others things on my plate now, but if no one takes it up I'd be up for looking into it.