Investigating AppImage version 2
Ciao.
In these days I was trying to investigate the possibility of creating a version 2 AppImage for the Linux distribution.
As already said before, this would give some benefits, among which the possibility of autoupdating the AppImage, #279707: Automatic updates on Linux
I tried from scratch inside an Ubuntu 14.04 64bit virtual image, with linuxdeploy tool https://github.com/linuxdeploy/linuxdeploy
Here is the log of what I did:
sudo apt-get update
sudo apt-get -y install git unzip libxslt1-dev libxrender-dev libxcomposite-dev libdrm-dev libgl1-mesa-dev libopenal1 gstreamer0.10-plugins-base software-properties-common
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get -y install g++-4.9 wget make curl alsa libasound2-dev portaudio19-dev libportmidi-dev libsndfile1-dev zlib1g-dev libfreetype6-dev libfontconfig1-dev lame libmp3lame-dev libegl1-mesa-dev libpulse-dev libnss3-dev libxss1 libnspr4-dev libxi-dev libxcursor-dev libxtst-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9
mkdir qt5
wget -q -O qt5.zip http://utils.musescore.org.s3.amazonaws.com/qt593.zip
unzip -qq qt5.zip -d qt5
rm -f qt5.zip
CMAKE_URL="http://www.cmake.org/files/v3.5/cmake-3.5.1-Linux-x86_64.tar.gz"
mkdir cmake
wget --no-check-certificate --quiet -O - "${CMAKE_URL}" | tar --strip-components=1 -xz -C cmake
export PATH="${PWD}/qt5/bin:$PATH"
export QT_PLUGIN_PATH="${PWD}/qt5/plugins"
export QML2_IMPORT_PATH="${PWD}/qt5/qml"
export LD_LIBRARY_PATH="${PWD}/qt5/lib:$LD_LIBRARY_PATH"
export PATH="${PWD}/cmake/bin:${PATH}"
wget --no-check-certificate --quiet https://github.com/linuxdeploy/linuxdeploy/releases/download/continuous/linuxdeploy-x86_64.AppImage
chmod +x linuxdeploy-x86_64.AppImage
git clone https://github.com/musescore/MuseScore
cd MuseScore
make release PREFIX=/usr
cd build.release/
make install/strip DESTDIR=../AppDir
cd ..
chmod +x AppDir/usr/bin/QtWebEngineProcess
cp build/Linux+BSD/portable/qt.conf AppDir/usr/bin/qt.conf
../linuxdeploy-x86_64.AppImage --appdir AppDir --output appimage -d AppDir/usr/share/applications/mscore.desktop -i AppDir/usr/share/icons/hicolor/scalable/apps/mscore.svg
The first block is used to prepare the environment for the compilation, which I copied straight from the Docker image used at the moment for MuseScore AppImage compilation.
First question: I see that since January the Docker image is based on Ubuntu 14.04; since also Travis gives an Ubuntu 14.04 machine, is the Docker file still necessary? Can't we create the AppImage directly inside Travis virtual machine?
(Previously, the Docker file was using Ubuntu 12.04, but I tried with an Ubuntu 12.04 virtual machine and got a series of errors)
The second block simply downloads the AppImage of linuxdeploy.
The third block fetches MuseScore source code.
The fourth block is the "make and make install" compilation core.
The fifth block are a couple of workarounds (already used in the current MuseScore AppImage) to avoid a crash at MuseScore AppImage startup.
The sixth block creates the AppImage itself.
As I can see, copying the "AppRun.sh", the "portable-utils", and the workaround for jack library before running linuxdeploy should create a working AppImage (see: https://github.com/probonopd/linuxdeployqt/wiki/Custom-wrapper-script-i… and linuxdeploy should avoid overwriting AppRun.sh https://github.com/linuxdeploy/linuxdeploy/commit/2f7ad201f1df3c9 - to be verified).
One of my concerns is the fact that with linuxdeploy the folders qt5/plugins and qt5/qml are not copied, but these can be manually copied before creating the AppImage.
Second question: Do we really need the whole contents of these folders? How can we understand what is really needed and what can be left out?
I know for sure that libqxcb.so is necessary otherwise the application won't start (and indeed the AppImage I managed to build runs only in the virtual machine in which it was created).
linuxdeploy creates in addition folder share/doc with the copyrights for the libraries it can find information on the system.
These are the libraries included in the current AppImage, but not copied by linuxdeploy in the new AppImage - are they maybe needed by Qt plugins?
libEGL.so.1
libopenal.so.1
libQt5Concurrent.so.5
libQt5DBus.so.5
libQt5Designer.so.5
libQt5Multimedia.so.5
libQt5MultimediaWidgets.so.5
libQt5OpenGL.so.5
libQt5Sensors.so.5
libQt5WebEngine.so.5
libQt5XcbQpa.so.5
These are additional libraries copied by linuxdeploy in the new AppImage, but not present in the current AppImage:
libasyncns.so.0
libdbus-1.so.3
libffi.so.6
libFLAC.so.8
libgbm.so.1
libgmodule-2.0.so.0
libgthread-2.0.so.0
libjson-c.so.2
libnsl.so.1
libogg.so.0
libpcre.so.3
libpulsecommon-4.0.so
libpulse.so.0
libsndfile.so.1
libvorbisenc.so.2
libvorbisfile.so.3
libvorbis.so.0
libwayland-client.so.0
libwayland-server.so.0
libwrap.so.0
libX11-xcb.so.1
libXau.so.6
libxcb-glx.so.0
libxcb-present.so.0
libxcb-sync.so.1
libxcb-xfixes.so.0
libXcomposite.so.1
libXcursor.so.1
libXdamage.so.1
libXdmcp.so.6
libXext.so.6
libXfixes.so.3
libXi.so.6
libXrandr.so.2
libXrender.so.1
libxshmfence.so.1
libXtst.so.6
libXxf86vm.so.1
I don't fully understand the chain (Travis + Docker + a series of custom scripts) used to build the current AppImage, so I don't think I could be able to write a PR for an AppImage version 2, but at least with linuxdeploy the work seems to be easier.
Comments
I have an update: the AppImage created with linuxdeploy was not working, even when I copied the plugins and qml fodlers inside the AppDir folder.
But then I tried with linuxdeployqt https://github.com/probonopd/linuxdeployqt/ and it worked like a charm; I didn't even have to manually copy qt.conf: linuxdeployqt automatically created it for me.
I had to add two additional dependencies needed by one of the plugins:
sudo apt-get install mysql-client libpq5
and then I followed the instruction from linuxdeployqt readme page:
wget -c -nv "https://github.com/probonopd/linuxdeployqt/releases/download/continuous/linuxdeployqt-continuous-x86_64.AppImage"
chmod a+x linuxdeployqt-continuous-x86_64.AppImage
and I ran (instead of the sixth block of my previous comment):
unset QTDIR; unset QT_PLUGIN_PATH ; unset LD_LIBRARY_PATH
../linuxdeployqt-continuous-x86_64.AppImage AppDir/usr/share/applications/*.desktop -qmldir=AppDir/usr/share/mscore-3.0/plugins -appimage
It automatically included
- some of the plugins in the plugins folder: the same as already included in the current AppImage and some more
- and some qml folders: it selected them by scanning the example qml scripts in share/mscore-3.0/plugins
For those who want to try, here is the resulting AppImage:
https://drive.google.com/open?id=1K7D9nDZCReOyf7xq3XpC7LRdrlISox4K
Notes:
a)
At the moment, a custom AppRun.sh (as the one used by MuseScore) can be included in two ways:
-1- use linuxdeployqt without the "-appimage" flag, and then use appimagetool to bundle the AppImage, see for example https://github.com/probonopd/cubelister/blob/baseline/.travis.yml#L26-L…
-2- use a recent linuxdeployqt build (after https://github.com/probonopd/linuxdeployqt/pull/335 )
b)
It copied twice QtWebEngineProcess, one in usr/bin and one in usr/libexec, as well as twice the folders qtwebengine_locales (one in usr/translations and one in usr/lib/qt5/translations) and resources (one in usr and one in usr/lib/qt5).
c)
It should be simple to add update info. From https://github.com/probonopd/linuxdeployqt/issues/202 :
"It should automatically detect if it is running on Travis CI, and it should automatically embed the required update information automatically if the environment variable GITHUB_TOKEN is set (which uploadtool also requires)."
In reply to I have an update: the… by ABL
As ordinary user: works fine for me (have never had such a fast download;). Wouldn't it be useful to discuss it further inside (shoogle's) issue tracker entry you mentioned?
Thanks for giving this a try.
> First question: I see that since January the Docker image is based on Ubuntu 14.04; since also Travis gives an Ubuntu 14.04 machine, is the Docker file still necessary? Can't we create the AppImage directly inside Travis virtual machine?
It was never necessary to use Docker, but it is highly useful for various reasons. One way it is helpful to you is that you can install Docker on your own computer and reproduce the build environment exactly, but benefit from your own computer being faster than Travis. On your local machine you can run incremental builds (i.e. run
make
without runningmake clean
so you don't have to start from scratch each time. If you can get comfortable with Docker you will find it much more convenient than Travis.Here is the command you would use with Docker on your own machine:
In reply to Thanks for giving this a try… by shoogle
There are three parts to this command:
-i -t
).-v
) the current working directory${PWD}
inside the container as/MuseScore
./bin/bash
inside the container.At this point your Terminal prompt will change from
$
to#
to indicate that you are running as root inside the container, and you can usels
andcd
to look around. You won't be able to access anything outside the container except the MuseScore directory that you explicitly mounted inside.Once inside the container you can do:
In reply to There are three parts to… by shoogle
If the build fails, don't close the terminal or exit the container, just edit the recipe and run the recipe again. You can edit the recipe in your normal text editor outside the container and it will be updated inside.
> One of my concerns is the fact that with linuxdeploy the folders qt5/plugins and qt5/qml are not copied, but these can be manually copied before creating the AppImage.
Second question: Do we really need the whole contents of these folders? How can we understand what is really needed and what can be left out?
I know for sure that libqxcb.so is necessary otherwise the application won't start (and indeed the AppImage I managed to build runs only in the virtual machine in which it was created).
Shared libraries
Firstly there are the basic library dependencies that you can detect with
ldd
. If you useldd
on those libraries you find that they have their own dependencies, so you get dependencies of dependencies, and so on. If these are not all available at runtime then the program will fail to start, so they need to be copied into the AppImage. The only libraries that don't need to be copied are ones that are installed by default on every Linux distribution.We used to use ldd-recursive to detect all of them and then manually copy them into the AppDir with copy-libs, but now
linuxdeploy
andlinuxdeployqt
can do all of this for you. They scan the executable to see which libraries are linked to, and then they copy those libraries (and their dependencies) into the AppDir.Plugins (and QML)
Secondly there are "plugins" (and QML is basically just one big plugin). These are libraries that are not loaded via the usual
ldd
mechanism but withdlopen
. This method of loading allows the program to run even if a particular plugin is not present on the system, but features that that depends on the plugin will not work and the program might crash if you try to use those features. It is a good idea to include plugins in the AppImage, but unfortunately it is much harder to detect plugin dependencies than shared library dependencies (you can't useldd
for example). In fact, as far as I know there is no way to detect which plugins are needed just by looking at the executable; you have to look at the source code or ask somebody who has.We used to base the list of plugins needed off of the list used on Windows and then add more with trial and error. Plugins can also depend on ordinary shared libraries, so we would run
lddrecursive
on each plugin. The plugins and their dependencies would be added to the list in copy-libs.However, as you found
linuxdeployqt
is able to take a very good guess at which plugins are needed and copy them into the AppDir for you. Whilelinuxdeployqt
doesn't know anything about MuseScore's source code, it does know about Qt's source code. It knows that if you link against certain Qt libraries (which it can detect using the ordinaryldd
mechanism) then it is very likely that you will need certain Qt plugins. This method allows it to find and copy all the plugins you need, plus some extra ones that you might not need but there is no harm in having.Linuxdeploy vs Linuxdeployqt
linuxdeploy
is a new version of / better replacement forlinuxdeploy
. Both programs can detect ordinary shared library dependencies and both can detect Qt plugin dependencies. The difference is thatlinuxdeployqt
can detect Qt plugins out of the box, whereaslinuxdeploy
needs a helper program called linuxdeploy-plugin-qt. However,linuxdeployqt
can only detect Qt plugins, whereaslinuxdeploy
is able to detect plugins for other frameworks like GTK+, providing you use the right helper program.TLDR:
linuxdeploy
and linuxdeploy-plugin-qt. They will do most of the work for you.linuxdeployqt
(it is slow).In reply to > One of my concerns is the… by shoogle
Sorry for the intrusion. Given my lack of confidence with Linux maybe there is something flashy that I do not see but...
Mint offered me the update (on a 32 bit PC) and now I have the v3 but no longer the 2.3.2.
It also used the link on the desk leaving the name MuseScore 2 but launched version 3.
I don't know if this is of any use to you. Regards.
In reply to Sorry for the intrusion… by Shoichi
@ Shoichi
In your case the update is coming through the repository of the package manager, but I don't see MuseScore 3.0 in the stable release repository: https://launchpad.net/~mscore-ubuntu/+archive/ubuntu/mscore-stable/+pac…
maybe I'm looking at the wrong one, or are you using the "unstable" (i.e. nightly build) repository?
By the way, the AppImage version is the equivalent under Linux of the Windows portable version, or the Mac version: it should work by simply downloading and running it, without installing additional dependencies or passing through the package manager.
Ciao! :-)
In reply to > One of my concerns is the… by shoogle
@shoogle : thank you for your explanations about docker and linuxdeploy/linuxdeployqt.
Regarding this latter topic, by trial and error I had found the qt-plugin of linuxdeploy, I briefly mentioned it in the other thread, but I found that with linuxdeploy + linuxdeploy-qtplugin:
* I could not prevent it to install Qt and QtWebengine translations, so that it resulted in a series of files copied twice (in two different directories) inside the AppImage, since MuseScore is already copying the translations in its folder under "share" folder;
* I could not prevent it to install the copyright files of the bundled libraries in the "doc" folder (they are not present in the current AppImage); then I found by searching inside its source code the action of the DISABLE_COPYRIGHT_FILES_DEPLOYMENT environment variable, but I haven't tested it yet;
* among the Qt plugins automatically installed, it did not copied the printsupport folder and its content inside the "plugin" folder, even if I played around with the EXTRA_QT_PLUGINS environment variable (i.e. using the plugin name, or its path, or the path to the .so file); this library is included in the current AppImage;
- under all the other aspects it was completely equivalent to using the original linuxdeployqt, and the resulting AppImage was working as well (but I didn't extensively check).
Regarding docker images: if I understand correctly, if I update the recipe it will not update locally the docker image, because that is the mirror of the online-stored image which on the other hands depends on the master bramch of MuseScore github; unless I create a personal docker image and link it to my fork of MuseScore and then to update this image I have to push the recipe change to my github account, wait for the online docker image to fetch the changes and rebuild, and then fetch the docker image again locally. Am I correct, or is there an easier way?
Thank you very much for your help.
In reply to @shoogle : thank you for… by ABL
> I could not prevent it to install Qt and QtWebengine translations, so that it resulted in a series of files copied twice (in two different directories) inside the AppImage, since MuseScore is already copying the translations in its folder under "share" folder
I don't know about Qt or QtWebEngine translations, but if MuseScore already has them then you should be safe to delete the extra ones. Just run linuxdeploy to copy everything and then add a command afterwards to delete the stuff you don't need:
Can you just delete the directory or are they mixed in with everything else?
> I could not prevent it to install the copyright files of the bundled libraries in the "doc" folder (they are not present in the current AppImage)
The copyright files are included for licensing reasons. I'm not sure it is strictly necessary given that we already display some licensing information under the Help menu, but it's best to leave them in. It shouldn't add much to the size since the AppImage is compressed.
> among the Qt plugins automatically installed, it did not copied the printsupport folder and its content inside the "plugin" folder, even if I played around with the EXTRA_QT_PLUGINS environment variable (i.e. using the plugin name, or its path, or the path to the .so file); this library is included in the current AppImage;
Maybe this is not necessary? Try running the AppImage and see if you can print. If not just add a command to copy that directory in.
When I created the original AppImage I just bundled everything that looked like it might be needed. In general
linuxdeploy-plugin-qt
should do a better job of guessing what is needed. If it doesn't then it is a bug and should be reported to https://github.com/linuxdeploy/linuxdeploy-plugin-qt/.In reply to @shoogle : thank you for… by ABL
> Regarding docker images: if I understand correctly, if I update the recipe it will not update locally the docker image, because that is the mirror of the online-stored image which on the other hands depends on the master bramch of MuseScore github; unless I create a personal docker image and link it to my fork of MuseScore and then to update this image I have to push the recipe change to my github account, wait for the online docker image to fetch the changes and rebuild, and then fetch the docker image again locally. Am I correct, or is there an easier way?
This is correct, the recipe does change the local image, but the change is not permanently stored, so when you exit the container it will revert back to the online image. However, this is not really a problem; just carry on updating the recipe and when your pull request is merged MuseScore's Docker image will be updated automatically.
if you need more dependencies than are provided in the image (e.g.
linuxdeploy
andlinuxdeploy-plugin-qt
) then you can just fetch them in the recipe. Notice how the recipe checks whether dependencies are already installed and only installs the ones that are not already present. You need to do a similar check for any new dependencies (e.g.linuxdeploy
andlinuxdeploy-plugin-qt
). The check will always fail until you update the Docker image, so the new dependency will be installed every time, but that won't take very long. When your pull request is merged and MuseScore's Docker image has been updated the check will pass and the dependency will no longer be installed (it will be provided by the Docker image).If you do want to build your own Docker image (you don't have to), there is probably a way to rebuild it locally so you wouldn't need to make an account on Docker Hub. The only reason to build your own image is to avoid waiting for your new dependencies to be downloaded each time you run the recipe in a fresh image. You would still have to wait for MuseScore to be built.
If you did want to create an account on Docker Hub you just need to create a repository called "musescore-x86_64" (here's mine) and set some variables in the Travis environment to tell it to use your Docker account instead of MuseScore's:
In reply to > * Regarding docker images:… by shoogle
At the moment I am still experimenting with Travis directly, not through a Docker image, yet.
Here is the appimage made with linuxdeploy and its qt-plugin:
https://bintray.com/antoniobl/musescore-custom-nightlies/download_file?…
It does not contain the printsupport folder inside the plugin folder: I quickly tested in my Linux environment and it prints, but I only have the "print to file" option, since I do not have a printer.
Can someone check if it prints?
I also found that linuxdeploy-qt-plugin at the moment does not automatically include libraries probably needed by the qml extensions I manually copied into the appimage (even if I copied them before running linnuxdeploy), that check-depends lists as "provided by neither" (i.e. system or the appimage).
In reply to At the moment I am still… by ABL
Good job! I can't test printing either, but I wouldn't worry about getting everything working on a first attempt. MuseScore has a much quicker release cycle now, so if you can get a version released that is 90% working then it would only be 2-4 weeks before the next version. Also, the AppImage could potentially be upgraded without waiting for another MuseScore release.
If
linuxdeploy-qt-plugin
thinks some libraries are not needed then I am inclined to believe it. Just exclude those libraries for now and see what happens. If MuseScore runs then they are not needed by it, but might be needed by a plugin. You should go to the Plugin Manager and enable all the plugins, then run them one by one. If they all work then the library is not needed by the plugins either.In reply to At the moment I am still… by ABL
Here is a diff of the output of
check-depends
from your AppImage vs the official AppImage for 3.0.0. The objective is not to make these equal (I would just go with whateverlinuxdeploy-plugin-qt
gives you for the time being), but you might find it informative.I generated this by running
check-depends
for both, which leaves you looking at the dependency list inless
. Presss
to save the output to a file (e.g. "AppImage-old.txt
"), then run this command to do the diff:You are already bundling more things than the official AppImage. Some things like
libXcursor.so.1
have moved from "Provided by system" to "Provided by both". If you are bundling those things manually then you needn't bother. Iflinuxdeploy-plugin-qt
is doing it then keep them in and consider submitting a bug report tolinuxdeploy-plugin-qt
when you have finished working on the AppImage.In reply to At the moment I am still… by ABL
The output has increased from "AppImage contains 1600 executables and 86 libraries" for the official AppImage to "AppImage contains 1820 executables and 158 libraries" in yours, but the actual file size of the AppImage has barely increased at all (by less than 1 MB and the total size is 148 MB) so the extra libraries is not really a serious concern.
By the way, the huge number of executables reported comes from the fact that when the AppImage is mounted, all the files inside it are given execute permission, so they all appear as "executables". Ideally this wouldn't happen, but it's not our bug to fix.
In reply to The output has increased… by shoogle
Actually libraries such as libX11-xcb.so.1 and libXcursor.so.1 and so on are copied into the AppImage by linuxdeploy before it starts the qt-plugin and they are left in the bundle since they do not appear in the exclusion list, which is taken from here: https://raw.githubusercontent.com/probonopd/AppImages/master/excludelist
You can see in the full compile log:
https://api.travis-ci.org/v3/job/479113513/log.txt
Section "-- Deploying dependencies for existing files in AppDir --" (it also listed as one of the needed libraries when the qt-plugin is run, section "-- Running input plugin: qt --").
As you can see in the compilation log, there are some Qt libraries listed as "provided by neither": I think these are searched for by some of the files in Qt qml folder, which I copied as a whole as it was done in the previous AppImage version. However, if I let linuxdeploy-qt-plugin scan MuseScore qml plugins, it copies only a fraction of these Qt qml folders (only those needed by the example MuseScore plugins, in principle).