Investigating AppImage version 2

• Dec 30, 2018 - 16:14

Ciao.
In these days I was trying to investigate the possibility of creating a version 2 AppImage for the Linux distribution.
As already said before, this would give some benefits, among which the possibility of autoupdating the AppImage, #279707: Automatic updates on Linux
I tried from scratch inside an Ubuntu 14.04 64bit virtual image, with linuxdeploy tool https://github.com/linuxdeploy/linuxdeploy
Here is the log of what I did:
sudo apt-get update
sudo apt-get -y install git unzip libxslt1-dev libxrender-dev libxcomposite-dev libdrm-dev libgl1-mesa-dev libopenal1 gstreamer0.10-plugins-base software-properties-common
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get -y install g++-4.9 wget make curl alsa libasound2-dev portaudio19-dev libportmidi-dev libsndfile1-dev zlib1g-dev libfreetype6-dev libfontconfig1-dev lame libmp3lame-dev libegl1-mesa-dev libpulse-dev libnss3-dev libxss1 libnspr4-dev libxi-dev libxcursor-dev libxtst-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9
mkdir qt5
wget -q -O qt5.zip http://utils.musescore.org.s3.amazonaws.com/qt593.zip
unzip -qq qt5.zip -d qt5
rm -f qt5.zip
CMAKE_URL="http://www.cmake.org/files/v3.5/cmake-3.5.1-Linux-x86_64.tar.gz"
mkdir cmake
wget --no-check-certificate --quiet -O - "${CMAKE_URL}" | tar --strip-components=1 -xz -C cmake
export PATH="${PWD}/qt5/bin:$PATH"
export QT_PLUGIN_PATH="${PWD}/qt5/plugins"
export QML2_IMPORT_PATH="${PWD}/qt5/qml"
export LD_LIBRARY_PATH="${PWD}/qt5/lib:$LD_LIBRARY_PATH"
export PATH="${PWD}/cmake/bin:${PATH}"

wget --no-check-certificate --quiet https://github.com/linuxdeploy/linuxdeploy/releases/download/continuous/linuxdeploy-x86_64.AppImage
chmod +x linuxdeploy-x86_64.AppImage

git clone https://github.com/musescore/MuseScore
cd MuseScore

make release PREFIX=/usr
cd build.release/
make install/strip DESTDIR=../AppDir
cd ..

chmod +x AppDir/usr/bin/QtWebEngineProcess
cp build/Linux+BSD/portable/qt.conf AppDir/usr/bin/qt.conf

../linuxdeploy-x86_64.AppImage --appdir AppDir --output appimage -d AppDir/usr/share/applications/mscore.desktop -i AppDir/usr/share/icons/hicolor/scalable/apps/mscore.svg

The first block is used to prepare the environment for the compilation, which I copied straight from the Docker image used at the moment for MuseScore AppImage compilation.
First question: I see that since January the Docker image is based on Ubuntu 14.04; since also Travis gives an Ubuntu 14.04 machine, is the Docker file still necessary? Can't we create the AppImage directly inside Travis virtual machine?
(Previously, the Docker file was using Ubuntu 12.04, but I tried with an Ubuntu 12.04 virtual machine and got a series of errors)

The second block simply downloads the AppImage of linuxdeploy.
The third block fetches MuseScore source code.
The fourth block is the "make and make install" compilation core.
The fifth block are a couple of workarounds (already used in the current MuseScore AppImage) to avoid a crash at MuseScore AppImage startup.
The sixth block creates the AppImage itself.

As I can see, copying the "AppRun.sh", the "portable-utils", and the workaround for jack library before running linuxdeploy should create a working AppImage (see: https://github.com/probonopd/linuxdeployqt/wiki/Custom-wrapper-script-i… and linuxdeploy should avoid overwriting AppRun.sh https://github.com/linuxdeploy/linuxdeploy/commit/2f7ad201f1df3c9 - to be verified).
One of my concerns is the fact that with linuxdeploy the folders qt5/plugins and qt5/qml are not copied, but these can be manually copied before creating the AppImage.
Second question: Do we really need the whole contents of these folders? How can we understand what is really needed and what can be left out?
I know for sure that libqxcb.so is necessary otherwise the application won't start (and indeed the AppImage I managed to build runs only in the virtual machine in which it was created).

linuxdeploy creates in addition folder share/doc with the copyrights for the libraries it can find information on the system.

These are the libraries included in the current AppImage, but not copied by linuxdeploy in the new AppImage - are they maybe needed by Qt plugins?
libEGL.so.1
libopenal.so.1
libQt5Concurrent.so.5
libQt5DBus.so.5
libQt5Designer.so.5
libQt5Multimedia.so.5
libQt5MultimediaWidgets.so.5
libQt5OpenGL.so.5
libQt5Sensors.so.5
libQt5WebEngine.so.5
libQt5XcbQpa.so.5

These are additional libraries copied by linuxdeploy in the new AppImage, but not present in the current AppImage:
libasyncns.so.0
libdbus-1.so.3
libffi.so.6
libFLAC.so.8
libgbm.so.1
libgmodule-2.0.so.0
libgthread-2.0.so.0
libjson-c.so.2
libnsl.so.1
libogg.so.0
libpcre.so.3
libpulsecommon-4.0.so
libpulse.so.0
libsndfile.so.1
libvorbisenc.so.2
libvorbisfile.so.3
libvorbis.so.0
libwayland-client.so.0
libwayland-server.so.0
libwrap.so.0
libX11-xcb.so.1
libXau.so.6
libxcb-glx.so.0
libxcb-present.so.0
libxcb-sync.so.1
libxcb-xfixes.so.0
libXcomposite.so.1
libXcursor.so.1
libXdamage.so.1
libXdmcp.so.6
libXext.so.6
libXfixes.so.3
libXi.so.6
libXrandr.so.2
libXrender.so.1
libxshmfence.so.1
libXtst.so.6
libXxf86vm.so.1

I don't fully understand the chain (Travis + Docker + a series of custom scripts) used to build the current AppImage, so I don't think I could be able to write a PR for an AppImage version 2, but at least with linuxdeploy the work seems to be easier.


Comments

I have an update: the AppImage created with linuxdeploy was not working, even when I copied the plugins and qml fodlers inside the AppDir folder.
But then I tried with linuxdeployqt https://github.com/probonopd/linuxdeployqt/ and it worked like a charm; I didn't even have to manually copy qt.conf: linuxdeployqt automatically created it for me.
I had to add two additional dependencies needed by one of the plugins:
sudo apt-get install mysql-client libpq5
and then I followed the instruction from linuxdeployqt readme page:
wget -c -nv "https://github.com/probonopd/linuxdeployqt/releases/download/continuous/linuxdeployqt-continuous-x86_64.AppImage"
chmod a+x linuxdeployqt-continuous-x86_64.AppImage

and I ran (instead of the sixth block of my previous comment):
unset QTDIR; unset QT_PLUGIN_PATH ; unset LD_LIBRARY_PATH
../linuxdeployqt-continuous-x86_64.AppImage AppDir/usr/share/applications/*.desktop -qmldir=AppDir/usr/share/mscore-3.0/plugins -appimage

It automatically included
- some of the plugins in the plugins folder: the same as already included in the current AppImage and some more
- and some qml folders: it selected them by scanning the example qml scripts in share/mscore-3.0/plugins

For those who want to try, here is the resulting AppImage:
https://drive.google.com/open?id=1K7D9nDZCReOyf7xq3XpC7LRdrlISox4K

Notes:
a)
At the moment, a custom AppRun.sh (as the one used by MuseScore) can be included in two ways:
-1- use linuxdeployqt without the "-appimage" flag, and then use appimagetool to bundle the AppImage, see for example https://github.com/probonopd/cubelister/blob/baseline/.travis.yml#L26-L…
-2- use a recent linuxdeployqt build (after https://github.com/probonopd/linuxdeployqt/pull/335 )
b)
It copied twice QtWebEngineProcess, one in usr/bin and one in usr/libexec, as well as twice the folders qtwebengine_locales (one in usr/translations and one in usr/lib/qt5/translations) and resources (one in usr and one in usr/lib/qt5).
c)
It should be simple to add update info. From https://github.com/probonopd/linuxdeployqt/issues/202 :
"It should automatically detect if it is running on Travis CI, and it should automatically embed the required update information automatically if the environment variable GITHUB_TOKEN is set (which uploadtool also requires)."

Thanks for giving this a try.

> First question: I see that since January the Docker image is based on Ubuntu 14.04; since also Travis gives an Ubuntu 14.04 machine, is the Docker file still necessary? Can't we create the AppImage directly inside Travis virtual machine?

It was never necessary to use Docker, but it is highly useful for various reasons. One way it is helpful to you is that you can install Docker on your own computer and reproduce the build environment exactly, but benefit from your own computer being faster than Travis. On your local machine you can run incremental builds (i.e. run make without running make clean so you don't have to start from scratch each time. If you can get comfortable with Docker you will find it much more convenient than Travis.

Here is the command you would use with Docker on your own machine:

cd /path/to/MuseScore  # your local copy of MuseScore's git repository
docker run -i -t -v "${PWD}:/MuseScore" "musescore/musescore-x86_64:latest" /bin/bash

In reply to by shoogle

There are three parts to this command:

  1. Run the container from musescore/musescore-x86_64:latest interactively (-i -t).
  2. Mount (-v) the current working directory ${PWD} inside the container as /MuseScore.
  3. Launch the executable /bin/bash inside the container.

At this point your Terminal prompt will change from $ to # to indicate that you are running as root inside the container, and you can use ls and cd to look around. You won't be able to access anything outside the container except the MuseScore directory that you explicitly mounted inside.

Once inside the container you can do:

cd /MuseScore  # go to the directory you mounted in the image
./build/Linux+BSD/portable/x86_64/Recipe  # run the recipe

> One of my concerns is the fact that with linuxdeploy the folders qt5/plugins and qt5/qml are not copied, but these can be manually copied before creating the AppImage.
Second question: Do we really need the whole contents of these folders? How can we understand what is really needed and what can be left out?
I know for sure that libqxcb.so is necessary otherwise the application won't start (and indeed the AppImage I managed to build runs only in the virtual machine in which it was created).

Shared libraries

Firstly there are the basic library dependencies that you can detect with ldd. If you use ldd on those libraries you find that they have their own dependencies, so you get dependencies of dependencies, and so on. If these are not all available at runtime then the program will fail to start, so they need to be copied into the AppImage. The only libraries that don't need to be copied are ones that are installed by default on every Linux distribution.

We used to use ldd-recursive to detect all of them and then manually copy them into the AppDir with copy-libs, but now linuxdeploy and linuxdeployqt can do all of this for you. They scan the executable to see which libraries are linked to, and then they copy those libraries (and their dependencies) into the AppDir.

Plugins (and QML)

Secondly there are "plugins" (and QML is basically just one big plugin). These are libraries that are not loaded via the usual ldd mechanism but with dlopen. This method of loading allows the program to run even if a particular plugin is not present on the system, but features that that depends on the plugin will not work and the program might crash if you try to use those features. It is a good idea to include plugins in the AppImage, but unfortunately it is much harder to detect plugin dependencies than shared library dependencies (you can't use ldd for example). In fact, as far as I know there is no way to detect which plugins are needed just by looking at the executable; you have to look at the source code or ask somebody who has.

We used to base the list of plugins needed off of the list used on Windows and then add more with trial and error. Plugins can also depend on ordinary shared libraries, so we would run lddrecursive on each plugin. The plugins and their dependencies would be added to the list in copy-libs.

However, as you found linuxdeployqt is able to take a very good guess at which plugins are needed and copy them into the AppDir for you. While linuxdeployqt doesn't know anything about MuseScore's source code, it does know about Qt's source code. It knows that if you link against certain Qt libraries (which it can detect using the ordinary ldd mechanism) then it is very likely that you will need certain Qt plugins. This method allows it to find and copy all the plugins you need, plus some extra ones that you might not need but there is no harm in having.

Linuxdeploy vs Linuxdeployqt

linuxdeploy is a new version of / better replacement for linuxdeploy. Both programs can detect ordinary shared library dependencies and both can detect Qt plugin dependencies. The difference is that linuxdeployqt can detect Qt plugins out of the box, whereas linuxdeploy needs a helper program called linuxdeploy-plugin-qt. However, linuxdeployqt can only detect Qt plugins, whereas linuxdeploy is able to detect plugins for other frameworks like GTK+, providing you use the right helper program.

TLDR:

  • Use linuxdeploy and linuxdeploy-plugin-qt. They will do most of the work for you.
  • Don't use linuxdeployqt (it is slow).
  • You can get rid of copy-libs.

In reply to by shoogle

Sorry for the intrusion. Given my lack of confidence with Linux maybe there is something flashy that I do not see but...
Mint offered me the update (on a 32 bit PC) and now I have the v3 but no longer the 2.3.2.
It also used the link on the desk leaving the name MuseScore 2 but launched version 3.
I don't know if this is of any use to you. Regards.

Attachment Size
18010702.png 152.34 KB

In reply to by Shoichi

@ Shoichi
In your case the update is coming through the repository of the package manager, but I don't see MuseScore 3.0 in the stable release repository: https://launchpad.net/~mscore-ubuntu/+archive/ubuntu/mscore-stable/+pac…
maybe I'm looking at the wrong one, or are you using the "unstable" (i.e. nightly build) repository?

By the way, the AppImage version is the equivalent under Linux of the Windows portable version, or the Mac version: it should work by simply downloading and running it, without installing additional dependencies or passing through the package manager.

Ciao! :-)

In reply to by shoogle

@shoogle : thank you for your explanations about docker and linuxdeploy/linuxdeployqt.
Regarding this latter topic, by trial and error I had found the qt-plugin of linuxdeploy, I briefly mentioned it in the other thread, but I found that with linuxdeploy + linuxdeploy-qtplugin:
* I could not prevent it to install Qt and QtWebengine translations, so that it resulted in a series of files copied twice (in two different directories) inside the AppImage, since MuseScore is already copying the translations in its folder under "share" folder;
* I could not prevent it to install the copyright files of the bundled libraries in the "doc" folder (they are not present in the current AppImage); then I found by searching inside its source code the action of the DISABLE_COPYRIGHT_FILES_DEPLOYMENT environment variable, but I haven't tested it yet;
* among the Qt plugins automatically installed, it did not copied the printsupport folder and its content inside the "plugin" folder, even if I played around with the EXTRA_QT_PLUGINS environment variable (i.e. using the plugin name, or its path, or the path to the .so file); this library is included in the current AppImage;
- under all the other aspects it was completely equivalent to using the original linuxdeployqt, and the resulting AppImage was working as well (but I didn't extensively check).

Regarding docker images: if I understand correctly, if I update the recipe it will not update locally the docker image, because that is the mirror of the online-stored image which on the other hands depends on the master bramch of MuseScore github; unless I create a personal docker image and link it to my fork of MuseScore and then to update this image I have to push the recipe change to my github account, wait for the online docker image to fetch the changes and rebuild, and then fetch the docker image again locally. Am I correct, or is there an easier way?

Thank you very much for your help.

In reply to by ABL

> I could not prevent it to install Qt and QtWebengine translations, so that it resulted in a series of files copied twice (in two different directories) inside the AppImage, since MuseScore is already copying the translations in its folder under "share" folder

I don't know about Qt or QtWebEngine translations, but if MuseScore already has them then you should be safe to delete the extra ones. Just run linuxdeploy to copy everything and then add a command afterwards to delete the stuff you don't need:

rm -rf /path/to/translations

Can you just delete the directory or are they mixed in with everything else?

> I could not prevent it to install the copyright files of the bundled libraries in the "doc" folder (they are not present in the current AppImage)

The copyright files are included for licensing reasons. I'm not sure it is strictly necessary given that we already display some licensing information under the Help menu, but it's best to leave them in. It shouldn't add much to the size since the AppImage is compressed.

> among the Qt plugins automatically installed, it did not copied the printsupport folder and its content inside the "plugin" folder, even if I played around with the EXTRA_QT_PLUGINS environment variable (i.e. using the plugin name, or its path, or the path to the .so file); this library is included in the current AppImage;

Maybe this is not necessary? Try running the AppImage and see if you can print. If not just add a command to copy that directory in.

When I created the original AppImage I just bundled everything that looked like it might be needed. In general linuxdeploy-plugin-qt should do a better job of guessing what is needed. If it doesn't then it is a bug and should be reported to https://github.com/linuxdeploy/linuxdeploy-plugin-qt/.

In reply to by ABL

> Regarding docker images: if I understand correctly, if I update the recipe it will not update locally the docker image, because that is the mirror of the online-stored image which on the other hands depends on the master bramch of MuseScore github; unless I create a personal docker image and link it to my fork of MuseScore and then to update this image I have to push the recipe change to my github account, wait for the online docker image to fetch the changes and rebuild, and then fetch the docker image again locally. Am I correct, or is there an easier way?

This is correct, the recipe does change the local image, but the change is not permanently stored, so when you exit the container it will revert back to the online image. However, this is not really a problem; just carry on updating the recipe and when your pull request is merged MuseScore's Docker image will be updated automatically.

if you need more dependencies than are provided in the image (e.g. linuxdeploy and linuxdeploy-plugin-qt) then you can just fetch them in the recipe. Notice how the recipe checks whether dependencies are already installed and only installs the ones that are not already present. You need to do a similar check for any new dependencies (e.g. linuxdeploy and linuxdeploy-plugin-qt). The check will always fail until you update the Docker image, so the new dependency will be installed every time, but that won't take very long. When your pull request is merged and MuseScore's Docker image has been updated the check will pass and the dependency will no longer be installed (it will be provided by the Docker image).

If you do want to build your own Docker image (you don't have to), there is probably a way to rebuild it locally so you wouldn't need to make an account on Docker Hub. The only reason to build your own image is to avoid waiting for your new dependencies to be downloaded each time you run the recipe in a fresh image. You would still have to wait for MuseScore to be built.

If you did want to create an account on Docker Hub you just need to create a repository called "musescore-x86_64" (here's mine) and set some variables in the Travis environment to tell it to use your Docker account instead of MuseScore's:

$DOCKER_USER # your username on Docker Hub
$DOCKER_TRIGGER_X86_64 # the secret key used to trigger docker builds in your repo

In reply to by shoogle

At the moment I am still experimenting with Travis directly, not through a Docker image, yet.
Here is the appimage made with linuxdeploy and its qt-plugin:
https://bintray.com/antoniobl/musescore-custom-nightlies/download_file?…
It does not contain the printsupport folder inside the plugin folder: I quickly tested in my Linux environment and it prints, but I only have the "print to file" option, since I do not have a printer.
Can someone check if it prints?

I also found that linuxdeploy-qt-plugin at the moment does not automatically include libraries probably needed by the qml extensions I manually copied into the appimage (even if I copied them before running linnuxdeploy), that check-depends lists as "provided by neither" (i.e. system or the appimage).

In reply to by ABL

Good job! I can't test printing either, but I wouldn't worry about getting everything working on a first attempt. MuseScore has a much quicker release cycle now, so if you can get a version released that is 90% working then it would only be 2-4 weeks before the next version. Also, the AppImage could potentially be upgraded without waiting for another MuseScore release.

If linuxdeploy-qt-plugin thinks some libraries are not needed then I am inclined to believe it. Just exclude those libraries for now and see what happens. If MuseScore runs then they are not needed by it, but might be needed by a plugin. You should go to the Plugin Manager and enable all the plugins, then run them one by one. If they all work then the library is not needed by the plugins either.

In reply to by ABL

Here is a diff of the output of check-depends from your AppImage vs the official AppImage for 3.0.0. The objective is not to make these equal (I would just go with whatever linuxdeploy-plugin-qt gives you for the time being), but you might find it informative.

I generated this by running check-depends for both, which leaves you looking at the dependency list in less. Press s to save the output to a file (e.g. "AppImage-old.txt"), then run this command to do the diff:

diff -u0 AppImage-old.txt AppImage-new.txt | grep "^[+-]" > check-depends_diff.txt

You are already bundling more things than the official AppImage. Some things like libXcursor.so.1 have moved from "Provided by system" to "Provided by both". If you are bundling those things manually then you needn't bother. If linuxdeploy-plugin-qt is doing it then keep them in and consider submitting a bug report to linuxdeploy-plugin-qt when you have finished working on the AppImage.

Attachment Size
check-depends_diff.txt 5.57 KB

In reply to by ABL

The output has increased from "AppImage contains 1600 executables and 86 libraries" for the official AppImage to "AppImage contains 1820 executables and 158 libraries" in yours, but the actual file size of the AppImage has barely increased at all (by less than 1 MB and the total size is 148 MB) so the extra libraries is not really a serious concern.

By the way, the huge number of executables reported comes from the fact that when the AppImage is mounted, all the files inside it are given execute permission, so they all appear as "executables". Ideally this wouldn't happen, but it's not our bug to fix.

In reply to by shoogle

Actually libraries such as libX11-xcb.so.1 and libXcursor.so.1 and so on are copied into the AppImage by linuxdeploy before it starts the qt-plugin and they are left in the bundle since they do not appear in the exclusion list, which is taken from here: https://raw.githubusercontent.com/probonopd/AppImages/master/excludelist
You can see in the full compile log:
https://api.travis-ci.org/v3/job/479113513/log.txt
Section "-- Deploying dependencies for existing files in AppDir --" (it also listed as one of the needed libraries when the qt-plugin is run, section "-- Running input plugin: qt --").

As you can see in the compilation log, there are some Qt libraries listed as "provided by neither": I think these are searched for by some of the files in Qt qml folder, which I copied as a whole as it was done in the previous AppImage version. However, if I let linuxdeploy-qt-plugin scan MuseScore qml plugins, it copies only a fraction of these Qt qml folders (only those needed by the example MuseScore plugins, in principle).

Do you still have an unanswered question? Please log in first to post your question.