Video Player Plugin
Today I decided I'm going to try to develop a video player plugin for MuseScore. I'm moving the old thread here from feature request, so far what I have, is I'm going to try to use the cursor object to get the video in sync with with the score, and I'm thinking of looking into Java video playback for the video player, if I can't find a simple open source video player in Java, I will try to write one.
Comments
I've been looking at the Java Media Framework (JMF) and I am going to work on making a simple video player, no controls accept a button to turn sound on and off. I'm going to probably be working on this for a while, never handled video in Java before, so it's new for me. I am also going to try to integrate Jffmpeg (Java ffmpeg) so that it can handle more formats other than AVI and MOV.
I'll keep posting my updates.
In reply to I've been looking at the Java by KoRnKloWn
The cursor object has nothing to do with the display of the score. As I mentioned before, it's just a plugin-ism to allow you to loop through the notes. So you'd have to write your own score display and animation routines as well.
In reply to The cursor object has nothing by Marc Sabatella
Not completely sure what you mean, couldn't I use the double time() function to get the current position and relay that to the video?
In reply to Not completely sure what you by KoRnKloWn
The time function will tell tell you the time position of the note the cursor is on, but again, that has nothing to do with what's being *displayed*. The score would just sit there not moving in any way while the video plays. There is no way to get to the score to play from the plugin architecture.
In reply to The time function will tell by Marc Sabatella
What I believe KoRnKloWn is talking about, is that the plugin itself wouldn't have a play button, but when the SCORE play button is pushed, if the video player plugin is open, then it would play along with it. Am I right?
There has to be something you can link it to. Maybe not the cursor since you really can't control that as it is now. But at least the score play button.
And what cursor are we talking about? Do you mean the blue bar that you follow across the score or the slider in the play panel?
In reply to If what i believe KoRnKloWn by Gabriel H
That's pretty much exactly what I'm talking about. And I'm talking about the Cursor object:
http://musescore.org/en/plugin-development/cursor-object
Basically what I have in mind is it would use time() from the cursor object to return the location on the score in milliseconds, and relay that to the video player to set the video at the same time, and yes, it would be the main play button on the score that would start it, and I'm not sure how that would work yet, I need to find out how to get the player to watch it, some kind of action listener or something. But the video player part doesn't seem like it will be too hard, the Java Media Framework makes video playback pretty simple. It would be way too hard (if not impossible) to make the video player straight out of Javascript.
In reply to That's pretty much exactly by KoRnKloWn
Actually now that I look at it, I'm not 100% sure the cursor object would work, the cursor is only created within the plugin, it's not referring to the blue cursor. But I'm still just going to worry about making the simple video player for now. If the cursor object doesn't work, there's got to be something that will.
In reply to Actually now that I look at by KoRnKloWn
That's what I mean when I say that the cursor object has nothing to do with what's displayed. There is no way to get any information about what is being played.
If you do create your own video player, you could certainly synchronize it from the beginning of the score. Or, if you've selected something, you can get at the selection. But I can't think of any way to start MuseScore's playback and your video player simultaneously. Synchronization would basically be limited to how fast you can manually press both "start" buttons.
In reply to That's what I mean when I say by Marc Sabatella
Yeah, I just realized that, I was getting ahead of myself a little bit. But I still think there may be possibilities, maybe something more on the Java side, if I can at least find a way to make some kind of external action listener that can get the signal to start the video when the button is pressed on MuseScore, do you think that may be possible?
In reply to Yeah, I just realized that, I by KoRnKloWn
I suspect that even that might not be currently possible, but you might be able to implement a short patch to hook something like that up for a future release. Have you looked around much at the musescore source? Adding a method to an existing object type appears to be relatively straightforward, if it's something already supported within the application.
In reply to I suspect that even that by Marc Sabatella
I'm really not sure, I haven't looked through the source, it's C or C++ right? I can try to look through it and see if I can make sence of it, but I'm very much a novice programmer, and mostly know Java, of course C and C++ are very similar to it. Besides, this will be a good learning experience for my self taught programming. I'll do whatever I can.
In reply to I'm really not sure, I by KoRnKloWn
OK, I managed to get the source code into eclipse correctly, however when looking through it, I'm afraid this is just too much over my head at the moment. As I learn more programming I'll continue to check in to see what I can do, and if someone else hasn't already done or started it. In the mean time let's try to convince the main developers to implement, if not a full video player, then at least the required objects for us to make a plugin. All we need is a way to track the BLUE cursor so we can get it's current play position, and input from the controls (play, pause, etc.). And I'll continue to make a simple Java video player that can eventually be used with this, unless of course someone else writes a better one sooner.
Sorry I don't have more formal training, I've never taken a class on programming (other than HTML/CSS, but that hardly counts), everything I've learned myself from a few For Dummies books and the Internet. I definitely have a long way to go.
I don't think a plugin is the right approach to this problem. There have been discussion about implementing Jack transport (http://musescore.org/en/node/11497) in MuseScore. It seems a more viable road. Xjadeo or VLC could be used as external video player and be synced with MuseScore.
Okay so maybe not a plugin, but what about implementing the video player in the play panel (shortcut key -F1, http://musescore.org/en/handbook/play-mode) and having it sync to the time display or slider bar?
I think the play panel would be the most obvious place to have a video window anyway. It saves on space.
What I am talking about is different then the blue cursor right?
In reply to Okay so maybe not a plugin, by Gabriel H
I think it does make sense to have the video player in the play panel, but that's just a location for where the video player can be displayed, unfortunately it doean't help us much. And about the blue cursor, all it is is a visual representation of what's being played, like the little tracker on a video or music player.
Also, I'm not sure Jack would work either, because as far as I know it doesn't necessarily have support for syncronizing play time, it just transports media from one app to another. Really, I don't think it would be too hard for the developers to just implement it as a feature, a full video player integrated into the program like what Sibelius has. I think it's just a matter of priorities, what they think is most important.
In reply to I think it does make sense to by KoRnKloWn
Getting the time from the score is not enough. Since you will want to move in the video and the score should move along too.
The first feature of Jack is indeed to pass audio or MIDI for one software to another but Jack Transport is exactly what you are looking for. A way to synchronize the time between several applications.
Jack transport is used for example between Ardour (a free digital audio workstation) and Blender (a free 3D animation software) to add music to the video. See for example http://www.youtube.com/watch?v=xvysT6Ak4y4
On the left Ardour, on the right Blender, and Jack doing the communication. If you move the cursor in Blender, it moves it in Ardour and the other way around. More info in this discussion
Enabling Jack transport features in MuseScore would make MuseScore compatible with ardour, Blender, XJadeo for the purpose of film scoring but for other purpose too !
Currently, Jack support in MuseScore should enable you to start the playback of a video and of the score at the same time (but no to stay synchronized)
Of course enabling jack transport in MuseScore doesn't mean a simple audio/video player embedded in MuseScore doesn't make sense.
In reply to Getting the time from the by [DELETED] 5
OK, I actually didn't know Jack had that function, thanks for the info, I'm sure that will come in handy! But about moving the score and video along, that is what I've been talking about, if they created plugin objects that could get a call from the main MuseScore play button being pressed, we could use that to start the video at the same time. But now that I see what you mean about Jack, I do think they could work pretty well with the transport feature. In fact we need several new plugin objects, you are actually extremely limited to what kind of plugins can even be made, all you can really do is alter the notes themselves, and text, but when it comes to lines, or dynamic markings like pianizzimo and forte, we are out of luck. I was thinking of trying to write a plugin to realize the dynamics of crescendos and diminuendos, but there's currently no way of having a plugin find the markings for them. It kinda sucks, they need to give us more freedom for plugin development.
In reply to OK, I actually didn't know by KoRnKloWn
I wrote half of the plugin framework :) and it's not easy at all to give more access to score element in the plugin framework. So it's done little by little and with precaution. In particular we don't want the plugins to slow MuseScore. As you may know, Musescore is open source software, you can also contribute to the plugin interface development if you want.
In reply to I wrote half of the plugin by [DELETED] 5
I see, I really wish I could contribute, I've already looked through the source and it's pretty much over my head, I'm a really novice programmer, I mostly know Java, and I've never taken classes (except for HTML/CSS). Hopefully I will learn enough to contribute, because I would love to contribute to several projects (especially MuseScore and LMMS).
I created a feature request in the issue tracker for either an integrated video player, or the plugin components required to do this through a plugin. Check it out and add your support for it so maybe they'll put it on their priority list! :]
http://musescore.org/en/node/11870
In reply to Created feature request in issue tracker by KoRnKloWn
Hi, just to mention I've been using jack transport to synchronize Ms , xjadeo and Ardour all together for some time, it works quite well. My issue is that when I open certain scores, Ms disconnects from jack (using it just for audio, no transport). So I have to go a lot to I/O tab to click ok and restore the connection. It's annoying. Anyone had that problem?
In reply to Well, jack transport works for me by pdro74
Could you describe your setup more in details? I'm pretty sure a lot of people will be interested to learn how you use MuseScore, Xjadeo and Ardour together ! Maybe a video?
In reply to Could you describe your setup by [DELETED] 5
I made a small video-tutorial about it, using Musescore, two synthesizers, xjadeo and Ardour. Must say Jack-Transport only does start-stop with Musescore, but I think the idea is there. It's also useful to record audio from MS with sounds not available inside MS (synthesizers). It will be ready tomorrow, more or less.
I forgot, the setup is on the video. Basically, TangoStudio (awesome music-distro) and MS thru jack.
Disregard, posted in error
Hi, I did a small video to show how to connect MS thru Jack to Xjadeo-video window, also to synthesizers and Ardour. As lasconic commented, Musescore only supports start-stop jack transport for the moment. It would be awesome to have full capabilities, like Ardour, Rosegarden, and so. The video is in spanish, but easy to follow. Feel free to comment,
http://www.youtube.com/watch?v=uUq4Hl2l2TY
Pedro
In reply to Video tutorial: Musescore+video window by pdro74
Totally awesome ! Thanks for this video! For those who can't speak spanish. The setup is as follow. Jack is the central part, it's a router for Audio, Midi and also a synchronizer for the transport (play/pause at the same time for every client for example).
MuseScore acts as a client for MIDI and Audio. The score is rendered to MIDI and sent via Jack to 2 instances of Yoshimi (a software synthesizer for linux only) to play with organ and strings. MuseScore's internal synthesizer generates also audio as usual but sends it to Jack. Yoshimi output is also sent to Jack again. The 3 audio streams (Organ & Strings from yoshimi, Piano from MuseScore) are redirected by Jack to Ardour (a DAW for Linux and MacOSX). Ardour is recording all these streams.
At the same time, a video is playing in Xjadeo. The playback of the video is started at the same time than the playback of MuseScore. As stated in a comment, pausing MuseScore would pause the video but seeking in the score would not seek in the video since the support of Jack Transport in MuseScore is incomplete. Any volunteer :) ?
(* btw, all the software is free as in free speech)
In reply to Totally awesome ! Thanks for by [DELETED] 5
and don't forget free beer :]
Hi can you please contact me via messenger if you have completed writing the video player for Musescore.
Thanks
In reply to Hi can you please contact me… by Emile Merten
It looks like this project has been abandonned. I myself recently tried to implement it. However, musescore's sound system messed with QtMultimedia, which caused the application to crash.
However, someone recently spoke in the dev forum about this: https://doc.qt.io/qt-5/qtqml-modules-cppplugins.html
Which might be worth a shot...
In reply to It looks like this project… by ecstrema
Thank you
Will check it out, I really hope Musescore or someone can get it working, it will really make scoring to film easier.