MSCX cat: a better way to join scores?
The albums feature in MuseScore 2.X has a facility to join multiple scores to create a single combined file. However, the feature is fussy about what it will accept in terms of input, and information is sometimes lost in the joining process. To address (or avoid) these concerns, MuseScore 3’s Albums feature will no longer have the ability to join scores. This means users looking to join scores will have to resort to copying and pasting content between files, which, besides being tedious and slow, also results in information being lost, such as tempo markings, breaks and spacers.
We need a better way to join scores.
Justification
OpenScore brings three use cases readily to mind, and I’m sure there are many other uses besides these:
- Joining scores transcribed by different people.
- Combining sections of a large score that were edited separately for performance reasons.
- Adding covers to scores.
Ideally it should be possible to join scores from the command line, as this would facilitate automation of one of the most labour-intensive aspects of OpenScore.
Current method
MuseScore’s existing join-scores feature attempts to fully read each score into MuseScore’s data structures (C++ classes, etc.), combine the data, and then dump it into a new file. The problem with this approach is that by interpreting the MSCX code the feature becomes vulnerable to bugs pretty much anywhere in libmscore.
Proposed method - “MSCX cat”
Perhaps a more reliable way to join scores would not attempt to interpret the data in the original scores, but simply take the raw MSCX code from the separate files and dump it into one file à la Unix cat (hence “MSCX cat”).
“Header” taken from first score:
<?xml version="1.0" encoding="UTF-8"?>
<museScore version="2.06">
<programVersion>2.1.0</programVersion>
<programRevision>871c8ce</programRevision>
<Score>
...
<Part>
...
</Part>
“Body” taken from all scores:
<Staff id="1">
... <-- dump staff 1 data from all scores here
</Staff>
...
<Staff id="N">
... <-- dump staff N data from all scores here
</Staff>
“Tail” inserted after final score:
</Score>
</museScore>
It is not necessary to interpret the MSCX into MuseScore’s internal structures, though a small amount of book-keeping may be required. As far as I can see, all that has to happen is beam numbers and ID numbers (for things like lines and measures) would need to be updated in later scores so that the numbers are unique.
Optional extras
A more ambitious implementation could, for example:
- Make room for instruments that only appear in later scores (removing the requirement that all instruments appear in all files)
- If scores have different stave space sizes (sp) the width and height of images and frames (vertical frames, horizontal frames, text frames) could be scaled so that they remain the same size in the joined score.
- Add text styles from later scores to the combined score's header. (This is a simple matter of looking for <TextStyle>...</TextStyle> and comparing names to see if a particular style was already present in the first file.)
- Connect ties, slurs and lines which carry over into the next score. (This would be an optional feature requiring one measure of overlap between scores. The algorithm would look for ties/slurs/lines that start in the penultimate measure of the previous score and attempt to connect them to ties/slurs/line that end in the 2nd measure of the next score.)
- Perform variable substitution (e.g.
$transcriber = JoeUser,$image = cover.png), with variables defined as metatags in Score Properties or a separate YAML or JSON file. This would automate the process of applying a cover to OpenScore files.
Implementation
This would be fairly easy to do in Python, but that would restricts its use to programmers. I would argue that this would be a valuable feature to include in MuseScore, but maybe as a plugin rather than in the main C++ code.
Questions/discussion
- Can plugins write to external files?
- Opinions on implementing in Python vs plugin vs in MuseScore’s C++ code.
- Would it be better just to fix the existing Join Scores feature instead and reintroduce it in MuseScore 3?
Comments
IMO there are two uses for using the album feature currently.
Creating an Album
A set of standalone pieces, not necessarily having the same set of instruments. As a transcriber from CD albums it would allow me to match up a single MuseScore distribution 'file'. Similarly bundling a suite would use this feature.
In my mind this (implementation wise) works on the mscz level and keeps the separate pieces within the Album as separate files, but merely displays one after the other (yes, no way to remove the page break between two scores in an album). The Album-file itself is "just" some metadata, perhaps with an additional coverpage/score.
Currently this usage is only possible by adding all instruments in all files and play with hide empty staves as well as forcing some staves to show via hidden notes. The current Album feature isn't quite up to this task imo, despite its name.
Joining scores
AFAIK this is actually the reason most people use the Album feature nowadays. Mostly used from performance or section based workflows where it was always the intention to end up with a single score anyway.
Joining scores should (have the option?) to automatically insert section breaks in between the joined scores. Each section is likely no longer linked to its origin file.
One can also keep debating whether each (musescore-)section should allow changing the instrument set; as in actual scores this happens often enough.
In reply to Yes, through the Qt file… by jeetee
At the moment I'm only interested in joining scores, so I don't really want to discuss the Albums feature in general here, but I will say this: I believe Werner only added the Albums feature and Join Scores facility as a hack to create the Open Goldberg and Open WTC scores. I think he wanted to edit the arias/fugues separately and then combine them into one score at the end for printing and playback. The Join Scores feature has proven buggy so it has been axed, but Albums are here to stay. I guess the idea is that Albums will offer a quick way to open lots of files at once, which the user is then free to play or print as they wish. With the proper handling there would be no need to actually join the scores together (or at least that's the idea anyway - I think I have shown that there are other reasons for wanting to join scores, beyond just for playback and printing).
In reply to I missed your reasoning for… by shoogle
Ah yes, forgetting to include one of the main points of the conversation: a classical side effect of posting past midnight :)
My preference/arguments for incorporating into MuseScore (and going C++) are the following:
As soon as you have to do anything more than basic XML concatenation you'd have to start interpreting the scores (which instruments, matching positions within a system). A joining scores feature comes with a lot of (future) options that require you to understand the score structure and as far as interpreting a MuseScore file goes, leveraging MuseScore itself seems… obvious :)
It's a feature that already is subject to feature requests to enhance its usage, implementing it outside of MuseScore would lead (in the long run) to a lot of duplicate work IMO.
Why not as a plugin then?
The most silly reason: it isn't one now.
Anything a plugin can do is dictated by what the plugin framework offers; I know it has been underdeveloped and some core things are changing there as well, but you'd be (currently) developing against a less stable API. I'm also not sure that this is the kind of functionality you'd want (parts of it) exposed towards the plugin framework (unless it uses a full-featured copy-and-paste, which still has to make it into the core code anyhow).
And the same basic reason again (also towards TPTB): it is a function currently included in MS2 and it is being used. MS3 is supposed to be the newest and better and whichever superlative you can come up with sales-wise (at least until MS4 comes into picture :p) and sometimes scrapping functionality (or moving it to plugins) makes sense. Currently I'm not convinced that for a relatively complex feature (implementation wise) that offers rather basic functionality this would be a good decision.
Yes, MS3 handles bigger scores better, but that doesn't mean people suddenly won't work on a section/part of a suite score while composing (even if just for easier navigation) and then wish to join / copy-paste those in the end.
In reply to Ah yes, forgetting to… by jeetee
> As soon as you have to do anything more than basic XML concatenation you'd have to start interpreting the scores (which instruments, matching positions within a system).
Interpreting the score is exactly what I am trying to avoid. A basic concatenation could be done with a simple Bash script and Unix tools like
sed, etc just looking for the regular expression'<Staff id="[0-9]*">'. Even my proposed "optional extras" only require looking for matching XML tags, which can be done using a generic XML interpreter rather than a specific MuseScore interpreter.> It's a feature that already is subject to feature requests to enhance its usage, implementing it outside of MuseScore would lead (in the long run) to a lot of duplicate work IMO.
None of the bugs/feature requests for albums or join scores require more than a basic XML interpretation of the file.
> implementing it outside of MuseScore would lead (in the long run) to a lot of duplicate work IMO.
We only care about a very limited number of XML tags (e.g.
'<Staff>','<TextStyle>','<... id="X">', etc) so the amount of duplication is small. In any case, my proposed solution deliberately avoids using any of MuseScore's existing score manipulation functions, so there would be the same duplication of work whether implemented within MuseScore's code or outside of it.> Anything a plugin can do is dictated by what the plugin framework offers; I know it has been underdeveloped and [...] you'd be (currently) developing against a less stable API.
The plugin API provides functions for manipulating score objects (i.e. files that have already been interpreted by MuseScore). By avoiding interpretation of the files I avoid the API altogether, so my plugin simply becomes a QML script (which is essentially just JavaScript). I believe any additional features that may be required (e.g. zipping and unzipping MSCZ) can be loaded in as JavaScript libraries.
> And the same basic reason again (also towards TPTB): it is a function currently included in MS2 and it is being used...
I agree that MuseScore 3 users will need a way to join scores.
I am not a computer expert at all and the details of your post escape me. But: If you join two scores without redoing the layout doesn't this mean that the beginning of each section will be on a new page regardless of the end of the previous section? So if you split up some piece among several typesetters you'd have to know where the page breaks are going to be and I don't think that can be predicted easily or at all.
Here is how I do (chamber music) works with movements: I set the score for each movement separately . I proofread diligently. I create parts in each file. I export the parts as separate mscx. files. I use the album function to join the files for each instrument separately. Then I arrange each part for page turns, collision avoidance and general good appearance. I can do the same for the scores at any point after proof reading. For publication on IMSLP I generate Pdfs for the score or each movement separately after setting the page numbers consecutively and allowing pages for title page and foreword (if existing). Those are only joined as Pdfs; every movement begins on a new page. For the parts I like to have one mscx. file for the all movements because sometimes it is quite convenient to have movements begin somewhere in the middle of a page. The loss of the album feature would be a bit of a blow to me. On the other hand: The way I do it is dictated by the slow work in large files. Typesetting chamber music is usually still quite fast for a one movement file but one file per (25 min) work is terribly slow. This problem is going to go a way with version 3 (so is the promise) so maybe things will work out without the album feature by just using one file for the whole piece.
In reply to I am not a computer expert… by azumbrunn
> If you join two scores without redoing the layout doesn't this mean that the beginning of each section will be on a new page regardless of the end of the previous section?
No, the next section can (and will) be on the same page as the previous section, providing there is still room on the page (although I will probably include an option to insert page breaks and system breaks between scores if desired). I'm not sure what you mean by "without redoing the layout". My proposed method operates on the MSCX file outside of MuseScore, so the layout will inevitably be recalculated the next time you load it in MuseScore (which is exactly what you would want to happen).
> If you split up some piece among several typesetters you'd have to know where the page breaks are going to be
I ask transcribers to add line breaks and page breaks to match the original, and then to reduce the stretch and stave space setting as necessary to make the music fit, so as a matter of fact I usually do know where the page breaks will be. I only relax the requirement to match line breaks and page breaks on scores where it would be impractical to match the original (e.g. if the original uses a page size very different to A4), but this has no real impact when it come to joining the scores together. It's true that some symbols might move around a bit if the line breaks change, but I always give the joined score to an engraver as a final step to correct for that sort of thing anyway.
<< copying and pasting content results in information being lost, such as tempo markings, breaks and spacers. >>
That's true but I strongly think that this copy behavior deserves a feature request of its own, and that would already helps you to join scores in several cases.
Often I want to copy a part of a score in the score itself or in another score, and I want to copy the FULL information, including incomplete measures, breaks, repeats, EVERYTHING.
Currently it isn't possible, only part of the content is copied pasted and lot of manual work has to be done to complete the copy.
While the current copy/paste makes sense and must be kept as it is, we really need a option "full copy" in addition to it.
In reply to << copying and pasting… by frfancha
> I strongly think that this copy behavior deserves a feature request of its own
I agree, feel free to make one ;)
But the point is any copy-paste feature within MuseScore's code inevitably risks missing information, whereas my solution of taking MSCX code from a bunch of files and dumping it into one large file will not miss anything.
In reply to > I strongly think that this… by shoogle
<< copy-paste feature within MuseScore's code inevitably risks missing information >>
Hum, I am confident that the MuseScore developers (I know, I know, can be me) are good enough to avoid that mistake.
In reply to << copy-paste feature within… by frfancha
You really shouldn't be, otherwise the existing copy-paste feature would work. ;)
But that's a bit unfair. What I was saying is that information lost is extremely difficult to avoid within MuseScore; something is bound to get overlooked somewhere no matter how good the developers are.
In reply to You really shouldn't be,… by shoogle
<< the existing copy-paste feature would work >>
I think that we perfectly agree on the joke behind that, but just to avoid any misunderstanding (of possible new readers e.g.), the current copy-paste feature does work well.
With current copy-paste: copy three 4/4 measures and paste in a 3/4 sections: the pasted notes fill four 3/4 measures.
This is what is wanted (at least in most cases).
With the new requested feature: copy three 4/4 measures and paste in 3/4 section: the three 4/4 measures appear as such where they are pasted, introducing 2 time signature changes: one just before them to go from 3/4 to 4/4, and one just after them to go back to 3/4.
In reply to << the existing copy-paste… by frfancha
When I copy more than one line of music I usually want to keep the time signature for the copy point and have any other system info (Tempo, key changes...) copied with it. This is for the purpose of joining scores rather than rolling the dice that I violated a rule to make a good album. Copying and pasting is faster and can be undone. I find this method easier. I'm totally in favor of an advanced copy feature.
It would be really nice if the join album feature were smarter. When scores are joined, use some sort of algorithm to line up the instruments in a reasonable fashion and enable the hide empty staves feature to help make only the desired instruments visible in each score added to the album.
In reply to > I strongly think that this… by shoogle
<< I agree, feel free to make one ;) >>
Done.
https://musescore.org/en/node/264845
I'm following this with interest but not much specific input. I will observe I have thought abut the idea of "unlinked parts" - parts with no specific links to the score that allow edits to be propagated both ways, but are nonetheless stored within the same MSCZ file. Seems like this could be useful for cases where you need a part to differ from the score in some significant way we can't otherwise handle, but don't want to complete give up the ability to have the part be, well, a "part" of the main score. Seems this same mechanism could come into play.