Change of Unicode codepoints for music fonts

• Aug 11, 2012 - 22:56

Here's something that came up in #17739: [MusicXML export] Lyrics do not handle flat and sharp signs but deserves its own discussion here.

I gather Unicode does define a few codepoints for music-related symbols. Well, three at least: flat, natural, and sharp (U+266d,U+266e,U+266f) are defined, and maybe a few others. And FreeSerifMscore will use these instead of the old Mscore1 codepoints. I just want to discuss the implications of this are.

If we are changing the Unicode codepoints for flat and sharp in FreeSerifMscore and the mapping for the text symbols palette, I guess I need to do this for MuseJazz too, yes? And then also update the chord style files that use this font? Not sure if I'd need to keep the old codepoints to avoid breaking older files that have these symbols embedded in text.

And I'd want to know the full list of moved symbols.

Comments

• Aug 12, 2012 - 10:16

Let's describe the current state in the nightly 6cc34f47d8

  • Indeed, Unicode defines several music elements. It defines some of them from U+2669 to U+266F (♩♪♫♬♭♮♯) in the "Miscellaneous Symbols" block. A lot of fonts support them, you might even see them in your browser. It defines also a lot of them in a dedicated block named "Musical Symbols". This is the block displayed in the F2 palette. The FreeSerifMscore has glyphs for this block and it's the font used in the F2 palette and by default for text (including tempo text, staff text, lyrics...).
  • Coda sign, DS, and the other symbolic markers use MScore font like they used to and not MScore1
  • Dynamics have two styles one with MScore1 and one with FreeSerifMscore, not sure why. The FreeSerifMscore one seems to be the default one but doesn't give good result. The palette dynamics are too thin.
  • Mscore1 was used to parse instrument name description and insert flat and sharp in instrument name. Now it's no more used but instrument name use a b instead of a ♭. I think that should be changed. Let's ask Werner for a confirmation. If it's the case, we should change the default chord description file too and use only one font by default instead of the "symbol" tag with associated font.
  • Except the glitch with dynamics, my understanding is that MScore1 remains for backward compatibility only. If it's the case, I wonder if we shouldn't match and replace characters to their equivalent for old scores instead of keeping MScore1 and then allow users to create 2.0 scores with this font.

Regarding MuseJazz, I guess it would indeed be better if it becomes more unicode. The remaining problem being compatibility. To handle it, I see 3 approaches.

  1. The character replacement approach describes in the last point above
  2. Duplicate the characters and then keep the old codepoint has you suggested
  3. Keep the font as it, and create MuseJazz2. But we end up in the same situation than we are right now with MScore1 and FreeSerifMscore

Any thoughts welcome. Let's wait for Werner for some more light and a decision.

Reply

• Aug 12, 2012 - 16:03

Regarding MuseJazz, my preference is not introduce a new font unless necessary. So the main question for me would be, to what extent would a score saved using MuseJazz flat signs et al still work if the codepoints were moved.

Chord symbols should be no problem because nothing codepoint-specific is saved in a score - only harmony elements. This can change with 2.0, I guess - in some cases, chord description info is written to the score - but existing 1.x scores shouldn't care if codepoints moved, as far as chords are concerned. Text elements - including lyrics and instrument names - are saved directly as UTF-8, however. So they wouldn't work.

I'm fine with either doing an automatic replacement on import or duplicating codepoints. But if character replacement is going to happen anyhow, then MuseJazz might as well just piggyback on that. FWIW, there are only a small handful of "generic" musical symbols actually implemented in MuseJazz. flat, sharp, natural, coda, segno. I didn't do the note glyphs (but should have and weill for 2.0!). There are other MuseJazz glyphs that are very specific to chord symbols, are not accessible via F2, and thus not likely to be an issue. I could move them to the musical symbols area if need, but since they are only likely to appear in chord symbols, no special replacement should be needed.

Reply

• Aug 13, 2012 - 09:30

The use of the Musical Symbols in Unicode was long time not possible in MuseScore due to bugs in the Qt implementation. The Musical Symbols are special as there codepoints are beyond the 16 bit range and must be composed out of two 16 bit codes.
Glyphs from the MScore (lilypond emmentaler) font cannot be easily mixed with glyphs from other fonts as the font metrics do not fit. The font is simply not designed for this purpose. The MScore1 font was an attempt to use some glyphs from the MScore font in normal text context with modified font metrics.

In the meantime the Qt bugs are fixed and it should be possible to use a "real" unicode font. The best solution imho would be to drop the MScore1 font completly as it has non standard code points and use a unicode encoded font for all text.
I tried this for tempotext with the GNU Free fonts. For my taste in FreeSerif the size of the note glyphs are too small and the vertical position not right. I modified some glyphs and the results can be seen in FreeSerifMScore.

Using unicode fonts has several advantages. First copy/paste of text between applications is possible. Second Text containing musical symbols can be implemented as a simple sequence of characters inside MuseScore. The handling of text using different fonts is much more complicated.

Reply

• Aug 13, 2012 - 14:44

Thanks for the explanations. Knowing the background behind the Mscore and Mscore1 fonts and their differing metrics helps a lot, and that much finally makes sense to me now.

And yes, I would much rather text be a simgle font where possible. I've run into all sorts of complications from my failure to include the note symbols in MuseJazz. Tempo texts now must, switch fonts, making resizing them difficult, etc.

So, are you thinking an automatic glyph replacement will be possible, so text saved on 1.X that uses the old codepoonts will still display properly (and then be saved using the new codepoints)? If so then I would like to move the glyphs in MuseJazz. I'd still want a list of what was moved, and an opinion on whether I need to move the glyphs that are unique to MuseJazz, like the "ma" glyph (hmm, maybe that should actually a ligature), some superscripted symbols, etc.

Also, will it be possible for the text symbols palette to somehow know if the font being used already supports the glyphs well enough to not require a switch to another font? In 1.X at least, even when inserting musical glyphs that are present in MuseJazz, the text symbols palette always inserts them as Mscore1, and I have to change them by hand. If there is any sort of flag I could set within MuseJazz to let MuseScore know it can use the MuseJazz glyphs, that would be most helpful.

Reply

• Aug 13, 2012 - 15:15

The text symbol palette is designed to handle only the FreeSansMscore font currently. There is nothing like a "current" font.
I'm checking if it's viable to do a match and replace in the text from MuseScore 1.2 score. I limit myself to the first row of 1.2 symbol palette (natural, sharp, flat, etc...) for the moment.

Reply

• Aug 13, 2012 - 15:49

Hmm, well somehow it works when inserting non-musical characters in 1.X. If I am typing in MuseJazz and hit one of the European characters, it is inserted directly using MuseJazz. Only the musical characters switch to Mscore1. So there must be something different about how the musical symbols are treated, amd *some* notion of a current font - maybe in the text editing code as opposed to the text symbol palette code?

In 2.0, it remains the case that inserting "ordinary" characters from the text symbols palette while in MuseJazz works fine - the characters are inserted as MuseJazz. It actually *appears* the same may be true of the musical symbols, but I can't be sure as there are no MuseJazz glyphs for most symbols so I just get a box. But the box claims to be AmuseJazz, leading me to believe if I provided glyphs, they'd work. Suggesting that the ext aymbols palette isn't actually going out of its way to use any particular font and is just trusting the font to have the necessary characters. Not sure that's a great idea if so. Seems the text editing code could somehow check to see if a glyph exists in the current font, at least, and if not, then switch to FreesansMscore or whatever. That way a font that wants to support music symbols - like MuseJazz - could implement the glyphs it really cares about, and expect the rest to default to something that works, just as is the case in 1.X.

BTW, the new text symbols palette is cool, but the music symbols are *way* too small.

Reply

• Aug 13, 2012 - 17:16

Nice start!

BTW, when I referred to the new palette symbols being too small, I didn't mean the characters inserted - I meant the symbols that appear on the palette icons themselves. They aren't just slightly smaller; they are so tiny and I can barely make out what button I am pressing. But seeing the results of the replacement, I do wonder about the slight size difference of the glyphs themselves. Seems this would be an issue particularly for coda and segno. Are the glyphs really different, is it different metrics, or what? Will the various templates and styles need to be updated to produce similar results? Any way the replacement mapping could fix this automatically for older scores, at least in the case of standalone text like the coda and segno?

Reply

• Sep 15, 2012 - 22:14

I'm trying to make my changes, but am not really understanding the substitution. Moving the accidentals is straightforward enough. But the rest of the symbols are apparently being turned into two-character sequences starting with 0xd834, and I'm not seeing anything in any of those slots in FreeSerifMscore. So I'm not quite understanding what I should do with those. I'd like to include characters for quarter note etc so tempo markings will work properly.

Reply

• Sep 18, 2012 - 00:32

Most Unicode points for music symbols are outside the so called BMP (Basic Multilingual Plane, i.e. the first 65536 positions), so they exceed 16 bits.

In a 16-bit coding (which is what Qt still uses, if I am not mistaken), those points are expressed with a pair of 16-bit chars (coded in a particular way), normally called "surrogates". I believe those two-char sequences you see are surrogate pairs.

Also, not all OSes give 'easy' access to those extra-BMP code points, via built-in tools; for instance, Windows Char Map do not show them in any font, even if they ARE there in the font. A specific font utility might be needed to check if a given font has glyphs for them or not. I have often found quite useful Babel Map .

It might be of interest to someone that a new Windows version of FontForge has been released a few weeks ago (after years!!) and should be available there: //sourceforge.net/projects/fontforge/ .

HTH,

M.

Reply

• Sep 18, 2012 - 01:46

Ah, that makes some sense about the >16 bit sequences.

I actually installed what I think is a different "new" version of FontForge for Windows a couple of months ago, but it seems to crash on me when I try to do anything much with it. Looking forward to seeng if I can get this newer one to work.

Reply

• Sep 22, 2012 - 22:14

Here is the pull request corresponding to this: https://github.com/musescore/MuseScore/pull/91. The flat, sharp, and natural glyphs have been copied to their Unicode locations, as have the coda and segno, and I added new glyphs for whole note through sixteenth note and dot at their Unicode locations.

As noted in the comments, I somehow messed up and included two unrelated commits by heuchi as part of my branch.

Reply

Do you still have an unanswered question? Please log in first to post your question.