In Windows Explorer, files with special characters in filepath don't open when double-clicked

Came up again in #290386: something is wrong

and again in https://musescore.org/en/node/290423

This problem is actually quite common and I have come across it in many different applications. Some managed to do some change in programming to support this with the use of Unicode support. An example is IrfanView.

Maybe someone could ask the author of IrfanView what changes he had made to the source code in order to support non-ANSI string in filename path.

This seems a plain MuseScore 3 issue, MuseScore 2 (I checked with 2.3.2) has no such problem. So it doesn't seem to be a plain Windows issue either.
Another working example is 7Zip.

Might be an issue with Qt (5.4 for MuseScore 2, 5.9/5.12 for MuseScore 3) on Windows?
Or something about the installer, registering file types differently?

I confirm that v. 2.3.0 works for me too.

All the v. 3.x (32 bit and 64 bit) all have this problem for me.

Came up again in https://musescore.org/en/node/291835

And again in https://musescore.org/en/node/293566

Hah! It is an MSVC issue, a MinGW build works!

Is that related to the compiler switches /execution-charset:utf-8 /source-charset:utf-8 (which could get shortened to just /utf-8 BTW)?
I found https://stackoverflow.com/questions/30829364/open-utf8-encoded-filename… which claims: On Windows, you MUST use 8bit ANSI (and it must match the user's locale) or UTF16 for filenames, there is no other option available

What actually is the difference between opening a score via File > Open and when passed as an argument into main()?
Maybe we just need to convert the commandline filenames into the same format that File > Open uses?

Came up again in #293843: CJK File names

Came up again in #296784: Cannot open file with filename written in Chinese characters

Came up again in https://musescore.org/en/node/283479

If this is an UTF-8/16 issue, could perhaps something like the patch attached below help? This is just a random guess but if someone on Windows could check this it would be great.

I'll check.
Edit: there is no QString::toUtf16() and the conversion from QByteArray to const short* is ambigous in that QString::fromUtf16()

I just spent a few hours on this, and I now have a working fix for this bug. (Actually, the bug — or if you prefer, design flaw — is in Qt itself, and what I've come up with is a workaround at the application level. I can share the gory details if anyone is interested.)

My fix requires platform-specific code in MuseScoreApplication::CommandLineParseResult() (inside an #ifdef Q_OS_WIN block). Is this acceptable? If so, I'll create a PR.

Sure, gory details and PR please

The gory details:

The problem is that the command-line arguments are passed to Qt as char *. On the Mac and Linux platforms, that's fine, because those platforms use UTF-8 everywhere for char *. Under Windows (whose design predates UTF-8), char * means Windows 8-bit “ANSI” in whatever the local code page is set to. Because of this, any characters that don't fit in the local code page are clobbered by the CRT before they even get to main().

The de facto way to get around this problem under Windows is to define an alternative entry point, wmain(), that takes the command-line arguments as wchar_t * strings that are encoded as UTF-16. I made this change to MuseScore, which worked as far as the entry point, but of course wouldn't build because the QApplication constructor wants char *.

OK, fine. So I put in calls to WideCharToMultiByte() to convert the UTF-16 wchar_t * strings back into UTF-8 char * strings. The application now built and ran, but failed at runtime because Qt was still interpreting the char * strings as Windows ANSI. Garbage in, garbage out.

Seemingly at a dead end, I went to see where the command-line arguments were actually being accessed by the app. The answer was just one place: MuseScoreApplication::parseCommandLineArguments(). So I put a Windows-specific shim right inside that function that queries the OS for the command-line arguments instead of asking Qt for them (since Qt's copies are already corrupted at this point) and then converts them to a QStringList before sending them off to the parser.

With that change, I was finally able to open scores with Chinese filenames on the command line.

Upon further reflection, though, I'm not happy with this workaround. What I'd really like to do is revisit the first approach and see if there's any way I can coax Qt into interpreting the char * strings as UTF-8 under Windows without having to touch the Qt source code.

So I'll play around with it some more and hold off on the PR at the moment. Once I either figure it out or give up I'll go ahead and create the PR, with either the current workaround or the hypothetical better one.

But also note that this is not a Windows nor a Qt issue, but an MSVC one, as a MinGW build does not have that issue. So is got to have something to do how MSVC handles that command line

The reason it works under MinGW is that MinGW is kind enough to transparently wrap the strings in a tidy UTF-16-to-UTF-8 conversion layer in order to make things more Unix-like, thereby making Qt happier by giving it Unicode strings in the only format that it correctly accepts.

MSVC also does a transparent conversion, but it's from UTF-16 to ANSI, which causes data loss. However, in MSVC's defense, this is by design and has been well-documented for decades. (Remember, this all predates the existence of UTF-8.) The designers of Qt, which came much later, should have taken this into account by being able to accept UTF-16 strings (or even UTF-8 strings) under Windows.

Whether you consider this situation Windows' fault, MSVC's fault, or Qt's fault depends on how you look at it. The way I see it, it's clearly Qt's fault.

Makes sense.

Figured out the better fix! PR #5694.

Fixed in branch master, commit fb63d3e128

_Fix #287955: In Windows Explorer, files with special characters in filepath don't open when double-clicked

Fixed a problem that caused MuseScore to fail to open files that contained non-ASCII Unicode characters under Windows._

Fixed in branch master, commit 89cb85e7f6

_Merge pull request #5694 from Spire42/287955-Windows-Unicode-command-line-arguments

Fix #287955: In Windows Explorer, files with special characters in filepath don't open when double-clicked_

Automatically closed -- issue fixed for 2 weeks with no activity.