Support screenreaders other than NVDA
Reported version
3.3
Priority
P1 - High
Type
Functional
Frequency
Once
Severity
S3 - Major
Reproducibility
Always
Status
closed
Regression
No
Workaround
No
Project
Most of the MuseScore UI is accessible at this point, but the reading of the scoreview only works with NVDA. The reasons for this are not entirely clear as there are many variables at play here. But there are things we can do to improve the ability of other screenreaders to read the score. This is issue is intended as a general catchall for the most important issues. I intend to submit a PR addressing these, and at that point there will no doubt be individual issues specific to each screenreader and particular aspects. We can track those specific issues separately once the major blockers are addressed.
Fix version
3.5.0
Comments
See https://github.com/musescore/MuseScore/pull/5592
I make no claims this is perfect. There are plenty of cases where JAWS stops responding and I cannot say why. WIndows Narrator will need score info only with Narrator+Tab shortcut (but without this change it won't at all, so it's still an improvment). And I have no idea how this might work with VoiceOver or Orca, nor have I tested with Qt 5.12.5 (which uses UIA rather than MSAA). But it's a few steps in the right direction.
Wrote today a request to the orca-mailing-list ((https://mail.gnome.org/mailman/listinfo/orca-list) without subscribing the list, it's still waiting for reviewing it from the list moderator for approval) - hope it will be published next time.
At time I don't get to run Orca with Qt-applications (even by setting the enviroment with "export QT_ACCESSIBILITY=1" before; gtk-applications seems to work fine (using OpenSuse 15.0 and XFCE)).
Thanks for looking into the Linux issues. I've been poking around a bit and my understanding is that setting QT_ACCESSIBILITY should no longer be necessary (in fact for me it seems to disable normal keyboard interaction), but yes, somehow the AT-SPI system seems to not be working, so that right now MuseScore does nothing under Orca, it isn't even seen as a running application. Do let me know if you figure out how to get it working, then I will be able to test my code to see if it makes a difference in navigation fo the score. Meanwhile I haven't given up on that, but I'm a little more focused on JAWS at the moment.
Not sure, but perhaps also appropriate to this topic: a (maybe known) Live-CD-System (also installable on hard disc or usb stick) for visual impaired people is the "knoppix adriane project" (http://www.knopper.net/knoppix-adriane/index-en.html). Last time I tried to run it seemed to failed with my hardware configuration. But maybe interesting for other, I could imagine it's offers a good accessibility base and could be an good environment to check MuseScore with it.
I've used "regular" knoppix, but didn't know about this one - thanks for the pointer!
Meanwhile, I've tested my PR with Qt 5.12.5 and unfortunately JAWS does not read the score automatically (it does if you explicitly use JAWSKey+I to read the current item). I posted to the Qt forum to try to get help: https://forum.qt.io/topic/110464/qaccessibilityevent-not-being-seen-by-…
I suspect we're missing something really basic in terms of setting up the accessibility interface, but I'm not sure what. Overall, I get much better accessibility results with Qt 5.12.5 than 5.9.8 (which is what our official builds use) - screen readers can access more object properties, etc. But the event we raise that is supposed to trigger screenreaders to read the changed text as you navigate just isn't getting through reliably.
Posted new to the Orca mailing list, in case someone will join the discussion: https://mail.gnome.org/archives/orca-list/2020-January/msg00046.html.
Maybe I'll find time (and skills ;) to update my versions of the Qt to Orca distribution packages soon.
Following the discussion (https://mail.gnome.org/archives/orca-list/2020-January/msg00061.html) on the orca mailing list hope I'll find time soon to upgrade my OS and so Orca (hopefully up to version 3.35.x). Will then report again.
FWIW looks like my version of Orca is older than yours, but it's what is available for Debian "stretch", which is what my Linux system - actually a Linux container within ChromeOS (aka, "crostini"). ChromeOS will supposedly update to "buster" next month and then I can try again. Of course, I'll also keep looking to see if there is anything more I can be doing with what I have, but so far no combination of playing with the various options suggested thus far has made a difference. I also posted in the "crostini" subreddit: https://www.reddit.com/r/Crostini/comments/eldthn/chromevox_or_orca_sup…, and I'm getting some useful information there (like, even we sort out the Linux issues in general, I may still struggle under crostini because it uses Wayland rather than X.
Today I upgraded OpenSuse to the actual version, but the Orca version within is still 3.26.0 and doesn't seem to work with MuseScore (and no newer one as 3.35.x seems to be available as needed to check the actual screen reader features). As far I've read I've to wait until May 2020, when the next distribution version will be published - very annoying.
@Marc Sabatella (and maybe also @shoogle ;): Do you follow the dicussion at the orca list? With some replies I'm stumped whithout having any development skills, see for example: https://mail.gnome.org/archives/orca-list/2020-January/msg00070.html.
(if required: the easiest way to reply is in my experience to subscribe the list)
I'm checking in on it from time to time. I have submitted the form to subscribe and am awaiting verification, at which point I will post to the discussion.
FWIW, Qt Creator itself doesn't work with Orca either on my system, which to me is a sign that whatever is going on, it's not specific to MsueScore but does have to do with proper setup of AT-SPI, DBus, or whatever. Or that we need a more recent version of Orca. I gather there is is a more recent version in the "backports" for Debian stretch, so I will try to learn how that works and see if that helps.
I managed to update orca to 3.30 (current stable version as far as I can tell) but no change. Neither MuseScore nor QtCreator show up in the list of applications at all, and not surprisingly then, they don't read.
Good news, courtesy of @shoogle - setting the environment variable QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 before starting MuseScore allows orca to see MuseScore, at least for me and some other developers. Not sure if it would also have worked before updating orca the other day. But anyhow, now it reads menus and dialogs, but still not the score itself. My PR - which managed to get JAWS working, at least when using Qt 5.9.8 (which is also what I'm using on Linux) - doesn't fix this, unfortunately. But at least now I'm at a point where I can start debugging.
Yes, works by setting the environment variable in this way for me too. Meanwhile I've found a community package of Orca 3.34.1 and installed it, so I don't also know how it would work with older versions.
What I noticed concerning the inspector: when I click into/hover over some fields (or switching via tab), it reads beyond the actual value also the description (for example stacking order or leading space), for other it reads only the value, but not the corresponding description (minimum distance, offset, and other). Orca reads indeed the description, when I hover over it with the mouse.
Further cross-referencing to LibreOffice (not sure if it's a Qt-application too), there Orca reads also some toolbox elements too (not sure if it's helpful)
Just played around al little bit with the actual nightly (revision: 3997e1f) and Orca reads also the toolbox and palettes items - great :-).
Just noticed the latest discussion about this topic on the Orca mailing list (please don't forget if you reply to a comment also to add the Orca mailing list adress in the address field, so that people can follow the discussion - a mistake I made the first time by replying, but probably you're more clever than me).
As far as I understand it, you got it run with Orca. But (beside the delay) completely in the same way as with NVDA? For me it reads (using the mentioned developement build) the menu, toolbar, symbols of the palettes (better descriptions of the symbols?), most (but not all) inspector entries, the context menu... (also when I hover over these elements),
But it doesn't read the palette labels themselves, and diddly-squat of elements inside the score (the only element Orca read so far whith much luck was seldom "Pause" (="rest"). More often it reads similar like "MuseSore invalid".
For me it would be nice to have a development build of the AppImage using Qt 5.12.x or above (tried to install it myself, but failed), maybe to exclude, that it isn't a QT bug.
My PR as I have it doesn't get Orca doing anything it doesn't already do, but yes, the code I am working on that I discuss on the Orca and Qt mailing lists is getting there. In particular, it does indeed read the score as you navigate. I'm still working out kinks, and Orca support will probably end up requiring a script to be installed (eventually it could be incorporated into a future Orca version directly, but I hope to have the PR updated within the next week or so.
Thanks for the tip about needing to explicitly reply to the list. I think there are settings somewhere to control that, can't remember if they are user or admin settings though.
BTW, for me, palette labels do read, at least using builds of 3.4.
FWIW, here is a demo with Orca:
https://drive.google.com/file/d/1jKAJNCsxPShp7kwUCu_JKmT_eA9XJgg4/view
Updated PR https://github.com/musescore/MuseScore/pull/5627 (I messed up the commit history on the original).
This one should work on NVDA and with Orca using either Qt 5.9 or 5.12 (Orca requires a script I will provide). It works out of the box with JAWS using Qt 5.9 but will currently require a script with 5.12. We do want to move to 5.12, especially for accessibility purposes, as 5.9 crashes with many palette operations.
Update: in the PR comments, I have provided scripts for both Orca and for JAWS, and they work with my PR built with either Qt 5.9 or 5.12. NVDA continues to work out of the box, Narrator continues to not read changes unless you ask it to, and it's easily confused about where focus is. No idea about VoiceOver.
Maybe I miss something in the Orca settings? Using the release candidate of MS 3.4.0 it's still reading menu items, elements of the palettes, parts of the inspector adjustments, toolbar and so on, but still nothing from the score itself, nor the labels of the palette items themselves, and nothing inside the score, if I lick on an element or use shortcuts (for example by using "reunion" it reads "reunion ungültig" ("ungültig"="invalid"), but then there's "silence").
Probably not important for development, but for documentation to setting up MuseScore with Orca.
That PR has not yet been merged, so ypou'd need to use the artifact from its build, nitz 3.4 RC or, in fact, 3.4 itself
Indeed, you need my PR to see the changes. But - palettes read just fine for me even without my PR. Can you explain how you are accessing them? I typically press Shift+Tab to move focus to the palettes, then tab and/or arrow keys from there. Everything reads as expected for me - the names of each palette, the name of each cell within the palette. Works just as well if I use the palette search, or use F9 to toggle the palette on and off. The only that does not read are the palette labels within the search results. Maybe that is what you mean?
What I meant was using the mouse/touchpad. It works with MS 3.4.1 much better, at least, if click on a palette section (it doesn't work for me with RC 3.4.0 and versions before). But it doesn't work for me when I hover over the palette sections (or only seldom random before mouse click) - in difference to the menu or toolbar items Orca doesn't read them.
But indeed (and i didn't notice it before), it works/worked with tab/shift+tab. (maybe a little bit OT: I think it would need a good documentation to navigate inside the palettes view (tab/shift tab to change to the palette section, arrow right to access the symbols inside the palette, ...). Above: Maybe it would be useful (and/documented) to access the palette via shortcut once it is open (without pressing again and again tab or disable and enable again the palettes view again via "F9")).
Maybe also useful: to access the score with a shortcut without enable the note input mode before.
Also with MS 3.4.1: For me Orca doesn't read still all elements for some adjustments inside the inspector.
And If I understand it correct: For reading the score itself it's necessary that your PR is merged (don't using self builds)? I extracted your zip file, but it doesn't change anything (and being a little bit confused in your commit about: anatoly-os added "modified the milestone: MuseScore 3.4.1" - it wasn't, was it?)
Indeed, the screen reader does not read on hover, the tooltips are not part of the accessibility info currently. I believe the PR https://github.com/musescore/MuseScore/pull/5474 may change that. But it's important to realize, the most common use of screen readers is in situations where the user is not using a mouse at all, so everything is normally optimized for keyboard usage.
So, as for how to navigate palettes using the keyboard - the same PR mentioned above changes this to be more standard, and I'm hopeful both PR's will be merged soon and make it into an upcoming release together. As it is, I would say, trial and error should give you a feel for how palette navigation works currently, but don't get too attached to it because hopefully it will change soon. The one thing I can tell you that's most relevant is that "Down" within a palette will literally move the cursor down, but that's not really the expected semantic, it should move through the palette cells one at a time. After all, a blind user won't realize the palette is laid out in a grid, the navigation needs to be linear. So that's one thing that will be changing with that PR. Other subtle details too.
There is documentation on how to use the accessibility features, just see the Accessibility section of the Handbook. But it's deliberately vague about the palette navigation because we know it may be changing soon. Note also there is a shortcut Ctrl+F9 to transfer control to the palette search box directly.
Which elements are not being read in the Inspector? The one I know about is "Minimum distance", it reads the value but not the label. I will be submitting a separate PR to add that info.
The ZIP I provide is the script that you need in addition to a build using my PR. If you have a build environment set up on your machine, you can build my PR yourself directly. Unfortunately I don't know of an easy way to make a test build available for Linux, too many dependencies to be sure that something built on my machine would work on yours.
FWIW concerning "hovering with mouse onto the labels": it affects after activating "speak objects under mouse" in Orca settings (orca -s). It works fine for the menu or toolbar elements, but not for palette labels.
concerning ctrl+F9: it doesn't work for me (it's defined in preferences->shortcut and even if I clear/delete it, it's not possible to define a new shortcut with ctrl+F9 - but maybe this is another topic (IIRC there was a bug report of ericfontainejazz some time before and is a QT bug)).
with the distribution package 3.3.4 and the AppImage 3.4.1 there are more sections inside the inspector, which won't be read correct. It works, if i want to change the value of the stacking order or leading space (it reads the actual value and beyond the section), but it doesn't work for me with other elements (for example if want to change the value of an offset, it doesn't read "offset" or the x-/y-axis, the description by entering a value if "fix to line" is activated and other value fields (tuning, velocitiy and so on). I didn't check all.
Thanks, I can confirm those specific fields don't read for me either, with Orca. Interestingly, some do on other screen readers, apparently because while they lack accessible names, they have "buddies" set up, or vice versa. Would you mind creating a new issue for the Inspector and listing the fields that don't read for you?
In reply to Thanks, I can confirm those… by Marc Sabatella
started here: #300287: Missing items Orca screenreader doesn't read in the inspector
Fixed in branch master, commit 9dc93c56fb
_fix #299387: support for more screen readers + collect_artifacts
MuseScore uses mostly standard Qt widgets and is thus accessible.
The SocreView widget is custom, however,
and we implement accessibility for it ourselves.
The support has long worked with NVDA only.
There are a number of reasons for this, and this PR addresses them.
It also improves the reading by NVDA.
The changes allow Orca and JAWS to work,
although we may also need to provide scripts for them.
The changes are:
The basic model hasn't changed - we set a value on the widget,
and send a value changed event.
This is not necessary the "correct" solution, but it works best.
I also included some code to facilitate changing to other approaches.
Different versins of Qt, different OS's, different screen readers
may require customization here.
But the code as I have enabled works well enough at the moment._
Fixed in branch master, commit 6bf7edafce
_Merge pull request #5627 from MarcSabatella/299387-new-screenreader-support
fix #299387: support for more screen readers_
"Fix version", @Jojo-Schmitz? :-)
;-)
Automatically closed -- issue fixed for 2 weeks with no activity.