Moving line anchor unwantedly moves endpoints of a line

As you can see in the gif, I'm trying to make a line point from a text box to a specific note. Specifically when I need to place the line over a barline in a vertical/diagonal fashion, it's extremely difficult and time-consuming to make it line up just right without moving the anchor point (which in turn moves other control points of the line).

I understand there might be a reason to have the endpoints move when the anchor moves in certain situations, but could this be perhaps disabled when "automatic placement" is ticked off? (I also suspect this might be because MuseScore doesn't like two anchors on the same note?)

Or maybe I'm missing something? This isn't a super big deal, but certainly something that could be addressed.

Thanks!