Fluctuating/inconsistent tempos as time signatures change

Hello all,

I'm facing a strange issue with this score, particularly it's playback. The first line sounds fine, but the rest have tempo morphs as the time signatures change. I've tried adding tempo markings, and while sometimes that has changed the results they haven't necessarily fixed the issue. Even when I remove the layout breaks and the double bars these issues still persist. How can I fix this?

Thanks,
Nate