Sounds like a similar problem I had recently with one of my MPPT chargers.
Turned out to be a problem with the MC4 connectors
Could you maybe swap around the panel/charger connections to see if the issue moves with panel or stays with the charger? A process of elimination?
Also, do you use any of the cheap torpedo shaped DC breakers? I had a problem some years ago where they would drop voltage under load, despite not getting warm. Only used quality Durite breakers since then with no more issues.