question

You don't say whether the GFI trip occurs when the inverter is using the AC input and passing through power to the loads, or if it is when the inverter switches away from AC input and is inverting.

GFIs measure the differential current between hot and neutral. This can be caused by:

1. leakage in some faulty device or wiring.
2. filter capacitors at the AC input to safety ground
3. a neutral to safety ground short which causes some load current to flow through the safety ground connection

I would investigate #3.

There should be EXACTLY one neutral/safety connection and that's at the source of the power. In the case of shore power, this happens at the service entrance. In the case of the inverter running without AC input, that "bond" occurs inside the inverter by closing the ground relay.

In a system with split-phase shore power and a single inverter, one leg is fed from the output of the inverter and the other is fed directly from shore power. The shore power neutral ends up connected to the inverter's AC output neutral in the power distribution panel.

When the inverter is passing AC input to it's output, it ties the input neutral to the output neutral through the AC input relay. The ground relay will be open in this case. So from a ground fault standpoint, all should be well. However, some neutral current will flow around the inverter via the shore power neutral. This could cause unwanted current flow inside the inverter.

When the inverter goes to invert mode, the AC input relay opens and the ground relay closes, but remember the shore neutral is tied to the load neutral in the power distribution panel. With the ground relay closed, there is now a connection between neutral and safety ground down stream of the GFI and the GFI will trip.

Disabling the ground relay will remove the neutral/safety ground connection, but there is a problem when shore power is not connected. With shore power disconnected, there is no neutral/safety ground connection for the inverter's output. This probably won't hurt loads but is against most electrical codes since voltage can appear on an unfused conductor.

The correct solution to this problem is to keep neutrals separate for each leg. It should be fairly easy to add a second neutral busbar inside the power distribution panel as I've shown below