question

Most probably a couple of possibilities;

First, check the common battery and cable connections to make sure no high resistance joints causing voltage drops.

Second, the supplied fuse and holder for the BMV's can give problems, I've had a fuse that was not of the correct rating, causing a voltage error on the BMV, as well the holder is poor mechanically. I replaced both with a poly switch (self-healing fuse by Littelfuse) of suitable rating. With a multimeter measure the voltage drop across between the battery and BMV, should be <0.1V.

I would also caution about the sensor/supply connections on the BMV, you need to exercise care when connecting the ferals, make sure you release the tension with a small screwdriver when installing to prevent damage to the receptacle, if this is damaged the connection becomes unreliable.

Have you got a good earth connection between the lithium and lead batteries?

From the way you describe the install, the advice you were given by the 'Victron Engineer' (put in quotes as not sure if it was someone actually from Victron?) seems odd. There is no problem I am aware of monitoring a Starter Battery on the Aux input no matter what type of Battery you have on the Primary input. I would also not agree with having to add a dedicated shunt to monitor the Starter.
May well not know all the facts, but TBH I find the advice as quoted suspect.

The only time I can think where you cannot do just what you have been doing is if you don't have a common ground on the batteries - in which case, I would expect to see no voltage rather than an overvoltage.

Once you have checked for loose conections and you still have a problem, what I would try if I were you is swap the two inputs on the Shunt and see what happens in terms of voltage reading. The Current/Ah/Watts/SOC reading will not make any sense when you do this, but it will let you know if the wrong voltage readings swap from Aux to Main (which points to an issue with the connection INTO the shunt) or if the Wrong readings stay on Aux (which points to an issue with the Shunt/BMV). Undervoltages are not uncommon with resistance in cabling/fuses, but overvoltage (especially in the area of 3V) is much more uncommon.