TXB0104-based converter (like SparkFun Voltage-Level Translator Breakout - TXB0104) works, but it’s rather “unstable” with long / extra wires. I’ve seen post somewhere stating that TXB0104 pretty good conditions to work.
For example, welding extra wire to VE.Direct TX output and connecting it to Picoscope works, but once you disconnect from Picoscope the TXB0104 goes haywire, producing gibberish 
.
BSS138-based SparkFun Logic Level Converter - Bi-Directional works too, and it seems it’s much simpler and more stable.
My guess that SoftwareSerial’s “invert” does something with pullup/down resistors of ESP and it makes it work in that way for some reason, while these proper decoupling/level converter solutions just work without any fiddling.
So the solution is to use proper 5v->3.3V logic level converter like BSS138-based from SparkFun.