I love the idea. No dedicated wiring, should be fairly easy to assemble.What you're now describing which could go through the "trailer battery charge" wire and return on the shared grounding wire of the Bargman cable (and plug) you already have. That's what I have, stepping up to 36V under the hood and running into a MPPT Solar controller (rather than a fixed voltage "step-down") - providing power to the entire Trailer (including battery charge), and not just dedicated to the fridge.
In my case, the SUV is switchable (so that I can tow other trailers), and defaults back to "standard voltage" whenever I turn the ignition off. In the Trailer, a 5-pin Relay detects the incoming TBC voltage and routes it into the SCC when it is "high".
Is that what you want to do? An MPPT Solar controller costs more than a dumb buck converter, but you wouldn't have to mess with switches when you finished towing. The Fridge is definitely wired to share +12 VDC for both the heater and the control board, the control board must also be powered up for AC operation at a powered campground.
My scheme is good for around 400 watts.
I assume a dedicated mppt charger would be in order, of course, as the panels operate on a different voltage, but a basic one should do.