You want the fridge to run on 12v which makes things much easier. You can turn the inverter off to save the power consumption and still have it charging the batteries. Since your 12v loads would come straight from the battery, the inverter would only need to be on when you needed
1220 err... 120v for something.
The huge benefit to using an AIO is that it's really, really easy to get things installed and running. With an AIO you get the MPPT charger for your panels, an AC charger to plug into a wall outlet, a transfer switch to prioritize between battery and wall power, the inverter to convert the battery power to wall power.
If you wanted to do components you'd need a lot of individual parts, wires, extra pieces, and would most likely cost more than the AIO was.
I did something similar with my
solar dolly where I run the AIO off the battery and a bunch of 12v stuff off the same battery. I can turn off the inverter and still have all my 12v stuff and USB stuff without having to eat up battery power.