Aaaahhhh, ok. There's your problem. Your batteries are wired up in a way that the batteries aren't charging or discharging evenly. Your battery with the charge controller are getting all the charge first and only passing a little power to the next battery, and a little less to the 3rd battery, and almost mothing to the 4th battery.
Your 4th battery is trying to feed the inverter load, asking for help from the other batteries and getting a little from battery 3, and a little less from battery 2.
Your charge controller will see that battery 1 is full when your inverter battery is crying for help. That unbalances the batteries pretty badly and short cycles the middle batteries, deep drains the inverter battery, and keeps the 1st battery mostly full.
Ok, hopefully that explains the basic concept. You've got to get all those batteries talking to a common point, all your Pos together and all your Neg together. From that common point ALL your loads get connected.
The easiest way is like Pic 1 above where you bring everything to a common point (bus bar) and go from there.
If you don't have access to a bus bar and
12v refrigerator, the next best option would be to connect your charge controller to Batt-1 Pos and Batt-4 Neg, and connect the inverter/fridge/lights/loads to Batt-1 Neg and Batt-4 Pos. That's not ideal but it will be better than what you have now.
