WOW! Looks like an excellent system and EXTREMELY CLEAN!!!! My first inclination is there is no way I'd want to upgrade anything, depending on if the batteries are still okay or near the end of their life.
No offence, but have you read the user manuals for each of the parts? That would be an extremely good place to start. If you don't have the printed manuals, you can find them all online as PDF files and then print them out to read and to have easy access to. It really is a must in my experience, to understand how each of your components works, how to input settings, etc., and even then it takes time working with them to do it right and remember the steps.
Next, YES check and tighten all the battery cable bolts if you are going to use the system. Again, the Victron battery manual should tell you how much to tighten the battery cable nuts and bolts. Do you have a torque wrench. Otherwise, clean every nut, bolt and washer and then snug them down tight but not forcefully. They must be tight for a good flat connection, otherwise poor connections can cause heat, damage and fires.
You would best test the batteries under "load" or stressing them via turning on a 500-1,000 watt appliance and watch the inverter and or charge controller reading of battery voltage as it changes. Again, you need to read the battery manual to find out what is normal for AGM batteries.
Are there any "loads" or appliances drawing power 24/7, or normally always on? Any corrosion built up anywhere that we can't see?
I'd clean and tighten all the battery bolts and nuts, check every other cable connection of the inverter and charge controller for tightness and then charge the batteries to full and then monitor them without a load and with small loads to see what happens, such as if they hold a full charge and how fast they might discharge with no load or small loads (an appliance on).
I don't have experience with AGM batteries vs. Lead Acid, but perhaps the Victrons are sulfated inside and so not holding a charge. Victron and their user forum should be able to advise you on how to figure that out. If they are at the end of their life, then get LiFePO4 batteries at whatever capacity you need and or that your system can provide. Ideally, build your own batteries, as it seems to me, you need to read, watch and so learn quite a bit more before you start changing or buying new parts.
Please read the user manuals and maybe do it several times. Victron also has a good document called Victron Energy Unlimited, which you can find in this forum's Resource area or on the Victron Energy website. Well worth reading as a start to understanding how things work together.
Have you done your Energy Audit yet?
Hopefully others with experience will add more to advise you.
