Correct. To increase only the capacity (not the voltage) you would add more batteries in parallel.
Can you add a cheaper 12v battery? Yes, you "can" but whether or not you "should" is another question. Generally, the recommendation is to stick with batteries of approximately the same size, age and condition.
It sounds like the heavy load of air conditioning may have worn down your existing battery. When you turn on your air conditioning, a load is placed upon your battery. This load causes the voltage of the battery to decrease. The bigger the battery (higher capacity) the less effect the load will have on the battery voltage.
As your battery wears out, it essentially gets smaller (loses capacity). It sounds like what has happened is that the capacity of your battery may have dropped (as a result of being used) to the point where it is no longer large enough to support the load imposed by the air conditioner. Remember, when the load is applied, the voltage drops, and it drops more with a small battery than a large one. Now that your battery has less capacity, the voltage drops far enough to cause your inverter to object.
I do not believe that you will find ultimate success by adding a cheaper battery to the bank. It might work for awhile, but the bottom line is that an air conditioner is a large load that requires a large battery. I think you need a new battery, properly sized for the job you are asking it to do.