A good rule of thumb is a lead-acid battery should not be drained at much more than 1/10th of C. If your battery is 100Ah at 12V, then 1/10 of C is 10Ah. At 12V that's a 120W max load to keep a happy battery. Since your frig is 440W, that is much higher than what your battery wants. So, yes, your system is too small.
Your low voltage numbers are an indicator that the load is too high, and that 120W panel is not keeping the battery fully charged. Is the battery laying flat on the roof of your RV? Flat, it might be putting out 60% of potential output. So, that 120W panel is likely only putting out 120W X 0.6X = 72W.
Let's do the math to design something workable. Getting back to 1/10C....
440W/12V = 36.6A X 10fold =366Ah battery.
Batteries like charging at 1/8C, so for a 366Ah battery you need... (366Ah/8) X 13V charging X 1.2FF = 714W. The 1.2FF is for a panel directly facing the sun. For a flat panel the FF is more like 1.6. So, (366Ah/8) X 13V charging X 1.6FF = 952W.
So, the answer is bigger batteries, and a lot more solar panels.