What brand batteries, and do you have a way to see the individual cell voltages? If you disconnect the 4 batteries, are they all charged to exactly the same voltage?
If the cells are out of balance, with one or two cells charged over 3.5V, and the other 2 cells below 3.5V, then a small change...