Last fire.. :-(

[Gazoo]

That is all well and good but you seem to be under the impression that I am getting into the way of that. This is the second time you have mentioned that. Maybe you should tell the OP rather than me.

I have nothing to tell the OP and this is his thread. But I did have a question for him. I asked fhorst to explain how the terminals got so nasty looking rather than both of us speculating. Then you came along and wrote your speculation novel after I asked him....... before he had responded. I am looking forward to an explanation from fhorst. Hopefully he will explain so all this needless speculation can be put to rest.

Not going to have any more tits for tats with you. But I am entitled to my opinion as you are yours.

 

[curiouscarbon]

Guys, play nice

 

[Posplayr]

There are a few other factors which may need to be considered, first is the time line of events, power goes off early in the morning before sunrise, then restores later after sunrise, hense charging has begun, then the fire occurs.

Secondly the possibility of rodent damage to insulation is extremely high in Thailand especially in closed spaces like between the stacked battery arrangement, this can go unnoticed resulting in faults or failure. One of the many sense wires, which may have been missing insulation, could have contacted a busbar at excessive voltage for a single cell, this could overwhelm the cell to the point of failure.

Protection against rodents, especially in the tropics, is something to taken seriously, on my pickup alone they disabled the wipers, indicators, a fuel injector, fuel door release, eaten the cabin air filter and the engine air filter, all in three years since new. My main battery cables of a solar system, which were in an unsealed PVC duct, were stripped if insulation until they shorted, thankfully the overcurrent protection did it's job, they have also disabled a system by chewing through a BMS sense wire.

There is no direct evidence of any over-current conditions other than through busbars on the main terminals of the cells. I only discount these other factors because none of them can create the excessive busbar current that apparently ran through the bus bars.

I don't want to make this too technical so I will not, however that said, in any formal FMEA you have to find a failure mode (e.g. a rat eating insulation) that associates with the observations (uniformly overheated battery posts). If you have a hypothetical fault mode that you can identify where a rat ate some insulation caused direct cabling short where this resulted in a battery terminal capable of 200 burning for over 24 hours I would like to see that. The flip side of this that people need to realize is that any likelihood of a failure mode root cause (i.e. rats eating insulation) is mitigated by lack of direct evidence of the effect that might have had.

Just because something happens quite often (e.g. that rates eat insulation or conductors get corroded), if there is no direct evidence of that root cause as a failure mode then that situation is deweighted as a possibility. In most circumstance there is ambiguity in the observations and so there is a weighting that occurs to adjust the probabilities of the direct observations with the typical conditions that can occur.

So if you are still awake, this is probability theory involving Bay's law. It involves the a priori probabilities (what you know before the fault like what commonly occurs) and a posteriori ( what probabilities are associated with the direct measurements/observation independent of the a priori).

So any theory of what might have happened must make a compelling case for something that matches the evidence. The following direct evidence matches the theory I posted.

• A single burned posted (single-point failure),
• many if not all heated posts,
• what looks like a fully discharged battery bank.
• A slow 24 burn duration

If there is anything that does not fit the a posteriori fault assessment that is an ambiguity that is addressed with a look at what is more likely to have happened independent of those direct evidence.
.

 

[curiouscarbon]

https://en.wikipedia.org/wiki/Bayes'_theorem

 

[Mcgivor]

To conduct an investigation the evidence should be examined in person in an undisturbed condition, so far be it for me to come to a conclusion based on a handful of poor resolution pictures, but I certainly wouldn't rule out indirect causes, after all if a wasp can bring down a passenger jet by plugging a pilot tube, Birgeniair flight 301, a rat could cause a battery failure.

 

[Frank in Thailand]

The theory the one cell developed an internal short and caused the meltdown and fire just doesn't seem plausible to me.
To my knowledge we don't have any examples of this happening before.

We have seen a lot of examples of a connection getting VERY hot if there is a loose connection or corrosion causing a bad connection ..... I think that it is the most likely explanation of what caused that cell to melt down.

@Bob B ,

2 things to mention.
First one.
Apparently I'm (almost) the only one who have bloated cells, and continued to use them.

This is a whole different ballpark.
What happens when you slightly compress bloated cells to prevent them from further expansion?

I can tell you, they de-bloat.
Slight compression is enough.
When tight enough not to be able to lift a cell out in a row of 6, a week later you can lift it out.

My mistake. (I guess) is to think "hey great, let's tighten a little more" same, just enough to not be able to lift a cell out of the 6.
After a few weeks the bloating was half.

For capacity, I've tested them all.
Between 144-151Ah after a year of usage.
Not so bad, consider they are bloated.

Was it a mistake?
I don't know. If it was, it's an understandable, without having deep knowledge of LiFePO4, reason for bloating, etc.
Where sure, we can all think of a few, over charge, over discharge, too high current in or out, short.

If none of this was the situation, just too many sheets in a housing...
You tell me. I don't know.

Just sharing what I did.

I wonder why people see 12v setup.
It's 48v S16, 5 sets in series.
So S16, 5*

The part that was installed, connected with Bus-bars was 1 X 280 S16 and 1 X 152Ah S16.

This was like this for many weeks, with regular checks for temperature.
Almost every day.

Due my last accident, I did not want to leave anything to chance.

At the moment of the fire, the cells where balanced, at +48v, between 2.9 and 3v
No crazy discharge.
Charge capacity was a few ampere, 3-400 watt on the 2 sets.

Like an airbag doesn't protect when there is no accident, BMS in this setup would not have done anything.
There was no trigger to act.

DIYBMS finished setup with smoke detector, that one would have triggered.
Finished DIYBMS with temperature sensor on EACH cell, that one would have triggered.

Standard Daly??
Only in the situation if the temperature sensor would have been at the cell that failed.

BMS stops charge or discharge at under/over charge and over capacity.
This was not happening!!
3-4A charge, 10-20% SOC.
Absolutely no reason to act.

I understand why people like to point a blame to my installation.
If I didn't do anything wrong...
It would make them feel unsafe about their own installation!

And you should.
Like the fire from Tesla showed...
It's never safe.

I'm absolutely certain that the wires where not the reason for the fire.

I'm even thinking about filming a full short with a wire for you guys to show how (not) spectacular that is.
Self fusing at 10A...
Like watching grass grow, that exiting.

Please let me know if you guys want this

So...
If it was
- not the wires..
- not the SOC (10-20%)
- not lack of BMS
- no lose contact or enough charge/discharge that would have made enough heat..

Possible options are greatly reduced.

Compression of bloated cells is apparently not a good idea ?
If you have them, leave them bloated, allow them to bloat even further if they want.
It doesn't hurt the capacity much, and compression can/ will create stress that can/will give a fire.

If you are a bit handy in making threads, buy cells without holes.

Use a forming tap, what makes way stronger threads.
Your end product will have a safer start then it would have been pre-drilled and tapped from China.

Grubscrews +Loctite red and primer
Before you even begin to do anything.

If sheets between cells can prevent accidents, use them.
I doubt Eve is placing them in their pre-build packs without reason.

Torque meter (!) As Equal torque will give more equal charge.
Less risks.

@Will Prowse
"Up in smoke" has also my vote
Alternative:
Magic smoke corner.

 

[Frank in Thailand]

Good point. Why don't we ask?

@fhorst What happened to the terminals of your cells? Maybe you posted it and I missed it. Why do all the cells terminals in the photos above look so nasty Thanks.

We didn't dump sand.
As it would not reach the lower layer where then fire was.

We thrown the dry sand between the 2 layers.
So it would have gone up / under and over all bus-bar/terminal.

What I found strange was the clogging up of the sand in the top layer, no bus-bar yet, but a nice dab of ox-guard.
Apparently it got liquid enough to be absorbed by the sand.

 

[Frank in Thailand]

I have nothing to tell the OP and this is his thread. But I did have a question for him. I asked fhorst to explain how the terminals got so nasty looking rather than both of us speculating. Then you came along and wrote your speculation novel after I asked him....... before he had responded. I am looking forward to an explanation from fhorst. Hopefully he will explain so all this needless speculation can be put to rest.

Not going to have any more tits for tats with you. But I am entitled to my opinion as you are yours.

Nasty looking??

If you talk about the sand??

Otherwise look at my thread repair post.

That is how they looked before installation.

Only thing Changed was cleaning and dab of Ox-guard.

Bad looking like this?



Not installed, just placed, no bus-bar, no wires, just cells, 6 in a row for the 152, 4 in a row for the 280Ah.


Or like this:


One terminal of one 280Ah that was installed and not bloated due the fire

 

[Frank in Thailand]

To conduct an investigation the evidence should be examined in person in an undisturbed condition, so far be it for me to come to a conclusion based on a handful of poor resolution pictures, but I certainly wouldn't rule out indirect causes, after all if a wasp can bring down a passenger jet by plugging a pilot tube, Brien air flight 301, a rat could cause a battery failure.

Rodents are a serious problem here in Thailand.

Cats are a solution, unless you are late in spaying....
Anyone want a cat?
We have over 50...
Now all girls and boys can no longer reproduce...

Dogs the same..
Not yet solved.. (too expensive) usually the family didn't have enough money to feed them quality food, so most just died of poor health.

Something changed there..
And from 7 we are now over 35...
5 from us (3 puppies) the rest mother, father, sister....
Good thing we have some space

The dogs and cats make lately more damage then rodents did before.

They can't get in the battery room.

Mice/rat gecko's, lizards, snake, scorpions, and many many more living creatures with or without legs/wings.

You cant stop them all.

No one would expect to find ants hatching farm in their printer/copier inside the house, without visual ants around. Yet, I did.

I can't rule out rodents.
That's why I used self fusing wire, that should fuse at 10A without damage the sleeve (right English word?)

The wires where no longer the rats nest, they where straight up.

Even if a rodent would have eaten a wire, it could not do much damage.

Then again.. if a wasp can bring down an airplane...

Anything Is possible, just not likely.

 

[Gazoo]

Nasty looking??

If you talk about the sand??

Yes and thanks for explaining. Since you didn't have busbars installed everything is clear.

 

[Hedges]

Are multiple series strings of cells connected in parallel? e.g. 16s2p, Such that, if one cell decided to short out and go to zero volts, other strings would dump current into that string.

Or, are multiple cells in parallel? e.g. 2p16s, such that if one cell decided to short out, its neighbor would dump current into it. Don't think this was the configuration because photo shows cells in series not parallel.

With 16s2p, if a single cell shorts and drains itself, other strings drain into this string. Could happen gradually. If instantaneously, you get 15s || 16s, so delta voltage is 1/16th cell per cell, or one cell per 16s string. With spec sheet resistance of 25 milliohms per cell, about (15 + 16) x 0.00025 ohms = 0.00775 ohms.

3.4V x 16 cells - 3.4V x 15 cells = 3.4V

3.4V / 0.00775 ohms = 440A

Some large packs of parallel/series flashlight sized lithium cells deal with this by fuse per cell (welded strips paralleling cells.)

For these large LiFePO4 packs, BMS isolating strings (which probably means BMS per string) might be best. Fuse per string could work above some current, but if fuse didn't blow wouldn't prevent overcharge of remaining cells in a short string.

We know that LiFePO4 normally releases vapor without self-igniting. Something external is thought necessary for ignition. Could be sparks if current-carrying wires are interrupted. Could be glowing metal if current through a bad connection caused heating and rupture. But isn't clear what initiated the failure - previously swelled cells having latent failure is a reasonable theory.

 

[Posplayr]

To conduct an investigation the evidence should be examined in person in an undisturbed condition, so far be it for me to come to a conclusion based on a handful of poor resolution pictures, but I certainly wouldn't rule out indirect causes, after all if a wasp can bring down a passenger jet by plugging a pilot tube, Birgeniair flight 301, a rat could cause a battery failure.

Speaking of a priori knowledge/probabilities (things we know without the occurrence of this event) is that paralleling strings require great care because a single point failure of a cell can cause a meltdown of the entire parallel string battery system.

Given the OP's battery architecture ignored all these known properties/risks and dangers of parallel strings, and the fact that a whole system meltdown is the most likely outcome from single-point cell failures (unless architecturally mitigated) this failure mode becomes the most likely a priori probability.

Both the photographic evidence as well as OP's observations are characteristic of the most likely fault even mentioned above we have strong a posteriori evidence that this is what in fact did occur (there is little if any evidence of alternate failure modes) . This is the fault analysis I previously postulated and is echoed in much greater detail in this document below.

Apparently, it bears saying, that the process I am going through to do this type of analysis is based on years of electrical engineering and systems engineering experience dealing with extraordinarily more complex systems than this. And yes I have a BSEE and an MSECE which is focused on advanced systems studies coupled with the decades of work experience indicated above. I have worked professionally in the area of fault detection, isolation, and prognostics (i.e. prediction of remaining EOL) or complex military systems. So it may not seem possible, but any experienced electrical engineer will immediately see the risks and the possibilities based on even a cursory review of the evidence come to the same conclusion.

Primary reference:

https://www.orionbms.com/manuals/pdf/parallel_strings.pdf

Key Warnings:

 

[Hedges]

So it may not seem possible, but any experienced electrical engineer will immediately see the risks and the possibilities based on even a cursory review of the evidence come to the same conclusion.

Which is exactly what I just did. Doesn't mean that was the failure mode in this case, but just by thought experiment realized it was a possibility.
Sometimes by analysis and measurement we can determine the risk is already mitigated. Sometimes not, and we may make changes or may accept the risk.

Having a way to let the house (and occupants) survive even a catastrophic failure is a way to address the known-unknowns and unknown-unknowns. OP did have this battery in a separate bunker. For RV users that may not be possible, although they could plan to have a way to "cut their losses" both literally and figuratively. Forum members have joked about a spring-loaded mechanism to expel a burning pack. For home users, although we think of LiFePO4 as relatively safe to have indoors, maybe should instead do an isolated, earth-sheltered bunker.

 

[Bob B]

@fhorst .... Do you have any pictures of the pack containing the cell with the worst damage after the sand was cleaned away, but before the pack was dis-assembled?

 

[Posplayr]

Which is exactly what I just did. Doesn't mean that was the failure mode in this case, but just by thought experiment realized it was a possibility.
Sometimes by analysis and measurement we can determine the risk is already mitigated. Sometimes not, and we may make changes or may accept the risk.

In failure analysis, it is about probabilities and risk. This fits in with decision theory and where Bayesian probability theory comes from.

Based on the reference linked, parallel strings (without mitigation measures) that have single-point cell failures will result in taking down the whole cell array with all parallel strings. So while we can not say with absolute certainty this is what happened it is by far the highest a priori possibility given the fact that there was no mitigation of the parallel string risks and a single point cell failure seems to have occured.

Although OP claims that a BMS would have not saved the system and that just as importantly he claimed he does not know what happened, he does conclude that a 15% SOC bank fully discharged and that there is a single point failure at a single post on a single cell. This can be confirmed from the pictures despite their low quality. This means that the preponderance of the a posteriori evidence (that which is observed) points to the characteristic pattern associated with the parallel string catastrophic failure.

So to assess this we look to Bayesian decision theoretics in what is called likelihood ratios. Without getting into the exact formulas, you basically compute the product of the a priori probability (what can occur without regard to this particular instance) times the a posterior probabilities (what is likely to have happened given the evidence) and select the highest product as most likely. No other fault mode that I can see and I can see no one else identifying one has a probability as high as the likelihood as the parallel string melts down.

( Note: identifying rodents as a possible issue is fine, but you would need to show a chain of events where that would have caused a single point failure in a single cell as supported by the evidence)

For the interested reader.

An Intuitive Introduction to Bayesian Decision Theory

Bayesian decision theory refers to the statistical approach based on tradeoff quantification among various classification decisions

www.analyticsvidhya.com

Bayesian Decision Theory

Oh! What are we if not patterns?

towardsdatascience.com

As far as having other mitigation factors are concerned in the overall safety design, OP could have ejection seats (to eject from the house) for all household members or he could flood his basement as he even had contemplated, but this is all part of a larger system solution that was absent and beyond the scope of failure analysis.

 

[Synopsis]

To me the area surrounding the terminal in the photo in the first post looks like a case of metal stress fatigue. I can posit that at some stage during freight or manufacture or post-manufacture that stud has copped a hard hit. That, alongside a slow bending moment potentially from expansion and contraction of the bank has lead that microscopic weak point to grow until it's eventually failed catastrophically and then you've got your conflagration.
Nonetheless, these posts are freaking me out as I have just bought some EVE cells and plan to put them in a hot, bushfire prone area. I've used other brands successfully for a long time in similar circumstances but you can bet I'm going to do a lot more research on these cells.

 

[Synopsis]

as for insulation between cells, I'm sure nearly nothing is going to prevent the cascading heat transfer, but for electrical insulation, these flexible teflon grill matts are outstanding. https://www.bunnings.com.au/bbq-buddy-nonstick-hot-plate-liner_p3180247

 

[Leon]

as for insulation between cells, I'm sure nearly nothing is going to prevent the cascading heat transfer, but for electrical insulation, these flexible teflon grill matts are outstanding. https://www.bunnings.com.au/bbq-buddy-nonstick-hot-plate-liner_p3180247

How thick are they?

 

[Synopsis]

Thin - like a sheet of thick paper. But they're teflon (PTFE) coated which kills two birds with one stone - no rubbing, and very electrically insulating.

 

[Synopsis]

How thick are they?

Leon - more about ptfe sheeting here. https://polyfluoroltd.com/blog/ptfe-battery-separators-an-essential-insulating-media/