Quantcast

Why don't altimeters have in-flight power-loss recovery?

The Rocketry Forum

Help Support The Rocketry Forum:

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
Its happened to almost all of us who fly electronic deployment. Something unexpected happens and the altimeter suffers a hiccup in power that causes it to reset.

My question is, why don't the altimeters have an in-flight recovery feature? I'm not aware of any commercial altimeter that has this feature, but it seems like it wouldn't be too hard to code up.

You could store a simple variable in EEPROM that tells the flight computer what the last event was. Read this variable at startup and if it wasn't "touchdown," then it could attempt to recover by quickly running an emergency recovery routine. This routine could sample the sensors, and then based on what was found, determine where the rocket is in flight. If the sensors determine the rocket is on the pad, then go back to the beginning and start with a normal powerup sequence.

One could also get more sophisticated with some inflight fault detection. If a bad e-match or short in the pyro wiring is causing a brownout, then it could choose to do other things. For example, if the apogee pyro is shorted and can't fire the ematch, on the second emergency recovery cycle the altimeter could fire the mains. I'd rather blow the main near apogee than come in ballistic.

I'm coding this up right now on my own flight computers. Considering how many times I've seen this happen to other flyers, and the potential increase to overall safety, I'm curious why this hasn't been pursued. Let me know if you have thoughts.
 

heada

Well-Known Member
Joined
Jan 18, 2009
Messages
3,863
Reaction score
1,057
Location
Indianapolis, Indiana
My ARTS2 supports dual batteries. My MAWD had a huge capacitor to buffer everything. Almost all the Eggtimer stuff supports dual batteries.

Rather than tweak code to try and recover from an unknown state mid flight which could be a safety hazard, create a power system that doesn't drop out under extreme cases.
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,144
Reaction score
1,407
I've actually thought about doing just that a number of times. You'd need a real-time clock, because you don't want to lose power and have the chutes blow when you power it up on the ground just because it thinks you're continuing a flight and you've gone below the main chute altitude threshold. It would have to know how long it's been since you lost power, and make a reasonable assumption about your flight length...

It's easier just to make sure that your connections don't come loose, and there are plenty of ways to prevent brown-outs (second battery, current limiting, etc.)
 

CoyoteNumber2

Original San Diego High Power Rocketry
Joined
Jan 30, 2009
Messages
623
Reaction score
115
For example, if the apogee pyro is shorted and can't fire the ematch, on the second emergency recovery cycle the altimeter could fire the mains.
I think some altimeters do/did fire the main channel at apogee if the apogee channel read open. I can't remember which. Olsen M1/M2, possibly.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
Brownout capacitors don't solve all the problems. Personally, I've had this happen 4 out of about 500 flights, even with a big capacitor onboard. One on a hi-alt45, one on a MAWD, one on a Stratologger, and once on my own device. Admittedly, only one or two of these could have been saved by a recovery routine.

You'd need a real-time clock, because you don't want to lose power and have the chutes blow when you power it up on the ground just because it thinks you're continuing a flight and you've gone below the main chute altitude threshold.
RTC is definitely a good way to do it. I was thinking of sampling the barometer for 1 second and see what the estimated velocity is. This could be paired with the last known event. Something like this:

Last event = motor burnout: If barometer estimated velocity is positive but decreasing, then we are ascending and apogee would be the next event. If its negative and increasing, fire apogee charge immediately.

Last event = apogee: If barometer estimated velocity is negative and we are above the main deploy altitude, then wait for the main deploy altitude. Fire mains immediately if velocity is negative and we are below the main deploy altitude. If velocity is roughly zero, then do nothing because we know the unit is on the ground.

I definitely want to ensure that the charges don't fire while on the ground. An approximately static altitude along with stable accelerometers reading 1G is a clear sign of this. Thoughts?
 

UhClem

Well-Known Member
Joined
Feb 6, 2009
Messages
1,746
Reaction score
235
You could store a simple variable in EEPROM that tells the flight computer what the last event was.
All such EEPROMs have a limited number of erase/write cycles. That number might seem large now...
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
All such EEPROMs have a limited number of erase/write cycles. That number might seem large now...
I'm not too concerned about erase/write cycles on the Teensy3.5. 100,000 write cycle life / 5 events written per flight = 20,000 flights. Even if I'm off by a factor of 100, thats more flights than any single unit will ever see.

There's also enough unused EEPROM that I can just switch to another location if I wear one out.
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,144
Reaction score
1,407
I2C EEPROM's are good for about 1,000,000 write cycles (and unlimited read cycles). If anyone manages to "wear out" an EEPROM on an Eggtimer product, we'll gladly replace it for your at our expense. It would take about 100,000,000 seconds, or about 3 continuous years of operation, before you'd hit that point. Your LiPo will probably die a lot sooner than that. :)
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
I'll code this up and start flying it. Hopefully it solves more problems than it creates. I'll report back after the first few flights.
 

OverTheTop

Well-Known Member
TRF Supporter
Joined
Jul 10, 2007
Messages
4,834
Reaction score
2,076
Location
Melbourne Australia
I had a Telemega go quiet, at apogee, on a flight last year due to switch bounce. I am not the only one to experience this I have since found out. I have suggested they write an emergency recovery into the code and they seem to think something based on baro pressure should be workable.

I would probably look at storing state variables in NV memory of some sort. EEPROMs can have a life down to 10k writes in some cases. The newer FRAM devices are around 10^14 write cycles, essentially limitless. We are switching to these in the products I design for work.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
EEPROMs can have a life down to 10k writes in some cases. The newer FRAM devices are around 10^14 write cycles, essentially limitless
I like the FRAM idea and may eventually go that way. Right now I’ve got enough free EEPROM where I can spread the variables out over several locations if needed. In fact, the MCU is already this which is why the EEPROM life is so good.
 

Saint_Tiki

Well-Known Member
Joined
Oct 6, 2020
Messages
54
Reaction score
55
What percentage of flights suffer a failure that this would prevent? And what percentage of flights now suffer a failure because of this added complexity until the code gets squared away?

I also don’t like the idea of turning the altimeter on and it deciding if it wants to enter ground mode self test or if it decides to enter launch detect and arms the pyros which might pop at the next gust of wind or flight of stairs.

The time spent working this feature would be better spent preventing power loss. It’s like why Crew Dragon doesn’t have propulsive landing as a backup to the parachutes. Time is better spent making the chutes better.
 

Saint_Tiki

Well-Known Member
Joined
Oct 6, 2020
Messages
54
Reaction score
55
I had a Telemega go quiet, at apogee, on a flight last year due to switch bounce. I am not the only one to experience this I have since found out. I have suggested they write an emergency recovery into the code and they seem to think something based on baro pressure should be workable.
But that’s clearly a switch problem? Get a different switch that doesn’t do that instead of inventing a new thing that could go wrong that only exists because of a different problem that wasn’t resolved.
 
Last edited:

timbucktoo

Well-Known Member
Staff member
TRF Supporter
Global Mod
Joined
Jun 13, 2014
Messages
8,069
Reaction score
1,256
Location
Cocoa Beach
Its happened to almost all of us who fly electronic deployment. Something unexpected happens and the altimeter suffers a hiccup in power that causes it to reset.
I’ve never suffered this hiccup you mention and all I fly is electronics & most are redundant. Have you investigated the cause of your hiccups? I have seen electronics fail on high G flights but seems to always be user error.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
I’ve never suffered this hiccup you mention and all I fly is electronics & most are redundant. Have you investigated the cause of your hiccups? I have seen electronics fail on high G flights but seems to always be user error.
Yes, I investigated and root-caused each. I agree with you about wiring and design, but I see it happen frequently with other flyers. I'd say I see it almost at every launch. This is why I created the topic.

On big rockets I always have redundant electronics so its typically not a big deal. Smaller ones sometimes only have 1 altimeter, so this is where the usefulness comes in. I had this happen recently to me because of a programming error on my own system, and thats why I thought about the ability to recover in-flight.
 

Saint_Tiki

Well-Known Member
Joined
Oct 6, 2020
Messages
54
Reaction score
55
A semi-random recommendation for a power switch.

I use these in combat robots that are subjected to significant shock and g-force. They are great super lightweight switches with a very reliable "On" connection. Reliable as in they are in hundreds of small combat robots that would instantly lose a match if they lost power. They can handle plenty of amps (40 cont, 100 burst) and weigh almost nothing. I put one in my dual deploy that I sent up for the first time this weekend. I also use them in 3 of my robots and haven't had an issue.

Shipping is from Canada. It's a great product though.

 

gtg738w

FlightSketch - flightsketch.com
TRF Sponsor
Joined
Dec 27, 2018
Messages
250
Reaction score
208
Most people only think of the no-fire failure mode for electronics but the unintended firing failure is probably more dangerous. Should an altimeter be given a higher probability of failure to try and compensate for other deficiencies in the system? Generally the approach is no, fix as much as you can and then add redundancy as needed.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
Most people only think of the no-fire failure mode for electronics but the unintended firing failure is probably more dangerous. Should an altimeter be given a higher probability of failure to try and compensate for other deficiencies in the system? Generally the approach is no, fix as much as you can and then add redundancy as needed.
I don't think it will dramatically increase risk. The added risk is only present immediately on powerup after the rare case of in-flight power loss. When this happens, there is indeed added risk that the recovery routine will misidentify the current flight status and fire charges at the wrong time. A 1-2 second sample of accelerometers & the barometer should be enough to make the misidentification risk very small. A gust of wind sitting on the pad should be easily distinguishable from a rapidly falling/ascending rocket.

I'm more concerned about what might happen if the power re-connects when a flyer picks up a quiet rocket that has failed. This is a more plausible scenario of flight event misidentification. I need to think about this scenario some more.
 

mikec

Well-Known Member
Joined
May 9, 2009
Messages
2,504
Reaction score
384
In spacecraft design, the behavior in fault conditions is often altered by the value of "mission phase bits", which are stored in very reliable NVM, much like what you are proposing. When flight values are tested on the ground there are all kinds of hardware and software inhibits on pyros, actuators, etc. And I don't think these bits are ever set autonomously, only by explicit ground command.

I appreciate what you're trying to do, but IMHO this failure mode 1) doesn't happen that often, 2) would be better addressed by simpler redundancy, and 3) introduces a series of bad failure modes and potentially unpredictable/unsafe behavior. As just one example, what happens if your sensors fail in some way?

FWIW, I personally would not use an altimeter that had this functionality.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
FWIW, I personally would not use an altimeter that had this functionality.
Then let the user decide. Make it an optional feature that could be turned on or off in setup, with the default "off." Akin to the "apogee delay" that is available in many altimeters.

That wouldn't be too hard to do.
 

Adrian A

Sponsor
TRF Sponsor
TRF Lifetime Supporter
Joined
Jan 22, 2009
Messages
2,317
Reaction score
289
Its happened to almost all of us who fly electronic deployment. Something unexpected happens and the altimeter suffers a hiccup in power that causes it to reset.

My question is, why don't the altimeters have an in-flight recovery feature? I'm not aware of any commercial altimeter that has this feature, but it seems like it wouldn't be too hard to code up.

You could store a simple variable in EEPROM that tells the flight computer what the last event was. Read this variable at startup and if it wasn't "touchdown," then it could attempt to recover by quickly running an emergency recovery routine. This routine could sample the sensors, and then based on what was found, determine where the rocket is in flight. If the sensors determine the rocket is on the pad, then go back to the beginning and start with a normal powerup sequence.
The Raven altimeters have a very large capacitor that allows the system to keep running even if there is a complete input power dropout for for 5+ seconds.

If an altimeter comes up from reset in the middle of the flight, it won't have enough information to safely deploy deployment charges based on the current state of its sensors. Non-volatile memory has limited write cycles, and if you are continuously writing state information like velocity during the flight, those will get used up fast. Making the altimeter electrically robust to power loss in-flight is IMO a much safer design strategy. There are so many corner cases to consider, even for in-flight deployments, that if an altimeter had to be prepared to fire charges any time it powered up, based on a few bits of stored data (how do you know was it stored properly from a real flight?) and a few seconds of sensor readings that don't include motor acceleration, there will be a lot of unintentional deployments.
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,144
Reaction score
1,407
In spacecraft design, the behavior in fault conditions is often altered by the value of "mission phase bits", which are stored in very reliable NVM, much like what you are proposing. When flight values are tested on the ground there are all kinds of hardware and software inhibits on pyros, actuators, etc. And I don't think these bits are ever set autonomously, only by explicit ground command....
All Eggtimer altimeters use "mission phase" indicators, and I would imagine that most others do too. In the case of the Quantum and Proton, power to the deployments is shut off until the rocket is in flight.
 

Saint_Tiki

Well-Known Member
Joined
Oct 6, 2020
Messages
54
Reaction score
55
I don't think it will dramatically increase risk.
A 1-2 second sample of accelerometers & the barometer should be enough to make the misidentification risk very small.
Define dramatically? 1/10? 1/100? Is that more likely than the current risk of failure you're trying to mitigate?

In any device I've ever used that has an accelerometer or a barometer, they require static conditions during start-up before they are ready to operate. Hot-starting an accelerometer in a random phase of flight seems very difficult to program. Same with a barometer. System would more than likely register that as a faulty sensor.

I think this software redundancy is possible but difficult to design and test. It would be a lot easier to just fix the bad thing (power loss) in the first place. Better components/switch. Redundant batteries or battery buses. Motor deploy drogue as backup. Backup altimeter.

If you can't fit backup altimeters or batteries inside (or don't wanna buy extras like me), then do it once, do it right. I hot glue every terminal block connection inside the AVBay. Old RC plane/quadcopter trick. Helps with vibrations.
 

Saint_Tiki

Well-Known Member
Joined
Oct 6, 2020
Messages
54
Reaction score
55
Making the altimeter electrically robust to power loss in-flight is IMO a much safer design strategy.
Yes, this is good. Fixing the loss of power situation in the first place should be 80% of the effort spent and 20% should be spent on a backup to that, like you're saying.

I don't know if capacitors would be enough though. If the system can run for 5 seconds on power loss, that's 5 seconds of what, 30mA draw? 100maA? Not much. Very unlikely there's enough juice in there to fire a pyro from the capacitor. I just imagine the power loss happening right before apogee, followed by a 1s pyro signal causing a brownout/shutdown of the altimeter.

It's like, your car is leaking fuel and you keep running out of gas. Yeah, you can put a jerry can in the trunk for when you run out. Or you could just fix the leak in the tank.
 

cerving

Owner, Eggtimer Rocketry
TRF Sponsor
TRF Supporter
Joined
Feb 3, 2012
Messages
4,144
Reaction score
1,407
A brownout cap may keep the electronics alive, but it's probably not going to fire the charges. If you don't get the laundry out, nothing else really matters... the best you're gonna get is the reason why, assuming that your electronics are still alive after you dig your rocket out of the ground. Good power connection and wire management, and redundancy, are the only way to prevent this... which is why it's required for a L3 cert.
 

mikec

Well-Known Member
Joined
May 9, 2009
Messages
2,504
Reaction score
384
All Eggtimer altimeters use "mission phase" indicators, and I would imagine that most others do too.
Obviously, but they are dynamic, not non-volatile, no? You always go back to the initial POR state when you power on.

Anyway, I think we're beating a dead horse here.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
Telemetry GPS is another scenario where in-flight recovery would be useful. Forget pyros - it would be useful to have the ability to resume telemetry in-flight so you know where to find it.

And the wear-out of NVM memory is still not an issue. You would only write when the phase changes, not continuously.
 

SparkyVTFlyer

Senior Member
TRF Supporter
Joined
Feb 8, 2009
Messages
203
Reaction score
27
Location
Yorktown, VA
Do we have any data on how often in-flight power loss happens? We know its relatively rare, but do any of the vendors collect meta-data of their users' flights? This would be very interesting analysis.
 
Top