Revision to Tripoli Rule Regarding Wireless Remote Switches

g.pitts said:

If Tripoli is interested in going down the path of a DFMEA analysis being a part of a "self-certification" process without owning a certification responsibility, I would think that a committee would be formed to put the process together, and the vendors would own the analysis of their designs. Is that what you were thinking as well, or am I off-base? But the offer stands to help in any way I can.

Click to expand...

I wish I knew more about the process and how it all works. I have no idea even if the manufacturers are willing to participate.

Steve Shannon said:

I wish I knew more about the process and how it all works. I have no idea even if the manufacturers are willing to participate.

Click to expand...

The process isn't that complicated, but it is tedious. In the case of an altimeter, a hardware DFMEA comprehends every "pin" of ICs and discrete components and what happens should the pin fail by opening, shorting (to ground or to V+ usually), short to an adjacent pin, etc. Spreadsheets work well for this purpose. Numerical scores are provided to each issue along three areas: (1) the probability that such an event will occur; (2) the severity of the event should it occur; and (3) and whether such an event is readily detectable (is it caught in manufacturing / test, or is it a silent issue). The "risk priority number", or RPN, of each issue is then scored by multiplying the three values together. Usually design mitigations are mandated for a particular failure scenario where the severity exceeds a certain value. The challenge will be to establish clear and unambiguous scoring criteria that is simple and ensures reasonably consistent (not necessarily exact) scoring between two designers.

Note that DFMEAs do not directly address software. That's another can of worms, but relevant since errant software (be it a bug in an algorithm or software that "gets lost" and begins executing random code or data) can fire a deployment charge with perfectly operational hardware.

g.pitts said:

Note that DFMEAs do not directly address software.

Click to expand...

I think Tripoli would be remiss if they didn't include software FMEA (from a black box perspective), and especially process based FMEA. Basically the hardware design could be "perfect" but hard for the user to understand how to exactly use it, leading to errors. I'd be willing to bet that we have several orders of magnitude more "process errors" with altimeters than hardware/software errors.

djs said:

I think Tripoli would be remiss if they didn't include software FMEA (from a black box perspective), and especially process based FMEA. Basically the hardware design could be "perfect" but hard for the user to understand how to exactly use it, leading to errors. I'd be willing to bet that we have several orders of magnitude more "process errors" with altimeters than hardware/software errors.

Click to expand...

The Tripoli BoD will need to decide how deep they want to go.

Should they become involved in the software aspect of an altimeter, is it requiring certain best practices, or more? The last program my team at the previous company I worked had to implement (by virtue of IEC 60730, which was adopted in-part by UL for the class product we were developing) an environment check to ensure the correct operation of RAM, FLASH, ADC, timers, etc. Does Tripoli want to go that far? Doing so would impose a lot of overhead for a small microcontroller that may have consequences ranging from obsoleting current products, price increases, to one or more vendors just not playing along with any incremental requirements. There are a lot of implications to these decisions, and I'm sure that's what Steve and company are thinking through.

I can think of no simpler way to destroy the hobby than impose this level of certification.

I guess we could all go back to using motor eject.

mikec said:

I can think of no simpler way to destroy the hobby than impose this level of certification.

I guess we could all go back to using motor eject.

Click to expand...

Indeed. That's the concern, no doubt.

g.pitts said:

Should they become involved in the software aspect of an altimeter, is it requiring certain best practices, or more?

Click to expand...

Personally I think no, because at a macro level, the only software bugs that I've seen in altimeters in the past few years are edge cases and not standard use.

I think the first answer of an FMEA should consider "we know there's a problem here, but we're not going to do anything about it" as a valid response. Like I alluded to above, I think process issues (how someone wires up an av bay and tests it, etc) are much more fault prone than the altimeter hardware or software.

djs said:

Like I alluded to above, I think process issues (how someone wires up an av bay and tests it, etc) are much more fault prone than the altimeter hardware or software.

Click to expand...

^^^^THIS^^^^
90% of problems are user error.

Speaknoevil said:

^^^^THIS^^^^
90% of problems are user error.

Click to expand...

I absolutely agree with that. Fortunately there are ways to reduce some of that, but never all.

mikec said:

I can think of no simpler way to destroy the hobby than impose this level of certification.

I guess we could all go back to using motor eject.

Click to expand...

I agree; we simply cannot impose that level of oversight and I don’t think we should. I’m simply wondering what we can do together to make incremental improvements and establish better definitions.

I liked term “inhibited”.

Regarding NFPA 1127 section 4.13.7:

Steve Shannon said:

But, again, it wasn’t a reinterpretation; we checked with the person who wrote the requirements to see what the correct original interpretation was.

Click to expand...

Did that person have any explanation or comments as to why the rule was written using the word "inhibited" rather than wording such as "physical break" or "physical disconnect"?

VernK said:

Regarding NFPA 1127 section 4.13.7:


Did that person have any explanation or comments as to why the rule was written using the word "inhibited" rather than wording such as "physical break" or "physical disconnect"?

Click to expand...

That’s a good question, Vern. Because I knew he works with missiles as a day job, I made the assumption that it came from that perspective. I’ll ask.

As someone who has worked on a lot of electronics and computer repair issues, I see far more faults in mechanical parts. I chose to switch to eggtimer quantum and proton as I felt the design led to safer setups. Both from switch issues and electrical issues. They handle reverse polarity without firing, have switching on both rails, and require a code entered over an encrypted wifi connection to test fire or arm for flight. I also test them with rapid power cycling and RF to ensure each build works under stress.

The proper pre flight in my mind connects bare ematches for testing on the day of flight. So you can be sure you don't have any shorted channels. Then power down, assemble charges and recovery gear. At this point I don't see any reason not to power the altimeters, but I would never arm them off the pad. However, if that's the rule I'll follow it. It may mean disassembling the ebay on the range as I avoid physical switches for the reasons mentioned. This adds a few minutes to the time I need to spend on the range.

For staging and air starts, I think more caution is needed and ignition devices should not be installed until on the pad and pointed safely. This might cause yet more range time. I think this makes more sense as I consider deployment charges going off to be mostly irritating, though there is some danger. Rockets should generally be pointed safely for deployment events, like a firearm, in my mind. However, we all know things happen. Even a smaller motor starting up is likely to cause injury.

Just a few thoughts from a member. Hopefully they came across respectfully as intended. I don't envy the BOD trying to herd us cats.

Seems like we are not trusting solid state devices when the rest of the world is going to them as quickly as possible. https://www.solidstatecontrolsinc.com/applications/powergen

It does remind me of the time when I was responsible for patching and debugging the operating system for a multi million dollar mainframe and it halted one day with an error code. The dumps from this machine take 2 boxes of paper and normally took me 2 to 3 days to decipher and figure out what the problem was. This particular case ended up with one bit changed in the opening system code and it was protected memory that nothing could write into. The conclusion I had to accept was that cosmic rays had hit the memory cell and changed it. This was the only time in my 20 years doing that job that I had seen that.

Update, the error code was an invalid instruction, looking at the source code and another dump showed that indeed the protected memory cell had flipped a bit, so something caused one bit to flip

Steve Shannon said:

That’s a good question, Vern. Because I knew he works with missiles as a day job, I made the assumption that it came from that perspective. I’ll ask.

Click to expand...

Also ask him if he ever considered that electronic switches were a possibility... he may not have even thought of them.

Better yet, invite him to a meeting with the manufacturers to understand how the phrasing is causing issues, and see if all can agree on a new phrasing that is clear to all without having to go through an "interpretation" phase.

I don't think a complicated failure analysis is necessary to determine if MOSFET/FET switching technology is a safe "off" mechanism for altimeters. Here is a resource that helps engineers understand basic MOSFET failure modes and perform basic reliability estimates an-976.pdf "Understanding and Using Power MOFSFET reliability data". Aside from manufacturing defects which would cause a failure in a very short time, MOSFET failures are largely driven by switching on/off cycles, overheating, and over voltage. A properly designed circuit and appropriately chosen component can mitigate these along with burn-in testing.

The article uses has an example with a target failure rate of 11.4 in 10^9 hours of operation. Come on, a rocket altimeter may at most be switched on/off a few thousand times and operated for a 1000 hours. We are talking about extremely low failure rates for these devices in properly designed circuits.

Outside of the cockpit, do we see mechanical switches in spacecraft, avionics, and automobiles in 2021? I suspect the answer is no. Granted these safety critical applications perform FEMA and use rigorous process for mechanical, electrical and software design. I am not advocating that we do that for the hobby. But they don't have a loophole on special circuits either. Yes, they may use hardened semiconductors. But I think the answer we are trying to drive toward is "how much is enough" and can a vendor demonstrate they meet the "enough" for solid state switches.

So take a look at the article, vendors and BOD. I think a few calculations from this article would go a long way in giving us confidence in an answer. Maybe vendors can get an qualified engineer to review their circuit designs (and respecting NDA and proprietary information). Code review. Probably not ... system behavior can be witnessed by demonstration. Again, we are not safety experts, nor do we particularly need to be, aside from a few basics. But we do have a lot of very talented engineers in this forum. Let's put our heads and collective wisdom together and work toward an answer.

I'm sure this has been mentioned previously, but I think the potential exposure of these devices to corrosive gasses should be considered, and I have it on good authority that some people will even fly an altimeter that might have experienced a hard landing at some point. What the vendors send out is not necessarily what is in the rocket.

Jim

there is also the probability that people design their own electronic devices (and I know some that do)

but my point is if the rest of the world trusts solid state switching, why can't we?

all computers use solid state switching, so do we now not trust any rocsim or finsim calculations?
are we back to using slide rules and paper and string tests?

you better not drive any modern car that has a computer inside or the gps or phone

jbr said:

if the rest of the world trusts solid state switching, why can't we?

Click to expand...

I can only speak to the specific case of the Featherweight Power Perch since it's the only one I've used and examined in detail.

I'm a professional EE and I have used and trusted the Power Perch since it came out. I've asked Adrian some questions about how it works and I was satisfied by those answers, and by my own test usage of the device, that it was as safe to use as a mechanical switch and much less prone to wiring errors. If it failed, it would either fail to power the altimeter (easy to detect) or fail to turn it off (also easy to detect in my initial prep.) Given how hard it is to trigger intentionally I think there's very little chance of doing so unintentionally, much less than some mechanical switches I've used.

I'd be happy to do an independent FMEA on it if that would help the situation.

The BOD hasn't explained what the path forward is, apart from asking the manufacturers to provide information (what information? To be used how?) I have already expressed my strongly-held opinion that NFPA 1127 with the word "inhibit" has empowered TRA to allow the magnetic switch, regardless what the author originally meant (that language was approved by committee so it doesn't come down to that one person's opinion under any circumstances.)

cerving said:

Also ask him if he ever considered that electronic switches were a possibility... he may not have even thought of them.

Click to expand...

He actually told me a bit of an anecdote about a problem with a solid state switch on a Tomahawk. It was one of the days I was sickest and I don’t remember the details, but I think it was found to be sensitive to interference from other nearby electronics. As a matter of fact, on March 5th at NARCON they’re going to have a presentation on electronics and interference I believe.

this apparently has caused a run on eggtimer wifi switches
my order has been on backorder

so expect a lot more of these at launches

Steve Shannon said:

it was found to be sensitive to interference from other nearby electronics

Click to expand...

Keep in mind that altimeters are already solid-state. Should we be expecting a directive from the BOD next that we can only use mechanical altimeters? Or we can't use radio tracking? You'll have a hard time closing that box if you open it.

mikec said:

Keep in mind that altimeters are already solid-state. Should we be expecting a directive from the BOD next that we can only use mechanical altimeters? Or we can't use radio tracking? You'll have a hard time closing that box if you open it.

Click to expand...

You like to go negative, don’t you?

I’m done with this conversation. I’ve given all the answers I have and some of you are saying or asking things that I’ve already responded to some time back.

Steve Shannon said:

You like to go negative, don’t you?

Click to expand...

I'm just trying to emphasize that IMHO there are some inconsistencies in what some decisions might imply. I'm sorry if I went over a line. Again, I appreciate the time you've taken to explain the BOD position, and my offer to do a detailed analysis on the Featherweight switch still stands. (And I do agree that this thread is repeating the same things, I'm certainly guilty of that.)

JimJarvis50 said:

I'm sure this has been mentioned previously, but I think the potential exposure of these devices to corrosive gasses should be considered, and I have it on good authority that some people will even fly an altimeter that might have experienced a hard landing at some point. What the vendors send out is not necessarily what is in the rocket.

Jim

Click to expand...

True, which is why I recommend powering up with bare ematches before every flight... no real safety issue if one pops. It's just common sense... like testing your batteries, making sure any terminal blocks are tight, etc. Electronic deployments take time to prep... going from motor deploy to electronic deploy requires a bump up in discipline.

mikec said:

(And I do agree that this thread is repeating the same things, I'm certainly guilty of that.)

Click to expand...

I think we can all agree the horse has been beaten into a buttery smooth paste at this point. There were plenty of us helping.

Safety agencies , ul, tuv, ce etc trust semiconductors. What they do not trust is software. Also safety agencies require devices to safely withstand 8kv electrostatic discharge while operating to stay safe.

It's not the reliability of the pn junction that is the risk here.