Deeply flawed NARAM 46 R&D report

The Rocketry Forum

Help Support The Rocketry Forum:

This site may earn a commission from merchant affiliate links, including eBay, Amazon, and others.

UhClem

Well-Known Member
Joined
Feb 6, 2009
Messages
2,052
Reaction score
443
While at NARAM 46 I made a point of attending the B division R&D oral reports because one title caught my attention:

"All-Fire/No-Fire Tests Estes Igniters using the Neyer D-Optimal Test" by Cody Steele

At the time I was fairly certain that I was likely the only person in the
audience who had even heard of this test let alone read the report:

https://neyersoftware.com/Papers/D-Optimal/D-Optimal.htm

(Note: Cody's report does not reference this report but another one which is
only a summary of different methods and does not detail the methods.
https://neyersoftware.com/Papers/EandP99/ISO14304.htm )


It had been a while since I read this document (from Mr. Neyer's web site) so I was a little fuzzy on the details. But after listening to Cody's oral
presentation I was skeptical that he had followed the Neyer D-Optimal method. Now that Cody's written report is available through NARTS I have confirmed that not only did Cody not use the D-Optimal method but the method he did use is deeply flawed and the results highly suspect at best but more probably totally worthless.


Some background.

The D-Optimal test is a variation on the Bruceton sensitivity test. The
variation being in how the exposure levels are determined. This type of test is used on igniters because exposure to currents lower than that required to fire them may alter their characteristics. Thus after one exposure the test article is thrown away regardless of if it fired or not.

The Bruceton test requires that you provide an estimate of the average and
standard deviation of firing current. The first test is conducted at the average
firing current. If the igniter fires, the current is reduced by one standard
deviation and the test repeated. If the igniter doesn't fire then the current is
increased by one standard deviation and the test repeated. This continues until an adequate sample has been gathered (forty or more).

The result of this test is a set of data points which consist of exposure levels
(currents) and a binary pass/fail. Obviously you cannot compute the sample
average and standard deviation from these by the normal methods. The method most commonly used is called Maximum Likelihood Estimate and the math is not simple.

The D-Optimal method uses a different algorithm (see the report) to determine the exposure levels. If you have estimated the average and standard deviation badly the Bruceton test does not work well. The D-Optimal test is insensitive to this sort of problem.


Did Cody follow this method? No.

The report states that an igniter was exposed to a given current and this
current was increased until the igniter fired. A procedure completely at odds
with any sensitivity test.

How were the computations done? I asked and was told that Microsoft Excel was used which cannot perform the required math.

Let's take a look at one of the data sets. This is the first set which had the power supply set to six volts (Yet another problem with the report. Not understanding how current limited power supplies and Ohms law work.)

1.21 pass
1.15 pass
1.13 pass
1.02 pass
1.01 pass
0.97 pass

0.94 fail
0.94 fail
0.94 fail
0.89 fail
0.89 fail
0.89 fail
0.80 fail

0.90 average
0.05 standard deviation

If the Neyer method had been used, the test results would have shown the passes and failures mixed up rather than grouped as we see here. But the data could have simply been sorted for presentation. This data also shows another problem. The Neyer test sequence is chosen to create data where the successes and failures overlap. In other words there should be a success (pass) at a level lower than at least one of the failures. This data set does not have this property so the Neyer method could not have been used. I duplicated the average and standard deviation from the report by computing the average and standard deviation of just the failures. An odd result.



Because the NARAM 46 R&D judges were not familiar with the Neyer D-Optimal method not only was Cody awarded first place in B division but he also received a large cash prize.

For work that is basically junk. Worse, the report shows that the author had at least read about the D-Optimal method and then knowingly presented test results that did not even come close to following that method while claiming that they did.
 
Worse, the report shows that the author had at least read about the D-Optimal method and then knowingly presented test results that did not even come close to following that method while claiming that they did.

Can you prove he did it knowingly? Since the report doesn't cite the document that you did, can you prove he actually had correct interpretation of the method and ignored it? I mean to do it knowingly would be a huge gamble, wouldn't it? What if you had been one of the judges?
 
Have I asked Cody about this? During the public Q&A portion of the R&D oral
presentations I asked (as near as my memory serves) if he used Neyer's software package as the computations are quite complicated. The answer was no. I was rather surprised at that. I suppose that I should have pushed him with further questions but time was limited.

I have expressed my concern about the report to Bill Spadafora (NARTS) in
explaining my eagerness to see this collection.


Is this just a case of misunderstanding or a deliberate fluffing up of a report?

Here is the description of the Neyer test from the cited reference:

"3.4.3 Neyer D-Optimal Test. This test was designed to extract the maximum
amount of statistical information from the test sample. Unlike the other test
methods, this method requires detailed computer calculations to determine the test levels. The Neyer D-Optimal test uses the results of all the previous tests to compute the next test level.

There are three parts to this test. The first part is designed to "close-in" on
the region of interest, to within a few standard deviations of the mean, as
quickly as possible. The second part of the test is designed to determine unique estimates of the parameters efficiently. The third part continuously refines the estimates once unique estimates have been established.

This test requires the user to specify three parameters, i.e., lower and upper
limits, and an estimate of the standard deviation. The first two parameters are used only for the first few tests (usually two (2) tests) to obtain at least one fire and one fail to fire. The estimate of the standard deviation is used only until overlap of the data occurs. Thus, the efficiency of the test is
essentially independent of the parameters used in the test design."

Note that there are no details of the test. Which requires a fairly complicated
flowchart to describe in another paper.

What is in the R&D report?

"In order to find these numbers I used a method called the Neyer D Optimal Method."

and

"Methods

The Neyer D-Optimal test was designed to extract the maximum amount of amount of (sic) statistical information from the test sample. The test requires the user to specify three parameters, i.e. lower and upper limits, and an estimate of the standard deviation. The first two parameters are used for only the first few tests to obtain one fire and one fail to fire. He (sic) estimate of the standard deviation is used until overlap of the data occurs. Thus, efficiency of the test is essentially independent of the parameters used in the test design.

1. Attach variable power output source to battery connectors on Estes Launch controller.
2. Turn on power.
3. Change voltage to 6 volts.
4. Separate igniters.
5. Attach controller clips to one igniter.
6. Press button on launch controller.
7. If igniter does not light add amps on variable output device.
8. repeat steps 4-7 until the igniter lights.
9. If igniter lights on the first try turn amps down and repeat steps 5-6 until
it does not light within one second.
10. repeat steps 5-9 until the lowest amps are found that ignite the igniter
within one second.
11. repeat steps 4-10 on volts of 9 volts and 12 volts with the quest controller and the 12-volt controller.
12 To find the no-fire situation find the highest amperage in which the igniter
will not light or smoke within thirty seconds.
13. Record all data."


In preparing this message I noticed that Cody quoted a long passage of the
referenced report without any attribution. This is a serious problem as well.
Plagiarism is the usual term although with the reference at the beginning an
appropriate response becomes fuzzy.


I would expect that in this report that the three parameters required by the
Neyer D-Optimal test would be included. They are nowhere to be found.

Compare this test method to the flowchart in Neyer's report and you will see
that they have nothing in common. It would appear that Cody used a simple but flawed test method and then fluffed up his report by claiming to have used a different method. The result being a report that looks more authoritative than it really is.

Or perhaps he really did use the complex Neyer D-Optimal test and coaxed MLE's out of Excel but wrote such a poor report that you can't tell that he did.
 
Are there not proper channels for this sort of thing?

It seems that presenting this on a forum that is not monitored by the person in question (namely, Cody Steele) in an environment where he will not respond is improper.

Just my humble opinion...
 
In preparing this message I noticed that Cody quoted a long passage of the
referenced report without any attribution. This is a serious problem as well.
Plagiarism is the usual term although with the reference at the beginning an
appropriate response becomes fuzzy.

I don't want to comment on the appropriateness of the report or of discussing it here. However, if by "without any attribution" you mean that exact words were used without quotation marks, then it's plagiarism whether there's a reference given or not. The reference indicates the source of the ideas, but it doesn't entitle the author to use the exact words of the source. Only quotation marks do that.
 
Originally posted by jflis
Are there not proper channels for this sort of thing?

It seems that presenting this on a forum that is not monitored by the person in question (namely, Cody Steele) in an environment where he will not respond is improper.

Just my humble opinion...
I tend to agree with Jim. I mean, we could discuss it here till we're blue in the face, but I'm not sure I see the point of such a discussion here on TRF.
 
David,

You need to bring this up to Cody Steele and the Contest Board, as this forum is really not the place for such matters. It serves no purpose to make a post here that would never be contested. Moreover, the forum is not a place to rant or vent your problems, although members tend to forget this at times. The primary purpose of this forum should be disccusions related on how to design, build and fly rockets, along with supporting equipment and vendor relations. We all need to remember it is a hobby, which is supposed to include fun in there somewhere. That's why they are called hobbies! :)

I suggest you visit the NAR site for contact information and continue from there; they would have to decide how to proceed.
 
Back
Top