A less well-known problem on Apollo 11 (as well as Apollo 12) was that erroneous data caused the LM’s engine thrust to fluctuate wildly. The landings were successful but telemetry coming back to Earth was alarming. The LM engine thrust was surging so vigorously, that the throttle control algorithm was only marginally stable!
The root of the problem was a design flaw that resulted from a miscommunication months prior.
The original version of the LM engine had a 0.3 second time delay between input command and throttle response and so the LM’s computer algorithm was written to cleverly overcome this.
Then a second brilliant programmer found an ingenious way to rewrite the software so there was only effectively a 0.2 second delay. So far so good.
HOWEVER, the LM engine had since been “redesigned” to cut this throttle response time to 0.075 seconds.
This small detail was simply ’missed’ by the programmers resulting in a near showstopper. Here’s the best part. Had the clever 2nd programmer not reduced the delay from 0.03 to 0.02 seconds [sic - should be 0.3 and 0.2], the throttle would have gone from metastable to COMPLETELY UNSTABLE and the landing(s) would have been impossible!
[snip]
Finally, a ten year old kept Apollo 11 from losing communication with Earth! As the three astronauts began their homeward journey, the Guam tracking station, which supplied communication on the final segment of Apollo’s flight, FAILED! A staff member employed his ten-year old son to do the necessary repairs that were only made possible with his very tiny hands!
The 10-year-old who helped Apollo 11, 40 years later
20 Jul 2009
(CNN) -- On July 23, 1969, as Apollo 11 hurtled back towards Earth, there was a problem -- a problem only a kid could solve.
It sounds like something out of a movie, but that's what it came down to as Apollo 11 sped back towards Earth after landing on the moon in 1969.
It was around 10:00 at night on July 23, and 10-year-old Greg Force was at home with his mom and three brothers. His father, Charles Force, was at work. Charles Force was the director of the NASA tracking station in Guam, where the family was living.
The Guam tracking station was to play a critical role in the return of Apollo 11 to Earth. A powerful antenna there connected NASA communications with Apollo 11, and the antenna was the only way for NASA to make its last communications with the astronauts before splashdown. But at the last minute on that night, a bearing in the antenna failed, rendering it nearly useless.
To properly replace the bearing would have required dismantling the entire antenna, and there was simply no time. So Charles Force thought of a creative solution: If he could get more grease around the failed bearing, it would probably be fine. The only problem was, nobody at the station had an arm small enough to actually reach in through the two-and-a-half inch opening and pack grease around the bearing.
And that's when Greg was called in to save the day. Charles Force sent someone out to his home to pick up Greg. Once at the tracking station, Greg reached into the tiny hole and packed grease around the failed bearing. It worked, and the station was able to successfully complete its communications role in the mission. Apollo 11 splashed down safely the next day.
At the time, Greg didn't think what he was doing was a big deal, and 40 years later, he's still modest about his role in the mission.
"That's all I did, was put my hand in and put grease on it," he says. If he hadn't been there, NASA would not have been able to make its last communications with the mission before splashdown, but Greg says "it wasn't life or death, [from] my understanding."
"My dad explained to me why it was important," he says, "but it kind of caught me by surprise afterwards, all the attention." iReport.com: Read Greg's firsthand account
That attention came from the media and even the astronauts themselves. Greg's small but important part in Apollo 11 was a story told by news outlets around the world. He even got a nice thank-you note from Neil Armstrong, whom he met when Armstrong went on a tour of NASA stations with the other astronauts to thank the staff after the mission. "To Greg," reads the note, which Armstrong wrote on a newspaper clipping of Greg's story, "with thanks for your help on Apollo 11. Neil Armstrong."
Further investigation on that throttle problem and actual Apollo 11 LM propellant remaining:
Don Eyles: "The Apollo 11 mission succeeded in landing on the moon despite two computer-related problems that affected the Lunar Module during the powered descent. An uncorrected problem in the rendezvous radar (RR) interface stole approximately 13% of the computer's duty cycle, resulting in five (1201 and 1202) program alarms and software restarts during the descent. In a less well-known problem, caused by erroneous data, the thrust of the LM's descent engine fluctuated wildly because the throttle control algorithm was only marginally stable."
Apollo 11 (LM-5) /5/
At landing, the quantity of propellants remaining in each of the four tanks was as follows.
1. Oxidizer tank 1 : 5.7 percent
2. Oxidizer tank 2: 2.7 percent
3. Fuel tank 1: 3.9 percent
4. Fuel tank 2: 3.9 percent
The minimum-usable quantities, based on the indication of oxidizer tank 2, allowed a remaining hover time of 63.5 seconds.
During the last 140 seconds of the PDI burn (hover phase), a series of large oscillating-throttle changes occurred (figure below). These changes were approximately 15 percent peak-to-peak about a nominal throttle setting of approximately 26 percent.
Large and rapid changes in vehicle attitude in pitch during the final phases of landing (Eyles: "Armstrong switched the autopilot from AUTO to ATT HOLD to manually fly over the rocky area") caused centrifugal accelerations on the inertial-measurement-unit accelerometers located at the top of the ascent stage high above the vehicle c.g., thereby giving a false indication of vertical acceleration. This false indication caused the lunar guidance computer to command a throttle change to compensate for an unreal change in vertical acceleration. Later, this problem was investigated under the analysis of the guidance and control system.
The oscillatory character of the P66 throttle command was apparently due to the actual value of the descent engine time constant being smaller than that assumed. And so it was: the performance of the descent engine had been improved, but the ICD was not modified accordingly. The actual time lag for the descent engine was only about 0.075 seconds when it was assumed to be 0.3 seconds. Despite of that both Apollo 11 and 12 flew with 0.2 seconds of compensation for a 0.3 second throttle delay. As a result the throttle was barely stable (until later missions 14, 15, 16 and 17).
Poster: indy91
The throttle-command routine of the LGC actually accounted for three different types of lags/delays. First the time between reading accelerometers and issuing throttle commands. This is due to AGC not being all that fast. Second the dynamic delay of the engine. This is the number they got wrong and was 0.2 seconds on Apollo 11 and was corrected to 0.08 seconds for Apollo 14. Thirdly the Descent Engine Control Assembly (DECA) converts the digital throttle commands from the computer to a voltage for the throttle actuator. And this voltages can only ramp up and down at about 85% thrust per second.
I can tell you exactly what happens if neither the engine or DECA would have any delay. The throttle oscillations become catastrophic. I know because I have simulated this. I am one of the developers of Project Apollo - NASSP (https://nassp.sourceforge.net), an open source project to simulate the vehicles of the Apollo program. And we are using the Virtual AGC emulator ( https://www.ibiblio.org/apollo/ ) for our simulation. The Apollo 11 source code is available through this emulator, so we are using 100% identical software as compared to the flown Apollo 11 mission. Don Eyles himself actually supplied the source code for most Lunar Guidance Computer (LGC) software versions to the Virtual AGC project.
So, when we first tried the Apollo 11 lunar landing, we did get catastrophic throttle oscillations in Program 66 (Rate-of-Descent landing mode). The first delay I mentioned above is simulated in the emulator, so it is accounted for. As the next step we added the throttle lag in our simulated DECA and that fixed most of the oscillations. So even without simulating the actual delay of the engine response (which was 0.075 seconds), we can get a fairly stable behavior. As compared to the graphs on Don Eyles website we still get slightly higher oscillations, but it is not a problem for landing. We haven't yet added the throttle delay of the engine itself, since we get decent results without it. Here is a video of the simulated Apollo 11 landing I made a while back: The video shows an automatic landing, which uses a different throttle algorithm and does not have any throttle oscillation. The video was recorded before the DECA delay was implemented that made Program 66 with the semi-automatic landing mode work. P66 is what all flown missions used to land, including Apollo 11.
We do have the ability to modify AGC versions. So what I will try is change the THROTLAG variable which is 0.2 seconds in the Apollo 11 AGC version and change it to the even more wrong 0.3 seconds. If the oscillations are still managable then we can conclude that even when accounting for the whole 0.3 seconds the throttle would still be stable enough to land. If the throttle fluctuation becomes catastrophic, then we need to also simulate the delay of the engine itself to come to a conclusion. Maybe I will record another video showing the throttle behavior with and without simulating the DECA and descent engine delays. But I will certainly report back what happens when the computer accounts for 0.3 second engine delay.
--------
So I have tried the landing with the full, assumed 0.3 seconds engine delay compensated. The oscillation has gotten much worse. It's now cycling between 10% thrust (the minimum possible) and 50%.
One of the important parts of the semi-automatic Rate-of-Descent (ROD) landing mode is that you need to be able to judge your altitude rate. Either from the computer display on the DSKY or direct Landing Radar measurements on the tape meters. With the knowledge of the altitude rate you can then adjust this rate with a switch, incrementing or decrementing the altitude by 1 feet per second with each switch actuation. This capability is severly degraded with oscillations this high. Due to the low gravity the cycling between 10% and 50% throttle doesn't immediately make you crash into the surface though. The altitude rate is cycling between about 1 and 5 feet per second. I levelled off at about 50 feet and then mostly had to judge my descent by looking outside, doing only a few adjustments with the ROD switch. I did not have to use manual throttle for the landing, 1-5 ft/s descent rate is in the acceptable margin. So I was able to land in the ROD mode despite the large oscillations, with a bit of trouble, but all in all still fairly safely.
If the Apollo 11 astronauts had encountered this situation then they might have opted to use manual throttle instead of using the computer controlled auto throttle mode. And as Don Eyles said in his "LM Tales", with the previous alarms they would have lost some confidence in the computer and might have aborted anyway.
There are (at least) two more effects that our simulation currently doesn't consider. First, as I mentioned in my previous post, the actual engine delay of about 0.075 seconds is not simulated. This delay would of course have reduced the oscillations, because it is closer to the 0.3 seconds estimate. So the throttle oscillation due to the wrong estimate would have been smaller. The other effect is "IMU bob", which was also mentioned in the article by Don Eyles. The IMU is not at the center of gravity of the LM and with large attitude rates your IMU therefore measures an acceleration. We also do not currently simulate this effect. Apollo 11 might have gotten the maximum throttle oscillation possible (10% to 100%) due to this. Just before the landing you wouldn't command large attitude rates anyway, but initially in Program 66 this can cause big issues. So the "IMU bob" effect makes an abort even more likely.
Sunburst and Luminary: An Apollo Memoir
by Don Eyles
In 1966 the author, newly graduated from college, went to work for the MIT laboratory where the Apollo guidance system was designed. His assignment was to program the complex lunar landing phase in the Lunar Module's onboard computer. As Apollo 11 approaches, the author flies lunar landings in simulators and meets the astronauts who will fly the LM for real. He explains the computer alarms that almost prevented Neil Armstrong from landing and describes a narrow escape from another dangerous problem. On Apollo 14 he devises a workaround when a faulty pushbutton threatens Alan Shepard's mission, earning a NASA award, a story in Rolling Stone, and a few lines in the history books. This memoir is a new kind of book about Apollo. It tells a story never told before by an insider the development of the onboard software for the Apollo spacecraft. It makes a vertical connection between technical details and historic events, but by broadening the story using his own experiences as he grows into adulthood in the 1960s the author draws a parallel between that era of successful space exploration, and the exploration, inner and outer, that was taking place in the culture.
A few weeks before the mission, United States intelligence agencies advised NASA security management that the Black September terrorist organization might be targeting the final lunar mission for some kind of attack. Cernan recounted his experiences in his excellent autobiography, The Last Man on the Moon. At first, NASA officials decided not to tell the crew of the threat since they already had an ambitious lunar mission to carry out — the longest and most challenging of all the Apollo flights. But then one day Cernan saw a noticeable change in security measures at the Cape.
