NASA Shares Initial Findings from Boeing Starliner Orbital Flight Test Investigation

Winston

Lorenzo von Matterhorn
Joined
Jan 31, 2009
Messages
9,546
Reaction score
1,709
NASA Will Only Tolerate So Much Danger
An investigation into what really went wrong with Boeing's last space mission turned up serious issues.
10 Feb 2020

https://www.theatlantic.com/science/archive/2020/02/boeing-nasa-starliner-software-problems/606361/

A lot went right during a recent attempt to reach the International Space Station. A lot went wrong too.

The rocket launched just before sunrise on a cool, late December day, cutting a streak of gold across the sky in Florida’s Cape Canaveral. The capsule it carried, which was designed and built for NASA by Boeing, was smoothly delivered past the edge of space. If the test had gone off without a hitch, the next time this spacecraft flew, it would have had astronauts inside. The capsule was supposed to stay in space for a week and dock to the ISS. But two days later, the capsule was back on Earth with its parachutes strewed across the New Mexico desert. It was healthy and in one piece, but its cargo was undelivered, and its mission cut short.

Now NASA says the mission could have gone even worse.

An investigation of the bungled mission has revealed more problems than officials and engineers alike expected to find. The flaws stem not from hardware, but from the flight software coded by Boeing engineers. The capsule, known as Starliner, turned out to be more dangerous than anyone realized.
“We don’t know how many software errors we have,” Doug Loverro, NASA’s associate administrator for human exploration and operations, told reporters on Friday. “We don’t know if we have just two or we have many hundreds.”

[snip]

Timing is everything, especially in spaceflight, and that’s where Boeing’s Starliner first had trouble. Thanks to a software glitch, Starliner incorrectly set its internal clock hours before it launched, which meant that after it separated from the rocket and reached space, the capsule missed the moment it needed to fire some thrusters and push itself into the right orbit. In a cruel twist, mission control lost contact with Starliner just then because, it seems, of interference from radio noise on Earth, possibly from cellphone towers. By the time engineers could command Starliner again, the capsule, disoriented and idling, had used up too much fuel to finish its climb toward the ISS.

With no choice but to return Starliner home, Boeing engineers started combing through the software and found another issue. Before Starliner begins its final descent to Earth, it must shed a service module that helped nudge it toward the atmosphere. But the way the software sequence was set up, the thrusters on this module wouldn’t have fired correctly. A rocky separation could have destabilized Starliner, causing it to tumble. The two spacecraft could’ve even bumped into each other, in which case the impact could have damaged the heat shield. Starliner needs that shield to survive the fiery drop of reentry, with astronauts on board or not.

“It’s hard to say where the service module would have bumped, but nothing good can come from those two spacecraft bumping,” Jim Chilton, the senior vice president of Boeing’s space and launch division, told me.

Boeing engineers rewrote the software and sent the new version to Starliner barely three hours before the capsule touched down in New Mexico. If they hadn’t intervened, NASA says, Starliner could have been lost.


NASA Shares Initial Findings from Boeing Starliner Orbital Flight Test Investigation
7 Feb 2020

https://blogs.nasa.gov/commercialcr...-starliner-orbital-flight-test-investigation/
 

Winston

Lorenzo von Matterhorn
Joined
Jan 31, 2009
Messages
9,546
Reaction score
1,709


Paraphrased content from that video:

"One thing that does actually really concern me [is where] these bugs were found. They were part of the standard software execution [sequence]. They weren't some weird contingency [code sequence]. These were in the main execution path. These things should have been tested a million times and the fact that they weren't tested leads me to believe that the less used bits of the software are probably even worse tested and that is very, very worrying."

The fact that the nominal sequence code should so obviously have been so very well tested leads one to suspect that possibly compromised code could be involved. If so, we'd never hear about it. I've read claims that we are doing or trying to do that to Iranian and NK missile programs.
 
Last edited:

Reinhard

Well-Known Member
TRF Supporter
Joined
Jan 18, 2009
Messages
1,321
Reaction score
617
Location
Austria
Boeing already understands the issue with the clock. However, the required fundamental change will be expensive and take time. Therefore they have opted to add another system that will detect and fix clock anomalies. It works in the background, so the Astronauts won't even need to know about it. It's called Mission Clock Adjustment System...

/s
Reinhard
 

RocketGeekInFL

Well-Known Member
Joined
Nov 22, 2013
Messages
3,062
Reaction score
2,335
I am predicting it now, Boeing doesn't make manned flight before the end of next year. They have way to many issues to correct. I also will not be surprised if NASA requires another un-manned test flight before they man rate Starliner.
 

Peartree

Cyborg Rocketeer
Staff member
Administrator
Global Mod
Joined
Jan 5, 2009
Messages
6,763
Reaction score
2,545
Location
Alliance, Ohio
At least one industry blogger is saying exactly that. He predicts that NASA will (or at least should) require Boeing to fly another qualification flight AT THEIR OWN EXPENSE, before man-rating it.

 

ThirstyBarbarian

Well-Known Member
Joined
Feb 11, 2013
Messages
10,953
Reaction score
5,128
I am predicting it now, Boeing doesn't make manned flight before the end of next year. They have way to many issues to correct. I also will not be surprised if NASA requires another un-manned test flight before they man rate Starliner.

That's what I think too. It's going to take awhile to fix, and NASA absolutely SHOULD require them to retest it.
 

Winston

Lorenzo von Matterhorn
Joined
Jan 31, 2009
Messages
9,546
Reaction score
1,709
Going from one of the top U.S. companies, a major component of the DJIA, to a tarnished reputation in less than a year.
And ALL due to bad software. I wonder if there have been some major changes at Boeing in that department. Overworked? Experienced personnel going elsewhere to the many other aerospace startups. To SpaceX?
 

Speaknoevil

Well-Known Member
Joined
Mar 18, 2018
Messages
1,144
Reaction score
548
Boeing already understands the issue with the clock. However, the required fundamental change will be expensive and take time. Therefore they have opted to add another system that will detect and fix clock anomalies. It works in the background, so the Astronauts won't even need to know about it. It's called Mission Clock Adjustment System...

/s
Reinhard
Wow, this is a fracking band-aid approach. Software rigor and quality is needed, not another layer of software to determine how to start/monitor a clock--Jeez, this is 1950's level stuff--just fix it!
 

TSMILLER

Well-Known Member
TRF Supporter
Joined
Aug 31, 2014
Messages
1,928
Reaction score
1,475
Location
SW WA
Going from one of the top U.S. companies, a major component of the DJIA, to a tarnished reputation in less than a year.
It's taken longer than a year, but it has lost it's reputation. Since the buy out by MD the company has gone down hill so far not even the employees have anything good to say about it.
 

mjennings

Well-Known Member
Joined
Jan 17, 2009
Messages
1,970
Reaction score
258
It's taken longer than a year, but it has lost it's reputation. Since the buy out by MD the company has gone down hill so far not even the employees have anything good to say about it.
Umm if MD did the buy out why is Boeing the name that survived. It's been 25 years now. Can we get over this?

I'm excited to see Starliners next attempt. Hopefully they have all the software and hydrazine valve issues worked out!
 

TSMILLER

Well-Known Member
TRF Supporter
Joined
Aug 31, 2014
Messages
1,928
Reaction score
1,475
Location
SW WA
Yes it's been 25 years of having MD shoved down every ones throat. BA is what it is today because they abandoned BA method of doing business is favor of MD.
Umm if MD did the buy out why is Boeing the name that survived. It's been 25 years now. Can we get over this?

I'm excited to see Starliners next attempt. Hopefully they have all the software and hydrazine valve issues worked out!
BA is what it is today because they were forced by the MD crowd to abandon the BA way of business in favor of the MD way of doing business. Prior to MD it was about airplanes, quality and safety.
After the (per the inside scoop, MD buy out) the MD way was forced upon the company. Profit over engineering, airplanes and safety.
Look at the BA logo, even the Logo is MD Logo in front of what the BA logo was.
 

Rob Campbell

Well-Known Member
TRF Supporter
Joined
Mar 14, 2014
Messages
2,729
Reaction score
865
Umm if MD did the buy out why is Boeing the name that survived. It's been 25 years now. Can we get over this?

I'm excited to see Starliners next attempt. Hopefully they have all the software and hydrazine valve issues worked out!
Sometimes, it depends on the laws of the state each company is headquartered in. Recall the Estes/Centuri merger. On paper, Centuri bought Estes because Arizona tax/merger laws in Arizona were more favorable than Colorado's.
 

mjennings

Well-Known Member
Joined
Jan 17, 2009
Messages
1,970
Reaction score
258
I've heard the same arguments from legacy Mac employees about Boeing. The truth is somewhere in the middle, and both cultures took some of the worst and beat practices forward. And the making money, over aircraft quality and safety is an industry wide issue. When we are reduced to effectively 2 commercial aircraft manufacturers world wide (Boeing and Airbus) and 3 US military aircraft manufacturers. There is only the stockholders to please. The USG has to keep more than a sole source supply base so contacts are decided by whose turn it is to win. I believe it was Kelly Johnson who said "soon we will have one manufacturer who makes one aircraft a year" (I may not have the wording exactly right). He wasn't wrong! My point in calling out the blame rhetoric is that it doesn't fix anything and at this point it just serves to alienate young engineers who weren't born yet when the merger happened and annoy mid career engineers who have heard it for to long. Complaining doesn't fix the issue, sure a single engineer or work group can only do so much in a large company, but make your part better and hope others will take note and spread it.
 

OverTheTop

Well-Known Member
TRF Supporter
Joined
Jul 9, 2007
Messages
8,005
Reaction score
6,300
Location
Melbourne Australia
I have been hoping they are learning from all these "process indicators" and fix things before they become major problems. These are not problems but indication of where attention is needed.
 

Bill S

Well-Known Member
Joined
Aug 6, 2019
Messages
1,450
Reaction score
1,119
The three scariest words in flight... Software by Boeing.

You mean software created in India, by the lowest bidder, dontcha? :)

"I also will not be surprised if NASA requires another un-manned test flight before they man rate Starliner. "

I would like the .govt to require that the CEO and board of Boeing ride aboard the next test flight to ensure that the code is indeed all correct. They should have to put their lives in jepardy to prove that their taxpayer funded product is safe and effective. :)
 

cbwho

Well-Known Member
Joined
Mar 20, 2019
Messages
380
Reaction score
248
Location
MN
This is a fundamental problem in Boeing and not easy to fix.

I work in the software industry. Writing code is all about corporate culture as that drives processes and best practices.

The problem with the their plane that crashed and Starliner is fundamentally the same.
 
Top