How to Nail Your Next Product Launch or Job Interview

We don’t rise to the level of our expectations. We fall to the level of our training.

—UNKNOWN

Millions of Americans had been waiting for this moment.1 A promise made by a young president, a revolution of cosmic proportions, was about to be realized.

The launch was woefully behind schedule. Months before the official launch date, concerns had been raised about readiness. Yet, officials turned a blind eye and hoped the glaring problems would somehow correct themselves. They were advised to delay or abort the launch altogether, but they passed. Stress tests performed just one day before the launch date revealed a lingering flaw that could compromise the entire mission.

But the test results were ignored. In a rush to launch by the tight deadline, officials pulled the trigger. As the data began to flow in, the engineers’ screens told a rapidly unfolding life-and-death story. They watched, with jaws dropped, as everything turned red.

Catastrophe ensued. Shortly after launch, it crashed and burned.

This was no rocket launch. Instead, it was the unveiling of healthcare.gov—a centerpiece of the Affordable Care Act—a landmark piece of legislation enacted during President Barack Obama’s term to provide affordable health insurance to Americans. The legislation was the promise, and the website was the fulfillment—or was supposed to be the fulfillment. Americans would use the website to shop for and purchase insurance.

Plagued by technical problems, the website crashed as soon as it launched. Users were unable to perform basic functions like create new accounts. The website miscalculated health-insurance subsidies and sent users into inescapable loops. Only six people were able to sign up for insurance on the first day of the website’s operation.

How was healthcare.gov—so critical to the success of the Affordable Care Act—so badly botched? Why did a platform that cost nearly $2 billion fail to perform basic commands?

Rockets and websites are different beasts, but they have at least one thing in common. Both need realistic testing, captured in the rocket-science principle test as you fly, fly as you test.

The ideas developed in the book’s first part (“Launch”) need tests that can reveal their weaknesses. That means checking the conditions, the interactions between components, and the testing equipment itself—as the $1.5 billion Hubble Space Telescope illustrates. The same questions apply to rehearsing an interview or trying a product with customers.

The Problem with Tests

Most of our decisions in life are based not on tests, but on hunches and limited information. We launch a new product, we change careers, or we try a new marketing approach—all without a single experiment. We blame a lack of resources for skipping the testing but don’t recognize the costs of new approaches that end up failing.

Even when we conduct tests, we perform superficial dress rehearsals that double as exercises in self-deception. We conduct tests—not to prove ourselves wrong, but to confirm what we believe is true. We tweak the testing conditions or interpret ambiguous outcomes to confirm our preconceptions.

Professors at Wharton and Harvard surveyed thirty-two cutting-edge retail companies to study their testing practices.2 The researchers found that 78 percent of the firms tested new products in stores before launching them. Although that number is impressive, the testing conditions were not. According to the researchers, the companies believed “their products will sell well despite unfavorable test results” and blamed “the weather (bad or good), the poor choice of test sites, the inferior execution of tests, and other factors for suboptimal sales.”3 In other words, the retailers made the test fit their expectations rather than adjust their expectations to fit the test results.

In a well-designed test, outcomes can’t be predetermined. You must be willing to fail. The test must run forward to shed light on uncertainty, rather than run backward to confirm preconceptions. Feynman said it best: “If it disagrees with experiment, it is wrong. In that simple statement is the key to science. It doesn’t make any difference how beautiful your guess is, it doesn’t make any difference how smart you are, who made the guess, or what his name is—if it disagrees with experiment, it’s wrong.”4

Self-deception is only part of the problem. The other part is the disconnect between testing conditions and reality. Focus groups and test audiences are often placed in artificial conditions and asked questions they would never get in real life. These “experiments” may measure reactions to the test setting rather than behavior in actual use.

Rocket science offers a way forward with a deceptively simple principle: test as you fly, fly as you test. According to the principle, experiments on Earth must mimic, to the greatest extent possible, the same conditions in flight. Rocket scientists test the spacecraft as the spacecraft will fly. If the test is successful, the flight must take place under similar conditions. Any significant deviance between the test and the flight can cause catastrophe—whether it’s a rocket, a government website, your job interview, or your next product.

A useful test checks whether a design works and actively looks for failure modes. Stress tests can help find its breaking point.

Breaking Point

The best way to determine an object’s breaking point is to break it. Rocket scientists use stress tests on Earth to expose flaws before they cause a failure in space. This objective requires exposing every component, down to the screws, to the same type of shocks, vibrations, and extreme temperatures awaiting them in space. Scientists and engineers must think through all the ways that they can trick these components and lines of computer code into committing fatal errors.

This approach also has the benefit of reducing uncertainty, to return to an earlier chapter. Testing can help turn unknowns into knowns. Each test, if performed under similar conditions to flight, can teach rocket scientists something new about the spacecraft and prompt them to tweak a piece of software or hardware.

But even in rocket science, the testing conditions often aren’t identical to the actual launch. There are certain things that we physically can’t test for on Earth. For example, we can’t mimic the exact same gravitational forces a rocket will experience during launch. We can’t completely simulate what it will be like to drive on Mars. But we can come close.

When I was working on the 2003 Mars Exploration Rovers mission, we periodically took a test rover for a spin around the Mars Yard—a tennis-court-sized area at JPL filled with the same types of rocks you might find on the red planet. The test rover was lovingly named FIDO, short for Field Integrated Design and Operations.5 We also took FIDO to places such as the Black Rock Summit in Nevada and Gray Mountain in Arizona. We put the rover through its paces to make sure it could do what it was supposed to do: avoid hazards, drill into rocks, take photos, and the like.

It’s one thing to drive a Mars rover on Earth. But it’s something else to operate it on Mars, where everything, from atmospheric density to surface gravity, is different from Earth. Some of the conditions can be reproduced at NASA’s Space Power Facility in Sandusky, Ohio, which houses an exceptionally large vacuum chamber. It can simulate the conditions of space travel, including high vacuums, low pressures, and extreme temperature variations.6

The chamber provided the ideal environment for testing the airbags we would use to land our rovers on the Martian surface.7 The Entry, Descent, Landing (EDL) team headed over to Sandusky to conduct some tests. They put a fake lander inside a set of airbags, pumped the vacuum chamber down to Martian pressures and temperatures, put some make-believe Martian rocks at the bottom of the chamber, and let it rip.

And rip they did. The rocks completely ripped through the bags, instantly deflating them. The resulting holes were big enough for a person to walk through. This one test revealed that the airbags we were planning to use were too weak.

One rock, ominously dubbed Black Rock, proved to be the perfect enemy. Adam Steltzner, who worked on the EDL team, described the rock as “the shape of a cow’s liver, with a light ridge running along its top.” It didn’t look particularly dangerous on the surface, but “it reached in and caused the bladder inside the bags to rupture.” Instead of dismissing Black Rock as an outlier—the type of rock we would be unlikely to bump up against on Mars—the EDL team members did the opposite.

They isolated the problem and then exaggerated it. They made replicas of Black Rock, scattered them across the chamber, and started flinging the airbags against them. Although the same airbags had successfully landed the Pathfinder on the Martian surface back in 1997, the success didn’t mean the airbag design was flawless. Luck may have intervened and prevented what could have been a catastrophic collision with the wrong type of rock. But our mission’s EDL team couldn’t count on luck and had to assume the worst: a field of Black Rocks on Mars waiting to tear our airbags to pieces.

The fix came from a seemingly unlikely place: bicycles. Many bicycle tires have an outer casing and an inner tube. Damage to the casing need not puncture the tube, although road debris can penetrate both. The EDL team compared apples and oranges, copying this design for our airbags and designing a double bladder for double protection. The second bladder provided a backup if the outer bladder failed. The new design was tested and retested until the airbags survived the punishment.

You don’t need a fancy vacuum chamber or a large budget to find the breaking point of your own widgets. You can run tests on prototypes or preliminary versions of your products or services using a representative group of customers. Include demanding conditions and plausible failure cases, then check whether the prototype still works.

Testing doesn’t end after the spacecraft is launched. Even after takeoff, we have to make sure that the instruments are operating properly in the unknown and volatile environment of space before we can begin trusting them.

We achieve that accuracy through a process called calibration. For example, each instrument on our Mars rovers had a calibration target. The fanciest target was built for our onboard camera, Pancam.8 The target was a sundial mounted on the rover deck. On its four corners, the dial had four different color blocks containing different minerals, along with gray areas of varying reflectivity. Peppered across the target was the word Mars written in seventeen languages (just in case the little green folks don’t speak English).9 The dial depicted the orbits of Earth and Mars and bore the inscription “Two Worlds—One Sun.” The center post of the sundial cast a shadow on the calibration target. Scientists used this shadow to adjust the brightness of the images.

Before we used any of the instruments, we would first point the instrument toward its calibration target. Pancam, for example, would snap a photo of the sundial and send it down to Earth. If the readings on Mars didn’t match our readings of the same target on Earth—if, for example, the green block on the sundial appeared red in the calibration photo—we would know the instrument was miscalibrated.

In our daily lives, we are miscalibrated far more often than we assume. We need a calibration target, preferably multiple trusted advisers, who can warn us when our reading of the events is off—when we’re looking at a green block but seeing red. Pick your calibration targets carefully, and make sure you can trust their judgment. If their judgment is off, yours will be too.

As we’ll see in the next section, it’s not enough to test the reliability of individual components. Without systems-level testing, you can unwittingly unleash Frankenstein’s monster.

Frankenstein’s Monster

In one sense, a spacecraft is no different from your business, your body, or your favorite sports team. Each is a system made up of smaller, interconnected subsystems that interact with each other and affect how the others operate.

Testing as you fly requires a multilayered approach. Rocket scientists begin testing with the subcomponents—for example, the individual cameras that will form a rover’s vision system, as well as the cables and connectors. Once the cameras are fully assembled, the vision system is tested again as a whole.

The reason for this approach is well summarized by a Sufi teaching: “You think that because you understand ‘one’ that you must therefore understand ‘two’ because one and one make two. But you forget that you must also understand ‘and.’”10 Components that otherwise function properly may refuse to play nice with each other after assembly. Put another way, systems may produce different effects than do the individual components standing alone.

These system-level effects can be disastrous. A medication may deliver terrific results on its own but prove lethal when it interacts with other drugs. Plug-ins on your website may work well in isolation but cause catastrophe as a system. Individually talented athletes may function horribly when assembled as a team.

We can call this problem Frankenstein’s monster. Its limbs come from human bodies. But once the pieces are stitched together, the result is unhuman.

Consider the case of another monster’s awakening. When Adolf Hitler came to power, the German constitution was one of the “most sophisticated” of its day.11 It contained two seemingly harmless provisions. One provision allowed the German president to declare a state of emergency—a declaration that the parliament could cancel by a simple majority vote. The other allowed the president to dissolve the parliament and call for new elections. German parliaments had a habit of fragmenting and deadlocking, so this second provision was intended to check against that problem. Although they seemed benevolent when viewed in isolation, these two provisions turned malevolent in combination, producing what constitutional law scholar Kim Lane Scheppele calls a “Frankenstate.”

In the early 1930s, President Paul von Hindenburg invoked his constitutional power to dissolve a hopelessly deadlocked parliament. Before elections were held to elect a new parliament, Hindenburg issued a state of emergency at the urging of Chancellor Hitler. The declaration suspended almost all civil liberties in Germany. Although the parliament had the constitutional power to override the emergency decree, there was no legislature in place to invoke that power.12 Schutzstaffel (SS) and Sturmabteilung (SA) agents immediately began a widespread cleansing of all opponents of the Nazi cause. Using the state of emergency as a pretext, the Nazis began to consolidate control and establish a one-party dictatorship with Hitler at the helm. The combination helped the Nazis exploit constitutional powers while dismantling democratic checks. Their consolidation of power also involved violence and repression; it cannot be reduced to a pair of design flaws.

A similar type of design flaw was also a possible cause of the Mars Polar Lander crash in 1999.13 As the lander was descending toward the Martian surface using its rocket motors, its three stowed legs popped into place 1,500 meters above the surface. Although we don’t know for certain what happened, the lander may have misinterpreted the jolt of the leg deployment as safe touchdown on the surface. But the lander hadn’t touched down. It was still descending. The computer prematurely cut the descent engines off and sent the lander on a fatal plunge.

The Mars Polar Lander team had tested the landing on the ground, including the deployment of the legs. When the team first ran the test, the electrical switch on the legs had been wired incorrectly and didn’t produce a signal. Members of the team discovered the miswiring and reran the test. But because they were running behind schedule, they focused only on the actual touchdown. They skipped the leg deployment, which would have happened before the touchdown during flight. Although the test showed that the switches were wired correctly, the fatal flaw remained concealed in the leg deployment. NASA didn’t retest the leg deployment phase with the proper wiring. The loss of the lander is consistent with that failure mechanism, though the exact sequence remains uncertain.

As these examples show, failure to conduct systems-level testing can produce unpredictable consequences. When you make a last-minute change to a product and ship it out the door without retesting the whole thing, you’re risking disaster. When you make a change to a section of a legal brief without examining how the change interacts with the whole, you’re dancing with malpractice. When you subcontract the design of a major government program to sixty contractors but fail to test the combined system—as happened with healthcare.gov—you’re courting catastrophe.14

In rocket science, there is another system that needs to be tested before takeoff. This system is far more unpredictable than the spacecraft itself. It panics. It forgets things. It tends to bump up against other objects or accidentally press the wrong button on the console. It can give in to fits of anger, develop a cold, or neglect important work to take in the cosmic scenery.

I am, of course, talking about the human beings on board.

The Right Stuff

The right stuff was the nickname given to the seven brave astronauts selected for NASA’s first manned space mission, Mercury. Yet equally deserving of this title is another group of volunteers, whose names you’ve never heard.15 NASA recruited these volunteers to take part in a series of tests on Earth to simulate flight conditions in space. In 1965, seventy-nine Air Force personnel, donning spacesuits, took rides on a space capsule attached to an impact sled. They rode the capsule “upside down and right side up, backward, forward, sideways, at 45-degree angles.” Human tolerance depends on the direction and duration of acceleration, posture, and support: sustained loads of about 5 g can cause loss of consciousness in some conditions. Here g denotes acceleration relative to gravity at the Earth’s surface. The volunteers experienced brief loads peaking at 36 g.16

The goal of these experiments was to test as you fly—to subject Air Force volunteers to the same types of shocks the astronauts would experience on the lunar journey. The volunteers damaged their eardrums and suffered compression injuries. One man ruptured his stomach after riding the capsule “with his rear end up in the air.” Another was found to have an eye that was “off a little bit.” Colonel John Paul Stapp, who ran the experiments, summed them up in a press release as follows: “At the cost of a few stiff necks, kinked backs, bruised elbows, and occasional profanity, the Apollo capsule has been made safe for the three astronauts who will have perils enough left over in the unknown hazards of the first flight to the moon.”

The test-as-you-fly rule explains why we sent our closest cousins to space before we sent humans.17 Because we had little idea about the effects of weightlessness on the human body, Ham the Chimp flew before American astronauts did. Ham survived the flight, suffering only a bruised nose, and later died of natural causes (he is buried at the International Space Hall of Fame, where he was eulogized by Colonel Stapp).

Ham had been trained to perform basic tasks such as pulling levers, which he successfully replicated during his sixteen-minute flight. Although Ham’s flight was a success, it bruised the fragile egos of the Mercury astronauts, who quickly realized that chimps were equally qualified to do their job. When President Kennedy’s daughter, Caroline, met the astronaut John Glenn, the disappointed four-year-old reportedly asked, “Where’s the monkey?”

We no longer send chimps to space or apply medieval torture techniques to Air Force volunteers. The methods have changed, but the underlying commitment to the test-as-you-fly rule remains. The day-to-day reality of an astronaut’s life is vastly different from the glamour you see in Hollywood movies. Astronauts are workhorses, not space adventurers. They don’t fly in space for a living. They train and prepare for spaceflight for a living. “I’ve been an astronaut for six years,” explained Chris Hadfield, “I’ve been in space for eight days.”18

The remainder is spent on preparation. By the time astronauts fly on their mission, they have flown the same route countless times on simulators.19 For example, the mock-up of the space shuttle was outfitted just like the real thing, with identical controls and displays. The astronauts operated the space-shuttle simulator as they would operate the actual spacecraft, working through different segments of the mission, from launch to docking to landing. The monitors on the simulator displayed the same scenes the astronauts would see in flight, and hidden loudspeakers generated the same noises—including vibrations, pyrotechnic explosions, and gear deployment—they would hear in flight.

But there’s one thing simulators can’t do: generate microgravity. That’s where the vomit comet comes into play.20 It’s the name for an airplane that flies in parabolas—kind of like a roller coaster—climbing and then diving to simulate weightlessness. At the top of each parabola, passengers experience about twenty-five seconds of microgravity. The airplane earned its name because these steep climbs and sharp dives tend to produce severe bouts of nausea among the passengers. Astronauts hop on board the vomit comet to practice moves like eating and drinking while floating in weightlessness.21

But twenty-five seconds isn’t enough to practice more complex moves. For longer practice sessions, astronauts dive into a giant indoor pool called the Neutral Buoyancy Lab. Carefully balanced buoyancy lets them rehearse spacewalk tasks, although water drag and the body’s experience of gravity make this different from actual microgravity.22 “I really feel like a full-fledged astronaut in the pool,” Hadfield writes, “I’m wearing a spacesuit, my breathing is assisted just as it is during a spacewalk.” In the pool, which contains mock-ups of the International Space Station, astronauts practice making the same types of repairs they’ll eventually conduct while floating in outer space (also called a spacewalk). They practice every step until it becomes second nature.23 For Hadfield, achieving this level of familiarity meant spending 250 hours in the pool to get ready for a 6-hour spacewalk.24

Astronaut simulations are directed at NASA by a simulation supervisor—or SimSup for short—who leads a team of instructors.25 One part of SimSup’s job is to teach the astronauts the correct procedures for every segment of the mission. The other part of the job is far more grim: kill the astronauts.

The simulation team plays its own version of the kill-the-company exercise we encountered earlier—where corporate executives play the role of a competitor seeking to put the company out of business. The goal of the kill-the-astronaut exercise is similar. It’s to push the astronauts to make the wrong moves in the simulator so they learn to make the right ones in space. When something goes astray in space, there’s often no room for prolonged deliberation. Testing as you fly requires reducing the response time as close to instantaneous as possible. For space shuttle missions, this preparation meant activating roughly 6,800 malfunction scenarios, throwing every imaginable failure—computer crashes, engine troubles, and explosions—at the crew.26 During the training of the Apollo astronauts, as author Robert Kurson explains, these simulations would run for days at a time. “The more catastrophic the better, until repetition began to groove instinct into all the participants, and dying helped the men learn to survive.”27

In many ways, these simulations are tougher than the actual flight. They follow the old adage “The more you sweat in peace, the less you bleed in war.” When Neil Armstrong first began walking on the lunar surface, he noted how the actual experience was “perhaps easier than the simulations at one-sixth g,” referring to the reduced gravity on the Moon.28 Sweating the small stuff on Earth ensured that the same stuff didn’t make Armstrong bleed in space.

Repeated exposure to problems inoculates astronauts and boosts their confidence in their ability to defuse just about any issue. When physics throws curveballs at them, their training kicks in. After Hadfield returned to Earth from a successful mission, he was asked if things had gone as planned. “The truth is that nothing went as we’d planned,” he responded, “but everything was within the scope of what we prepared for.”29

Apollo astronaut Gene Cernan spoke in similar terms about his training. “If the [spacecraft] went somewhere we didn’t like or the ground didn’t like,” he said, “I could flip a switch and I could control over 7.5 million pounds of rocket thrust [and] fly the thing to the Moon myself.” Cernan was the commander of the Apollo 17 mission and the last person to leave footprints on the lunar surface. He continued: “I had practiced it and trained for it so many times, I almost dared her, I almost dared her to quit on me.” After repeated practice, the astronaut and the spacecraft had fused into one. “Every breath she breathed,” Cernan recalled, “I breathed with her.”30

When the oxygen tank exploded on the Apollo 13 mission—literally taking the astronauts’ breath away—their training kicked in. The movie Apollo 13 displays a chaotic environment on the spacecraft and in mission control, with rocket scientists and astronauts scrambling to improvise solutions. Because the service module was damaged from the explosion, they had to figure out how to use the lunar module—intended only to shuttle two astronauts to the lunar surface—as a lifeboat for returning all three astronauts back to Earth.

But the reality was much calmer than its Hollywood depiction. Gene Kranz, the flight director for the mission, had conducted regular dress rehearsals to train mission controllers to solve complex problems in stressful situations.31 In fact, a similar type of contingency requiring the astronauts to use the lunar module as a lifeboat had been simulated before. “No one had ever simulated exactly what happened,” Apollo astronaut Ken Mattingly explains, “but they had simulated the kind of stress that could be applied to the system and the people in it. They knew what their options were, and had some ideas already in place about where to go.”32

This training strategy is useful far beyond rocket science. Consider, for example, oral arguments before the US Supreme Court. The highest judicial body in the land, the court hears less than one hundred cases every year, with a small number of the country’s premier attorneys getting the privilege to present arguments before it.

I remember when I first walked into the courtroom as a visitor. The first thing I noticed wasn’t the grandeur, the tall ceiling, or the marbled walls. No, it was how frighteningly close the attorney’s lectern is to the mahogany bench where the nine Supreme Court justices sit. As attorneys present their arguments to the Court, they get interrupted by sharp, and often confrontational, questions from the justices. For every half hour of argument, a lawyer can expect an average of forty-five questions.33 Questions often pour in before the attorneys even finish their first sentence. Given the short distance between the lectern and the bench, lawyers literally get blindsided by justices out of their vision.

Emotional appeals may work before a jury, but not before nine of the greatest legal minds in the country. The lawyers must be cool and collected while giving instantaneous responses to a barrage of questions. “You have to think not just how the answer to this question is going to work,” explains frequent Supreme Court advocate Ted Olson, “but what that’s going to mean for other yet unasked questions. And you don’t want to please one Justice and alienate two others at the same time.”

It takes a rocket-science mindset—and rocket-science preparation—to master this mental roller coaster. Before he became a judge, John Roberts—who later became chief justice of the Supreme Court—was widely considered one of the best oral advocates to ever have appeared before the court. To prepare for arguments, Roberts would draft hundreds of questions he could conceivably receive from the judges. He would prepare answers for every single one, but he knew that simply writing the answers wouldn’t be enough. On argument day, the questions would be thrown at him in random order from different judges. To bring the test closer to flight, he would “write the questions on flash cards, shuffle them, and test himself, so he’d be prepared to answer any question in any order.”

When Roberts stepped up to the lectern to deliver his argument, he looked like a natural. Jonathan Franklin, a former colleague, recalls the effect: “He was able to take complicated points, distill them to their essence and respond with an absolute minimum of verbiage, and make it seem that his argument is so obviously correct that you have no choice but to agree with him.” His delivery was so smooth that it seemed to unsuspecting observers Roberts had heard the questions before and knew exactly how to respond.

Another lawyer applied the same mindset to her athletic training. When she started competing, Amelia Boone was a lawyer at a major Chicago law firm. On a typical training day, Boone would go for a run in a wetsuit, dunking herself in and out of the icy waters of Lake Michigan, with the frigid winter wind whipping against her face.34 Observers decked out in layers of thick winter clothing would assume these were the delirious actions of a masochist. But no, the Queen of Pain—as Boone has come to be known—was getting ready for the World’s Toughest Mudder.

The World’s Toughest Mudder makes a marathon look like a casual stroll. The race is run for twenty-four hours nonstop. Participants must fight off sleep while conquering roughly twenty of the “biggest, baddest” obstacles scattered across a five-mile course. It’s survival of the fittest: Whoever completes the most laps wins.35

Some of the obstacles are in water, which can drop down to freezing temperatures. To prevent hypothermia, all participants run in wetsuits. While the runners are on land, the wetsuit helps preserve the body’s warmth since body heat tends to dissipate over the grueling twenty-four hours.

When Boone first started training, she had little strength. She spent six months trying to do a single pull-up, but failed miserably. In her first race, she fell off all the obstacles. “I was really bad at that,” Boone told herself after the race. “Let’s try and get better.” And get better, she did. By the time of this account, she was a four-time world champion with results that placed her among the leading obstacle racers across gender categories.

Boone’s secret is the same as any self-respecting astronaut: test as you fly. Train in the same environment you’ll experience on race day—while your competition trains from the comfort of a gym because it happens to be raining outside. “You don’t race on a treadmill with Netflix in front of you,” Boone says, “so you shouldn’t be doing your training like that.”

The rain, the snow, the dark, the cold, the wetsuit—they all beckon Boone. By the time the race rolls around, she has been desensitized to the brutal conditions awaiting her. She greets them with a smile that seems to say, “Nice to see you again. Let’s dance.”

In our lives, we don’t do what Roberts and Boone do. We train in conditions that don’t mimic reality. We practice a major speech in the comfort of our home, when we’re fully rested and awake. We do mock job interviews in our sweatpants with a friend using a predetermined set of questions.

If we applied the test-as-you-fly rule, we could practice our speech in an unfamiliar setting with a time limit and planned interruptions. We would do mock interviews while wearing an uncomfortable suit, with a stranger ready to throw curveballs at us.

Businesses can also benefit from this principle. Corporate simulations, if they follow the test-as-you-fly rule, can “enhance an organization’s ability to make high-stakes decisions,” as three business school professors write in the Harvard Business Review.36 For example, Morgan Stanley conducts drills to determine how to respond to various threats, including hackers and natural disasters. One aerospace firm holds dress rehearsals to determine how to respond to moves from their competitors, such as mergers or alliances. “By engaging in dress rehearsals,” the researchers explain, “participants get to know each other’s strengths or weaknesses, and informal roles become clear.”

The test-as-you-fly rule, as we’ll see in the next section, can also help everyone, including companies and comedians, run focus groups and gauge public opinion on their next product or brand-new joke.

The Rocket Science of Public Opinion

An early survey of multifunction devices illustrates why stated preferences can be a poor guide to a product people have not used.

One of the most profitable consumer products in modern history, the iPhone was a flop in surveys conducted before it was released. When asked in a survey whether they “like the idea of having one portable device” to fulfill all their needs, only about 30 percent of Americans, Japanese, and Germans said yes.37 They seemed to prefer carrying around a separate phone, a separate camera, and a separate music player instead of a single device that could perform all three functions. Echoing the survey results, then Microsoft CEO Steve Ballmer said, “There’s no chance that the iPhone is going to get any significant market share. No chance.”

The iPhone didn’t prove the survey wrong. As author Derek Thompson explains, the survey accurately measured the participants’ “indifference to a product they had never seen and did not understand.” In other words, the survey had failed to follow the test-as-you-fly rule. Hypothetically thinking about the iPhone was nothing like seeing it in person. Once consumers saw the iPhone in an Apple Store—once they stepped into the brand and held the revolutionary new device in their hands—they couldn’t let it go. Their indifference quickly morphed into desire.

There’s a question that’s commonly asked by companies to customers in pricing experiments: How much would you pay for this pair of shoes? Think about it. When was the last time someone asked you this question in real life? My guess is never. It’s one thing for customers to say they’ll buy a hypothetical shoe at a hypothetical price point. But it’s another for them to reach into their wallet, pull out their hard-earned dollars, and hand over the money to a cashier. The shoe company is far better off building an actual prototype, placing it in an actual store, and selling it to an actual customer—in other words, testing as they fly.

George Gallup, whose name became associated with public-opinion polling, also studied the gap between reported and observed behavior.

George Gallup was interested in finding an objective way to determine reader interest in newspapers.38 He decided to write his PhD dissertation on the topic, appropriately titling it “An Objective Method for Determining Reader Interest in the Content of a Newspaper.” For Gallup, the operative word was objective. He was deeply skeptical of subjective methods of determining reader interest, particularly the use of surveys and questionnaires. He believed—correctly, it turned out—that when it comes to reporting their own behavior, people tend to bend the truth. Readers would claim in surveys that they read the front page of the newspaper in full, but in reality, they would skip to the sports or the style section.

Put differently, these surveys failed the test-as-you-fly rule. Filling out a survey about reading a newspaper, and the actual act of reading a newspaper, are two different things. Gallup knew that for the test to work, it had to closely resemble the flight.

So what did Gallup do to remedy the problem? He sent a team of interviewers into people’s homes to watch them read newspapers and mark each part of the paper as read or unread. Awkward? Yes. For checking which sections people actually read, direct observation provided a useful check on their recollections. “Almost without exception,” Gallup wrote, “later questioning proved… preliminary statements [in surveys] false.” Gallup’s analog experiment was the precursor to modern digital tracking. If you think his approach was creepy, remember that Netflix knows exactly what you watch, when you watch it, and whether you stopped the last season of House of Cards before it ended. Viewing records show behavior that a self-report may miss, although they do not by themselves explain why someone watched or stopped.

Great comedians also think like rocket scientists and test their material before an actual audience to observe their reaction. They pop into comedy clubs unannounced to test their material in a low-stakes environment filled with strangers. For example, before hosting the Oscars in 2016, Chris Rock dropped by the Comedy Store, a comedy club in Los Angeles, to test his material.39 Ricky Gervais and Jerry Seinfeld also visit tiny comedy clubs and adjust their jokes—or drop them altogether—in light of audience reaction.40

It’s one thing to drop into random comedy clubs or watch people as they read newspapers. But it’s something else entirely to ask people to let a stranger walk into their bathrooms and watch their children as they brush their teeth. The global design firm IDEO did exactly that after being tasked by Oral-B with designing a better toothbrush for children. Oral-B executives initially rolled their eyes at IDEO’s unorthodox and mildly disturbing request. “It’s not rocket science,” the executives protested. “We’re talking about kids brushing their teeth.”41

It turns out that it is rocket science. Designing a great toothbrush, just like designing a great rocket, requires synergy between test and flight. Let’s set aside the amusing image of an IDEO staffer busily taking notes while a five-year-old tries to concentrate on the already-challenging task of brushing teeth. Instead, let’s focus on what IDEO uncovered. Before IDEO came along, manufacturers of children’s toothbrushes assumed that children, who have smaller hands, needed smaller toothbrushes. So they took adult-sized toothbrushes and made them skinnier.

This approach sounds intuitive, but it completely missed the mark. IDEO’s field research uncovered that children brush their teeth differently than how adults do it. Unlike adults, children grab the entire toothbrush with their fist. They lack the type of dexterity that adults have to move the toothbrush around with their fingers. Skinny toothbrushes make the children’s job even harder, since the toothbrush tends to move around in their hands as they try to brush. As a result, what children needed were big, fat toothbrushes. Despite their initial skepticism of IDEO’s approach, Oral-B executives went with IDEO’s recommendations, producing a toothbrush that became the best seller in its category.

IDEO used the same strategy in redesigning the patient experience in a hospital. A hospital room can feel bleak to a patient confined to bed: plain walls, fluorescent lights, and long stretches with little to look at.

When one health-care organization brought in IDEO to redesign the patient experience, the executives were probably expecting a stylish PowerPoint presentation with new, creative designs for the hospital rooms. Instead, what they got was a mind-numbing six-minute video clip. The video showed nothing but the ceiling of a hospital room. “When you lie in a hospital bed all day,” IDEO’s chief creative officer Paul Bennett explained, “all you do is look at the roof, and it’s a really shitty experience.”42

What Bennett describes as “a blinding glimpse of the bleeding obvious” came after IDEO employees put themselves in a patient’s shoes. An IDEO designer checked in to the hospital as a patient and lay in an actual patient bed for hours, getting wheeled around, staring at the ceiling tiles, and capturing the abysmal experience on a video camera. That six-minute clip of the dull tiles was a small glimpse of the overall patient journey—a “mix of boredom and anxiety from feeling lost, uninformed, and out of control,” as IDEO’s CEO Tim Brown said.43

Six minutes of footage was sufficient for the hospital employees to spring into action. They decorated the ceilings, put up whiteboards for visitors to leave messages to the patients, and transformed the style and color of the patient rooms to make them more personal. They also put rearview mirrors on hospital stretchers to allow patients to see and connect with the doctors and nurses wheeling them around. IDEO’s presentation ultimately kick-started a broader discussion to improve the overall patient experience so that patients were “treated less like objects to be positioned and allocated, and more like people in stress and pain,” Brown explained.44

As these examples show, instead of creating artificial testing environments disconnected from reality, we’re better off observing customer behavior in real life. If you want to design a better newspaper, watch people read the paper. If you want to design a better kid’s toothbrush, watch kids brush their teeth. If you want to see if people will love the iPhone, put an iPhone in their hands. “If you want to improve a piece of software,” as IDEO’s founder David Kelley explains, “all you have to do is watch people using it and see when they grimace.”45

This approach provides a vast improvement over subjective self-reporting in artificial conditions. But it doesn’t completely eliminate the distance between the test and the flight. Observing people, it turns out, tends to affect how they behave.

The Observer Effect

The observer effect is among the most misunderstood concepts in science. It has given rise to pseudoscientific claims that the conscious mind can magically alter reality and make a spoon move across the dinner table. But the scientific concept is simple. By observing a phenomenon, you can affect that phenomenon. Let me explain.

I started wearing glasses when I became a professor. But true to the absent-minded professor stereotype, I tend to misplace them. If I’m looking for my glasses in a dark room, I do what everyone else does: turn on the light. The act of turning on the light sends a flood of photons toward my pair of glasses, which reflect off the glasses and into my eyes.

But now assume that, instead of my glasses, I’m trying to find an electron. To observe an electron, I do the same thing and send some photons in its direction. My glasses are relatively big objects, so when photons collide with them, the glasses don’t move. But when photons collide with an electron, they displace the electron. You can also think of it as a coin lodged between the cushions of a couch.46 The very act of trying to grab the coin pushes it further out of reach.

The act of observation disturbs humans in a different way. When people know they’re being observed, they behave differently.

Let’s suppose you’re in a test audience for a new television show. Watching the show as part of a focus group is a different experience from watching it in your living room. The test isn’t identical to the real flight. In a focus group, you may find numerous flaws in a show—since you’re being observed by people who asked you to critically evaluate it—even though the same show would have proved binge-worthy in your living room.

For example, the TV show Seinfeld performed abysmally before test audiences.47 When creating the show’s premise, the producers asked a question we encountered in an earlier chapter: “What if we did the opposite of what everyone else is doing?” At the time, the sitcom playbook was set in stone. A group of characters would run into problems, resolve those problems, learn something from the experience, and hug each other.

From the get-go, Seinfeld producers were clear on their mission. They would flip the script. There would be no hugging. There would be no learning. The characters on Seinfeld would draw laughs from constantly repeating their mistakes and overlooking their own faults. In case there was any confusion, the writers wore jackets that said No Hugging, No Learning. But test audiences, accustomed to the standard sitcom playbook, were expecting lots of hugging and learning. As a result, Seinfeld was a spectacular failure in focus groups. Yet the show went on to become one of the most popular sitcoms of all time.

The observer effect is often an unconscious process. Even when we assume we’re not affecting the participants—even when we’re careful to not dislodge that coin in the couch cushion—we might be cuing them in subtle but significant ways.

Consider Clever Hans the horse.48 Hans was the closest thing a horse could come to a rocket scientist. He became a worldwide sensation for his ability to perform basic math. Its owner, Wilhelm von Osten, would ask the audience for a math problem. Someone would shout out, “What is six plus four?” and Hans would tap his hoof ten times. His ability went beyond addition. He could subtract, multiply, and even divide. People suspected fraud, but independent investigators found no foul play.

It was a young psychology student named Oskar Pfungst who figured out what was really going on. Hans could find the right answer only if he could see the human questioner. His mathematical genius disappeared if he was wearing blinders or otherwise couldn’t see the human intermediary. In the end, it was the human questioners who were unwittingly providing cues to the horse. As Stuart Firestein writes, “People would tense the muscles of their body and face at the beginning of Hans’s answer and release the tension when he arrived at the correct hoof tap.” Remarkably, even after Pfungst discovered Hans’s secret, he couldn’t prevent himself from unconsciously giving cues to the horse. As long as he knew the answer, Pfungst’s demeanor would involuntarily shift when Hans arrived at the correct hoof tap.

The distortions introduced by the observer effect are significant. Observer effects can distort test results, as Hans’s responses show. The disappointing reception of a television pilot may also reflect the audience, the format, or other differences from later viewing; it does not isolate observation as the cause.

One way to mitigate this effect is to put blinders on both the human questioner and the horse by conducting what’s called a double-blind study. For example, in drug trials, both the participants in the study and the scientists running it are kept in the dark—hence double-blind—about whether the participants are getting the actual drug or the fake one, called a placebo. If the methods aren’t double-blind, scientists may insert their hopes and prejudices into the study, treating the participants differently or unconsciously cuing them like Hans’s human questioners.

You can also take a cue from the best-selling author Tim Ferriss.49 Most authors, in picking the title and cover design for their book, simply go with their gut or, at best, consult a few friends. The more astute ones run a survey of their audience. But Ferriss took this analysis to the rocket-science level with his first book.

To select a title, Ferriss applied the test-as-you-fly principle. He bought domain names for roughly a dozen book titles and ran a Google AdWords campaign to test click-through rates. When a user typed certain keywords into a Google search related to the book’s content, an ad would pop up with the book title and subtitle that pointed to a dummy web page for a book that didn’t yet exist. Google would automatically randomize and mix and match the book titles and subtitles that were displayed to the user, allowing an objective analysis of popularity. Within a week, it became apparent that The 4-Hour Workweek title attracted by far the most attention. Ferriss took the data to his publisher, which didn’t need much convincing that the title was the right one.

But Ferriss didn’t stop there. To choose a cover for his book, he went to a bookstore with alternative cover designs in hand. He picked up a book from the new-arrivals section and wrapped it in one of his covers. He then sat back and watched how many times the book was picked up by unsuspecting customers, repeating the exercise for each version of the cover for thirty minutes at a time until he settled on a winner.

One final piece of the testing puzzle is often overlooked. Perfectly planned tests can spit out perfectly incorrect results if the testing instrument itself is flawed.

Multiple Testers

The irony was hard to escape. A space telescope built to produce distortion-free images was spitting out distorted images.50 The Hubble Space Telescope was launched in 1990 with the promise of taking detailed, high-resolution images of the cosmos—ten times sharper than what earthbound telescopes could produce. The size of a school bus, the telescope would hover above the Earth free of the distortions introduced by the atmosphere, providing the clearest view humanity has ever had into the cosmos.

But the first set of images that came down from the Hubble weren’t nearly as clear as astronomers had hoped. The $1.5 billion telescope was suffering from myopia and sending fuzzy photos back to Earth.

It turned out the primary mirror of the telescope had been ground to the wrong shape because the testing device used to ensure correct grinding hadn’t been set up properly. One of the lenses on the testing device—called the reflective null corrector—was out of position by 1.3 millimeters (or 0.05 inches). This positioning produced a flaw on the mirror that was one-fiftieth the thickness of a piece of paper. That may seem like a minor flaw, but when it comes to sensitive instruments, millimeters can be mountains. Over the course of five years of grinding and polishing, the mirror had been very precisely ground to the wrong shape.

The commission that was convened to investigate the mirror fiasco criticized the use of a single instrument to test the mirror. Because of cost and scheduling concerns, the team had dismissed the need for independent testing by a second instrument.

Here’s the moral of the Hubble story. If you’re going to rely on a single instrument for testing—and put all your proverbial eggs in the same basket—you must test the basket to make sure it won’t cave. But this wasn’t done in the case of the Hubble. No one had tested the testing device to ensure a correct setup and accurate spacing of its lenses.

Thankfully, there was a contingency in place, and the telescope could be serviced in space. Astronauts did what you do when your vision is blurry. They put eyeglasses on the Hubble. Because the flaw on the Hubble’s primary mirror was perfectly wrong, the perfect prescription could right the wrong. In a servicing mission in 1993, astronauts outfitted the Hubble with spectacles, restoring the telescope to its promised glory and returning it to its mission of producing dazzling images that now decorate computer backgrounds across the globe.

Consider another example from outside rocket science.51 The version of Facebook’s website described here dated to 2006, when the “web was a lot more text heavy,” as Julie Zhuo, then Facebook’s vice president of product design, told me. With the rise of camera phones, the company wanted to create a more visual experience. After six months of work, the Facebook team created a modern, cutting-edge website. They tested the new website internally, and it worked beautifully. They hit publish and waited for all the praise to pour in.

But the company was in for a rude awakening. The metrics showed the redesign was a colossal failure. “People started using Facebook less. People were commenting and engaging with other people less,” Zhuo explained to me.

It took the Facebook team several months of fieldwork to figure out what had gone wrong. The team had tested the new website using high-tech computers in Facebook offices. But the vast majority of Facebook users lacked access to top-of-the-line equipment. They were accessing the website with old computers that didn’t support all the fancy imagery that came with the redesign. Put differently, for most Facebook users, the flight was far different from the test. Only when the Facebook team switched out its testing instruments—and used low-tech instead of high-tech equipment—did the group create a redesign that worked for its user base.

These examples hold important lessons for us all. Treat your testing instruments like your investments and diversify them. If you’re building a website, test it using different browsers and different computers. If you’re designing a children’s toothbrush, watch many children brush their teeth—lest you get the one miracle child who uses a toothbrush like an adult. If you’re deciding which job offer to take, consult multiple calibration targets. One person’s opinion might provide only a fuzzy perspective. It’s only through independent validation and multiple testing sources that you get closer to twenty-twenty vision.

Before relying on a test, ask which conditions it reproduces and which it leaves out. Check the equipment and the complete system, and use the results to revise the design or preparation.

Test results are most useful when the conditions, instruments, and observers have also been checked.