What Aren’t You Seeing?

Below is an image from World War II I think every business owner should spend a few minutes looking at.

It shows an aircraft covered with red marks representing damage from enemy gunfire and antiaircraft fire. Those marks are concentrated heavily across the wings, portions of the fuselage, and the tail, while other areas, particularly around the engines and some of the aircraft’s most critical systems, show considerably less damage.

Take a good look at the image before going much further and ask yourself a simple question: if you were responsible for getting more of those aircraft and their crews home safely, where would you add armor?

Most people immediately look at all those red marks and arrive at what seems like the logical answer. You reinforce the areas getting hit the most. That thought is exactly what makes the story so powerful.

During World War II, mathematician Abraham Wald worked with the Statistical Research Group at Columbia University on the problem of aircraft survivability. Military planners had damage data from aircraft returning from combat and wanted to determine which areas were most vulnerable so that limited armor could be placed where it would do the most good.

Wald looked at the problem differently because he recognized something unusual about every aircraft represented in the available data: every one of them had made it home. Once you recognize that, the picture begins to look completely different.

The damage scattered across the wings and fuselage was not necessarily showing where the aircraft most desperately needed additional protection. In many cases, it was showing the opposite. Those were areas where an aircraft could sustain damage and still survive long enough to return from the mission.

The more interesting question was what appeared to be missing. Why were there so few marks around certain critical areas? Perhaps those sections were not being struck less often at all. Perhaps aircraft taking serious damage there simply were not returning to be counted. The missing damage was represented by the missing aircraft.

Wald’s actual analysis was considerably more sophisticated than the simplified version of the story that gets repeated today. His wartime work examined the likelihood of aircraft surviving hits to different sections and demonstrated that survivability could vary dramatically depending on where damage occurred. The famous red-dot aircraft image commonly used to illustrate the idea was not part of Wald’s original 1943 report, but rather a later visualization loosely based on his work.

Knowing this does not weaken the lesson, however. If anything, the image has endured because it allows us to experience the mistake before someone explains it. We see what appears to be the obvious answer, and then suddenly realize why the obvious answer may be completely wrong.

This became one of the most famous examples of survivorship bias, and once you understand it, I want you to look at your business a little differently because you may be making the same mistake without realizing it.

Think about the information you use to make decisions every day. Your sales reports show what sold, inventory reports show what moved, range management data tells you who booked a lane, class rosters identify who registered, membership reports show who joined, and your website records the customers who completed purchases. Employees also contribute information based on the questions and requests they hear across the counter.

All of that information is useful, but there is an important question hiding underneath it: who had to make it through the process successfully in order to appear in that data?

The customer in your sales report found something he wanted, accepted the price, navigated the buying process, and completed the transaction. The person on your class roster found the class, understood what was being offered, decided it was worth the money, registered successfully, and showed up. Your range member understood the membership options, saw enough value to join, and completed enrollment. These are your returning aircraft.

You should absolutely study them because they tell you a great deal about what is working. What you should not do is assume they represent everyone who encountered your business, because there is another group of customers who may be able to teach you just as much, and in some cases even more. They are the people who never made it into the report.

Suppose you pull a category sales report and notice that you sold almost none of a particular type of firearm during the previous quarter. The obvious conclusion is that your customers are not interested in that category.

Maybe that is true, but Wald should be sitting on your shoulder asking another question: what aren’t you seeing?

What if dozens of people searched your website for that category and found nothing? What if customers repeatedly asked employees whether you carried a particular model and were told you did not? Perhaps the products people wanted were technically in your catalog but did not appear when customers searched using the terminology they naturally use. Maybe the price was not competitive, inventory appeared unavailable when the customer was ready to purchase, or the customer found exactly what he wanted somewhere else. Suddenly, zero sales can mean something entirely different.

A traditional sales report might lead you to conclude there is no demand, while the missing customers could be telling you that demand exists but your business is failing to capture it. Those are completely different business problems even though the final sales report can look exactly the same.

That is the retailer’s version of staring at an aircraft wing covered in damage and assuming the blank area around a critical component must not matter.

There is another form of survivorship bias that can be especially dangerous in firearms retail because this industry can feel intimidating to someone who does not already understand it.

Walk into a firearms store as an experienced owner and many things seem obvious. You understand the terminology, know how transfers generally work, recognize what a Form 4473 is, understand ammunition types, and probably feel comfortable asking to handle a firearm or discussing what you are looking for.

Now imagine walking through that same front door as someone purchasing a firearm for the first time. The business has not changed, but the experience certainly has.

That customer may not know where to begin, what questions to ask, what terminology employees are using, or what happens next in the purchase process. Product descriptions may assume knowledge the customer does not have, while transfer instructions that seem perfectly clear to someone who has completed ten transfers may be confusing to a person doing it for the first time. Some customers will ask for help, but others will quietly leave.

This is where the data can become misleading. If you ask your regular customers whether your store is easy to shop, many may tell you that it is, and from their perspective they are telling the truth. They have learned your store, understand your processes, know your employees, and have figured out how you operate. They survived the process.

The customer who could not figure it out probably is not standing at your counter explaining what went wrong. He may already be standing at somebody else’s counter instead. You do not see the damage because that aircraft never returned.

None of this means your best customers give bad advice. It means their experience represents a very specific group of people.

Imagine asking twenty longtime range members why they continue to renew. You might hear that they appreciate the staff, enjoy the facility, value member pricing, like the atmosphere, and believe the benefits justify the cost. Some may have even forgotten they’re paying. That is useful information, but it is important to understand exactly what question those answers have actually addressed. You learned why members remain members. You did not learn why hundreds of other range customers may have visited during the year and never become members in the first place.

Perhaps the membership is priced incorrectly, the benefits are not explained clearly, or employees are presenting it inconsistently. Maybe occasional shooters do not believe they will visit enough to justify the expense, the enrollment process contains too much friction, or the benefits really are excellent but people who are not already members never fully understand what those benefits are.

If your objective is improving retention, existing members are an important source of information. If your objective is growing membership, however, you also need to understand the people who considered joining and chose not to.

Businesses get themselves into trouble when they ask one group of people a question and then assume the answer applies equally to another group.

Customer complaints feel like strong evidence because they are visible. They arrive through email, chat, reviews, phone calls, or conversations across the counter, which naturally causes management to pay attention to the people making noise.

Now consider the customer who encountered exactly the same problem and never complained. Maybe he could not find the firearm he wanted online, became confused during the transfer process, felt uncomfortable during an interaction with an employee, experienced a frustrating checkout, waited too long for help, or could not make sense of the range reservation process. He never sent an email, asked for a manager, or left a one-star review. He simply left. From the retailer’s perspective, it can look as though nothing happened. From the customer’s perspective, plenty happened.

That creates a significant blind spot because businesses are very good at measuring events. A sale is an event. A return, support request, complaint, membership cancellation, or abandoned cart is something that can be counted and investigated. Quiet abandonment, on the other hand, may produce no obvious event at all.

There may be no transaction, ticket, complaint, or cancellation request, which makes the absence of data feel like the absence of a problem. Wald’s aircraft should remind us that sometimes the absence of data is exactly what deserves our attention.

This is where retailers can turn an interesting World War II lesson into something useful inside the business.

Most companies devote enormous amounts of time to studying what happened. More retailers should also study what almost happened.

Look at what customers searched for when the website produced no useful result, which products received significant attention but rarely converted into purchases, and where online shoppers most often disappeared during the buying process. Examine how many first-time range visitors never came back, how many class attendees failed to engage with the business again, and how many customers repeatedly rented firearms without ever making a purchase.

You can extend the same thinking throughout the operation. How many people purchased a firearm but never returned for ammunition, optics, cleaning supplies, training, or range time? How many potential members asked about joining but never enrolled? How many customers began filling out a form but never submitted it? How many people opened a promotion, clicked the offer, showed obvious interest, and disappeared before purchasing? The questions are endless and you and your team should explore them.

Any one of those numbers by itself may not tell you much, but together they begin showing the outlines of the aircraft that did not come home.

That may be where some of your biggest opportunities are hiding.

Retailers may need to broaden the way they think about customer data.

From an accounting perspective, the customer who completed a transaction is obviously the one who generated revenue. From a learning perspective, however, someone who did not buy can sometimes reveal more about the business than someone who did.

A successful transaction demonstrates that your process worked for that particular person, under those circumstances, at that moment. A failed or abandoned transaction can expose where the process begins to break down.

Suppose 100 people successfully purchase a firearm online while another 40 repeatedly abandon the process at roughly the same point. The 100 completed transactions prove that finishing the process is possible, but the other 40 may be showing you exactly where revenue is leaking.

Congratulating yourself on the successful orders without investigating the customers who disappeared would be the same mistake as focusing entirely on the damage visible on Wald’s returning aircraft.

The red marks were not meaningless. They contained extremely valuable information. The mistake would have been assuming they told the entire story.

Your completed transactions are valuable for the same reason. They are essential pieces of information, but they are not the whole story.

The next time you are looking at a report inside your business, try asking one additional question:

What would have had to happen for someone to be missing from this report?

If you are reviewing sales, consider the customers who wanted something but did not buy it. When looking at memberships, think about the people who considered joining but decided against it. If you are studying website conversions, investigate the shoppers who started the process but never finished, and when reviewing customer feedback, think about the people who were unhappy enough to leave but not motivated enough to tell you why.

The same thinking applies to support tickets. How many customers encountered the same problem but found a workaround, gave up, or simply decided dealing with it was not worth their time? When analyzing your best customers, ask what they successfully navigated that may have caused someone else to disappear.

That single question can completely change the way you interpret the information in front of you.

Wald did not look at the returning aircraft and decide the available data was wrong. He recognized that it was incomplete. Those aircraft could tell him where surviving aircraft had sustained damage, but understanding vulnerability required him to also account for the aircraft that were not there.

That is a much more useful lesson than simply memorizing the definition of survivorship bias. It gives you a different way to think about the information your business produces every day.

Retailers are surrounded by more data today than most small businesses could have imagined having access to twenty years ago. Sales reports, inventory data, website analytics, marketing metrics, customer records, range activity, memberships, classes, support conversations, and countless other measurements can tell us an enormous amount about what is happening inside a business.

The challenge is no longer simply collecting more data. It is learning how to question the data we already have.

Sometimes the most important thing on a report is the largest number, a trend that has begun moving in the wrong direction, or an unusual exception that deserves attention. Other times, the most important thing is something that never appears on the report at all.

That is why I keep coming back to the aircraft.

Once you understand what you are looking at, the image becomes difficult to see the same way again. The areas covered in red no longer simply tell you where damage occurred. The blank spaces begin demanding an explanation.

Your business has blank spaces too. They include customers who did not buy, products that could not be found, memberships that were considered but never started, carts that were abandoned, first-time visitors who never returned, problems customers never reported, questions people were too uncomfortable to ask, and opportunities that disappeared without ever creating a transaction for you to analyze.

Your reports will do a very good job showing you the aircraft that came home. The next level of running your business is learning to account for the ones that did not, because the thing you are not seeing may be the thing you most need to understand.

Coreware helps firearms retailers and ranges connect more of the information happening across their business so they can better understand customer behavior, inventory, sales, marketing, memberships, operations, and the opportunities that might otherwise remain hidden. To learn more about how Coreware can help you see more of what is happening inside your business and make better decisions from it, visit coreware.com.