Customers & Bullet Holes

In talking with painter James Christensen, Photoshop engineer Jerry Harris picked up an anecdote I found interesting:

During World War II, Allied bomber losses were high, so the powers that be demanded a fix. The engineers set out to eyeball every bomber they could, gathering great statistics for each bullet hole. After a long study they decided to add more armor plating to the areas that had the highest concentrations of holes. A bit after these improved planes were deployed, they received some startling news: more planes were going down than before. At this point I thought, “Did they make them too heavy?”

Then the light bulb went on for someone: they had measured every bullet hole in every plane at their disposal, but they’d failed to realize it was the ones that they did not have access to that mattered. It was the ones that did not return that needed to be scrutinized. They needed to improve the armor in the places that the returning planes had no bullet holes.

Sounds like it might generally relate to product marketing, and user studies: go investigate the customers that don’t return for seconds, i.e. upgrade.

I can’t vouch for the story’s veracity, but the lesson seems sound. This is part of why one needs to listen to customers, but only up to a point: they’ll tell you how to please the customers you have–but not the ones you don’t have.

13 thoughts on “Customers & Bullet Holes

  1. The story is about Abraham Wald, a Hungarian mathematician and his research papers are available on Scribd under the title of A Method of Estimating Plane Vulnerability Based on Damage of Survivors.
    [Wow–excellent find, Ken. Thanks for the very quick suggestion. –J.]
    I believe Clayton Christensen of the “Innovator’s Dilemma” made a similar comment in one of his books about department stores. During the 80’s they probably had more data about their customers than almost any other industry and responded to that data. What they missed was the flood of customers no longer coming into their stores and why. Again, they were responding to the wrong data sets.

  2. The analogy is instructive, but only to a point. Certainly, Adobe should be seeking information from users of competing products as to why they chose the competing products, but it is even more productive to seek views from present users of Adobe products, particularly unhappy customers, BEFORE they become ex-customers. It is usually easier to keep customers than to get new ones.
    [Hence this blog. –J.]

  3. uh, sorry! is this the After Effects Blog?
    A car brings you from A to B, regardless from speed.
    A war brings you…
    A shore brings you peace of mind.
    A ssholes bring you nowhere.
    A dobe will rule the world one day.
    A customer is King, as long he behaves like a good King.
    A dobe should treat its customers like a KING!
    We have the money to buy, not you.
    Thanks for reading.

  4. On the theme of listening to customers, may I direct you to a thread on the Adobe forums where Adobe seems intent on antagonizing both current and prospective customers: N-Trig Tablet Not Recognized.
    There are a couple of rants in there, but I think it’s worthwhile to read them all, even the lengthy ones. There are a lot of really good points made, that seem to be completely getting ignored by Adobe.
    Remember the Carbon64 blowup? Well, this one isn’t there yet, but it definitely has that potential. Would be interested in seeing a response.
    [Thanks for the heads-up. I’ll check it out. –J.]

  5. Everyone is singing the same tune here guys. All John was trying to say is we need to both listen to those actively engaged in discussions, and seek out those who are not (Such as customers who have not chosen to upgrade as an example) Without both perspectives, you have an incomplete conceptual model of how to improve our product.
    When I spoke with James, it was in the context of the imagination where some individuals more or less suffer from paralysis by analysis of what they can see versus what they can imagine.

  6. Give customers what they want, not what they ask for. Those are two different things. Apple does this really well for its consumer products, but somewhat less effectively with its pro apps.

  7. This is part of why one needs to listen to customers, but only up to a point: they’ll tell you how to please the customers you have–but not the ones you don’t have.
    Um, reading this one might be forgiven in believing that Adobe actually cares what their customers think as opposed to their share holders/investors and the like.
    Two examples immediately spring to mind:-
    (1) Customer Services. How often does one read on the User to User forum of how absolutely hopeless and unhelpful users have found this experience to be.
    (2) Adobe’s pricing policy, especially as applies to their European customers. How Adobe can possibly justify a European customer living say, in Germany, being unknowingly connected to a server in Ireland from which to download an Adobe product.
    If Adobe addressed these two issues alone not only would they attract more customers but equally hold onto the ones that they already have. But of course, as experience shows, they don’t and won’t listen buy hey, nothing new in that.

  8. “Give customers what they want, not what they ask for. Those are two different things. Apple does this really well for its consumer products, but somewhat less effectively with its pro apps.”
    Apple gives customers what Jobs wants and whatever cuts costs [hence the dreadful lack of choice of kit], is a more accurate statement.
    I had to buy a unwieldy 17″ laptop to replace a far more useful 13″ laptop as that’s the only one that doesn’t have an appalingly reflective screen. How many customers need mirrors on their computer screens?
    Not only that, how many customers need a laptop with abrasive sharp edges to rest their forearms on – not comfy!

  9. Great story. Thank you.
    It is similar (but inverted) to the data communication/analysis problem that led to the Space Shuttle Challenger disaster. In that case, the analysts in favor of launch analyzed the data of all flights with damaged O-rings and calculated the correlation between the extent of O-ring damage and the temperature at launch. The correlation appeared weak.
    The engineers closest to the problem felt so strongly that it was unsafe that one said to his daughter before the fated launch, Do you want to see a major disaster tomorrow? Come up to my office and watch the launch with me.
    So why didn’t it show up in the data? The correlation analysis had excluded all O-rings that had zero breaks — that were undamaged. All of the undamaged flights had been in warm weather. Including the large number of undamaged, warm-weather flights made the correlation extremely strong.
    My thanks to Prof. Jenny Chatman of Haas, who teaches a masked version of this story in a way that leaves an indelible impression on impressionable MBA students — lessons in not just data anlaysis, but also data communication, group dynamics and groupthink.

  10. I think the more pertinent point here is that it’s up to Adobe to sift through the drudges that is badly-articulated customer feedback and demands, and figure out exactly what solution will work. Present this back to customers and see if it gels. If not, back to the drawing board.
    That’s exactly what Chris Cox in the tablet thread didn’t do.

  11. Just a quick note to say that I enjoy your blog, John. I was a PM in the software industry for 10 years and being the PM of a successful product is like juggling 10 chainsaws at once. Lots of enhancement requests = lots of tough decisions and it is inevitable that you will drop something that ultimately disappoints some customer(s) who will want to cut your arm off – I feel your pain. 🙂
    [Heh–thanks, James. –J.]

  12. Longtime user here, and got ACE certification about ten years ago. I used Photoshop for work at major firms; I guarantee most of you have seen pics I’ve retouched for various ads and publications.
    With that said, Photoshop has gone downhill. I used to *love* Photoshop, and thought its interface was the model of perfection. Now, with each new release, the interface keeps getting *worse.* And it’s not because it’s different. It’s actually worse.
    And now, CS4 and CS5 actually *removed* functionality! Give me a project, and I can crank out the work faster and with less problems with CS2. I work in the real world, and the little things Adobe has been doing since CS2 has been making the product worse.
    With the exception of CS3’s Intel Mac compatibility and Smart Filters feature, CS2 was the last REAL, tried-and-true Photoshop release. The ones that came after are shameful.
    [I’m sorry you feel this way. Unfortunately you haven’t given me any examples of what, specifically, has made you unhappy, and your pseudonym and fake email address make it impossible for me to follow up with you. I hope you’ll write again and offer some constructive criticism. Otherwise it’s just kind of a burning bag left on the doorstep. –J.]

Leave a Reply

Your email address will not be published.