Understand Your Market AKA Normal Distribution vs. Power-Law Distribution

Just phenomenal article in the 13-20 Feb, 2006 article in the New Yorker by Malcolm Gladwell.

Basically, using nothing more than basic statistical theory, the article summarizes three examples where instead of focusing on the broad middle of the population, you could focus on the far left of the curve and nail the vast majority of the problem. The items listed were:
  • The LAPD: Over a 5 year period (1986 to 1990), 1800 officers were charged with use of excessive force. When you look closer at the data, it turned out that the vast majority (1400) had only one or two allegations (it is important to note that these are all allegations, and not actual findings). In fact, the deeper you looked, it showed that 44 cops basically made up the VAST majority of complaints. The 8500 LAPD officers weren't the problem, less than 0.5% of them were.

  • Homelessness: Analysis of the homeless found that 250,000 people were homeless over the five years previous to the early 1990's. But when you looked closer, it was just 2500 people that accounted for the MAJORITY of the costs; in specific, those 2500 people accounted for roughly $62M a year in health care and other costs. In Boston, tracking of 119 homeless showed they accounted for 18,834 emergency room visits at a minimum cost of $1,000 a piece. One homeless individual had been to the emergency room 87 times.

  • Car emissions: 5% of all vehicles produce 55% of the auto pollution. A poorly tuned (either engine neglect or extremely high mileage) car can produce 200x more pollution than a new well-tuned car.


I am entirely sure you could find these examples of these power-law distributions all over the place. But Mr. Gladwell goes on to identify the real problem... it's not that smart people aren't doing the analysis to identify these, it's that we don't know what we're after. The example solution for homelessness, as an example, is to find the top people (maybe 500 a year) and get them off the street by showering them with attention and the tools to do so. This would entail free apartment, daily counselor visits, job placement, etc. This may seem costly, but it'll actually be MUCH cheaper than trying to solve their problems through existing methods (just giving them enough healthcare to get them walking out the door).

My thoughts are basically to clarify what you are looking for in the clearest possible examples. If your goal is to reduce pollution, don't bother requesting that everyone gets their car checked just because it's egalitarian. That's meaningless! It doesn't do any additional significant good. Just find the worst polluters and solve their problems (pay for it if you have to)! If your goal is to solve homelessness, then solve it. Pay for the work to get the people who tax the system off the street, and use the savings to help the broad middle. If your goal is to stop police brutality, don't bother continually training the common police officer, they're not doing anything bad. Focus on the worst and get rid of them. Equal treatment only works if everyone is the same. Everyone, in these cases, are most definitely NOT the same.

And for those who are worried because someone who spends a week on the street before they find a new job gets significantly less than someone who has cirrhosis of the liver, that doesn't matter, since the first person doesn't cost the system as much. If you simply constrain the lists (make it just the top 500 out of the top 1000 cases), people will not wait around or drink themselves silly just to get to the top of these special lists. People are more rational than that.