1,000 Simultaneous Experiments

Lots of insights to unpack in Gigi Levy-Weiss's latest post on the NFX blog. And perhaps a few provocative ideas that don't apply everywhere all the time.

But it captures the zeitgeist and zeroes in on something essential: complex (software-based) experiments can be orchestrated cheaply, at scale by a handful of people.

In 1998, I spend some time working with Jim Donehey, then the CIO at Capital One, on growing an IT organization from 50 to 1,800 in just a couple of years. Why the crazy headcount growth? Because of the company's then-subversive (and now embraced by all retail financial institutions) "Information-Based Strategy", that is, the use of data to match specific products to individual customer risk and behavior.

It's hard now to believe that this was a disruption to the credit-card industry, but they were called crazy for running these insane information-gathering mailing campaigns, sending different offers to tens of thousands of people, measuring their responses and the subsequent credit behavior of those who accepted offers. The strategy was extreme multivariate testing and it proved extremely successful at uncovering niches that no other credit card company had touched. One example: foreign nationals working in the US who didn't have a credit record. The rest is history, Capita One became the largest player in credit cards and expanded to other forms of credit.

By running tens of thousands of experiments simultaneously.

But only a well-capitalized bank with 1,800 IT employees and 400 contractors could afford it. Not anymore. 3 people on a shoestring budget can do it.