As christmas approached in 2015, the price of pumpkin-pie spice went wild. It didn’t soar, as an economics textbook might suggest. Nor did it crash. It just started vibrating between two quantum states. Amazon’s price for a one-ounce jar was either $4.49 or $8.99, depending on when you looked. Nearly a year later, as Thanksgiving 2016 approached, the price again began whipsawing between two different points, this time $3.36 and $4.69.
We live in the age of the variable airfare, the surge-priced ride, the pay-what-you-want Radiohead album, and other novel price developments. But what was this? Some weird computer glitch? More like a deliberate glitch, it seems. “It’s most likely a strategy to get more data and test the right price,” Guru Hariharan explained, after I had sketched the pattern on a whiteboard.
The right price—the one that will extract the most profit from consumers’ wallets—has become the fixation of a large and growing number of quantitative types, many of them economists who have left academia for Silicon Valley. It’s also the preoccupation of Boomerang Commerce, a five-year-old start-up founded by Hariharan, an Amazon alum. He says these sorts of price experiments have become a routine part of finding that right price—and refinding it, because the right price can change by the day or even by the hour. (Amazon says its price changes are not attempts to gather data on customers’ spending habits, but rather to give shoppers the lowest price out there.)
It may come as a surprise that, in buying a seasonal pie ingredient, you might be participating in a carefully designed social-science experiment. But this is what online comparison shopping hath wrought. Simply put: Our ability to know the price of anything, anytime, anywhere, has given us, the consumers, so much power that retailers—in a desperate effort to regain the upper hand, or at least avoid extinction—are now staring back through the screen. They are comparison shopping us.
They have ample means to do so: the immense data trail you leave behind whenever you place something in your online shopping cart or swipe your rewards card at a store register, top economists and data scientists capable of turning this information into useful price strategies, and what one tech economist calls “the ability to experiment on a scale that’s unparalleled in the history of economics.” In mid-March, Amazon alone had 59 listings for economists on its job site, and a website dedicated to recruiting them.
Not coincidentally, quaint pricing practices—an advertised discount off the “list price,” two for the price of one, or simply “everyday low prices”—are yielding to far more exotic strategies.
“I don’t think anyone could have predicted how sophisticated these algorithms have become,” says Robert Dolan, a marketing professor at Harvard. “I certainly didn’t.” The price of a can of soda in a vending machine can now vary with the temperature outside. The price of the headphones Google recommends may depend on how budget-conscious your web history shows you to be, one study found. For shoppers, that means price—not the one offered to you right now, but the one offered to you 20 minutes from now, or the one offered to me, or to your neighbor—may become an increasingly unknowable thing. “Many moons ago, there used to be one price for something,” Dolan notes. Now the simplest of questions—what’s the true price of pumpkin-pie spice?—is subject to a Heisenberg level of uncertainty.
Which raises a bigger question: Could the internet, whose transparency was supposed to empower consumers, be doing the opposite?
If the marketplace was a war between buyers and sellers, the 19th-century French sociologist Gabriel Tarde wrote, then price was a truce. And the practice of setting a fixed price for a good or a service—which took hold in the 1860s—meant, in effect, a cessation of the perpetual state of hostility known as haggling.
As in any truce, each party surrendered something in this bargain. Buyers were forced to accept, or not accept, the one price imposed by the price tag (an invention credited to the retail pioneer John Wanamaker). What retailers ceded—the ability to exploit customers’ varying willingness to pay—was arguably greater, as the extra money some people would have paid could no longer be captured as profit. But they made the bargain anyway, for a combination of moral and practical reasons.
The Quakers—including a New York merchant named Rowland H. Macy—had never believed in setting different prices for different people. Wanamaker, a Presbyterian operating in Quaker Philadelphia, opened his Grand Depot under the principle of “One price to all; no favoritism.” Other merchants saw the practical benefits of Macy’s and Wanamaker’s prix fixe policies. As they staffed up their new department stores, it was expensive to train hundreds of clerks in the art of haggling. Fixed prices offered a measure of predictability to bookkeeping, sped up the sales process, and made possible the proliferation of printed retail ads highlighting a given price for a given good.
Companies like General Motors found an up-front way of recovering some of the lost profit. In the 1920s, GM aligned its various car brands into a finely graduated price hierarchy: “Chevrolet for the hoi polloi,” Fortune magazine put it, “Pontiac … for the poor but proud, Oldsmobile for the comfortable but discreet, Buick for the striving, Cadillac for the rich.” The policy—“a car for every purse and purpose,” GM called it—was a means of customer sorting, but the customers did the sorting themselves. It kept the truce.
Customers, meanwhile, could recover some of their lost agency by clipping coupons—their chance to get a deal denied to casual shoppers. The new supermarket chains of the 1940s made coupons a staple of American life. What the big grocers knew—and what behavioral economists would later prove in detail—is that while consumers liked the assurance the truce afforded (that they would not be fleeced), they also retained the instinct to best their neighbors. They loved deals so much that, to make sense of their behavior, economists were forced to distinguish between two types of value: acquisition value (the perceived worth of a new car to the buyer) and transaction value (the feeling that one lost or won the negotiation at the dealership).
The idea that there was a legitimate “list price,” and that consumers would occasionally be offered a discount on this price—these were the terms of the truce. And the truce remained largely intact up to the turn of the present century. The reigning retail superpower, Walmart, enforced “everyday low prices” that did not shift around.
But in the 1990s, the internet began to erode the terms of the long peace. Savvy consumers could visit a Best Buy to eyeball merchandise they intended to buy elsewhere for a cheaper price, an exercise that became known as “showrooming.” In 1999, a Seattle-based digital bookseller called Amazon.com started expanding into a Grand Depot of its own.
The era of internet retailing had arrived, and with it, the resumption of hostilities.
In retrospect, retailers were slow to mobilize. Even as other corporate functions—logistics, sales-force management—were being given the “moneyball” treatment in the early 2000s with powerful predictive software (and even as airlines had fully weaponized airfares), retail pricing remained more art than science. In part, this was a function of internal company hierarchy. Prices were traditionally the purview of the second-most-powerful figure in a retail organization: the head merchant, whose intuitive knack for knowing what to sell, and for how much, was the source of a deep-seated mythos that she was not keen to dispel.
Two developments, though, loosened the head merchant’s hold.
The first was the arrival of data. Thomas Nagle was teaching economics at the University of Chicago in the early 1980s when, he recalls, the university acquired the data from the grocery chain Jewel’s newly installed checkout scanners. “Everyone was thrilled,” says Nagle, now a senior adviser specializing in pricing at Deloitte. “We’d been relying on all these contrived surveys: ‘Given these options at these prices, what would you do?’ But the real world is not a controlled experiment.”
The Jewel data overturned a lot of what he’d been teaching. For instance, he’d professed that ending prices with .99 or .98, instead of just rounding up to the next dollar, did not boost sales. The practice was merely an artifact, the existing literature said, of an age when owners wanted to force cashiers to open the register to make change, in order to prevent them from pocketing the money from a sale. “It turned out,” Nagle recollects, “that ending prices in .99 wasn’t big for cars and other big-ticket items where you pay a lot of attention. But in the grocery store, the effect was huge!”
The effect, now known as “left-digit bias,” had not shown up in lab experiments, because participants, presented with a limited number of decisions, were able to approach every hypothetical purchase like a math problem. But of course in real life, Nagle admits, “if you did that, it would take you all day to go to the grocery store.” Disregarding the digits to the right side of the decimal point lets you get home and make dinner.
By the early 2000s, the amount of data collected on retailers’ internet servers had become so massive that it started exerting a gravitational pull. That’s what triggered the second development: the arrival, en masse, of the practitioners of the dismal science.
This was, in some ways, a curious stampede. For decades, academic economists had generally been as indifferent to corporations as corporations were to them. (Indeed, most of their models barely acknowledged the existence of corporations at all.)
But that began to change in 2001, when the Berkeley economist Hal Varian—highly regarded for the 1999 book Information Rules—ran into Eric Schmidt. Varian knew him but, he says, was unaware that Schmidt had become the CEO of a little company called Google. Varian agreed to spend a sabbatical year at Google, figuring he’d write a book about the start-up experience.
At the time, the few serious economists who worked in industry focused on macroeconomic issues like, say, how demand for consumer durables might change in the next year. Varian, however, was immediately invited to look at a Google project that (he recalls Schmidt telling him) “might make us a little money”: the auction system that became Google AdWords. Varian never left.
Others followed. “eBay was Disneyland,” says Steve Tadelis, a Berkeley economist who went to work there for a time in 2011 and is currently on leave at Amazon. “You know, pricing, people, behavior, reputation”—the things that have always set economists aglow—plus the chance “to experiment at a scale that’s unparalleled.”
At first, the newcomers were mostly mining existing data for insights. At eBay, for instance, Tadelis used a log of buyer clicks to estimate how much money one hour of bargain-hunting saved shoppers. (Roughly $15 was the answer.)
Then economists realized that they could go a step further and design experiments that produced data. Carefully controlled experiments not only attempted to divine the shape of a demand curve—which shows just how much of a product people will buy as you keep raising the price, allowing retailers to find the optimal, profit-maximizing figure. They tried to map how the curve changed hour to hour. (Online purchases peak during weekday office hours, so retailers are commonly advised to raise prices in the morning and lower them in the early evening.)
By the mid-2000s, some economists began wondering whether Big Data could discern every individual’s own personal demand curve—thereby turning the classroom hypothetical of “perfect price discrimination” (a price that’s calibrated precisely to the maximum that you will pay) into an actual possibility.
As this new world began to take shape, the initial consumer experience of online shopping—so simple! and such deals!—was losing some of its sheen.
It’s not that consumers hadn’t benefited from the lower prices available online. They had. But some of the deals weren’t nearly as good as they seemed to be. And for some people, glee began to give way to a vague suspicion that maybe they were getting ripped off. In 2007, a California man named Marc Ecenbarger thought he had scored when he found a patio set—list price $999—selling on Overstock.com for $449.99. He bought two, unpacked them, then discovered—courtesy of a price tag left on the packaging—that Walmart’s normal price for the set was $247. His fury was profound. He complained to Overstock, which offered to refund him the cost of the furniture.
But his experience was later used as evidence in a case brought by consumer-protection attorneys against Overstock for false advertising, along with internal emails in which an Overstock employee claimed it was commonly known that list prices were “egregiously overstated.”
In 2014, a California judge ordered Overstock to pay $6.8 million in civil penalties. (Overstock has appealed the decision.) The past year has seen a wave of similar lawsuits over phony list prices, reports Bonnie Patten, the executive director of TruthinAdvertising.org. In 2016, Amazon began to drop most mentions of “list price,” and in some cases added a new reference point: its own past price.
This could be seen as the final stage of decay of the old one-price system. What’s replacing it is something that most closely resembles high-frequency trading on Wall Street. Prices are never “set” to begin with in this new world. They can fluctuate hour to hour and even minute to minute—a phenomenon familiar to anyone who has put something in his Amazon cart and been alerted to price changes while it sat there. A website called camelcamelcamel.com even tracks Amazon prices for specific products and alerts consumers when a price drops below a preset threshold. The price history for any given item—Classic Twister, for example—looks almost exactly like a stock chart. And as with financial markets, flash glitches happen. In 2011, Peter A. Lawrence’s The Making of a Fly(paperback edition) was briefly available on Amazon for $23,698,655.93, thanks to an algorithmic price war between two third-party sellers that had run amok. To understand what happened, it seemed sensible to talk to the man who helped develop the software they were using.
Guru hariharan uncapped a dry-erase marker in a conference room at Boomerang’s headquarters in Mountain View, California. He was talking about what had led retailers to this desperate place where it’s necessary to change prices multiple times a day. On a whiteboard, he drew a series of lines representing the rising share of online sales for various kinds of products (books, DVDs, electronics) over time, then marked the years that major brick-and-mortar players (Borders, Blockbuster, Circuit City and RadioShack) went bankrupt. At first the years looked random. But the bankruptcies all clustered within a band where online sales hit between 20 and 25 percent. “In this range, there’s a crushing point,” Hariharan said, clapping his hands together for emphasis. “There’s a bloodbath happening.”
Beyond this crushing point, traditional retailers with both a brick-and-mortar and an online presence feel compelled to compete purely on price. Hariharan talked wistfully of the days when he’d walk into RadioShack and have a salesperson direct him to the exact connector cable he needed. But once retailers enter the crushing zone, expenses like staff, training, and customer support typically are slashed. Profit margins keep falling nonetheless—why go to the store at all if no one there can help you?—and a death spiral ensues. (RadioShack traced just this path before filing for bankruptcy in 2015.)
“It didn’t have to be that way,” Hariharan said. Now he’s helping retailers fight back.
We can’t process every piece of price information thrown our way. So we judge a store’s prices based on a handful of products we know well. Grocers have recognized this for decades, which is why they keep the price of eggs and milk consistently low, making their profits on other goods whose markups we don’t notice as easily.
When he was at Amazon, Hariharan, who has a degree in machine learning, helped invent and patent the Amazon Selling Coach, a system that helps third-party vendors optimize their inventory and prices. He and his team at Boomerang have built a massive system that tracks prices and has informed billions of pricing decisions for clients ranging from Office Depot to GNC to U.S. Auto Parts. But its software engine isn’t built to match the lowest price out there. (That, Hariharan notes, would be a simple algorithm.) It’s built to manage consumers’ perception of price. The software identifies the goods that loom largest in consumers’ perception and keeps their prices carefully in line with competitors’ prices, if not lower. The price of everything else is allowed to drift upward.
Amazon long ago mastered this tactic, Hariharan says. In one instance, Boomerang monitored the pricing shifts of a popular Samsung television on Amazon over the six-month period before Black Friday. Then, on Black Friday itself, Amazon dropped the TV’s price from $350 all the way to $250, undercutting competitors by a country mile. Boomerang’s bots also noticed that in October, Amazon had hiked the price of some HDMI cables needed to connect the TV by about 60 percent, likely armed with the knowledge, Hariharan says, that online consumers do not comparison shop as zealously for cheaper items as they do for expensive ones.
What’s interesting is how other retailers are now beginning to adapt. To show me this, a Boomerang employee opened up the dashboard seen by the firm’s clients. Scrolling through a menu of premade algorithms, he selected a rule, “Beat Competitor by 10%,” for certain items meeting the following criteria:
If (comp_price>cost) and (promo_flag = false) then set price = comp_price*0.90
That is: If the competitor’s price is greater than the cost of making the item, and the competitor isn’t running a onetime promotion, then undercut the competitor by 10 percent. The rule was implemented with a click, and onscreen, I could see a healthy drop in the client’s Price Perception Index.
But that’s not the end of the story. The price cuts will register on competitors’ pricing sonars. Whether or not to respond in kind depends, in part, on how theiralgorithms interpret the signal. Is this the first shot in a pricing war? Or is the retailer just trying to clear inventory from its warehouse? In practice, it’s hard to tell. So an innocuous, temporary price cut may set off a machine-against-machine price war that, if left unchecked, could quickly devastate a retailer’s bottom line. Boomerang clients are prompted to select “Guardrails”—further rules that provide a check on the initial set of rules—and establish a certain amount of human oversight. Faisal Masud, the chief technology officer at Staples, one of Boomerang’s first customers, thinks human involvement makes sense only in rare cases. “We want to make sure the software makes the decisions, not the human being,” he says. “It’s all automatic. Otherwise you’re losing.”
The complexity of retail pricing today has driven at least one of Boomerang’s clients into game theory—a branch of mathematics that, it’s safe to say, has seldom found practical use in shopping aisles. Hariharan says, with a smile: “It lets you say, ‘What is the dominant competitor’s reaction to me? And if I know the reaction to me, what is my first, best move?’ Which is the Nash equilibrium.” Yes, that’s John Nash, the eponymous Beautiful Mind, whose brilliant contributions to mathematics now extend to the setting of mop prices.[“Source-theatlantic”]