The biggest U.S. food makers, already dealing with increased competition and shifting consumer tastes, now face an additional threat: supermarkets are taking away prime shelf space.
Grocers are relying on their own proprietary research to decide how and where to shelve certain products, rather than relying on companies that sell well-known brands to tell them what to put on what shelf at what price. The shift is resulting in less space for traditional supermarket staples from companies such as General Mills Inc. and Clorox Co. in favor of niche items and store brands that deliver higher margins and are often in higher demand.
"Our retailers have better information now," General Mills Chief Executive Jeff Harmening said. "So more of our conversation is about 'How do we drive growth together?'"
Retailers such as Kroger Co. and Walmart Inc. are using increasingly sophisticated software to decide where to place items and which products to shelve next to one another—factors that can move sales up or down several percentage points—according to food-industry executives. The software, which can incorporate video-surveillance and other data, helps them create so-called planograms of the products on their shelves.
Grocers are incorporating into the software's algorithms such metrics as "walk rates," which measure how much time a typical customer is willing to spend looking for certain products before giving up and leaving without buying anything. High-selling products with short walk rates get to be in the "strike zone," just below eye level, where retailers often put products they want consumers to notice, said Phil Stanley, chief sales officer at Hershey Co.
"There's a lot more precision in how we think about the aisle today," said Mr. Stanley, who added that Hershey is stepping up its own analysis capabilities.
Kroger, the largest U.S. supermarket chain, has invested in beefing up its ability to collect and analyze data from customers. That is changing the grocer's relationships with suppliers and the way it lays out stores, said Michael McGowan, vice president of the company's data analytics division.
"We're also looking at customer loyalty metrics to assess how folks engage with brands," Mr. McGowan said.
The increased use of software and proprietary data by grocers is dealing another blow to the makers of decades-old packaged foods, which in many cases have fallen out of step with trends toward more natural and healthy foods and lost some influence over grocers.
The diminished power of "category captains"—the top sellers of products such as soup or cereal—is the biggest change to the way food is sold since Walmart Inc. expanded its grocery offerings 30 years ago, industry veterans say.
Retailers once relied on big consumer-goods companies when making decisions about allocating shelf space because the companies were the experts in their respective food categories, such as Campbell Soup Co. in soup, Kellogg Co. in cereal, and Kraft Heinz Co. in cheese and condiments. Grocers also didn't want to invest in consumer insights, and they were happy to take the hefty slotting fees big brands pay for prime space.
Now, retailers are more focused on doing what it takes to maximize sales growth even if it means giving up some of those fees by stocking more of their store-branded products. "Retailers began to realize that when they rely heavily on category captains, they are at a disadvantage because there's an inherent bias," said AlixPartners managing partner Jonathan Greenway, who consults for brands and retailers.
Over the past 18 months, as retailers started to develop their own systems, the category-captain role began dissolving, said packaged-food analyst Alexia Howard of Bernstein.
"They have a big brand telling them to do it one way, and their own analytics telling them to feature more private-label or challenger brands," she said.
Now, grocers are devoting more space to new products to show customers that they offer, for example, the trendiest varieties of healthy snacks and infused beverages. Some $17 billion of annual packaged-food sales, roughly 3.5% of the market, has shifted from incumbents to startups since 2013, according to Nielsen data.
Large, publicly traded food makers have faced declining retail distribution over the past year. In January, their total points of distribution were down 2.8% from a year earlier, according to Nielsen data.
General Mills' sales of baking mixes and ingredients have declined over the past five years. The company's Betty Crocker, Bisquick and Gold Medal flour brands are losing shoppers to smaller rivals and store brands. For example, Kodiak Cakes LLC, a family-run maker of high-protein pancake mix, has taken market share from Bisquick and other competitors, according to market-research firm Spins. Even though Kodiak's annual revenue is less than one-tenth of General Mills' baking sales, Kodiak is gaining shelf space.
"We are the ones with the growth," said Joel Clark, Kodiak's founder and chief executive.
Clorox said earlier this month that its quarterly sales were hurt by retailers' decisions to cut back on shelf space for Glad products in response to higher prices .
General Mills said it is working harder to provide more insights than retailers or smaller brands can in order to remain relevant to grocery executives. It is also trying not to be biased.
"Our competitive advantage doesn't come from looking at two different versions of truth and trying to talk them into something they shouldn't do," Mr. Harmening said.
Supermarkets are also gaining leverage over retailers with generic products sold under their own brands at cheaper prices than name-brand goods. Kroger owns 33 manufacturing plants to make various store-branded products such as maple syrup and flour. Those products make up a growing share of its sales and shelf space.
A former Kroger executive said the relationship between retailers and food makers has changed. "It used to be more of a personal relationship," he said. "It's become cold, hard facts that make decisions."