David Lindsay still remembers the chaos. As a machine learning engineer on DoorDash’s merchant team, he stared at menu data that refused to behave. One restaurant called a dish “spicy Sichuan noodles.” Another listed the same item as “mala beef chow fun, extra heat.” Images showed red chili flakes in both, yet traditional tagging systems flagged them inconsistently. At the scale of millions of unique items and hundreds of thousands of daily menu changes, the problem wasn’t theoretical. It blocked search, personalization, and restaurant success.
So Lindsay, along with colleagues Shuang Liu and Ying Yang, built something different. The team at DoorDash detailed their approach in a July 2, 2026, post on the company’s engineering blog. They combined vision-language models, a jury of large language models acting as evaluators, autonomous agents that rewrite prompts on the fly, and heavy distributed computing. The result? Metadata generation that runs at scale, costs far less, and beats human reviewers on accuracy. Careers at DoorDash laid out the full architecture.
Food data resists structure. Dishes cross cultures. Descriptions vary by chef, region, even the time of day. Names collide. Photos tell part of the story that text misses. DoorDash’s catalog reflects this mess. The company serves vastly different merchants, each expressing items in unique ways. Without clean attributes — spicy or not, cuisine type, preparation method, dietary flags — recommendations falter and search returns junk.
Traditional fixes fell short. Human labelers couldn’t keep up. Domain experts who could distinguish Nepalese from North Indian dishes or decode “Sichuan-style” heat levels were too few and too expensive. Scaling annotation across billions of catalog entities looked impossible. So the team flipped the script. They let AI generate the metadata, then judge its own work.
Enter the LLM jury. Instead of one model offering a verdict or one human rater, multiple strong evaluators review each proposed tag independently. They vote. They explain. The system aggregates those votes into a consensus. It checks tags one by one — protein content, preparation style, health attributes — rather than scoring the entire item at once. The approach lifted annotation accuracy by roughly 20 percent over typical human reviewers. “Our automated LLM-based consensus evaluation system — LLM juries — replaces slow, costly, and inconsistent human validation,” the authors wrote.
That jury sits at the heart of a closed loop. Generated tags feed into evaluation. Failures feed back into prompt improvement. The system doesn’t just label. It learns what it gets wrong and adjusts.
Context optimization agents drive the learning. Prompt engineering by hand is brittle. Tiny wording shifts produce wildly different outputs. Edge cases multiply as new menus arrive. The DoorDash team built agents that treat prompt refinement like reinforcement learning. They scan failure cases from the jury’s high-quality evaluation set. They propose targeted changes to the context provided to the generator model. They test, measure precision and recall on hold-out data, and iterate.
The agents improved precision more than 20 percent on hold-out sets. They accelerated prompt development tenfold. Humans no longer spend weeks tweaking prompts. The loop runs in minutes. And it favors failure signals over success stories. “Failure cases carry more signal than successes,” the engineers observed after testing combinations. Data quality matters above all. A noisy evaluation set sends the agent chasing ghosts.
They rejected population-based methods such as evolutionary algorithms that mutate random prompt variants. Those require scoring many candidates per round and tuning extra hyperparameters. The failure-driven agent reads specific mistakes and suggests precise rule additions. Each step stays purposeful. Convergence happens fast because the prompt changes mirror weight updates in model training, yet cost far less.
Multimodal signals add another layer. Text alone misses visual cues — the glistening oil that signals deep-fried, the green herbs that hint at freshness. Vision-language models ingest both menu descriptions and dish photos. The team also pulls signals from broad web searches when local data stays ambiguous. All of it flows into the generator before the jury weighs in.
But high-quality generation still demanded cheap, fast models for production. Frontier LLMs delivered accuracy yet carried prohibitive inference costs at DoorDash scale. The answer lay in AI-led annotation. The same jury-and-agent pattern creates training labels automatically. Dedicated generation and evaluation agents label thousands of examples across many tags. No humans required. The resulting data trains small language models that match frontier performance at roughly 10 percent of the inference cost. Inference drops by about 90 percent overall while quality holds.
Scale still loomed. Millions of unique items. Billions of options. Daily updates. A naive system calling APIs one item at a time would take over a month for a full backfill. Costs would explode. The team engineered a distributed inference pipeline built on Spark. First, deduplication removes identical items shared across merchants. Exact feature matches prevent redundant calls. Remaining unique payloads split across a cluster of workers. Batch APIs group requests for higher throughput. For fine-tuned models, data shards across GPUs. After inference, results map back to every original entity. Backfill time collapsed from over a month to just a few days. The pipeline now refreshes metadata continuously.
Structural validation catches obvious errors early and triggers retries. Merchants retain override rights. They can correct tags when the AI misses nuance. That human-in-the-loop safeguard keeps trust high. Yet the bulk of the work stays automated.
The payoff shows up for customers. Structured attributes power better search. Filters surface exactly the spicy dishes or vegan options someone wants. Personalization grows sharper because the system understands that “Sichuan-style” carries heat and that a photo of bubbling cheese means comfort food. Analytics improve. Restaurant discovery gets easier. Downstream applications that once fought noisy data now run on clean signals.
This work builds on earlier DoorDash experiments with LLM judges. In May 2026 the company published details on using LLM-as-a-judge to evaluate natural language search. Authors Xiaochang Miao and Heather Song described decomposing queries into binary facets — does this match “tacos under $20”? — and calibrating against human golden sets. They enriched judge context with item-level pricing, customization flags, and store attributes. The facet-based approach exposed gaps hidden by aggregate metrics and reduced label noise. Careers at DoorDash captured that effort. The food metadata project extends the same evaluation discipline into catalog building.
Recent discussions on X highlight growing interest. Engineers noted DoorDash’s LLM evals in early July 2026. One machine learning roundtable in Japan referenced the company’s offline LLM carousel generation alongside multimodal work from other firms. A Spanish-language post praised the jury system for delivering 20 percent higher precision at 90 percent lower cost. The ideas travel fast.
Yet challenges remain. Model collapse risks rise when systems train repeatedly on AI-generated data. The DoorDash team guards against that through continuous jury oversight and merchant overrides. Data quality still anchors everything. Low-quality evaluation sets derail optimization agents. Cultural nuances in cuisine demand careful rubric design.
The architecture offers lessons beyond food delivery. Any domain with messy, high-volume, contextual data — retail catalogs, medical records, legal documents — faces similar hurdles. Consensus evaluation, autonomous prompt agents, AI-generated training labels, and distributed inference form a template. Companies can replace slow human loops with measurable, fast, cheap AI loops. But only if they treat evaluation as foundational.
Lindsay, Liu, and Yang positioned their platform as more than a fix for menus. It creates a semantic understanding of every item and store. Unstructured text becomes rich attributes. Generative AI moves from experimental to production-grade. The infrastructure supports today’s search while laying groundwork for tomorrow’s hyper-personalized experiences.
DoorDash isn’t alone in applying AI to food. China’s Meituan, often called the local DoorDash equivalent, open-sourced a 1.6-trillion-parameter model in late June 2026 that beat certain OpenAI benchmarks on coding tasks. That effort, trained on domestic chips despite U.S. export controls, shows how food platforms now push frontier AI boundaries. Yet Meituan’s work targeted general capabilities. DoorDash focused on the gritty, practical problem of catalog quality. Both signal the same trend: operational data problems at massive scale now drive serious AI research inside non-AI companies.
The jury system itself echoes broader shifts in evaluation. Single-model judges carry bias and inconsistency. Multiple evaluators with voting reduce variance. Structured rubrics and chain-of-thought prompting improve alignment with human judgment. DoorDash calibrated its juries against expert sets, then let them run continuously. The approach scales where periodic human annotation cannot.
Look closer at the optimization loop and parallels to software engineering emerge. The agent reads failure logs much like a developer reads stack traces. It proposes context changes the way an engineer adds guard clauses. Metrics guard against regression the way unit tests do. The tenfold speedup in prompt development mirrors productivity gains from AI coding assistants. Engineers now spend less time on brittle manual tuning and more on system design.
Cost numbers tell a stark story. Small fine-tuned models slash inference expense by 90 percent. Distributed batching and deduplication shrink compute further. What once required a month of wall-clock time now finishes in days. That practicality matters. Many AI projects stall at prototype because inference costs or latency make production impossible. DoorDash cleared those barriers.
Merchants benefit directly. Accurate metadata helps surface their dishes to the right customers. Overrides let them correct the record when AI misses a family recipe’s secret. The system respects their expertise while handling the vast majority of routine tagging.
Customers notice better recommendations without realizing the machinery underneath. They search for “quick healthy lunch” and receive options that actually match. Filters for dietary needs work reliably. Discovery improves because the platform finally understands what each photo and description truly signals.
The team emphasized one lesson repeatedly. Evaluation quality bounds everything else. A weak jury produces noisy signals that mislead the optimization agents. Low-quality training data from annotation loops produces weak fine-tuned models. Context matters more than raw model size. Enriching prompts with pricing flags, customization options, or display logic often fixed “judge failures” that were really missing information. The May search evaluation post made the same point: many failures traced back to incomplete context.
So the platform keeps evolving. New menu patterns appear. Cultural dishes gain popularity. Photos change with seasonal ingredients. The jury monitors. The agents adjust. Distributed pipelines refresh. The loop stays alive.
DoorDash has turned a chaotic catalog into structured knowledge. In doing so, it showed how AI systems can self-improve at scale without constant human oversight. The methods travel. Other industries will copy the jury, the agents, the distributed inference. Food just happened to be the proving ground. And the results taste pretty good.
WebProNews is an iEntry Publication