Building to Learn: What Three Weeks of Prototyping Taught Us About Agentic Marketing Optimization

A product description can win every comparison in a lab and still never appear in a shopping assistant.

The score answers a narrower question: does this description address the needs expressed in a set of test scenarios, according to the evaluators I chose? It does not establish whether a shopping service will retrieve the product, rank it, or change what a customer buys.

That gap shaped the three-week prototype I built in February 2026 to explore product discoverability in AI shopping. The work produced a more useful result than a ranking predictor: a clearer account of what simulation can test, what requires observation outside the lab, and what would be needed to connect the two.

Three questions, three kinds of evidence

“AI discoverability” can refer to several outcomes. They need different tests.

Question	What it asks	Evidence that can answer it
Does the content address the stated need?	Are relevant features, benefits, constraints, and use cases present and clear?	A structured review, scenario comparison, or model-assisted content evaluation.
Does a shopping surface show the product?	Does the product appear for a defined query, account, market, time, and context?	Repeated observation of that surface or reporting provided by the platform, with its limits recorded.
Does the change improve a business outcome?	Did the change increase qualified visits, purchases, margin, or another intended result?	Downstream analytics and, for causal claims, a suitable controlled comparison.

A lab can help with the first question. It can generate hypotheses about the second. It cannot answer the second or third using its own semantic score alone.

I started with a hypothesis: richer context could help a model represent what a shopper needs, and those representations could help assess product content. I used the working name Context-Conditioned Intent Activation, or CCIA, also explored in The Geometry of Intention. The questions differ: Geometry asks whether context improves interpretation of a person’s goal; this lab assessed content against supplied or generated needs. Better product-fit judgments would not validate inference of a real shopper’s intention. CCIA remains a hypothesis, not an established theory or optimization method.

Research on intent classification supports a narrower proposition: language models can use intent descriptions in some task-oriented classification settings. Those results do not establish that a system can reliably infer an individual shopper’s goals, predict a commerce assistant’s choices, or improve rankings. The distance between those claims is the experiment.

How the lab changed over three weeks

In the first week, I planned a chat interface for exploring intent, an alignment score, and basic content suggestions. That design assumed a score could tell a team what to change. When an embedding-based score returned a number without showing which product need was or was not addressed, I moved toward decomposable signals: whether a benefit, use case, or constraint appeared in the description. The change made a result easier to inspect. It did not make the result a ranking signal.

The first query generator exposed a different problem. Starting with product features tended to produce queries that echoed the brand’s language or became too specific. I added top-down, bottom-up, and hybrid ways to form scenarios, along with category checks and input quality gates. These controls made the test set easier to examine; they could not establish that it represented what shoppers actually ask.

I also moved from one model judge to several. Different judges sometimes disagreed, which showed how much an evaluation depended on the model doing the judging. Agreement offered a comparison across evaluators, not independent confirmation that a product would appear in a shopping service. That distinction led me to separate synthetic judgments from observations made on external shopping surfaces, and to keep recommendations based only on lab scores provisional.

By the third week, the planned scorer had grown into a research workflow. It generated intents and scenarios, assembled query batteries, compared content variants, retested them with multiple judges, and kept synthetic judgments separate from observations of external shopping surfaces. I also described scoped memory and calibration functions in the build account. The learning layer was Bayesian-style design guidance, not exact Bayesian updating by the language models.

These details record how I understood the February prototype; no repository commit or reproducible run report accompanies this account. The Commerce Readiness Stack discusses a related public experimental architecture, but its current repository cannot retrospectively verify an unpinned February version. This is my build account, not an independent implementation audit. Its lessons do not establish improved visibility or sales.

What building exposed

Three lessons changed how I read the system’s results.

A model’s preference is a proxy

In the first version, I treated a language model’s alignment judgment as a possible predictor of what a shopping system would recommend. That assumption did not hold up as evidence. A judge can assess whether a product description appears to match a scenario. Its score does not reveal how a production shopping service retrieves or ranks products.

Those services can also rely on product feeds, structured specifications, availability, price, merchant information, reviews, shipping, policies, and their own retrieval and ranking logic. Improving one description may make its information clearer while leaving other constraints unchanged.

They may also learn from behavior: what is shown, clicked, purchased, or returned. An outside lab usually cannot see the platform’s full impression-level decisions or the path from a recommendation to a purchase. Downstream analytics can reveal some customer actions, but they do not expose the internal ranking process or prove which change caused an outcome.

I shifted toward decomposing a judgment into visible signals, such as whether a benefit or use case was stated. This makes the output easier to inspect. It does not make the judge a proxy for the whole shopping system. Running several models can reveal disagreement; agreement can still reflect a shared blind spot.

Scores move when the test moves

In my tests, absolute scores varied with prompts, context, evaluation order, model settings, and model updates. A score such as 0.71 can look more exact than the underlying judgment warrants.

Pairwise comparisons can be easier to interpret: under the same scenario, did one version win more often than another? But a win rate is not automatically stable or representative. It still depends on which scenarios were selected, how often tests were repeated, which judges were used, and how ties or disagreements were handled.

A useful report should keep those conditions visible. It can show the comparison set, repeat count, judge identities and versions, prompt, order, and disagreement. Statistical intervals are helpful only when the measurement process and assumptions support them. They cannot turn a synthetic judgment into evidence of market impact.

Weak inputs create confident noise

I also found that generated scenarios depended on the quality of their starting information. Sparse product details, ambiguous categories, or missing specifications could produce malformed or overly specific queries. In some cases, generated questions reflected the brand’s existing language rather than a shopper’s likely need.

I added multiple ways to generate query sets, checks for category confidence, normalization of product specifications, and quality gates before saving generated results. Those are sensible controls for a research tool. They do not prove that a generated query represents real customer demand.

One gate has to precede scoring altogether: category eligibility. A laptop description should not earn a high match score for a shoe query because both mention comfort or portability. A confirmed category mismatch should stop that comparison before any soft assessment of benefits or use cases. If the category itself is uncertain, the system should ask for review or withhold the result. A confident fit score cannot repair an invalid comparison.

The practical lesson is to inspect the test set before interpreting the score. A large battery of plausible-sounding questions can still encode the wrong audience, omit important constraints, or repeat the same underlying idea.

Follow a finding through the lab

The architecture becomes clearer when I follow what happens to a result. The important connections are between records, not between framework names:

Stage	Record passed to the next stage
Intent and scenario generation	A need, category, constraints, and the source of the scenario, within a client, brand, and product scope
Experiment design	A reviewed query battery and named content variants, fixed for that comparison
Simulation and judging	A run linked to its battery, variants, prompts, models, settings, and decomposed judgments
External validation	A separately recorded shopping-surface observation linked to the prediction it can test, with query, market, date, and coverage
Review	A finding that states its supporting evidence, uncertainty, scope, and permitted reuse
Memory and maintenance	A reviewed lesson available to later generation, with evidence links and conditions for correction, expiry, or withdrawal

For a product page missing sizing information, the lab could compare the existing copy with a version containing a sizing table. Both must first pass the same category check. If judges prefer the revised version across the fixed scenarios, the finding is still narrow: those judges found it more useful under those conditions. An external observation, if available, belongs in a separate record. It might test a visibility prediction, but it cannot on its own establish that the table caused more appearances or sales.

Keep stronger claims locked until their evidence exists

I called the original control soft-locked insights. A result could be stored and inspected while remaining unavailable as accepted strategic guidance. The useful distinction is between recording a hypothesis and allowing it to guide later work.

A finding should begin as provisional, with an explicit question it has not yet answered. A claim about external visibility needs relevant external evidence before promotion. A claim limited to content coverage can be reviewed against the content and test set. The promotion record should identify the evidence, reviewer, scope, and limits of reuse. No universal accuracy threshold can make all these claims valid, and agreement between judges cannot unlock an external-performance claim by itself.

Let memory carry the limits with the lesson

Memory & Agency explains what a reusable lesson must retain about its evidence and limits. In this lab’s design, beliefs and memory were scoped by client_id, brand_id, and product_id. A lesson about one product should not become a rule for another client merely because retrieval finds similar language.

Before reuse, the system should check support, input quality, unresolved contradictions, scope, and freshness. Weak or contradictory artifacts can remain in the research record while being excluded from generation context. For the sizing example, an accepted lesson might be limited to making fit information explicit for that product and scenario set. It would not become “sizing tables improve AI rankings.”

Later evidence must be able to restrict or withdraw the lesson. Maintenance includes reviewing stale observations, correcting findings, and tracking prediction agreement: how often did a predicted external result match the recorded observation, under which conditions? The original build account called this calibration. Probability calibration is a narrower test: predictions assigned a given confidence should be correct at a corresponding rate across suitable cases. Agreement counts alone do not establish it. Retrieving a reviewed note changes a later run’s context; it does not train the underlying model or establish a causal rule.

What the build account establishes

The February account described a Next.js interface, FastAPI services, SQLite storage, and a bring-your-own-key model gateway separating generation and validation models. Those choices locate the work, but the evidence boundary remains:

Status	What belongs here
Described in my original build account	Scoped experiments, multi-judge runs, validation logging, belief weighting, memory distillation and retrieval, and manual or scheduled maintenance
Not independently demonstrated here	Correct enforcement of every boundary, calibrated predictions, reliable reuse, or improved visibility and business outcomes
Further work identified in that account	Automated shopping-surface probing and stronger links between lab predictions, observations, and downstream analytics

The architecture explains what would need to be inspected in a reproducible version. It should not be read as a claim that every control was delivered and verified in three weeks.

The platform evidence changed after the prototype

My original reporting review was dated 8 February 2026. It found limited direct access to platform-level observations at that time. That snapshot needs updating before it is used to describe the current market.

Google has since introduced AI Performance Insights in Merchant Center. Its current documentation describes visibility and share-of-voice reporting for conversational shopping queries across AI Mode and AI Overviews, with availability, traffic scope, and data limitations. This is meaningful evidence about visibility within Google’s defined report. It is not a universal view of shopping assistants or proof that a content edit caused a business outcome. Google’s reporting documentation lists its present scope and limitations.

OpenAI’s current shopping documentation describes how product results may be selected and how merchants can provide product feeds. The public documentation reviewed for this revision does not describe a comparable merchant performance report. Perplexity’s merchant programme is another platform-specific route for product information and shopping discovery; its current access and reporting details should be checked directly before planning a measurement study. OpenAI’s shopping documentation describes product selection and merchant data, while Perplexity’s merchant page describes its programme.

These differences matter. Platform reporting can make some observations easier, but the data may cover only a subset of markets, queries, products, or traffic. A platform’s metric is evidence within its own definition. It is not automatically comparable with another platform’s metric, downstream analytics, or a causal estimate.

What the prototype can support

The lab’s most defensible use is to help people inspect product information against explicit needs. It can identify missing specifications, unclear benefits, or use cases that a chosen scenario set does not cover. It can compare versions under recorded model and prompt conditions and show where evaluators disagree.

Those outputs can guide a content review. They do not support claims that a product will rank higher, receive more AI referrals, or produce more sales. Even an observed appearance in a shopping answer does not show that a copy change caused it. Product availability, feeds, location, time, personal context, and other system signals may have changed too.

The distinction is useful to buyers of optimization tools as well as their builders. Ask what the tool observed directly, what it simulated, how often its measurements repeat, and whether any reported relationship has been tested against independent outcomes. “The model preferred this copy” and “the product received more qualified orders” belong in different columns of a report.

Design an experiment that can teach you something

An experiment should state what evidence could change my mind before I see a favourable score.

1. Choose one question. For example: does adding complete sizing and fit information make the product page easier to assess against a defined set of shopper needs? Keep this separate from whether an assistant displays the product.
2. Define the comparison. Record the product version, scenario set, prompts, models, settings, and time of test. Change one meaningful element at a time where possible.
3. Check the scenarios. Review them for coverage, duplication, category fit, and unsupported assumptions. Keep the source of each scenario visible, such as customer research, platform query data, or a generated hypothesis.
4. Repeat and report disagreement. Compare versions across multiple runs and judges. Report the results as simulation-based content evidence, including variation and exceptions.
5. Observe the external surface separately. Where access permits, record whether and how products appear under a defined market, account, query, and date. Note personalization and sampling limits.
6. Connect downstream outcomes carefully. Track referrals and customer actions where analytics allows. Treat correlations as correlations. Use a controlled test before claiming that the content change caused a lift.
7. Revisit the claim as systems change. Models, product feeds, shopping interfaces, platform reports, and eligibility rules change. Keep the date and configuration attached to results.

This process will not remove uncertainty. It will show which uncertainty I have reduced and which remains.

Build to learn, then name what you learned

The three-week effort did not establish a way to predict AI shopping rankings. It made the gap between a lab judgment and an external outcome harder to ignore. That is a worthwhile result if the system reports it plainly.

Use model-assisted comparisons to improve clarity and coverage. Use platform observations to study visibility within a defined surface. Use independent outcome data and appropriate controls to examine business impact. Keep the three connected, but do not collapse them into one score.

My prototype’s strongest contribution is a disciplined question for the next experiment: what did I observe, from which system, under what conditions, and what conclusion does that observation actually support?

References

• Hong et al., “Exploring the Use of Natural Language Descriptions of Intents for Large Language Models in Zero-shot Intent Classification”, SIGDIAL 2024. The study concerns intent classification in task-oriented dialogue, not commerce ranking.
• Arora et al., “Intent Detection in the Age of LLMs”, EMNLP Industry Track 2024. The study compares intent-detection approaches and does not test product discoverability.
• Google Merchant Center, AI Performance Insights, current scope and reporting limitations.
• OpenAI Help Center, Shopping with ChatGPT Search, product selection and merchant data.
• Perplexity, Merchant Program, programme overview.