Ferremundo: Chief AI Officer, from strategy to a governed buying agent

What I own

Four things, in the order the business feels them.

The AI strategy and roadmap: what gets built, in what sequence, and what it is worth. The AI engineering function: the people, the workflow, and the review culture that turn that roadmap into shipped features. The trust model: what the agent is allowed to do, what it decides to do inside those limits, and how both hold up in front of an auditor. And the retrieval and agent stack end to end: catalog modelling on Postgres, the embedding strategy for text and image, the FastAPI matching surface, the conversational agent on self-hosted Agno, the incremental warehouse sync that keeps the catalog and embedding matrices fresh, and the production posture across Datadog APM, structured JSON logs, AWS CDK, and ECS rolling deploys. On top of matching, the agentic purchase flow that takes a reseller from natural-language intent to a confirmed order without leaving the conversation.

Strategy: rescoping before building

The original brief asked for an AI layer that reads inventory from ERP systems on both sides of every transaction, and it led with a computer-use agent that would read any retailer’s ERP off the screen. I rewrote the plan before writing any code, because the most expensive mistakes in this kind of engagement are scoping mistakes.

Three changes did most of the work. Screen-scraping became the last resort rather than the first method. A computer-use extraction session costs tens of cents to dollars and fails often; an API call or a CSV export costs effectively nothing and almost never fails, and nearly every ERP can export a spreadsheet. So the data access order became API, then direct database read, then file import, then computer use only where nothing else works. Features that need data waited for the data. Demand forecasting needs months of clean history from both sides and network recommendations need a network, so both moved out of the first phase instead of shipping as demos that could not be trusted. Ordering started as a suggestion, not an action. Automatically placing purchase orders from AI-extracted data is a financial liability until accuracy is proven on real traffic, so the first version proposes and a person confirms. That last constraint still holds today, and the whole authorization design is built around it.

Building the AI function

There was no AI team when this started. I built it: onboarding and mentoring five engineers from a partner agency, then establishing the engineering culture, development workflow, and code review practice around them. Since April 2026 that team has merged 130+ pull requests into the AI platform.

The culture is written down, because the team is remote and anything we do not write down does not exist. The charter makes one point above the rest: we are judged by what a retailer can actually do, not by what we shipped. Before building anything, an engineer writes one sentence describing what a retailer will be able to do afterwards that he cannot do today. Each person owns an outcome in production rather than a ticket, every problem area has exactly one accountable name, and a defect someone knows about never leaves their hands quietly. Agency work rewards staying inside the scope and closing the ticket, which is rational and exactly wrong for an agent that spends a retailer’s money, so the charter is also a commitment in the other direction: if I ask people to own outcomes, I owe them the context, the access, and the standing to own them.

That scaffolding is what makes the accuracy numbers below repeatable rather than a one-off. A retrieval system only its author can safely change is a liability, so the review practice and the standing benchmark matter as much as the model does. Decisions that would be expensive to revisit, such as the agent framework or the basket’s forward-compatibility seams, are recorded as architecture decision records with the alternatives that lost.

The roadmap: from answering to initiating

I own the AI roadmap and translate the founder and CTO’s requirements into phases. The arc is a single sentence: turn an assistant that answers into an agent that initiates.

The phases are sequenced so each one ships value alone and makes the next cheaper. Search and disambiguation came first, because nothing downstream works without it. Conversational search and image search followed, then full cart operations against the live B2B API, with stock and quantity rules matching the app’s own. Purchase and credit came next: the agent now quotes several items at once, builds a basket, checks out on credit or pre-paid terms, places the order only after an explicit confirmation, and reports the invoices for each shipment. WhatsApp is the current frontier, deliberately sequenced after purchase, because a retailer arriving on WhatsApp should be able to buy rather than only build a cart. Its evaluation set is built from real salesman chats, where a single request arrives spread over several messages, in retailer shorthand, often as a photo of a box. Since both channels mount the same agent definition, that is a channel mount rather than a rebuild.

The destination is a replenishment agent: detect when a store is due to reorder, assemble the complete basket with the right SKUs, quantities, and that customer’s live prices, and propose it over WhatsApp for one-tap approval. Four constraints shape it, and all four are deliberate. A human always confirms, and confirmation fills the cart rather than placing an order, so every purchase rule and credit decision stays exactly where it is today. The agent works only inside the retailer’s own catalog and prices, never across tenants. Prediction comes first and the language model second: a statistical engine decides what and when, and the LLM only composes and explains. And every action is a permissioned tool rather than logic buried in a prompt.

Trust, governance, and compliance

An agent that spends a retailer’s money needs two separate answers: what it is able to do, and what it decides to do inside those limits.

The first is the authorization design. The agent signs an assertion with a key it cannot export and trades it for a short-lived token that names one retailer, one channel, and a list of permitted operations. Every business call carries that token through a dedicated agent gateway, and no business call carries a retailer’s own credential. The actions an attacker would want, such as charging a stored card, shipping to a new address, or changing contact details, are outside the credential entirely, so no amount of persuasion reaches them. The design started from what an agent platform needs in three years (several channels, actions taken while the retailer is absent, more than one agent) and only then reconciled with what the backend provides today, because authorization decisions spread into every endpoint and are unusually expensive to revisit.

The second is behaviour governance. The position is that scopes are the ceiling, action validation is the control, and instructions to the model are the weakest layer. The agent may only write a SKU resolved through our own catalog lookup, a quantity the B2B side accepts, and an address from the retailer’s saved list, so a malicious instruction fails at the point of action without needing to be recognised as malicious. Controls that must hold regardless of the model live in code rather than in the prompt, because a prompt rule was observed to hold on one model and not on another.

Both feed a SOC 2 readiness program kept from the first commit, at the CEO’s request, so that a future audit is an exercise in evidence collection rather than remediation. It is deliberately honest about its gaps: it records which controls are enforced, which are only proposed, and what must close before an auditor is engaged.

The flagship delivery: why not OpenSearch

The incumbent was an OpenSearch-backed app search, and it failed in the specific way lexical search always fails in this market. Hardware store owners search from a job site on a phone, in one or two words. They type “tubo” when the product is “Tubo PVC presión 1/2” pulg PN10”, or “plastig” for a brand, or a dimension in whichever of four formats they happen to use. Before this shipped, 15.8% of searches returned nothing at all, about 22,000 dead ends a month.

The deeper problem is that 60% of queries are a single word, so the bottleneck is disambiguation, not retrieval. When someone types “tubo” the catalog has dozens of tubes and the entire job is ranking the right one first. That is a ranking problem, and it is won with signals OpenSearch has no access to: ABC velocity, view popularity, and this customer’s own purchase history.

So the decision was to build custom rather than adopt it, and the decision was measured afterwards rather than asserted: Top-1 59.1% against OpenSearch’s 46.5% on the same query set, at roughly a tenth of the monthly cost. What we gave up is real, just not binding at this size: automatic horizontal scaling, sharding for million-item catalogs, and built-in dashboards. At 11,500 SKUs the catalog fits in RAM.

Current production sits at Top-1 64.8%, Top-5 82.3%, p50 571 ms on a standing 385-query labelled benchmark. The sections below are how that pipeline works.

The retrieval pipeline

A hybrid pipeline with a fine-tuned multilingual E5-base as the first-class encoder. BM25 anchors rare drug-name-style tokens, the dense encoder handles paraphrase and Spanish slang, a BGE cross-encoder reranks the merged pool, and a synonym and alias table corrects long-tail miscalls. Embedding-first routing skips the heavier fuzzy and reranker work for queries that confidently resolve at the encoder layer, which is what produced the +3.5 pp top-1 and -190 ms latency move. A separate Doc2query enrichment pass, which concatenates synthetic queries into catalog text, gave another lift to 81.5% top-5.

Embedding finetune (text)

Backbone intfloat/multilingual-e5-base, fine-tuned with CachedMultipleNegativesRankingLoss on a four-source training set: GPT-4o-mini synthetic Spanish queries (~46k), labeled benchmark positives, synonym-derived positive pairs, and hard negatives mined by nearest neighbors over catalog embeddings in Postgres. Best checkpoint chosen by cosine NDCG@10 on a held-out slice. The frozen baseline lands at Acc@1 45.3%, Acc@5 64.0%, NDCG@10 55.0%; the fine-tuned model is what unlocks the production accuracy and latency posture.

Image search (vision)

A separate pipeline trains a contrastive image-text encoder on top of google/siglip2-base-patch16-256 using SigLIP’s sigmoid loss, with image-to-text Recall@1/5/10 on a deterministic held-out 10% split. Training data is 6,251 active-SKU thumbnails downloaded from the staging API and aligned with synthetic-query text. The fine-tuned weights are exported to ONNX and serve at about 99 ms per query on CPU. INT8 quantization was tried, recorded, and rejected: Recall@5 went from 64% to 13% for only a 1.3x speedup. This is the canonical example I now reach for when someone proposes “just quantize the encoder.”

The conversational agent, and why it moved to Agno

The agent handles product search, image search, follow-ups, a conversation basket, and the purchase flow against the live B2B API, with stock and quantity validation matching the app’s own rules. It routes deterministically first: small talk, SKU-shaped strings, and confident follow-ups are answered with zero LLM calls, and the model runs only when reasoning is genuinely needed. On an ordering surface the agentic path costs a couple of seconds for a fraction of a point of accuracy, which is the right trade there and the wrong one for search-only, so the same service exposes both.

It started as a hand-built LangGraph ReAct loop with its own checkpointing across in-memory, Postgres, and Redis backends. That worked, and I migrated off it anyway. LangGraph ships excellent primitives, but primitives were the problem: WhatsApp, scheduling, human-in-the-loop approvals, tracing, and evals were each going to be a separate infrastructure project, and every one of them is on the roadmap above. Agno makes all five first-class, mounts the WhatsApp channel from the same agent definition as the app channel, and self-hosts on our own AWS with sessions, runs, and traces in our own Postgres.

The migration replaced plumbing only. Routing rules, cart rules, the search pipeline, and the prompts all lived in framework-free modules and moved across unchanged, which is the entire reason swapping the framework was tractable at all. The trade-off is accepted rather than hidden: Agno is young and has shipped breaking API changes, so versions are pinned and business logic stays out of framework classes. Alternatives were evaluated and rejected on record, including Google ADK (optimized for Gemini on Google Cloud when we run OpenAI and Claude-class models through our own gateway), LangChain deep agents (a planning harness for open-ended work, where ours is tightly guardrailed and transactional), and the multi-agent frameworks (they optimize agent-to-agent collaboration, and there is one agent here with a small tool set and hard business rules).

The operator console

Built rather than bought. Any conversation can be replayed end to end: chat as a real customer, see product cards and live cart state, read per-reply execution traces, inspect individual tool calls, browse session history, and switch across local, staging, and production. When someone asks why the agent proposed something, the answer is in the same place the team already works, and no conversation data leaves our infrastructure to a third-party tracing vendor.

Where the work runs

The split matters more than it sounds. Retrieval and ranking run in the API process against in-memory structures, because at 11,500 SKUs the catalog fits in RAM and the retrieval logic itself is a few milliseconds. Every model inference (query embeddings, image embeddings, cross-encoder reranking) is a call to a remote GPU serving stack, because that is the latency that actually matters and it does not belong in an API container.

At startup the match service hydrates catalog rows, ranking signals, the alias map, the BM25 index (pickled from Postgres or rebuilt), the text embedding matrix with its SKU ordering, and the vocabulary and synonym tables. A nightly sync job upserts warehouse data, embeddings, image embeddings, and BM25 artifacts, and operators roll the API tasks afterward so workers reload from the new state. Treating “the cache is reloading fat matrices after batch sync” as a first-class concern rather than a side effect is what keeps the system honest in production.

Production posture

Multi-stage Docker images, ddtrace-run wrapping the entrypoint when Datadog is enabled, AWS RDS Postgres, Alembic migrations on startup with a SKIP_MIGRATIONS escape hatch, S3 model sync from a ferremundo-models bucket, CircleCI for black/pytest plus ECR build and ECS rolling deploy to a dev cluster, and structured JSON logs with Datadog trace correlation. The health endpoint exposes embedding coverage and image index size, which is what I look at first when something has shifted.

Lessons

The highest-leverage work in this role happened outside the code. A rescoped brief and a written team charter cost less than any model change, and they decide which problems the models are ever pointed at. Governance is the same: designing what the agent may do before it can do anything made the purchase flow an extension rather than a rewrite, and it is the reason a human confirmation could be promised to the business on day one and kept.

On the technical side, B2B hardware search fails in ways leaderboard models never measure. Customers type one word or a slang synonym, and the right SKU is often buried behind a popular default, so you spend more time on disambiguation signals and alias learning than on raw retrieval tricks. Fine-tuned E5 only pays off once you stop treating embeddings as a side channel: embedding-first routing was the largest accuracy move and it lowered latency by skipping fuzzy work most of the time. Hard negatives from your own catalog matter as much as synthetic positives; otherwise the cross-encoder and the alias table fight you when expansion changes surface forms across query and title. Agents look like a small wrapper, but they need checkpointed state and deterministic bypasses for short inputs, or you burn seconds on LLM routing for every SKU lookup. Image search is not “quantize and forget”; INT8 wrecked recall with almost no speed win, which is the kind of result you only get from measuring on real product photos. The “cache” is mostly reloading fat matrices after batch sync, which is simple, but it means deployments and cron are part of the correctness story, not an afterthought.

This engagement is ongoing; this page will be updated as the system evolves.