Building a multi-agent Graph RAG system on AWS: what actually broke
Most RAG write-ups stop at a Jupyter notebook: a vector store, a handful of chunks, a demo query that happens to work. I wanted to know what breaks when you actually deploy something like this — real AWS services, a real dataset, a router deciding between specialist agents at query time, images as a query modality, and a frontend a stranger could actually use. So I built one: a multimodal, multi-agent RAG assistant over 5,000 real Amazon products (the McAuley Lab “Amazon Reviews 2023” dataset), entirely on AWS-native services, with every architectural decision either forced by a real constraint I hit or resolved by an experiment against real data.
Try the live demo or read the full architecture write-up and code.
The setup
The system answers natural-language product questions — typed, or posed as an uploaded photo — by routing each query to whichever combination of semantic search, knowledge-graph traversal, and visual similarity search actually applies, fusing the results with an LLM-judged ranking, and generating an answer whose every claim gets checked against the source record before the user sees it.
flowchart LR
A["Shopper's question<br/>(+ optional photo)"] --> B["Router: which specialists apply?"]
B --> C["SearchAgent<br/>hybrid text+vector"]
B --> D["GraphAgent<br/>knowledge-graph traversal"]
B --> E["ImageAgent<br/>visual similarity"]
B --> F["LookupAgent<br/>exact id"]
C --> G["LLM-judged ranking + reason"]
D --> G
E --> G
F --> G
G --> H["Streamed answer,<br/>every claim citation-checked"]
Four things made this a genuinely different exercise than a RAG demo: it had to run on AWS end to end (not local models), it needed a real knowledge graph (brand, category, and co-review relationships, not just vector similarity), it needed to accept a photo as a query, not just text, and it had to be cheap enough to actually leave running — under $100/month, real AWS Budget alerts configured, not just a target on paper.
Orchestration: who decides which agent runs?
The interesting design question wasn’t which AWS service to lean on — it’s who decides, at query time, which specialist agents actually get involved. I hand-rolled that: a router classifies each incoming question and dispatches only the relevant specialists — SearchAgent (semantic search), GraphAgent (knowledge-graph traversal), ImageAgent (visual similarity), LookupAgent (exact id) — in parallel, each running its own Claude tool-use loop. I’d originally planned to lean on a managed multi-agent orchestration feature for this, but it turned out to be closed to new accounts, and its newer replacement is a hosting platform rather than a declarative orchestrator — so the router and dispatch logic had to be hand-written regardless of which platform hosted it. That turned out to be the right amount of ownership anyway: the behavior that actually matters — which agents apply to a given question, how their results get fused, what happens when one comes back empty — lives entirely in that hand-written layer, not in a managed service’s opinions about it.
The knowledge graph is a DynamoDB adjacency-list rather than a dedicated graph database (the obvious managed option wasn’t available on this AWS account’s plan) — every relationship written in both directions at ingestion time, so a traversal (same brand, same category, co-reviewed — all 1–2 hop lookups) is a single-partition query. The interesting part isn’t the storage engine, it’s that those 1–2 hop relationships were the actual shape of the questions worth answering — which is what let a plain adjacency-list stand in for a graph database at all.
The experiment that found my hybrid search wasn’t hybrid
SearchAgent’s plan was a standard bool query combining a knn clause and a match clause — OpenSearch’s default summed scoring. Before shipping it, I ran it against a real evaluation: 300 real reviews, 30 synthetic queries, precision@5 and hit-rate@5 measured against a real OpenSearch index, four combination methods compared side by side.
Two of those four scored identically to three decimal places: the “hybrid” query and pure BM25 keyword search.
That’s not a coincidence, and a direct score comparison on one query showed exactly why. BM25 scores in this corpus ranged from single digits to over 20, unbounded. The k-NN cosine similarity scores sat in a tight 0.6–0.7 band. Add them together and the vector clause barely moves the total — the top “hybrid” score was just the top BM25 score plus a roughly one-point nudge. The naive combination wasn’t combining anything. It was BM25 wearing a hybrid-search costume. That silently defeats the entire reason a vector clause gets added in the first place: catching a semantic match with no keyword overlap, the “good bass” query that never shares a word with the review that answers it.
I fixed it with OpenSearch’s native hybrid query and a real score-normalization pipeline (min_max, then a weighted average), then ran a follow-up experiment to actually tune the weighting rather than guess at it. The punchline there was almost funny: none of the fusion methods I tried, weighted or not, actually beat plain BM25 on that eval. I dug into why instead of just picking the highest number, and found a structural blind spot in my own methodology — my synthetic queries were generated by asking Claude to paraphrase the same review they’d be graded against, which structurally favors a keyword-matching method regardless of how good the vector search actually is. Shipping BM25-only because it won a benchmark biased toward BM25 would have been the wrong lesson to take from my own experiment. I shipped the real hybrid mechanism anyway, with equal weighting, because it’s the documented fix for the scale-mismatch bug, its shortfall in that specific eval is small and explainable, and dropping the vector clause removes a capability the system genuinely needs — just flagged as “best available mechanism, not yet proven to beat the naive alternative on this query style” rather than a closed question.
The finding that actually mattered: an independent judge that didn’t trust my own verifier
Every one of my earlier live-verification checks — and I ran a lot of them, real API calls against real deployed infrastructure at every stage — passed. Each specialist agent, individually invoked, did the right thing. The router correctly fused multi-specialist results. The generator produced grounded, cited answers. A second, cheaper model checked each citation against its source record and dropped anything unsupported. By every test I’d built, the system worked.
Then I built a genuinely independent evaluation harness — real queries against the live, deployed API, scored by a separate LLM judge that never sees the production verifier’s own reasoning, so a bug in that verifier can’t hide behind its own approval. Twenty real end-to-end queries later: 85% recall (the system found the right product most of the time, and honestly said “I don’t know” rather than guessing when it couldn’t), but only 45% of answers scored as predominantly grounded by the independent judge.
That gap is the whole point of building an independent check instead of trusting your own tests. The judge’s notes explained exactly what it was catching: “the citation snippet is circular — it’s just the answer text itself, not actual product information.” My streaming redesign (plain text with inline [[product_id]] markers, so the answer could stream token-by-token instead of waiting on one JSON blob) had quietly changed what a “citation” actually was. The snippet shown next to each claim was now the sentence the generator itself had just written — not a quote pulled from the underlying product record the way the original design intended. My own server-side verifier was still doing its job, checking the claim against the real record before letting it through — but the evidence I displayed back to the user had stopped being independently checkable, and none of my component-level tests were built to notice, because each one only ever checked the system’s output against itself.
I hadn’t shipped a broken feature. I’d shipped a feature that quietly stopped proving what it claimed to prove, and the only thing that caught it was deliberately building a check with no access to my own system’s reasoning.
The bug a real question found that no test did
Weeks after the retrieval pipeline shipped, a real question surfaced something none of my scripted tests had: “find a charcoal soap bar and tell me what other products that brand makes.” GraphAgent got dispatched correctly, the router’s routing logic worked exactly as designed — and GraphAgent came back with zero results. Every time.
The cause was almost embarrassingly simple in hindsight: GraphAgent and LookupAgent can only traverse or fetch by an exact product id. Neither has any by-name search capability of its own. Every one of my own test queries, the ones I’d written to verify the router, happened to include an explicit id because I already knew the id I was testing with. A real user describing a product the way people actually talk — by what it is, not by its catalog number — hit a gap my own test suite structurally couldn’t have found, because I’d never once written a test the way an actual stranger would phrase a question.
The fix: when the router dispatches GraphAgent or LookupAgent without an explicit id already present in the question, it now runs SearchAgent first — even if the router itself never asked for it — purely to resolve a real product id from the shopper’s own description, then hands that id forward. Verified against a real multi-product brand: the resolved id let GraphAgent correctly return its actual siblings, where it had silently returned nothing before.
Where that leaves things
| Metric | Value |
|---|---|
| Products / reviews / chunks indexed | 5,000 / 10,500+ / 20,500+ |
| Product images embedded (visual search) | 5,000 |
| Knowledge graph edges | ~19,500 |
| Production eval recall (real queries, live API) | 85% |
| Production eval citation grounding (independent judge) | 45% — the open problem above |
| Monthly AWS cost target | Under $100, budget alerts configured and live |
The grounding-rate gap is still open. The fix is scoped — have citation resolution pull a genuine excerpt from the product’s own record instead of restating the generator’s sentence — but not yet built, and I’d rather say that plainly than quietly patch it before anyone reads this. I did add real customer reviews to every cited product’s card as a partial, honest mitigation in the meantime: independent evidence a shopper can check for themselves, sitting right next to the model’s own claim.
The full architecture write-up has every agent’s exact prompts and settings, the reasoning behind each one, and the complete experiment record — including the ones that confirmed a decision instead of overturning it, which don’t make for as good a story but were just as necessary to actually trust the system.