Multimodal Graph RAG Shopping Assistant

A multimodal, multi-agent RAG assistant over a real e-commerce catalog, built entirely on AWS. A router dispatches specialist agents for semantic search, knowledge-graph traversal, and image similarity search, fuses their results with an LLM-judged ranking, and streams a citation-checked answer.

Overview

A shopping assistant over 5,000 real Amazon products (McAuley Lab “Amazon Reviews 2023”) that answers questions typed as text or posed as an uploaded photo. Every retrieval decision, model choice, and index setting is backed by either a real constraint hit against AWS (a service in maintenance mode, an unsupported region, a blocked API) or a real experiment against real data — recorded, not assumed.

Architecture

Graph RAG, multi-agent orchestration — a router classifies each question and dispatches only the relevant specialist agents (semantic search, knowledge-graph traversal, image similarity search, exact-id lookup) in parallel, hand-rolled Claude tool-use loops on Bedrock AgentCore Runtime, after classic Bedrock Agents turned out to be closed to new accounts.
A real knowledge graph — a DynamoDB adjacency-list (Neptune isn’t available on this AWS account’s plan), traversing brand, category, and co-review relationships in single-partition queries.
Hybrid semantic search — OpenSearch native hybrid query with real score normalization, after an experiment caught the naive bool(knn, match) combination silently collapsing into keyword-only search.
Image search via vector embeddings — Cohere Embed v4, one real embedding per product, k-NN matched against an uploaded photo.
Streamed, citation-checked answers — every claim in the generated answer is checked against its source record by a second model call before the user sees it; an independent evaluation harness (a judge with no visibility into the production system’s own verifier) is what actually caught a real gap in citation traceability.

What I’d tell someone building this

The full write-up covers the real surprises: the AWS service I planned to build on turned out to be closed to new accounts, a real experiment proved my “hybrid” search wasn’t actually hybrid, and an independent LLM judge caught my own citations quietly restating claims instead of proving them — a gap every one of my own component-level tests had missed, because each one only ever checked the system’s output against itself.