Comparing prompting techniques for intent classification
I started a small side project to build intuition for prompting techniques: not just whether they help, but by how much, and what they cost. This is the first experiment from it.
The question was simple. Take a small open model. Ask it to classify a customer support message into one of 77 fine grained banking intents, using seven different prompting techniques. Measure accuracy. Measure cost. See what actually matters.
Try the live dashboard or read the full write-up and code.
The setup
I used Banking77, a dataset of customer support queries across 77 banking intents. Its classes are close in meaning on purpose. card_not_working and virtual_card_not_working are different intents, and telling them apart takes real understanding of the query, not just topic spotting.
The model was llama3.1:8b, run locally through Ollama. That wasn’t the plan going in. I started on Hugging Face’s Inference Providers, ran out of free credits partway through, considered Claude next, and realized a Claude Pro subscription doesn’t include API credits. A free local model turned out to be the right call anyway. No cost to track as a variable, and anyone can rerun the whole thing.
flowchart LR
A["Banking77 test set"] --> B["Stratified sample of 231 queries"]
B --> C{"7 prompting techniques"}
C --> D["llama3.1:8b via Ollama"]
D --> E["Parsed response: coarse group and intent"]
E --> F["Quality metrics"]
E --> G["Operational metrics"]
Every technique ran on the same 231 queries, sampled once, so none of them got an easier set to work with.
The seven techniques
- Zero-shot: just the label catalog, no examples
- Zero-shot, schema-guided: same, but every label gets a one line definition
- Few-shot, fixed: 10 static examples, one per coarse group, in every prompt
- Few-shot, semantic retrieval: 5 examples chosen per query, by embedding similarity
- Chain-of-thought: reason step by step, then answer
- Self-consistency: chain of thought sampled 3 times, answers vote
- Hierarchical: two calls, first pick the group, then pick the intent from only that group
What won, and what lost
| Technique | Accuracy | Mean latency |
|---|---|---|
| Zero-shot | 59.7% | 1.37s |
| Zero-shot, schema-guided | 67.1% | 1.59s |
| Few-shot, fixed | 64.5% | 1.65s |
| Few-shot, semantic retrieval | 90.5% | 3.22s |
| Chain-of-thought | 63.2% | 3.18s |
| Self-consistency | 62.3% | 9.69s |
| Hierarchical | 44.2% | 2.04s |
Three things stood out to me, and for each one I wanted a real reason, not just a number.
Retrieval won by a landslide. Choosing examples per query, by how similar they are to the actual query, beat every other technique by 20+ points. With 77 classes, no fixed prompt can demonstrate all of them, so a fixed set of examples only helps when a query happens to resemble one of them. Retrieval manufactures that resemblance every time, on purpose. There is a mechanical reason this works so well: in-context learning seems to run partly on induction heads, attention patterns that scan the context for something the current input resembles and complete the pattern from there (Olsson et al., 2022). Relevant examples give that mechanism something real to latch onto. Irrelevant ones do not, and can even bias the answer toward whatever labels happen to be shown (Zhao et al., 2021), which is likely why the fixed few-shot technique barely beat zero-shot despite costing more tokens.
Splitting the decision into two steps made it worse, not better. I expected “pick the group, then pick the intent” to help, the way breaking a hard problem into smaller ones usually does. Instead it scored lowest on every metric, including the group accuracy itself. In one joint call, the model can use the query’s actual wording to pick the specific intent, and the correct group falls out of that for free. Split into two calls, the first one has to guess the group in isolation, using only abstract category names, without the concrete wording that would make the answer obvious. Get that first guess wrong, and the second call is stuck solving the wrong problem with no way back.
I want to be honest that this part is not a new discovery on my part. Hierarchical classification research has documented this exact failure mode for decades, usually called error propagation or the “blocking problem”: a top down classifier that decides the coarse category first suffers when that first decision is wrong, in a way a flat classifier deciding the specific label directly does not (Silla and Freitas, 2011). My hierarchical technique is exactly that design, just built from two LLM calls instead of two trained classifiers, so getting a worse result is the expected outcome, not a surprise. It’s also why I skipped Tree of Thoughts and Graph of Thoughts as techniques here: both are built for problems with a real branching solution space worth searching and backtracking through, like puzzles or planning (Yao et al., 2023; Besta et al., 2023). This task is a flat, one-step decision, and the hierarchical result is a small piece of evidence for that: even a cheap two-step split, with no search at all, already lost accuracy, which suggests the expensive search machinery in ToT or GoT would not have fixed it either.
Chain of thought didn’t clearly help, and self-consistency didn’t fix that. A recent meta-analysis of over 100 chain of thought papers found its benefit concentrates almost entirely on math and symbolic tasks (Sprague et al., 2024). That tracks with what I saw. Intent classification is closer to a single lookup than a multi-step derivation, so writing reasoning text before the answer doesn’t give the model extra computation to spend, it mostly just gives it room to talk itself into a plausible-sounding wrong answer. Self-consistency samples that reasoning three times and votes, which only helps if the three mistakes are independent and random. Mine weren’t. The model had a consistent blind spot for certain classes, so sampling it three times mostly produced the same mistake three times, at three times the cost.
The full write-up has the complete results table, exact prompts for every technique, and the caveats worth knowing before trusting these numbers (the short version: this is one model, one run, and my own taxonomy for the 77 intents, not an official one).
Read the details, or poke around the dashboard yourself.