Structure-aware retrieval is having a moment. A steady stream of papers, product launches, and polished blog posts is making the case for embedding document structure — section hierarchies, breadcrumbs, parent-child relationships — directly into the vector index, rather than shredding documents into flat chunks. The headline claims tend to be dramatic: near-perfect accuracy, production-ready, drop-in replacements for standard RAG.
The underlying idea is worth taking seriously. The headline claims deserve caution. These are not the same judgment, and conflating them is how enterprise AI initiatives quietly die in year two.
Here's the frame that actually holds up. We evaluate any retrieval approach on three things: the soundness of the underlying idea, the quality of the evidence behind the claims, and the honesty of the production path from demo to deployment. Structure-aware RAG passes the first test easily. It struggles with the second. The third is where most organizations get hurt.
The Intent Is Right, and It Matters
Strip away the branding and the benchmark theater. The underlying insight is simple: documents carry meaning in their structure, and standard vector RAG throws that meaning away the moment it chunks.
That's correct, and it's the single most underappreciated failure mode in enterprise RAG today. When you shred a 200-page technical manual, a master services agreement, or a 10-K into 400-token windows, you're not just losing context — you're deleting the hierarchy a human reader uses to know what they're looking at. "Cash flow" means something different under Liquidity and Capital Resources than it does inside a footnote about pension obligations. A flat embedding index cannot tell the two apart. A human reading the table of contents can, instantly.
Every enterprise document class our clients care about — compliance manuals, regulatory filings, contracts, SOPs, engineering standards, clinical protocols — is structured. That structure is free signal. Leaving it on the table is the retrieval equivalent of ignoring primary keys in a database. The direction these approaches point toward — embed the breadcrumb, respect the section boundaries, retrieve the full section rather than a fragment — is exactly where enterprise RAG needs to go.
Where the Evidence Falls Short
The value is real. The evidence supporting the headline claims usually isn't — and the gap between demo and deployment is where enterprises lose years.
Benchmarks are demos dressed as proofs. Small, curated question sets on a handful of well-formed documents, often with the authors writing both the questions and the ground truth. When a system is reported to "outperform the ground truth" on edge cases, that's a polite way of saying the answer key was soft. A benchmark your system beats is no longer measuring accuracy; it's measuring narrative.
LLM-as-judge compounds the problem. The dominant evaluation methodology — use a capable LLM to grade answers against reference answers — has well-known failure modes that rarely get disclosed. Judge models inherit the biases and blind spots of whichever model is scoring. They reward fluency over correctness, favoring confidently-stated wrong answers over hedged right ones. They can be gamed by surface features like formatting or length. And when the same model family is being evaluated and doing the evaluation, the feedback loop produces numbers that look rigorous and aren't. None of this means LLM-as-judge is useless. It means a reported accuracy number with no disclosure of the judge model, the prompt, or the inter-judge agreement is not evidence. It's a claim.
Baselines are usually missing. The most important number in any RAG evaluation is "what does a competent standard pipeline score on the same questions?" That number is almost always absent. We rarely learn how much of the reported accuracy comes from the architecture versus a capable LLM working on any reasonably-retrieved context. Without the baseline, the comparison isn't evaluation — it's marketing.
The approach leans on clean inputs. Most structure-aware pipelines assume a parser produces well-formed headings. On SEC filings and modern technical docs, they do. On the scanned vendor contract from 2004, the Confluence page where every heading is styled as bold text, or the engineering spec where section titles are "Details," "Notes," and "Appendix A" — they don't. When re-rankers operate on breadcrumb paths rather than content, they're betting entirely on headings being descriptive. In most enterprises, they aren't.
Single-document retrieval is the easy case. Real enterprise RAG means cross-document reasoning across thousands or millions of files, often with identical-looking breadcrumbs from different teams, regions, or fiscal years. These demos rarely touch that regime, and it's where production deployments actually fail.
LLM-based filtering at index time is a silent-failure risk. Letting a model decide which sections are "noise" without logging and human review is not something we'd sign off on for a regulated client. In regulated environments, auditability isn't a polish item. It's a gating criterion — the difference between a clever architecture and one you can actually deploy. "The model decided the appendix was irrelevant" is not a defensible audit answer.
None of this makes the underlying techniques wrong. It makes them reference implementations, not production blueprints. Treating them as the latter is the expensive mistake.
What We Actually Recommend
For clients asking "should we adopt this?" — the answer is consistent.
Adopt the idea, not the artifact. Structure-aware retrieval is the right direction. Breadcrumb-enriched embeddings, section-bounded chunking, and pointer-based context assembly are all patterns worth building into your RAG stack regardless of which specific implementation you start from. Hierarchical and parent-document retrievers in the mainstream frameworks already implement most of this. Anthropic's contextual retrieval — prepending document-level context to each chunk before embedding — has been public for over a year and is straightforward to replicate. You don't need to bet on a specific vendor to get most of the value.
Invest upstream, not downstream. The biggest gains in enterprise RAG come from document understanding — parsing, structure extraction, metadata tagging — not from the fifteenth variation on re-ranking. If your headings are bad, no retrieval architecture saves you. Spend the budget on parsing and taxonomy before the vector database.
Benchmark on your own corpus, blind-graded. Any approach claiming a headline accuracy number should be tested against a baseline, on your documents, with questions you didn't write and answers a human graded without seeing which system produced them. This is boring, slow, and the only thing that actually tells you what to deploy. If a vendor resists this — and many do — that resistance is information.
Design for auditability from day one. Every retrieved chunk should carry its structural trace. Every filtered-out section should be logged. Every re-ranker decision should be inspectable. The goal is a glass-box pipeline — not because it's elegant, but because when something goes wrong in production, you need to know why. For regulated workloads, this isn't a preference. It's what makes the system deployable at all.
Stop chasing 100%. A system that scores 93% on a broad, representative, blind-graded benchmark is almost always more valuable than one scoring 100% on a curated set of sixty questions. The former tells you something about reality. The latter tells you something about the benchmark.
The Broader Pattern
Most of this advice generalizes beyond structure-aware RAG. Adopt the idea, not the artifact. Invest upstream. Benchmark on your own data, blind-graded, against a real baseline. Design for auditability. Stop chasing benchmark perfection. These principles apply to whichever category of enterprise AI technique is having its moment six months from now, and they're the ones that separate teams that ship durable systems from teams that chase demos.
Structure is the signal. Respect it. But respect your own data, your own users, and your own audit trail more than any headline number.
Most of this advice generalizes beyond structure-aware RAG. Adopt the idea, not the artifact. Invest upstream. Benchmark on your own data, blind-graded, against a real baseline. Design for auditability. Stop chasing benchmark perfection. These principles apply to whichever category of enterprise AI technique is having its moment six months from now, and they're the ones that separate teams that ship durable systems from teams that chase demos.
Structure is the signal. Respect it. But respect your own data, your own users, and your own audit trail more than any headline number.