AI retrieval · RAG · Azure · Personal project
Grounded Doc Q&A
A question-answering and search service over public Azure Container Apps documentation. Every answer cites its sources, the service refuses when the sources don't support an answer, and an eval harness fails the build when quality drops. I built it to see what RAG quality looks like in numbers instead of demos.
What it does
Ask a question about the docs and get an answer you can check, or a plain refusal.
Search and answer
Hybrid retrieval pulls the best passages. The answer API writes a grounded answer with citations, or refuses and says why when the sources don't support one.
Three ways in
A React UI shows the answer, its cited sources and the retrieved chunks with scores. An MCP server exposes search_docs and ask_docs so an MCP client can use the docs as a tool. The API is plain HTTP.
Measured, not assumed
Hand-written questions with expected sources score retrieval, answers, citations and refusals. The same checks run in CI against Azure and fail the build when a number drops too far.
How it fits together
Docs are chunked and embedded into Azure AI Search. The answer API retrieves, grounds, cites and refuses. The eval harness and the Bicep template wrap the whole thing.
Technology stack
C# and ASP.NET Core end to end, Azure AI Search and Azure OpenAI for retrieval and answers, React for the UI, Bicep for the infrastructure.
What the numbers showed for technical reviewers
Forty hand-written questions over 236 Container Apps docs. Small on purpose, so the caveats come with the results.
Hybrid beat keyword-only
The right doc was in the top 3 results for 93% of questions with vector or hybrid retrieval, against 78% for keyword-only. Hybrid gave the best top-5 rate at 97%. With 40 questions one question is 2.5 points, so differences of a few points are noise. The gap between keyword and the other two is not.
The semantic ranker didn't help
Adding Azure's semantic ranker lowered the top-3 and top-5 rates on this set. I haven't tuned it, so I'm reporting what happened and drawing no wider conclusion.
Smaller chunks ranked better, at a price
Chunking at 600 characters instead of 1,800 put the right passage first 10 points more often (4 questions) and raised the mean reciprocal rank from 0.817 to 0.856, for 66% more chunks to embed and store. With 40 questions that is suggestive, not conclusive, so the default stayed at 1,800.
Refusing is a feature
Three of the 43 answer questions have no answer in the docs, and all three were refused in every run. That is weak evidence at n = 3. One answerable question is refused every run even though a relevant doc is retrieved,
The evals caught a real bug
When the model refused, it sometimes returned null citations, which crashed the service. The eval run surfaced it. It's fixed, with regression tests.
Run-to-run variance is real
An early run answered only 80% of the answerable questions. I couldn't reproduce it in four later runs and never found the cause, and the answer model has no pinned temperature. So CI tolerances are loose (5 to 10 points), except refusal, which has zero tolerance.
Nothing to leak, nothing left running
The infrastructure choices that keep a personal project cheap and safe.
No keys to store
Everything authenticates with Entra identities, so there is deliberately no Key Vault: nothing to put in it. The app uses a managed identity. CI signs in with GitHub OIDC, and that identity can only call OpenAI and read the search index.
Cheap to leave up
The container app scales to zero, is capped at two replicas, and rate-limits per IP, because each request spends model tokens. Azure AI Search bills hourly while it exists, so the whole environment tears down with one command.
The template is the environment
I first built the environment by hand, then wrote the Bicep template and deployed it. The new deployment reproduced the eval numbers exactly, so I deleted the hand-built one. The template deploys from a developer login. A manual deploy workflow is in the repo but hasn't been run.
A small gotcha
GitHub's OIDC subject claims use an immutable form when a repo has that setting on. A name-based federated credential did not match, and the fix was to use the immutable subject.
What this is, and isn't
A personal project built to learn and measure. The limits are part of the result.
Questions I wrote myself
The 40 questions were written by me, not sampled from real users, over one slice of one documentation set.
An unvalidated judge
An LLM judge scores groundedness. It is a different model from the one that writes answers, but the same family, and I haven't checked it against human labels.
Capabilities demonstrated
What building and measuring this proves I can deliver.