How to Build an Internal AI Assistant Over Company Documents

Introduction

Learning how to build an internal AI assistant over company documents has become a practical priority because employees already use AI to find information. A March 2026 Census Bureau survey reported that 37% of workers used AI to search for information or technical help. That behavior describes exactly the job a document assistant performs, only with your own policies, wikis, tickets, and contracts as the source material. The engineering pattern is now well understood and rests on retrieval, which means finding the right passages before a language model writes a single word. The hard parts are less glamorous and include messy files, permission rules, evaluation, and the ongoing cost of keeping answers current. This guide walks through the architecture, the key decisions, the risks, and a seven-step build, with real deployments from Morgan Stanley, Uber, LinkedIn, and DoorDash. By the end you will know what to build, what to measure, and which mistakes to avoid before the first employee types a question.

Quick Answers on Building an Internal AI Assistant

What is the fastest way to build an internal AI assistant over company documents?

Index your documents, retrieve the most relevant passages for each question, and have a language model answer only from those passages with citations. This retrieval-augmented pattern avoids retraining and keeps answers current.

Do internal AI assistants need document permissions?

Yes. An internal assistant must filter retrieved passages by the asker’s access rights before the model sees them. Otherwise it can reveal confidential company documents to employees who could never open the original files.

How do you know an internal AI assistant is accurate?

Build a golden set of real employee questions with expert-approved answers, then score every release against it. Track retrieval recall, answer correctness, citation accuracy, and refusals, and rerun the suite daily after launch.

Key Takeaways

• Retrieval, not retraining, is the default way to ground an internal assistant in private company documents.
• Document preparation, chunking, and hybrid search with reranking usually decide answer quality more than the model choice does.
• Permissions must be enforced at query time, because an assistant that ignores access rights becomes a data leak.
• Ship only after a golden-set evaluation passes, and keep a daily regression suite running after launch.

What Is an Internal AI Assistant Over Company Documents?

How to build an internal AI assistant over company documents starts with a definition: an internal assistant retrieves passages from your private files, then a language model answers only from those passages and cites each source.

Why Company Knowledge Stays Hard to Find

Most organizations do not lack knowledge; they lack a reliable way to reach it at the moment a question appears. Policies sit in shared drives, decisions hide in chat threads, and technical answers live in tickets that nobody remembers to search. Each tool has its own search box with its own ranking quirks, so employees often ask a colleague instead. That habit works until the colleague is on leave, changes teams, or gives an answer based on a policy that was replaced last quarter. The cost shows up as repeated questions, slow onboarding, and avoidable mistakes that nobody traces back to a missing document.

Traditional keyword search solves only part of the problem because it matches words rather than meaning. A new hire who searches for time off carryover may never find a handbook page titled Annual Leave Rollover Rules. Language models close that vocabulary gap, and enterprise search combined with LLMs is changing how organizations manage what they know. An assistant can also read several passages together and write one direct answer instead of returning ten blue links. The real value of an internal assistant is not novelty but the minutes it returns to every employee who would otherwise hunt through folders. Those minutes compound across thousands of questions asked every single month.

An assistant also changes the economics of documentation work inside a company. Once people can see which questions go unanswered, writers learn exactly which pages are missing, outdated, or contradictory. Support and platform teams can then fix the source material rather than answer the same question for the hundredth time. The assistant becomes a mirror that reflects the quality of the knowledge base back to its owners. Teams that treat the project as a content improvement program, and not only a software project, tend to see better results. That mindset shapes every decision in the rest of this guide.

How Retrieval Turns Documents Into Answers

Any plan for how to build an internal AI assistant over company documents has two phases, and keeping them separate makes the whole system easier to reason about. In the offline phase, a pipeline reads each document, splits it into passages, converts every passage into a numerical embedding, and stores the result in an index. In the online phase, the system embeds the employee's question, searches the index for the closest passages, and assembles them into a prompt. The language model then writes an answer from that prompt and returns it with links back to the sources. If you want a lighter hands-on walkthrough of the same loop, the open-source RAG chatbot tutorial covers it with free tools.

Every quality problem in an assistant can be traced to one of those stages, which is why diagnosis starts with the retrieved passages rather than the model. If the right passage was never retrieved, no prompt can rescue the answer. If the right passage was retrieved but buried among noise, the model may ignore it or blend it with the wrong one. When an answer is wrong, read the retrieved passages first and the generated text second. Teams that log both the question and the retrieved passages for every request debug in minutes instead of days. The architecture is simple enough to draw on a whiteboard, yet each stage hides decisions that deserve careful attention.

Choosing Between Retrieval, Fine-Tuning, and Long Context

Building on that two-phase picture, the first design decision is whether retrieval is even the right tool. Fine-tuning changes the weights of a model, which teaches style and format well but stores facts poorly and cannot be updated by editing a page. Retrieval keeps knowledge outside the model, so a corrected policy becomes visible the moment the index refreshes. For most internal use cases, where facts change and answers must cite a source, retrieval wins on freshness and auditability. The comparison of retrieval-augmented generation and fine-tuning explains the trade-offs in more depth.

Long context offers a third path, and for small collections it can be the simplest one. Anthropic's engineering team notes that if a knowledge base is smaller than about 200,000 tokens, roughly 500 pages, you can place the whole thing in the prompt. A company with a single employee handbook may never need a vector database at all. The approach stops scaling when documents multiply, because every request pays to process the full text again. Prompt caching softens that cost, but it does not remove the limits on attention that appear in very long inputs.

A practical rule of thumb follows from all of these facts. Use long context for small, stable collections and prototypes where speed of delivery matters most. Use retrieval for large or changing collections, and whenever answers must show their sources. Reserve fine-tuning for behavior such as tone, output format, or domain vocabulary, never as a substitute for a document index. Many mature systems combine all three, using retrieval for facts, light fine-tuning for style, and generous context windows for the assembled passages. Start with the simplest option that meets your accuracy target and move up only when measurements justify it.

Preparing and Cleaning Your Source Documents

Shifting from architecture to raw material, the quality of your documents sets the ceiling for the quality of your answers. Start by inventorying every repository the assistant should cover, including wikis, shared drives, ticket systems, slide decks, and PDF policy libraries. For each source, record the owner, the update frequency, the access model, and whether the content is authoritative or merely informal discussion. Sources without a named owner should usually wait, because nobody will be accountable when their pages go stale. This inventory also gives security and legal teams a concrete list to review before any data moves.

Extraction is where many projects quietly lose accuracy without noticing. Scanned PDFs need optical character recognition, slide decks lose meaning without their speaker notes, and spreadsheets turn into nonsense when flattened row by row. Tables deserve special care, since a policy that depends on a grid of thresholds is unreadable once the cell structure disappears. Uber's engineers reported that standard PDF loaders did not work universally on their policy documents, especially tables with nested cells. They moved documents into a structured format and wrote a custom loader that preserved headings, tables, and the table of contents. Budget real engineering time for this step, because it rarely works well with defaults.

Cleaning goes beyond extraction and includes removing duplicates, boilerplate, and obsolete versions. Five copies of the same travel policy will produce five near-identical passages that crowd out other useful results. Headers, footers, and cookie notices repeat on every page and pollute the embeddings with meaningless text. Mark superseded documents as archived instead of deleting them, so the index can exclude them while auditors can still trace history. Version metadata such as effective dates lets the assistant prefer the newest policy when two documents disagree. Teams that build automated checks for these problems keep the index healthy as content changes.

Document-processing tools can shorten this work considerably, and a good overview is available in the guide to intelligent document processing for business operations. Treat document preparation as a product with its own owner, backlog, and quality metrics rather than a one-time import script. A small team that reviews the lowest-scoring documents each week will improve answers faster than any model upgrade. Record every transformation applied to a file so you can reproduce the index from scratch. Reproducibility matters when an auditor asks why the assistant said what it said last month. Good preparation is unglamorous, but it is where most of the reliability gains come from.

Chunking and Metadata That Preserve Meaning

Beyond cleaning, the next decision is how to split each document into passages that retrieval can find and the model can use. Chunks that are too small lose context, so a sentence such as "the limit is 500 dollars" no longer says which limit or for whom. Chunks that are too large dilute the embedding, because a single vector must now represent several unrelated ideas. A common starting point is a few hundred tokens with a modest overlap between neighbors. Splitting along natural boundaries such as headings, sections, and table rows works better than cutting at fixed character counts. Treat any default as a hypothesis that your evaluation set can confirm or reject.

Metadata is the second half of chunking and often matters just as much as the text itself. Each chunk should carry its document title, section path, owner, last-modified date, source system, and the access-control information needed later for permission filtering. Anthropic described a technique called contextual retrieval in which a short explanation of where the chunk sits in its document is prepended before embedding. They reported that each added context usually runs 50 to 100 tokens, and that it helps chunks that would otherwise be ambiguous. A passage reading "revenue grew 3%" becomes searchable once it names the company and the quarter. Uber's team went further and attached generated summaries, keywords, and likely questions to each chunk.

Tables, lists, and code need their own rules so that structure survives the split. Keep a table with its header row, and never cut a numbered procedure in the middle of a step. Store the original document location with every chunk, because citations are only trustworthy when employees can click through to the exact page. Keep the chunking configuration under version control so that every index can be tied to the settings that produced it. When you change chunk size, rebuild and re-evaluate, because retrieval behavior can shift in ways intuition will not predict. A disciplined approach here prevents the slow drift that makes assistants worse over time.

Embeddings and Vector Stores for the Retrieval Layer

Next, the retrieval layer needs a way to compare the meaning of a question with the meaning of each passage. Embedding models turn text into lists of numbers so that similar meanings land close together in a high-dimensional space. If the idea is new to you, the explanation of how word embeddings capture meaning provides useful background. Pick a model that handles your languages and your domain vocabulary, then test it on real questions rather than public leaderboards alone. Larger vectors generally cost more to store and search, so compare accuracy gains against index size before committing.

The vector store itself matters less than teams expect, provided it supports the features your design requires. Look for metadata filtering, since permission checks will depend on it, along with hybrid search, incremental updates, and sensible backup options. Managed services reduce operational work, while a database you already run may be enough for a few million chunks. Choose the boring option your team can operate confidently, and switch only when a measured bottleneck forces the change. Plan for re-embedding from the start, because upgrading an embedding model means rebuilding every vector in the index. Keep the raw chunks in durable storage so that rebuilding is a scheduled job rather than an emergency.

Hybrid Search and Reranking for Better Answers

Building on embeddings, the strongest retrieval setups combine two kinds of search instead of trusting one. Vector search finds passages that mean the same thing as the question, even when the wording differs. Keyword search, usually the BM25 algorithm, finds exact terms such as product codes, error numbers, and policy identifiers that embeddings often blur. Anthropic measured the difference carefully on its own internal test sets. Failed retrievals at the top 20 fell from 5.7% with plain embeddings to 2.9% with contextual embeddings plus BM25. Adding a reranking step brought the failure rate down to 1.9% in the same experiment. Those are one vendor's results on its datasets, so treat them as a direction to test rather than a promise.

A reranker is a second model that reads the question together with each candidate passage and scores how well they fit. It is slower than vector search, so the usual pattern retrieves a generous set, perhaps 100 or more candidates, and keeps only the best 10 to 20 after reranking. Reranking is usually the cheapest accuracy improvement available once your basic retrieval works. It also gives you a natural place to apply business rules, such as boosting newer documents or demoting archived ones. Log the scores at each stage so you can see whether a missing passage was never retrieved or merely ranked too low.

Some questions need relationships across documents rather than similarity to a single passage. Asking which projects depend on a retired service, for example, requires following links between records. Graph-based retrieval addresses this need, and the comparison of GraphRAG against traditional RAG shows when the added complexity pays off. Most internal assistants should master hybrid search and reranking before reaching for graphs. Add the heavier machinery only when your evaluation set shows a class of questions that simpler retrieval cannot answer.

Enforcing Document Permissions at Query Time

Turning to access control, the most dangerous failure of an internal assistant is not a wrong answer but a correct answer shown to the wrong person. Employees can see only some of the documents in a company, yet a naive index mixes everything into one pool. If the assistant retrieves a confidential compensation memo for an intern's question, the language model will happily summarize it. The risk is not hypothetical, since the OWASP list of LLM risks places sensitive information disclosure second and adds vector and embedding weaknesses as a new entry. Permissions therefore belong inside the retrieval step, before any text reaches the model, and never in the prompt as a polite request.

The standard design stores access-control metadata with each chunk and filters results by the identity of the person asking. Microsoft documents several patterns in its Azure AI Search access control overview, including security filters that compare a user or group identifier against each document. Its newer options inherit permissions from sources such as SharePoint, Azure Data Lake, and sensitivity labels. Many of those options are still in preview, and Microsoft states that preview features are not recommended for production workloads. Whatever platform you use, the principle is identical: the query carries the user's identity, and the index returns only what that identity may read.

Synchronization is the part of the design that most teams forget. The same Microsoft page notes that changes to group membership or document permissions affect results only after a synchronization step runs. If someone leaves a project on Monday and the index refreshes on Friday, that person can still retrieve project documents all week. Decide how fast permission changes must propagate, then design refresh jobs, event hooks, or query-time checks against the source system to meet that target. Test access control with real accounts at different privilege levels before launch, and repeat the test after every pipeline change. Treat a failed permission test as a release blocker, not a bug to schedule.

Logging closes the loop and protects both employees and the company. Record who asked what, which passages were retrieved, and which documents were cited, while following your retention and privacy rules. These logs let security teams investigate suspected leaks and help owners see which documents are used most. They also expose over-broad sharing, because an assistant often reveals folders that were accidentally open to everyone for years. Several companies discover their worst permission mistakes only when an assistant makes them visible. Fixing those source-system permissions is a benefit of the project, even though it can be uncomfortable. The related guide on handling data privacy and security covers the wider governance context.

Prompting and Grounding the Answer Generator

Given the retrieved passages, attention moves to how the generator uses what it receives. The prompt should state the assistant's role, instruct it to answer only from the supplied passages, and require a citation for every claim. Tell the model what to do when the passages do not contain the answer, since saying "I could not find that" is far better than a confident guess. Place the retrieved passages in a clearly labeled block and keep the employee's question separate so that instructions cannot be confused with content. Short, direct instructions usually outperform long lists of rules that contradict each other.

Context placement deserves attention because models do not read long inputs evenly. Researchers found that performance can degrade significantly depending on where relevant information sits, with the best results when it appears at the beginning or end of the input. Order the passages so the strongest evidence lands at the edges, and keep the total number of passages modest. More passages are not automatically better, since extra context adds noise and cost. The broader practice of context engineering for LLM agents offers more patterns for assembling prompts deliberately.

Evaluating Answer Quality Before Launch

Rounding out the build phase, nothing replaces a structured evaluation before employees see the assistant. Start by collecting real questions from help desks, chat channels, and subject-matter experts, then ask those experts to write or approve the correct answer and its source. A golden set of a few hundred questions is enough to reveal systematic problems, and it grows over time as failures are added. Uber's team curated more than 100 expert questions tested against over 40 policy documents when it improved its on-call assistant. Cover easy lookups, multi-step reasoning, questions with no answer, and questions that touch restricted material.

Score retrieval and generation separately so you know where to act. For retrieval, measure whether the correct passage appears in the top results, using recall at a fixed cutoff. For generation, measure whether the answer is correct, whether it is faithful to the retrieved text, and whether the citations actually support each claim. Human grading is the gold standard but expensive, so many teams add an automated judge model for scale. The guide on evaluating LLM outputs with a judge model explains how to calibrate such a judge against human labels. Never trust an automated judge until you have compared its scores with expert grades on a sample of the same answers.

Evaluation continues after launch, and Morgan Stanley's approach offers a useful pattern. OpenAI's account describes an evaluation framework that tests each use case before deployment and a daily regression suite of sample questions that checks for weaknesses and compliance. Daily runs catch silent breakage from model updates, index refreshes, and document changes. Pair them with a feedback button so that employees can mark answers helpful or wrong in one click. Review the flagged answers weekly and convert each confirmed failure into a new test case. Techniques for catching ungrounded claims are covered in the guide to reducing hallucinations in production.

Risks and Failure Modes to Plan For

Looking at what can go wrong, an internal assistant carries risks that differ from those of a public chatbot. Hallucination is the best known, where the model states something unsupported by any retrieved passage. Stale content is quieter and more common, because an outdated policy retrieved faithfully still produces a wrong answer. Over-confident answers to questions outside the knowledge base mislead employees who assume the assistant knows everything. Each of these failures looks plausible to the reader, which is exactly why they need systematic measurement rather than occasional spot checks.

Security risks deserve their own attention, starting with prompt injection. A document in your index can contain hidden instructions, such as text telling the assistant to ignore its rules or reveal other files. The OWASP Top 10 for LLM applications ranks prompt injection first and lists system prompt leakage and vector and embedding weaknesses among its new 2025 entries. Defenses include treating retrieved text as untrusted data, limiting what tools the assistant can call, and filtering outputs. The detailed guide to defending against prompt injection attacks walks through practical mitigations.

Operational risks round out the overall picture of what can fail. Index poisoning occurs when an untrusted contributor edits a wiki page to plant misleading content that the assistant then repeats. Cost overruns happen when a popular assistant sends very long prompts for every question. Dependency risk appears when a model provider changes behavior or retires a version without much notice. Write down each risk with an owner, a detection signal, and a response plan before launch, so incidents are handled by procedure instead of improvisation. Revisit the register every quarter, since new features such as tool calling and file uploads introduce new attack surfaces.

Ethics, Privacy, and Employee Trust

Stepping back from the technology, an internal assistant raises ethical questions that decide whether employees will actually use it. People worry, reasonably, that their questions are being watched and that a poor answer might be held against them. Be explicit about what is logged, who can read the logs, and how long they are kept. Publish those rules in plain language before launch, and involve works councils or employee representatives where local law expects it. Trust is earned by restricting how question logs can be used, not by promising that they never exist.

Privacy duties extend to the documents themselves, and so does accountability for the answers. Personnel files, medical leave records, and customer data may sit in repositories the assistant could reach, and regulations such as GDPR and HIPAA can apply to them. Exclude sensitive categories by default, and add them only with a documented purpose and a legal review. Governance structures help here, and the overview of enterprise agent governance in Microsoft Agent 365 shows how large vendors frame ownership and oversight. Label answers as machine-generated, show their sources, and point to a human owner for legal, medical, or disciplinary topics. Decide in advance which topics require a human handoff and encode that rule in both the prompt and the evaluation set. Employees should never be penalized for relying on a cited answer that later proved wrong, provided they followed the process.

Cost, Latency, and the Monthly Budget

Turning to economics, an internal assistant has four cost centers that behave differently as usage grows. Indexing costs are mostly one-time per document, covering extraction, embedding, and the occasional rebuild. Storage costs scale with the number of chunks and the size of each vector. Query costs are the largest line item for busy systems, because each question involves an embedding call, a search, possibly a reranker, and a generation with a long prompt. Staff time for content owners, evaluation, and support is the fourth center, and it is frequently underestimated.

You control query cost through prompt design and caching more than through model choice alone. Sending fewer, better passages shortens the prompt and often improves accuracy at the same time. Caching repeated prefixes helps a great deal, and Anthropic reports that prompt caching can cut costs by up to 90% and latency by more than half in its retrieval setup. Route simple questions to smaller models and reserve the most capable model for hard ones. The practical guide to reducing LLM inference costs lists further tactics. Track cost per answered question from the first day, because a budget you cannot measure cannot be defended.

Latency shapes adoption just as strongly as cost does in practice. Employees abandon tools that take more than a few seconds to start responding, so stream tokens as soon as they are available. Each extra stage, such as reranking or a guardrail check, adds delay. DoorDash noted that its guardrail added latency, and it defaulted rare problem responses to human agents as a trade-off. Measure the slowest five percent of requests, not only the average, since long tails shape how people feel about the tool. Set a service target, such as a time to first token, and review it monthly alongside quality and cost.

Putting the Assistant Into Daily Work

Moving on from architecture, adoption depends on where the assistant lives and who champions it. Meet employees in the tools they already use, such as Slack, Microsoft Teams, the intranet, or the help desk portal. Uber placed its on-call assistant in Slack channels where engineers already asked questions, and it offered buttons to mark answers resolved or escalate to a human. Start with one team that has a clear pain, such as IT support or HR policy questions, and a named owner who cares about the outcome. A narrow launch produces sharper feedback than a company-wide release of an unfinished tool.

Training, communication, and operations then decide whether early enthusiasm lasts. Show employees what the assistant does well, what it cannot do, and how to phrase questions with enough detail. Name content owners who receive a weekly digest of unanswered or poorly rated questions, and celebrate small wins such as a policy page rewritten after the assistant exposed a contradiction. Someone must own refresh schedules, incident response, model upgrades, and the quarterly review of risks and costs. Document how to roll back an index or a prompt change when quality drops. Define service levels that match the stakes, since an HR benefits assistant needs different guarantees than a developer helper. Review weekly active users, questions per user, resolution rate, and escalation rate to see whether the tool is becoming part of daily work.

The Future of Internal Knowledge Assistants

Looking ahead, internal assistants are moving from answering questions toward completing tasks across several systems. Agentic designs let a model decide which tools to call, rewrite its own queries, and check its work before responding. Uber's enhanced agentic approach already uses a query optimizer, a source identifier, and a post-processing agent to improve answers. Such designs raise capability and risk together, since every tool the assistant can call is a new permission to govern. Expect governance, audit trails, and deterministic guardrails to become standard parts of the stack.

Memory and structure will also change how retrieval works inside companies. Knowledge graphs capture relationships between people, projects, systems, and policies in a form that pure text search cannot represent. The article on semantic knowledge graphs for LLM agents explores how such structures supplement vector search. Assistants will increasingly remember user preferences and past conversations, subject to the privacy rules discussed earlier. The durable advantage will belong to organizations whose knowledge is well maintained, well permissioned, and well measured, not to those with the fanciest model. Models will keep improving, but your documents and your evaluation discipline are the assets that compound.

Evaluation itself is becoming more automated and more continuous every year. Judge models, simulated users, and synthetic test generation let teams test thousands of scenarios before each release. Standards and regulation are also maturing, with risk frameworks asking organizations to document how AI systems are tested and monitored. Teams that already keep golden sets, logs, and risk registers will find compliance a modest extension of existing practice. The next few years will reward organizations that treat their internal assistant as a living product with owners and metrics. Anyone learning how to build an internal AI assistant over company documents should start small, measure honestly, and expand as the evidence supports it.

How to Set Up the Pipeline Step by Step

Beyond the design decisions, the fastest way to learn is to build a small working pipeline and then harden it. The seven steps below show how to build an internal AI assistant over company documents with Python and open-source libraries, and every stage maps to a managed service. The code is deliberately compact so that you can read it in one sitting and adapt it to your stack. It omits authentication, retries, and scaling concerns that a production system must add. Treat it as a skeleton for a pilot with a single team and a few thousand documents. Replace each placeholder with your own sources, identity provider, and model endpoint before showing it to employees.

Step 1 - Define the scope and prepare the project

Begin with a narrow, valuable slice, such as IT support documentation or HR policies for one region. Write down the 10 to 20 questions that the pilot team asks most often, along with the approved answers and the documents that contain them. These become the first entries in your golden evaluation set, and they keep the project honest later. Identify the repositories involved, name an owner for each, and confirm with security which of them may be indexed. Create a clean virtual environment so the pipeline can be reproduced on another machine. The commands below install the libraries used in the rest of this walkthrough.

python3 -m venv .venv
source .venv/bin/activate
pip install pypdf sentence-transformers rank-bm25 numpy
mkdir -p data/raw data/index eval

Step 2 - Extract and clean the documents

Place the first 50 approved files under the raw data folder and write a loader that returns text plus metadata for each one. Record the title, source path, owner, modified date, and the groups allowed to read the file. If your source system exposes permissions through an API, map them into a list of group names at this stage. Strip repeated headers, footers, and page numbers so they do not pollute the embeddings. Keep a log of any file that fails extraction, because silent failures create invisible gaps in the knowledge base. Pro tip: treat a document with no known access list as private by default and exclude it until an owner approves it.

from pathlib import Path
from pypdf import PdfReader

def load_documents(folder, acl_map):
    docs = []
    for path in Path(folder).rglob("*"):
        if path.suffix.lower() == ".pdf":
            reader = PdfReader(str(path))
            text = "\n".join(page.extract_text() or "" for page in reader.pages)
        elif path.suffix.lower() in {".md", ".txt"}:
            text = path.read_text(encoding="utf-8")
        else:
            continue
        groups = acl_map.get(path.name)
        if not groups:
            print(f"SKIPPED (no ACL): {path.name}")
            continue
        docs.append({"title": path.stem, "source": str(path),
                     "groups": groups, "text": text})
    return docs

Step 3 - Chunk the text and attach metadata

Split each document along its headings first, then break oversized sections into overlapping windows of about 300 words. Carry the document title and section heading into every chunk so that each passage can be understood on its own. Copy the access list onto each chunk, since permission filtering will run against chunk metadata at query time. Keep chunk identifiers stable between runs so that evaluation results can be compared over time. Store the chunking parameters in a configuration file instead of burying them in code. The function below shows a simple word-window approach that you can replace with a heading-aware splitter.

def chunk_document(doc, size=300, overlap=50):
    words = doc["text"].split()
    chunks, start, n = [], 0, 0
    while start < len(words):
        piece = " ".join(words[start:start + size])
        chunks.append({
            "id": f'{doc["title"]}-{n}',
            "text": f'{doc["title"]}: {piece}',
            "source": doc["source"],
            "groups": doc["groups"],
        })
        start += size - overlap
        n += 1
    return chunks

Step 4 - Embed the chunks and build the hybrid index

Load an embedding model, convert every chunk into a normalized 384-dimension vector, and save the matrix next to the chunk metadata. Build a keyword index over the same chunks so that exact terms such as error codes can be matched directly. Normalized vectors allow cosine similarity to be computed as a simple dot product, which keeps the search code short. For a pilot, an in-memory matrix is enough, but plan to move to a vector database as the collection grows. Record the model name alongside the index, because mixing vectors from different models silently breaks retrieval. Rebuild the index whenever the embedding model or chunking settings change.

import numpy as np
from sentence_transformers import SentenceTransformer
from rank_bm25 import BM25Okapi

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

def build_index(chunks):
    texts = [c["text"] for c in chunks]
    vectors = model.encode(texts, normalize_embeddings=True)
    bm25 = BM25Okapi([t.lower().split() for t in texts])
    return {"chunks": chunks, "vectors": np.array(vectors), "bm25": bm25}

Step 5 - Retrieve with permission filtering and fusion

At query time, filter the candidate chunks by the asker's groups before any scoring influences what the model sees. Run vector search and keyword search over the permitted subset, then merge the two ranked lists with reciprocal rank fusion. This method rewards passages that rank well in either list and needs no score calibration between them. Return the top 8 chunks with their sources so that the generator can cite them. Log the question, the user identity, and the identifiers of the returned chunks for debugging and audits. Add a cross-encoder reranker after fusion once you have measured the baseline.

def search(index, question, user_groups, k=8):
    allowed = [i for i, c in enumerate(index["chunks"])
               if set(c["groups"]) & set(user_groups)]
    if not allowed:
        return []
    qvec = model.encode(, normalize_embeddings=True)[0]
    vec_scores = index["vectors"][allowed] @ qvec
    bm_scores = index["bm25"].get_scores(question.lower().split())[allowed]
    fused = {}
    for ranking in (np.argsort(-vec_scores), np.argsort(-bm_scores)):
        for rank, pos in enumerate(ranking[:50]):
            fused[pos] = fused.get(pos, 0) + 1 / (60 + rank)
    best = sorted(fused, key=fused.get, reverse=True)[:k]
    return [index["chunks"][allowed[p]] for p in best]

Step 6 - Generate grounded answers with citations

Assemble a prompt that contains the instructions, the numbered passages, and the employee's question in clearly separated blocks. Tell the model to answer only from the passages, to cite passage numbers, and to say so plainly when the answer is missing. Send the prompt to whichever model endpoint your organization has approved, and stream the response to the user interface. After generation, verify that every citation number refers to a passage that was actually supplied. Reject or regenerate answers that cite nothing, because uncited claims are the hardest for employees to check. Keep the temperature near 0 so that repeated questions receive consistent answers.

PROMPT = """You are an internal assistant. Answer using ONLY the passages below.
Cite passage numbers like [1]. If the passages do not contain the answer,
reply: "I could not find that in the company documents."

Passages:
{passages}

Question: {question}
Answer:"""

def answer(index, question, user_groups, call_llm):
    hits = search(index, question, user_groups)
    passages = "\n".join(f"[{i+1}] ({h['source']}) {h['text']}"
                         for i, h in enumerate(hits))
    reply = call_llm(PROMPT.format(passages=passages, question=question))
    return {"answer": reply, "sources": [h["source"] for h in hits]}

Step 7 - Evaluate, launch, and monitor

Load the golden set and measure retrieval recall before you look at generated text. A question counts as a hit when at least one expected source appears among the retrieved results. Block the launch if recall falls below the 90% threshold your team agreed on, and investigate the misses one by one. Add generation checks for correctness and citation accuracy using expert grades or a calibrated judge model. After launch, schedule the same script to run daily and alert the owners when scores drop. Feed confirmed user complaints back into the golden set so the suite grows with real usage.

import json

def recall_at_k(index, golden_path, k=8):
    golden = json.load(open(golden_path))
    hits = 0
    for case in golden:
        results = search(index, case["question"], case["user_groups"], k)
        sources = {Path(r["source"]).name for r in results}
        hits += bool(sources & set(case["expected_sources"]))
    return hits / len(golden)

Key Insights

• The U.S. Census Bureau's March 2026 worker survey found that 37% of workers used AI to search for information or technical help, so demand for document search is mainstream.
• OpenAI's Morgan Stanley customer story reports that advisor access to firm documents rose from 20% to 80% and that more than 98% of advisor teams actively use the assistant.
• A LinkedIn study of knowledge graph retrieval reported a 77.6% improvement in mean reciprocal rank and a 28.6% drop in median resolution time for customer service issues.
• Anthropic's contextual retrieval research cut failed top-20 retrievals from 5.7% to 2.9% with hybrid search and to 1.9% with reranking, so retrieval design deserves attention equal to model choice.
• Uber's enhanced agentic RAG work raised acceptable answers by a relative 27% and cut incorrect advice by a relative 60% on its security and privacy help channels.
• The OWASP 2025 list of LLM risks moved sensitive information disclosure from sixth to second place and added vector and embedding weaknesses, putting permission design at the center of risk.
• Researchers behind Lost in the Middle showed that accuracy falls when relevant facts sit in the middle of long inputs, so passage order and passage count measurably affect answers.
• Anthropic notes that knowledge bases under 200,000 tokens, roughly 500 pages can skip retrieval and fit directly in the prompt, which makes small collections simple to launch.

Taken together, these figures describe an assistant whose success depends on retrieval quality, access control, and measurement more than on the language model itself. Employees clearly want AI to find information, and large deployments show real gains in document access and resolution time. The same sources show that gains come from careful engineering choices such as hybrid search, reranking, structured parsing, and guardrails. They also show that risks concentrate in permissions, long-context behavior, and unverified answers. Teams that respect both sides of that evidence ship assistants that employees trust and auditors can defend.

Dimension	Long context	Retrieval (RAG)	Fine-tuning	Agentic RAG
Best fit	Small, stable collections and quick prototypes	Large or changing collections that need citations	Style, tone, format, and domain vocabulary	Multi-step questions that need several lookups
Freshness of answers	Updated when the prompt text is replaced	Updated when the index refreshes	Frozen until the model is retrained	Updated when the index refreshes
Setup effort	Lowest; no index required	Moderate; ingestion, chunking, and search	High; curated examples and training runs	Highest; orchestration and guardrails on top
Source citations	Possible but depends on prompt structure	Natural; every passage carries its source	Weak; facts are stored inside weights	Natural, with traces of each retrieval step
Permission enforcement	Hard; the whole text sits in the prompt	Strong when filters run before the model	Poor; trained knowledge cannot be filtered per user	Strong, but each tool needs its own checks
Running cost	High per request because every call resends the text	Moderate; retrieval keeps prompts short	Low per request after upfront training	Highest; several model calls per question
Main risk	Attention loss on very long inputs	Retrieval misses and stale or poisoned documents	Confident wrong facts that cannot be traced	Compounding errors and excessive tool permissions
Evaluation difficulty	Moderate; compare answers to a golden set	Moderate; score retrieval and generation separately	High; hard to isolate what the model learned	High; must grade the whole trajectory

Internal Assistants in Practice

Morgan Stanley's Advisor Knowledge Assistant

Morgan Stanley built an internal assistant for its financial advisors on top of GPT-4, and the firm describes it as able to answer questions from a corpus of 100,000 documents. The bank implemented an evaluation framework that tests each use case before deployment, with advisors and prompt engineers grading summaries for accuracy and coherence. According to OpenAI's customer story, more than 98% of advisor teams now use the assistant, and access to documents rose from 20% to 80%. A daily regression suite of sample questions checks for weaknesses and compliance after launch. The main limitation is that the account comes from the vendor and publishes no error rates or baseline comparisons, so the time savings are described mostly in qualitative terms. Advisors are also still required to review and edit AI outputs before using them with clients.

Uber's Genie On-Call Copilot

Uber deployed Genie, a Slack-based copilot that answers on-call questions using internal documents such as its engineering wiki and an internal question-and-answer site. The team chose retrieval over fine-tuning because it needed no curated training examples, which shortened the time to market. Since its September 2023 launch, Uber reported that Genie runs in 154 Slack channels, has answered over 70,000 questions, and saved about 13,000 engineering hours by the company's own estimate. Answers cite source URLs, and users rate each response as resolved, helpful, not helpful, or not relevant. The helpfulness rate stood at 48.9%, which means that more than half of the responses were not rated helpful. Uber also limited indexing to sources widely available to its engineers, a trade-off that protects privacy but narrows coverage.

LinkedIn's Knowledge Graph for Support Retrieval

LinkedIn built a customer service question-answering system that combines retrieval-augmented generation with a knowledge graph constructed from historical support tickets. Instead of splitting tickets into plain text chunks, the team preserved the structure within each issue and the relationships between issues. When a query arrives, the system parses it, retrieves related sub-graphs, and generates an answer from them. In the published study, the method improved mean reciprocal rank by 77.6% over the baseline, and median per-issue resolution time fell by 28.6% after about six months of use. A key limitation is that the approach required a large archive of structured tickets, which many organizations do not have. The paper reports results for one customer service team, so other teams should treat the figures as encouraging evidence and not a guaranteed outcome.

Lessons From Costly Deployments

Case Study: DoorDash's Guardrails for Dasher Support

DoorDash described a support challenge in 2024 because delivery contractors, called Dashers, needed quick answers about pay and policies, and live agents could not scale to every question. The company built a retrieval-augmented chatbot that condenses a conversation into an issue summary, finds similar resolved cases tied to knowledge base articles, and drafts a reply from that material. Its engineers knew that such a system can hallucinate, so they added an online guardrail that checks every output for grounding, coherence, and policy compliance. The guardrail has two tiers, with a fast semantic similarity check first and a slower language-model evaluator only when the first check raises a flag. A response must pass every test before it reaches the Dasher, and otherwise the conversation defaults to a human agent.

According to DoorDash's engineering blog, the guardrail reduced overall hallucinations by 90% and cut potentially severe compliance issues by 99%. A more sophisticated guardrail model was tested first but dropped because of slow response times and high token costs. The remaining drawbacks are added latency, since generating, checking, and retrying take time, and the continued need for live agents on complex cases. Summarization errors can also corrupt retrieval, so DoorDash noted that the conversation summary must be highly accurate. The lesson for internal assistants is to treat verification as a separate component that earns its place through measurable results.

Case Study: Air Canada's Chatbot Liability Ruling

Air Canada faced a legal problem after its website chatbot gave a grieving customer, Jake Moffatt, bad advice. The chatbot said he could request a reduced bereavement fare within 90 days of ticket issue after traveling. Moffatt bought full-price tickets, but the airline's actual policy did not allow bereavement rates for completed travel, and the airline later acknowledged that the chatbot's words were misleading. The airline argued that the chatbot was a separate legal entity responsible for its own actions, and that Moffatt should have checked the separate bereavement travel webpage. A tribunal rejected both arguments and found that Air Canada had not taken reasonable care to ensure the chatbot was accurate. As CBC News reported, Moffatt recovered $812, which was the fare reduction he had expected.

The sum was small, but the reasoning matters for every company that deploys an assistant. A business is responsible for what its assistant says, even when the correct policy exists elsewhere on its own site. The solution implied by the ruling is a controlled one, which grounds answers in a single authoritative policy source and tests them against it. The case also shows the limit of disclaimers, since the tribunal saw no reason why a customer should know which part of a website was accurate. Employees deserve the same protection when they rely on an internal assistant for policy answers.

Case Study: Samsung's Ban After a Code Leak

Samsung Electronics faced a data protection problem in April 2023 when employees uploaded sensitive source code to ChatGPT, according to Bloomberg's report. Public AI services keep submitted text on external servers, where it is hard to retrieve or delete and could be disclosed to other users. A memo to employees at one of its largest divisions introduced a ban on generative AI tools such as ChatGPT, Google Bard, and Bing. The impact was an immediate ban rather than a measurable reduction in risk, and Bloomberg's report gave no figures on how many employees were involved. A ban alone cannot remove the demand for quick answers, which is the argument for a sanctioned internal assistant that keeps documents inside the company boundary. Such an assistant still requires the permission controls, logging, and retention rules described earlier in this guide.

Common Questions About Building an Internal AI Assistant