The model is only one part of the system
A generative AI application can produce fluent answers while relying on incomplete, outdated or inappropriate information. The challenge is whether the application can retrieve the right content, respect access rights and provide enough context to answer reliably. A pilot built on one carefully selected folder may fail when expanded to thousands of documents and conflicting versions.
Readiness must be assessed across information architecture, quality, permissions, retrieval, evaluation and operational ownership—not only model selection.
Start with a focused knowledge domain
Avoid connecting every repository at once. Choose a domain with a clear user need, manageable scope and accountable business owner, such as IT guidance, approved product documentation or a defined set of operating procedures. Specify which questions the application should answer and which it must decline or route elsewhere.
This scope reveals authoritative sources, content gaps and relevant document types. A focused, maintained knowledge domain is often more valuable than a broad collection of documents with unclear ownership.
Improve quality, structure and currency
Retrieval works better when documents have clear titles, meaningful sections, stable identifiers and enough context to be understood outside their original location. Remove duplicates where appropriate, archive obsolete versions and identify conflicting guidance. Preserve metadata such as owner, publication date, business unit, document type and effective status.
Quality is not simply a technical score. A readable policy may be obsolete; a current procedure may be incomplete; two documents may use different terms for the same concept. Business owners need a process for reviewing content and resolving conflicts. AI cannot reliably compensate for unclear source material.
Treat permissions as part of retrieval
An AI application should not expose information a user could not otherwise access. Permission controls need to follow content into indexing and retrieval, not just be checked at the front door. This is difficult when repositories use different identity systems, inherited permissions or document-level restrictions.
Test with users who have different access rights, including users who must not see sensitive content. Check how permission changes and deletions reach the index. Security testing should include indirect attempts to retrieve unauthorised information, not only obvious access-denied cases.
Choose an architecture that preserves context
Retrieval-augmented generation (RAG) retrieves relevant source passages and supplies them to a model alongside the user’s question. Search may combine keywords, semantic retrieval and metadata filters. Results can be reranked, restricted to approved sources and presented with citations so users can verify an answer.
Chunking matters: chunks that are too small lose context; chunks that are too large may include irrelevant information. Metadata and document structure help preserve relationships. Configuration should be tested against real questions and content rather than selected by convention alone.
Evaluate retrieval and answers separately
Create realistic questions with expected answers, relevant sources and known edge cases. Include questions that cannot be answered from the approved corpus, ambiguous wording and conflicting or outdated source material. Evaluate retrieval quality separately from answer quality: a model cannot cite the right passage if retrieval never finds it.
Measure groundedness, correctness, citation relevance, refusal behaviour, latency and usefulness. Have subject-matter experts review samples and document failure modes. Repeat evaluations when prompts, models, indexes or source content change.
Govern the information lifecycle
Define who can publish, approve, update and retire content. Establish how changes reach the index, how often content is refreshed, how errors are reported and how owners respond. Track source lineage so an answer can be traced to the material that informed it.
Consider privacy, retention, data residency, logging and sensitive information in prompts or outputs. Governance should be practical enough to operate continuously, not just a document created for project approval.
A staged path to readiness
Inventory candidate sources, select one domain, clarify ownership, clean high-value content and map permissions. Build a small retrieval prototype and test representative questions. Fix the largest source and retrieval issues before expanding to more repositories or consequential workflows.
Readiness is not a one-time certification. It is the ongoing ability to maintain information quality, access control and evaluation as the business changes. Invest in that capability alongside the application itself.
The takeaway
Before scaling generative AI, make sure the right information can be found, trusted, permission-checked and continuously maintained.
This article presents general perspectives for business and technology decision-making. Recommendations should be adapted to each organization’s context, requirements and risk profile.