An AI context window is the amount of information, measured in tokens, that a large language model (LLM) can process at one time. Think of it as the model’s short-term memory. Everything you provide in a prompt—instructions, questions, and code examples—must fit within this window for the AI to “see” it and generate a relevant response.

Tokens are the building blocks of text for an AI, representing characters, words, or parts of words. For code, a token could be a variable name, an operator, or a bracket. Different models have different context window sizes, ranging from a few thousand to hundreds of thousands of tokens. While larger windows are becoming more common, they still face limitations when dealing with the vastness of enterprise-scale codebases.

Token limits pose a significant challenge because modern software projects are often too large to fit into a single context window. An enterprise codebase can contain millions of lines of code spread across thousands of files. When you ask an AI assistant to refactor a function, explain a feature, or find a bug, it can only analyze the code you provide directly in the prompt.

This limitation has several practical consequences:

• Incomplete understanding: The AI can’t see the full picture. It might not know about dependencies, related modules, or inheritance structures that exist outside the provided snippet.
• Incorrect suggestions: Without full context, the AI might suggest changes that break other parts of the application. It might introduce a bug, misunderstand the business logic, or use an outdated pattern.
• Poor debugging: When trying to solve a complex bug, the root cause might be in a different file or service. If that code isn’t in the context window, the AI is effectively working with one hand tied behind its back.
• Manual overhead: Developers have to manually find and copy-paste relevant code snippets into the prompt, which is time-consuming, error-prone, and often incomplete.

Essentially, you can’t just ask an AI to “find the bug in our codebase” by pasting in a single file. You need a more sophisticated strategy to give it the right information at the right time.

To work effectively with AI on large projects, you need to engineer a system that manages the context for the model. The goal is to provide the most relevant information within the token limit. Three common strategies are context chunking, semantic search, and Retrieval-Augmented Generation (RAG).

The simplest approach is to break your code into smaller, logical pieces, or “chunks.” Instead of feeding the AI an entire service, you might provide a single file, class, or even just a function. This is a manual but often effective starting point. You could, for example, provide the function you want to refactor along with the class it belongs to and any directly imported modules.

This method requires developer intuition to identify which chunks are most relevant to the task.

A more advanced technique is semantic search, which finds code based on conceptual meaning rather than just keywords. This process involves using an embedding model to convert code chunks into numerical representations (vectors) that capture their semantic meaning. These vectors are then stored in a specialized vector database.

When you have a question or a piece of code to analyze, you convert your query into a vector and use the database to find the code chunks with the most similar vectors. This curated selection of highly relevant code is then passed to the LLM.

For example, a search for “user authentication logic” could find relevant files like auth_controller.py, user_model.js, and session_manager.rb, even if they don’t contain those exact keywords.

Retrieval-Augmented Generation, or RAG, formalizes these concepts into a powerful, automated system. A RAG system gives an AI assistant what feels like perfect recall of your entire codebase by connecting it to a knowledge base—in this case, a vector database of your code.

Here’s how it typically works:

1. Indexing: An automated process runs through your entire codebase, breaking it into chunks and converting each chunk into a vector embedding using an AI model. These embeddings are stored in a vector database.
2. Retrieval: When a developer asks a question (e.g., “How do we handle payment processing?”), the RAG system converts the query into an embedding and searches the vector database for the most relevant code chunks.
3. Augmentation: The retrieved code chunks are automatically added to the developer’s original prompt as context.
4. Generation: The combined prompt (original query + retrieved code) is sent to the LLM, which uses the provided context to generate a highly accurate and relevant answer.

This approach effectively gives the AI a “search engine” for your code, ensuring it always has the most relevant information to answer questions and perform tasks.

While building a custom RAG system is powerful, it’s not a trivial undertaking. It introduces new infrastructure and complexity that requires careful planning and maintenance.

Key challenges include:

• Infrastructure management: Setting up and maintaining a vector database requires specialized knowledge.
• Keeping the index fresh: Codebases change constantly. You need a robust pipeline to watch for changes, re-index modified files, and ensure the vector database is always up-to-date.
• Chunking strategy: Deciding how to split the code (by file, class, function, or even smaller logical blocks) has a major impact on the quality of the search results.
• Tuning for relevance: The retrieval step is crucial. Fine-tuning the search algorithm and embedding models to ensure the most relevant code is retrieved is an ongoing process of experimentation.

As you develop custom AI solutions like RAG systems, you are essentially creating new internal tools and APIs that interact with your most valuable intellectual property: your source code. Securing these new endpoints is critical to prevent unauthorized access and maintain control over your codebase.

This is where a robust authentication and authorization platform like Kinde can help. When you build a RAG system, it likely exposes an API that developers query. You need to ensure that only authorized team members or services can access this API.

By using Kinde, you can quickly add a security layer to your custom AI tools. You can protect your internal APIs by requiring a valid JWT access token for every request. This ensures that every call to your RAG system is authenticated and authorized, giving you peace of mind that your codebase remains secure.

For more details on implementation, see our guide on how to protect your API.

• Protect your API