# Welcome Welcome to the ColiVara documentation! Here you'll get an overview of all the features ColiVara offers to help you build a state of the art retrieval system. Colivara is a suite of services that allows you to store, search, and retrieve documents based on their ***visual*** embeddings.
Why visual embeddings? Documents are visually rich structures that convey information through text, as well as tables, figures, page layouts, and charts. While legacy document retrieval systems exhibit good performance on query-to-text matching, they struggle to pass visual cues efficiently to large language models, hindering their performance on practical document retrieval applications such as Retrieval Augmented Generation.
It is a web-first implementation of the [ColPali paper](https://arxiv.org/abs/2407.01449) using ColQwen2 as the LLM model. It works exactly like RAG from the end-user standpoint - but using vision models instead of chunking and text-processing for documents. No OCR, no text extraction, no broken tables, or missing images. What you see, is what you get. ### Performance {% hint style="info" %} ColiVara performance is near **state of the art** for **Retrieval-Augmented Generation on the vidore leaderboard.** We significantly outperfomed currently methods for document parsing and processing such as OCR and captioning. {% endhint %} Our detailed [**Benchmark Performance Evaluation**](https://app.gitbook.com/o/DHjMRQs4Vfjvi2m062WT/s/r0RGXIvkomMWAuxgus2N/~/changes/31/getting-started/about/colivara-detailed-evaluation-result) have illustrated Colivara's performance across diverse benchmarks. Metrics like **NDCG\@5** score (Normalized Discounted Cumulative Gain at rank 5) and **Latency** were recorded for a comprehensive analysis. | Benchmark | Colivara Score | Avg Latency (s) (lower is better) | Num Docs | | ----------------------- | -------------- | --------------------------------- | -------- | | Average | 86.8 | N/A | N/A | | ArxivQA | 87.6 | 3.2 | 500 | | DocVQA | 54.8 | 2.9 | 500 | | InfoVQA | 90.1 | 2.9 | 500 | | Shift Project | 87.7 | 5.3 | 1000 | | Artificial Intelligence | 98.7 | 4.3 | 1000 | | Energy | 96.4 | 4.5 | 1000 | | Government Reports | 96.8 | 4.4 | 1000 | | Healthcare Industry | 98.5 | 4.5 | 1000 | | TabFQuad | 86.6 | 3.7 | 280 | | TatQA | 70.9 | 8.4 | 1663 |
### **Key Findings** * ColiVara dominated *visual-heavy benchmarks* like ArxivQA and InfoQA with NDCG\@5 score of **88.1**, *double* the performance of captioning-based systems. * Even for on *text-centric benchmarks*, ColiVara outperformed traditional methods by up to **30%** on benchmarks like DocQA and multimodal benchmarks like InfoQA. * For more comprehensive benchmarks, where a holistic approach of visual and textual analysis is key to query generation, such as the key for queries in specific domains (Sustainability, Energy, AI, Government Report, Healthcare), ColiVara shines overwhelmingly over competitions, scoring in the **high 90s** for all benchmarks. This is due to ColiVara's Holistic Multimodal Integration and Spatial Context Awareness.
### Jump right in
Getting StartedCreate your first RAG pipeline with 2 lines of codeQuickstart
GuidesLearn all what ColiVara have to offerGuide
API ReferenceTry the API live
# Quickstart
ColiVara is an web API that abstracts all the difficult parts about visual RAG. It embeds and saves documents, and then returns the highest matching pages when a user makes a query. {% hint style="info" %} We use the Python SDK in this quickstart, but since ColiVara is an API, you can use any language by making standard API calls. {% endhint %} ### API Keys Get an API Key from the [ColiVara Website](https://colivara.com) or via self-hosting. ### Install the Python SDK ```bash pip install colivara-py ``` ### Index a document Colivara accepts **a file url, or base64 encoded file, or a file path**. We support over 100 file formats including PDF, DOCX, PPTX, and more. We will also automatically take a screenshot of URLs (webpages) and index them. ```python import os from colivara_py import ColiVara rag_client = ColiVara( # this is the default and can be omitted api_key=os.environ.get("COLIVARA_API_KEY"), # this is the default and can be omitted base_url="https://api.colivara.com" ) # Upload a document to the default collection document = rag_client.upsert_document( name="attention is all you need", url="https://arxiv.org/abs/1706.03762", metadata={"published_year": "2017"} ) ``` ### Search You can filter by collection name, collection metadata, and document metadata. You can also specify the number of results you want. ```python results = rag_client.search(query="What is the role of self-attention in transformers?") print(results) # top 3 pages with the most relevant information ``` ### FAQ
Do I need a vector database? No - ColiVara uses Postgres and pgVector to store vectors for you. You DO NOT need to generate, save, or manage embeddings in anyway.
Do you convert the documents to markdown/text? No - ColiVara treats everything as an image, and uses vision models. There are no parsing, chunking, or OCR involved. This method outperforms chunking, and OCR for both text-based documents and visual documents.
How does non-pdf documents or web pages work? We run a pipeline to convert them to images, and perform our normal image-based retrieval. This all happen for you under the hood, and you get the top-k pages when performing retrieval.
Can I use my vector database? Yes - we have an embedding endpoint that only generates embeddings without saving or doing anything else. You can store these embeddings at your end. Keep in mind that we use late-interaction and multi-vectors, many vector databases do not support this yet.
# About RAG (Retrieval Augmented Generation) is a powerful technique that allows us to enhance LLMs (Language Models) output with private documents and proprietary knowledge that is not available elsewhere. For example, a company's internal documents or a researcher's notes). However, it is limited by the quality of the text extraction pipeline. With limited ability to extract visual cues and other non-textual information, RAG can be sub-optimal for documents that are visually rich. ColiVara uses vision models to generate embeddings for documents, allowing you to retrieve documents based on their visual content. Read more about the [original ColPali Model here](https://app.gitbook.com/o/DHjMRQs4Vfjvi2m062WT/s/r0RGXIvkomMWAuxgus2N/~/changes/18/getting-started/about/colpali-model) *From the ColPali paper:* > Documents are visually rich structures that convey information through text, as well as tables, figures, page layouts, or fonts. While modern document retrieval systems exhibit strong performance on query-to-text matching, they struggle to exploit visual cues efficiently, hindering their performance on practical document retrieval applications such as Retrieval Augmented Generation. [Learn More in the ColPali Paper](https://arxiv.org/abs/2407.01449)

Credit: helloIamleonie on X

### Key Features * **State of the Art retrieval**: The API is based on the ColPali paper and uses the ColQwen2 model for embeddings. It outperforms existing retrieval systems on both quality and latency. * **User Management**: Multi-user setup with each user having their own collections and documents. * **Wide Format Support**: Supports over 100 file formats including PDF, DOCX, PPTX, and more. * **Webpage Support**: Automatically takes a screenshot of webpages and indexes them even if it not a file. * **Collections**: A user can have multiple collections. For example, a user can have a collection for research papers and another for books. Allowing for efficient retrieval and organization of documents. * **Documents**: Each collection can have multiple documents with unlimited and user-defined metadata. * **Filtering**: Filtering for collections and documents on arbitrary metadata fields. For example, you can filter documents by author or year. Or filter collections by type. * **Convention over Configuration**: The API is designed to be easy to use with opinionated and optimized defaults. * **Modern PgVector Features**: We use HalfVecs for faster search and reduced storage requirements. * **REST API**: Easy to use REST API with Swagger documentation. * **Comprehensive**: Full CRUD operations for documents, collections, and users. * **Dockerized**: Easy to setup and run with Docker and Docker Compose on your infrastructure. ### Evals: The ColPali team has provided the following evals in their paper. We have run quick sanity checks on the API and the Embeddings Service and are getting similar results. We are working on own independent evals and will update this section with our results. Updates: * 11/6/2024: Our ArxivQ score is 86.6 - matching state of the art results in the vidore leaderboard.
### Components: 1. Postgres DB with pgvector extension for storing embeddings. [ColiVara repo](https://github.com/tjmlabs/ColiVara) 2. REST API for document/collection management. [ColiVara repo.](https://github.com/tjmlabs/ColiVara) 3. Embeddings Service. This needs a GPU with at least 8gb VRAM. The code is under [`ColiVarE`](https://github.com/tjmlabs/ColiVarE) repo and is optimized for a serverless GPU workload. > You can run the embedding service separately and use your own storage and API for the rest of the components. The Embedding service is designed to be modular and can be used with any storage and API. (For example, if you want to use Qdrant for storage and Node for the API) 4. Language-specific SDKs for the API (Typescript SDK Coming Soon) 1. Python SDK: [ColiVara-Py](https://github.com/tjmlabs/colivara-py) ### License This project is licensed under Functional Source License, Version 1.1, Apache 2.0 Future License. For commercial licensing, please contact us at [tjmlabs.com](https://tjmlabs.com). We are happy to work with you to provide a license that meets your needs. # ColPali Architecture Traditional document parsing techniques rely heavily on extracting plain texts, at the cost of overlooking graphical elements. However, complex documents such as technical papers and presentations provide much of their context visual cues such as such as tables, images, charts, and the layout structure. As such, visual context is vital to extract, interpret, and prioritize information in order to generate contextually accurate response. In short, ColPali is an advanced document retrieval model that leverages Vision Language Models to integrate both **textual** and **visual** elements for highly accurate and efficient document search {% hint style="success" %} **ColiVara** uses **ColQwen**, an improved model from ColPali. {% endhint %} ## Standard Document Retrieval technology Document Retrieval technology is central to many applications, either ass a standalone ranking system, or as part of a complex Retrieval Augmented Generation (RAG) pipeline.
There are 2 phases in a Standard Retrieval :
An Offline Indexing phase, where all the documents from the corpus are indexed.
An Online Querying phase, where a user query is matched with a low latency to the pre-computed document index.
To index a standard PDF document, the Offline Indexing process might look like this: * PDF parsers or Optical Character Recognition (OCR) systems extract text from document pages. * Layout detection models segment the document into structured parts. * Optional captioning step, which provides natural language descriptions for visual elements, making them more compatible with embedding models. * A chunking strategy groups related text passages to maintain semantic coherence * Text embeddings are created by mapping vectors meant to represent the text's semantic meaning. Finally, the documents - with its generated embeddings - are ready to be queried. A Query follows a similar process to be converted into its vector presentation. The query's vector can be used on the document's vector to produce an answer. {% hint style="warning" %} While there have been large improvements in text embedding models, practical experiments have shown that the performance bottleneck lies in the ingestion pipeline in which the *visually rich* document was processed, prior to being consumed by an LLM. Additionally, this process is slow, difficult, and prone to propagate errors. {% endhint %} *** ## ColPali Model **ColPali** is a novel approach to Document Retrieval, by doing away with the textual indexing phase, and replacing this process by generating embeddings on the images (or screenshots of documents) directly.
ColPali was built on previous technologies:
PaliGemma A Vision Language Model that processes images by breaking them down into smaller sections, or *patches*. These patches are analyzed by a vision transformer (*SigLIP-So400m)* in order to produce detailed vector representations, known as *patch embeddings*. These *embeddings* are fed into a language model (*Gemma 2B*) to create representations of each patch within the language model's space. This results in a multi-vector representation for each page image, with both visual and contextual information.
ColBERT with Late Interaction A specialized search model that combines *Multi-Vector Representation* with a *Late Interaction* approach * **Multi-Vector Representation**: Unlike traditional retrieval methods that summarize an entire document with a single vector, ColBERT creates separate vectors for each word or phrase in a document and query. As a result, ColBERT can retain more information and identify nuances in contexts. * **Late Interaction Mechanism**: Instead of performing complex interactions between vectors for the query and document upfront, ColBERT initially calculates a broad similarity score to *cheaply* narrow down potential matches. Detailed similarity comparisons are only one on the most relevant parts of the document.
{% hint style="success" %} ColPali, a Paligemma-3B extension that is capable of generating ColBERT-style multi-vector representations of text and images, optimizes the ingestion pipeline of visually rich document by creating embeddings on the visual elements using Vision Language Model. This yields much greater performance boost than optimizing the text embedding model. {% endhint %} *** ### Performance ColPali exhibits high efficiency: * **High retrieval performance**: The use of Vision Language Model creates high-quality contextual embeddings from document images, facilitating quicker retrieval. * **Low querying latency**: Late interaction mechanism significantly decreases the number of comparisons needed during querying. * **High indexing speed***:* Simpler indexing process by directly processing document images, eliminating the need for complex text extraction and segmentation pipelines.

Both visually demanding and text centric document are better retrieved using the ColPali model

*** ## Citation Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., & Colombo, P. *ColPali: Efficient Document Retrieval with Vision Language Models*. Illuin Technology, Equall.ai, CentraleSupélec, Paris-Saclay, ETH Zürich. Available at: # ColiVara Benchmark Evaluation ## Overview Colivara stands out as a practical, user-focused tool that combines state-of-the-art technology with real-world adaptability. Built on the colqwen2 model, Colivara is optimized for operational environments, offering reliable, scalable, and actionable insights. Benchmark comparisons show ColiVara is markedly better than current methods for data extraction and query generation from documents. Whether dealing with raw text, OCR-enhanced data, or captioned inputs, Colivara consistently outperforms by a wide margin. When compared to the top research models on the ViDoRe benchmark, ColiVara results stay competitive.

NDCG@5 scores comparison of ColiVara and Traditional Unstructured Methods

Our comparisons are done using the systems’ **NDCG\@5** score, which is the standard used by the ViDoRe benchmark. *Normalized Discounted Cumulative Gain at rank 5* evaluates how a retrieval model delivers relevant results to a query. NDCG gives higher importance to results that are both relevant and appear at the top of the list. It focuses on the quality of the top 5 results, which are usually the most important to users
*** ## ViDoRe We are measuring our data using the **ViDoRe Benchmark**, which was introduced alongside the [**ColPali model**](/getting-started/about/colpali-architecture). The benchmark serves as a platform to evaluate and compare different retrieval models, highlighting the importance of visual elements in document understanding and retrieval {% hint style="info" %} **ViDoRe**, short for Visual Document Retrieval Benchmark, is a comprehensive evaluation framework designed to assess the performance of retrieval systems in matching user queries to relevant document pages. It emphasizes the integration of both textual and visual information, reflecting the multifaceted nature of real-world documents. {% endhint %} *** ## Technical Comparisons of Traditional Methods Retrieval-Augmented Generation (RAG) integrates retrieval systems with generative models, enabling contextual, grounded, and informed responses. However, most retrieval systems today still rely heavily on traditional document parsing methods like text-only processing, OCR-enhanced retrieval, and captioning strategies. These methods, while robust and widely used, face significant limitations when applied to the complex and dynamic requirements of RAG systems.
Unstructured (Text only) This refers to a document retrieval system where only textual elements of the documents are utilized. Non-text elements like images, figures, tables, and charts are filtered out as "noisy" information. The focus is solely on extracting and processing text for retrieval tasks
Unstructured + OCR (Optical Character Recognition): This approach incorporates OCR technology to process visual elements. The OCR extracts textual content from these visual elements, which is then chunked and added to the retrieval pipeline. This expands the system's capability to handle and retrieve text embedded in non-textual document components. OCR has been a game-changer for processing scanned PDFs and image-heavy documents.
Unstructured + Captioning In this setup, visual elements are processed using a captioning strategy. Advanced Vision Language Models (VLMs) generate detailed natural language descriptions of these elements, which are integrated into the retrieval pipeline. This enables the system to retrieve documents based on visual context described in text form, enhancing multimodal document retrieval capabilities.
*** ## ColiVara significantly outmatches traditional methods in technical benchmarks ColiVara was tested head-to-head against traditional methods in technical benchmarks such as: * ArxivQA: Text-heavy dataset * DocQA: Pages-heavy dataset * InfoQA: Inforgraphics-heavy dataset * TabFA: Tabular data-heavy dataset * TAT-Q: Tabular data-heavy dataset with Numeric reasoning queries Traditional methods like Unstructured (Text-Only), OCR, and Captioning pipelines fall short when tasked with understanding the nuanced relationships between text, visuals, and structured information, leaving gaps in retrieval accuracy and relevance. In contrast, ColiVara’s multimodal approach enables it to excel across all these domains, demonstrating the versatility and precision needed for technical document retrieval. Across all technical benchmarks, ColiVara outperforms traditional approaches by leveraging its ability to integrate textual content with contextual layout features. This allows it to identify not just matching text but also the structural relevance of document sections, ensuring higher retrieval accuracy. Of Note, ColiVara outperforms even in text-heavy benchmark where visual elements are not the main focus. ColiVara holistically integrates contextual cues that traditional text-only pipelines often miss. In other words, traditional methods treat the document as a linear stream of text, disregarding structural elements such as page breaks or paragraph breaks. While ColiVara incorporates both the text and its broader layout representation, leading to more accurate retrieval. *** ## ColiVara outperforms traditional methods in domain-specific benchmarks These benchmarks often involve highly specialized and multimodal documents that combine dense textual information, data tables, charts, and figures: * Shift: environmentally-themed dataset * AI: Artificial Intelligence algorithms and research dataset * Energy: technical and administrative reports in the Energy sector * Government: technical and administrative governmental reports * Health: medical records, reports, and studies-heavy dataset
These benchmarks do not focus on specific technical evaluation. Instead, they are drawn from real-world knowledge domains. Depending on the domain, the retrieval system might need to perform heavily on one or more technical aspects, such as tabular data or charts. These benchmarks more so represent real-world usages where users are more likely in need of analyzing overlapping, and sometimes even contradicting, data points to answer a specific question. As the data proves, ColiVara excels on domain-specific benchmarks because of its ability to seamlessly integrate textual, visual, and structural data into a unified retrieval framework, which is particularly important for specialized documents that combine multiple modalities. ColiVara’s end-to-end multimodal architecture enables it to process entire pages as cohesive entities, capturing the interplay between text and visuals while understanding spatial relationships and layout structures. This allows ColiVara to interpret trends in visual data, link numerical reasoning to textual descriptions, and retrieve the most contextually relevant content. Furthermore, its late interaction mechanism enhances fine-grained matching between user queries and both text and visual elements, ensuring precision even in highly complex domains. In short, ColiVara consistently outperforms traditional methods and sets a new standard for domain-specific document retrieval. *** ## ColiVara scored the highest Average score, indicating consistent performance across all Tasks The Average score serves as an indicator of overall reliability across diverse tasks. Colivara scored **87.6,** demonstrating solid and consistent performance. This steady performance makes Colivara ideal for organizations needing a dependable system for a variety of tasks, including document search, retrieval, and management. Unlike traditional retrieval systems that excel in either textual or visual contexts but falter in handling both simultaneously, ColiVara demonstrates a robust capability to adapt and perform optimally regardless of the document type or domain. This consistency stems from its end-to-end multimodal architecture, which integrates text, visuals, and layout into a unified framework, ensuring it captures all relevant information across diverse formats ColiVara's ability to maintain high retrieval accuracy across tasks with varying complexities and requirements makes it a standout performer, setting a new standard for reliability and effectiveness in document retrieval systems. # Self-hosting ColiVara is made up of multiple services. The first is an embedding service that turns images and queries into Vectors. This is bundled separately as it needs a GPU. The second is [a Postgres with a pgvector](https://github.com/pgvector/pgvector) extension to store vectors. A [Gotenberg](https://gotenberg.dev/) service that handles document conversions to PDFs. And finally, a [Django-Ninja](https://django-ninja.dev/) REST API that handles user requests. Other than the Embedding service, everything else is bundles together via docker-compose to run seamlessly on a typical VPS. For production workloads - you may consider a managed Postgres instance for automatic security updates and regular backup. ### Embedding Service 1. Git clone the service repository ```bash git clone https://github.com/tjmlabs/ColiVarE ``` 2. Optional: download uv and install it in your environment. We use uv, however you can also use pip to install the requirements. ```bash pip install uv uv venv # or python -m venv .venv source .venv/bin/activate #.venv/Scripts/activate on windows ``` 3. Compile requirements based on your environment. As this services uses pytorch under the hood- requirements will be different depending on your OS and Nvidia GPU availability. We use a mac for development and a Linux in production. 4. Install the requirements ```bash uv pip compile builder/requirements.in -o builder/requirements.txt ``` 5. Download the models from huggingface and save them in the `models_hub` directory before building. See src/download\_models.py for more details. ```python from colpali_engine.models import ColQwen2, ColQwen2Processor import torch model_name = "vidore/colqwen2-v1.0" if torch.cuda.is_available(): device_map = "cuda" elif torch.backends.mps.is_available(): device_map = "mps" else: device_map = None model = ColQwen2.from_pretrained( model_name, cache_dir="models_hub/", # where to save the model device_map=device_map, ) processor = ColQwen2Processor.from_pretrained(model_name, cache_dir="models_hub/") ``` 6. Run the service locally using the following command ```bash python3 src/handler.py --rp_serve_api ``` 7. The Embedding service is now running on `http://localhost:8000/`. You can test it using the following command. Remember - you do need a GPU and at least 8gb of VRAM available. The performance on a M-series of Macs is also acceptable for local development. ```bash curl --request POST \ --url http://localhost:8000/runsync \ --header 'Content-Type: application/json' \ --data '{"input": {"task": "query","input_data": ["hello"]}}' ``` You may consider running this service in an "on-demand" fashion via Docker for cost-savings in production settings. ### REST API 1. Clone the ColiVara repository ```bash git clone https://github.com/tjmlabs/ColiVara ``` 2. Create a .env.dev file in the root directory with the following variables: ``` EMBEDDINGS_URL="the serverless embeddings service url" # for local setup use http://localhost:8000/runsync/ EMBEDDINGS_URL_TOKEN="the serverless embeddings service token" # for local setup use any string will do. AWS_S3_ACCESS_KEY_ID="an S3 or compatible storage access key" AWS_S3_SECRET_ACCESS_KEY="an S3 or compatible storage secret key" AWS_STORAGE_BUCKET_NAME="an S3 or compatible storage bucket name" ``` 3. Run all the services via docker-compose ````bash ``bash docker-compose up -d --build docker-compose exec web python manage.py migrate docker-compose exec web python manage.py createsuperuser # get the token from the superuser creation docker-compose exec web python manage.py shell from accounts.models import CustomUser user = CustomUser.objects.first().token # save this token ``` ```` 4. Application will be running at and the swagger documentation at `http://localhost:8001/v1/docs` 5. The swagger documentations page is also a playground - where you can try all the endpoints using the token created earlier
### Development Follow the steps above to get the service up and running. 1. To run tests and type checking - we have 100% test coverage ```bash docker-compose exec web pytest #mypy for type checking docker-compose exec web mypy . ``` 2. Make a branch with your changes and additional code 3. Open a Pull request on Github. We have CI/CD and pre-commit hooks to format and test your changes We welcome contribution and discussion. # Retrieval Augmented Generation (RAG) The common problem ColiVara solves is RAG over visually rich documents where text extraction pipelines fail or presents incomplete data. For example, heavy tabular data with many charts such as policies for a medical facility. You can find the complete code for this demo below. {% embed url="" %} *** ## Overview
What you'd need * A *ColiVara API key* * An *LLM API key* of your choice * Appropriate documents for RAG queries
High-level steps overview 1. Prepare your environment 2. Prepare your documents 3. Sync (upload) documents to ColiVara server 4. Transform your search query 5. Search processed document for relevant embeddings 6. Generate a factual, grounded response
*** ### 1. Prepare your environment First, install the ColiVara Python SDK. If using Jupyter Notebook: ```bash !pip install colivara_py ``` If using the command shell: ```bash pip install colivara_py ``` *** ### 2. Prepare your documents This is the start of our RAG pipeline. We start by preparing the documents. That could be a directory on your local computer, a S3 bucket, a google drive. Anything you can think of will work with ColiVara. For the purposes of this guide - we will use a local directory with some documents in them. If using Jupyter Notebook: ```bash !pip install requests ``` If using the command shell: ```bash pip install requests ``` Then download the documents from Github. For the purposes of this demo, we will download the smallest 2 files. But, feel free to try with your own documents or all the documents in our demo repository. ```python import requests import os def download_file(url, local_filename): # Send a GET request to the URL response = requests.get(url) # Check if the request was successful if response.status_code == 200: # Ensure the 'docs' directory exists os.makedirs('docs', exist_ok=True) # Write the content to a local file with open(local_filename, 'wb') as f: f.write(response.content) print(f"Successfully downloaded: {local_filename}") else: print(f"Failed to download: {url}") # URLs and local filenames files = [ { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/Work-From-Home%20Guidance.pdf", "filename": "docs/Work-From-Home-Guidance.pdf" }, { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/StaffVendorPolicy-Jan2019.pdf", "filename": "docs/StaffVendorPolicy-Jan2019.pdf" } ] # Download each file for file in files: download_file(file["url"], file["filename"]) ``` *** ### 3. Sync your documents We want to sync our documents to the ColiVara server. So, we can just call this as our documents change or updated. ColiVara logic automatically updates or inserts new documents depending on what changed. The `wait=True` parameter ensures this process is synchronous. ```python from colivara_py import ColiVara from pathlib import Path import base64 # set base_url and api_key rag_client = ColiVara( base_url="https://api.colivara.com", api_key="your-api-key" ) def sync_documents(): # get all the documents under docs/ folder and upsert them to colivara documents_dir = Path('docs') files = [f for f in documents_dir.glob('**/*') if f.is_file()] for file in files: with open(file, 'rb') as f: file_content = f.read() encoded_content = base64.b64encode(file_content).decode('utf-8') rag_client.upsert_document( name=file.name, document_base64=encoded_content, collection_name="demo collection", wait=True ) print(f"Upserted: {file.name}") sync_documents() ``` *** ### 4. Transform your query Next - we we want to to transform a user messages or questions into an appropriate *RAG question*. In retrieval augmented generation - user and AI take turns in a conversation. In each turn, we want to get a factual context - that the AI can use in providing the answer. We will need an LLM to help us with this transformation. For this guide, we will use gpt-4o but lighter models are also effective. If using Jupyter Notebook: ```bash !pip install openai ``` If using the command shell: ```bash pip install openai ``` Here is the code for the transformation. ```python from openai import OpenAI llm_client = OpenAI(api_key="your-api-key") def transform_query(messages): prompt = """ You are given a conversation between a user and an assistant. We need to transform the last message from the user into a question appropriate for a RAG pipeline. Given the nature and flow of conversation. Example #1: User: What is the capital of Brazil? Assistant: Brasília User: How about France? RAG Query: What is the capital of France? Somewhat related query, however, if we simply use "how about france?" without any transformation, the RAG pipeline will not be able to provide a meaningful response. The transformation took the previous question (what the capital of Brazil?) as a strong hint about the user intention Example #2: User: What is the policy on working from home? Assistant: User: What is the side effects of Wegovy? RAG Query: What are the side effects of Wegovy? The user is asking for the side effects of Wegovy, the transformation is straightforward, we just need to slightly adjust. The previous question was about a completely different topic, so it has no influence on the transformation. Example #3: User: What is the highest monetary value of a gift I can recieve from a client? Assistant: User: Is there a time limit between gifts? RAG Query: What is the highest monetary value of a gift I can recieve from a client within a specific time frame? The user queries are very related and a continuation of the same question. He is asking for more details about the same topic. The transformation needs to take into account the previous question and the current one. Example #4: User: Hello! RAQ Query: Not applicable The user is simply greeting the assistant, there is no question to transform. This applies to any non-question message, Coversation: """ for message in messages: prompt += f"{message['role']}: {message['content']}\n" response = llm_client.chat.completions.create( model="gpt-4o", messages=[{"role": "assistant", "content": prompt}], stream=False, ) query = response.choices[0].message.content return query messages = [{"role": "user", "content": "What is the work from home policy?"}] transform_query(messages) ``` *** ### 5. Search for context using the RAG pipeline Finally, with our document and query prepared - we are ready to run our RAG pipeline with ColiVara. ```python def run_rag_pipeline(query): # clean the query that came from the LLM if "not applicable" in query.lower(): # don't need to run RAG or get a context return [] if "rag query:" in query.lower(): query = query.split("ry:")[1].strip() if "" in query: query = query.split("")[0].strip() print(f"This what we will send to the RAG pipeline: {query}") results = rag_client.search(query=query, collection_name="demo collection") results = results.results # example result: """{ "query": "string", "results": [ { "collection_name": "string", "collection_id": 0, "collection_metadata": {}, "document_name": "string", "document_id": 0, "document_metadata": {}, "page_number": 0, "raw_score": 0, "normalized_score": 0, "img_base64": "string" } ] }""" context= [] for result in results: document_title = result.document_name page_num = result.page_number base64 = result.img_base64 # base64 doesn't have data: part so we need to add it if "data:image" not in base64: base64 = f"data:image/png;base64,{base64}" context.append( { "metadata": f"{document_title} - Page {page_num}", "base64": base64, } ) return context query = 'RAG Query: What is the work from home policy?' context = run_rag_pipeline(query) ``` Let's peek what our context looks like:
*** ### 6. Generate answer With your context in hand - now you pass this to an LLM with multi-modal/vision capabilities and get a factually grounded answer. ```python def draft_response(messages): query = transform_query(messages) context = run_rag_pipeline(query) content = [ { "type": "text", "text": f"""Use the following images as a reference to answer the following user questions: {query}. """, }, { "type": "image_url", "image_url": {"url": context[0]["base64"]}, }, { "type": "image_url", "image_url": {"url": context[1]["base64"]}, }, { "type": "image_url", "image_url": {"url": context[2]["base64"]}, }, ] last_message = messages[-1] # the last message is the user question, we enhance it with the RAG query last_message["content"] = content # pop the last message and append the new message messages.pop() messages.append(last_message) response = llm_client.chat.completions.create( model="gpt-4o", messages=messages, stream=False, ) answer = response.choices[0].message.content return answer messages = [ {"role": "user", "content": "Can I work from home on Fridays?"} ] answer = draft_response(messages) print(answer) """ This what we will send to the RAG pipeline: What is the policy on working from home on Fridays? Based on the documents provided, the policy on working from home at Mount Sinai is as follows: 1. **Eligibility and Requirements**: Remote work arrangements can be made available to employees in appropriate positions as determined by Mount Sinai. These arrangements require departmental approval and are subject to periodic review. 2. **Days of Remote Work**: The specific days for remote work, such as Fridays, would be set in the Remote Work Schedule and agreed upon with the manager, as there is a blank section to be filled regarding the days of the week. 3. **Compliance and Reporting**: Employees are required to comply with existing policies and notify their manager when they are working remotely. 4. **Frequency**: Employees engaged in full-time remote work must work at least one scheduled day at a Mount Sinai location in New York State annually. 5. **Approval and Review**: Remote work involving supervisory roles, such as Principal Investigators, requires approval from the Dean’s office. There is no specific mention of Fridays, implying that any remote work schedule, including Fridays, needs to be established and agreed upon with management. """ ``` And with that - your RAG pipeline is complete. # Data Extraction ## Retrieval-Augmented Generation (RAG) vs Data Extraction While similar, RAG and Data Extraction are fundamentally different in their use cases and desired goals. **RAG** is combines *information retrieval* with *generative AI* to create responses based on relevant context. RAG is ideal for answering specific questions or generating **context-aware responses** from a large corpus, like handling FAQs, policy summaries, or dynamic information. Let’s consider a practical example using an employee policy document that contains information about work-from-home guidelines, employee benefits, and leave policies. For a RAG-based system, the user might ask a question, such as: > *How much paid time off does an employee with 5 years of service receive?* The system might then respond with an answer: > *Employees with 5 years of service are eligible for 20 days of paid time off per year. Leave requests should be submitted at least two weeks in advance, and PTO is restricted during company-wide blackout periods* {% hint style="info" %} **RAG** provides **contextual responses** in natural language, suitable for conversational or FAQ-type applications. {% endhint %} **Data extraction** aims to extract **structured data** information from documents (e.g., names, dates, specific values) *without generating responses*. The goal is to capture and structure data for easier access and analysis. Using the previous example when querying from an employee policy document containing work-from-home policy, an user might query the document for "PTO detail". After which, a data extraction system might give back a JSON data as follow: ```json "pto_details" : { "years_of_service": 5, "pto_amount": "20 days", "notice_requirement": "2 weeks", "black_out_applied?": true } ``` {% hint style="info" %} **Data Extraction** gives **structured data** outputs, ideal for populating databases, automating forms, or generating reports where individual fields are needed without additional narrative context. {% endhint %} *** ## Data Extraction using ColiVara ### Step 1: Client Setup Install the `colivara-py` SDK library If using Jupyter Notebook: ```bash !pip install --no-cache-dir --upgrade colivara_py ``` If using the command shell ```bash pip install --no-cache-dir --upgrade colivara_py ``` ### Step 2: Prepare Documents {% stepper %} {% step %} **Download Files**: Download the desired files to your machine. This code specifically will download them into your `docs/` folder ```python import requests import os def download_file(url, local_filename): response = requests.get(url) if response.status_code == 200: os.makedirs('docs', exist_ok=True) with open(local_filename, 'wb') as f: f.write(response.content) print(f"Successfully downloaded: {local_filename}") else: print(f"Failed to download: {url}") # URLs and local filenames files = [ { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/Work-From-Home%20Guidance.pdf", "filename": "docs/Work-From-Home-Guidance.pdf" }, { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/StaffVendorPolicy-Jan2019.pdf", "filename": "docs/StaffVendorPolicy-Jan2019.pdf" } ] # Download each file for file in files: download_file(file["url"], file["filename"]) ``` {% endstep %} {% step %} **Upload Files to be procesed**: Sync documents to the ColiVara server. The server will process these files to generate the necessary embeddings ```python from colivara_py import ColiVara from pathlib import Path import base64 rag_client = ColiVara( base_url="https://api.colivara.com", api_key="your-api-key" ) new_collection = rag_client.create_collection( name="my_collection", metadata={"description": "A sample collection"} ) def sync_documents(): # get all the documents under docs/ folder and upsert them to colivara documents_dir = Path('docs') files = [f for f in documents_dir.glob('**/*') if f.is_file()] for file in files: with open(file, 'rb') as f: file_content = f.read() encoded_content = base64.b64encode(file_content).decode('utf-8') rag_client.upsert_document( name=file.name, document_base64=encoded_content, collection_name="my_collection", wait=True ) print(f"Upserted: {file.name}") sync_documents() ``` {% endstep %} {% step %} **(Optional) Verify that documents have been processed by ColiVara**: Documents have been convert into "screenshots" to generate embeddings If using Jupyter Notebook: ```python from IPython.display import display, HTML, Image def display_image_from_document(document): pages = doc.pages for page in pages: base64_img = page.img_base64 if base64_img.startswith('data:image'): base64_img = base64_img.split(',')[1] image = Image(data=base64.b64decode(base64_img)) display(image) document_name = "Work-From-Home-Guidance.pdf" doc = client.get_document(document_name=document_name, collection_name="all", expand="pages") display_image_from_document(doc) ``` If using the a Python code editor (such as VSCode): ```python from IPython.display import Image from io import BytesIO from PIL import Image def display_image_from_document(document): pages = doc.pages for page in pages: base64_img = page.img_base64 if base64_img.startswith('data:image'): base64_img = base64_img.split(',')[1] image_data = base64.b64decode(base64_img) image = Image.open(BytesIO(image_data)) image.show() document_name = "Work-From-Home-Guidance.pdf" doc = rag_client.get_document( document_name=document_name, collection_name="my_collection", expand="pages") display_image_from_document(doc) ``` {% endstep %} {% endstepper %} ### Step 3: Extract data {% stepper %} {% step %} **Install the LLM of choice**. Here, we are using OpenAI's GPT model If using Jupyter Notebook: ```bash !pip install openai ``` If using the command shell: ```bash !pip install openai ``` {% endstep %} {% step %} **Extract the JSON data** ```python import json from openai import OpenAI llm_client = OpenAI(api_key="your-api-key") # if your document is too big (20+ pages), or you want to extract data across multiple documents, # consider a pipeline where you search first and get top 3 pages, and then do this step. # here since our document is small - we are passing the whole document at once. def extract_data(data_to_extract, colivara_document): string_json = json.dumps(data_to_extract) content = [ { "type": "text", "text": f"""Use the following images as a reference to extract structured data with the following user example as a guide: {string_json}.\n If information is not available, keep the value blank. """, } ] pages = colivara_document.pages for page in pages: base64 = f"data:image/png;base64,{page.img_base64}" content.append( { "type": "image_url", "image_url": {"url": base64} } ) messages = [ {"role": "system", "content": "Our goal is to find out when a policy was issued to remind our users to review it at regular intervals. Always respond in JSON"}, {"role": "user", "content": content} ] completion = llm_client.chat.completions.create( model="gpt-4o", messages=messages, response_format= { "type": "json_object" }, temperature=0.25, seed=123 ) return completion.choices[0].message.content document_name = "Work-From-Home-Guidance.pdf" doc = rag_client.get_document( document_name=document_name, collection_name="my_collection", expand="pages") data_to_extract = {"year_issued": 2014, "month_issued": 3} data = extract_data(data_to_extract, doc) print(data) ``` {% endstep %} {% endstepper %} The result output data should be: > ```json > { > "year_issued": 2020, > "month_issued": 3 > } > ``` # Advanced filtering ## You can use ColiVara to filter documents based on their metadata **Colivara** offers fine-grained control over data retrieval by implementing various **Query Filters**, which boosts precision and lowers latency in response generation by filtering by document's metadata (or collection's metadata). Queries running on document can be filtered by: * **Direct Matching (**[**`key_lookup`**](#id-1.-key_lookup)**,** [**`has_key`**](#id-4.-has_key)**,** [**`has_keys`**](#id-5.-has_keys)**,** [**`has_any_keys`**](#id-6.-has_any_keys)**)**: Allows for exact matches, making it ideal for queries where the exact value matters (e.g., retrieving specific data where exact metrics are critical). * **Partial Containment (**[**`contains`**](#id-2.-contains)**,** [**`contained_by`**](#id-3.-contained_by)**)**: These enable partial matches, so if a user asks for a general topic (like "budget plans"), the model can pull relevant documents even if they do not perfectly match but contain related content. Advanced filters can be used on metadata from a single document, or from the entire collection This leads to *higher relevance*, as users get nuanced responses tailored to both broad and specific information requests. *** ## How to utilize query filter To use Advanced Filtering on your documents or collection, pass in a `query_filter` parameter to the `search` function call. First - add relevant metadata to your documents (or collections) on document insertion. ```python from colivara_py import ColiVara rag_client = ColiVara( # this is the default and can be omitted api_key=os.environ.get("COLIVARA_API_KEY"), # this is the default and can be omitted base_url="https://api.colivara.com" ) # Upload a document to the default_collection document = rag_client.upsert_document( name="sample_document", url="https://example.com/sample.pdf", metadata={"author": "John Doe"}, # optional - specify a collection collection_name="user_1_collection", # optional - wait for the document to index wait=True ) ``` Then - use pass in a `query_filter` parameter to the `search` function call. ```python query_filter = { "on": "document", # or collection "lookup": "contains", # or any of the supported lookup method "key": "auther", "value": "John Doe" } search_results = rag_client.search( query="my_query", query_filter=query_filter, collection_name="user_1_collection", top_k=3 ) ``` *** ## Available Filters #### 1. `key_lookup` * **Description**: Performs an exact match between a specified metadata key and a value. Useful when a precise key-value pair match is required. * **Parameters**: * `key (str)`: Metadata key to match. Must be a single string. * `value(str, int, float, bool)`: Exact value for key. Required. * **Example Usage**: ```python # Retrieve documents where the author is exactly "John Smith" query_filter = { "lookup": "key_lookup", "on": "document", # or "collection" "key": "author", "value": "John Smith", } ``` *** #### 2. `contains` * **Description**: checks if a specified metadata key contains a particular substring or element. Useful for partial matches. * **Parameters**: * `key (str)`: Metadata key in which to search for the substring. * `value (str)`: Substring to find within the key’s value. Required. * **Example Usage**: ```python # Find documents where tags include the substring "science" query_filter = { "lookup": "contains", "on": "document", "key": "tag", "value": "science", } ``` *** #### 3. `contained_by` * **Description**: verifies if a metadata value is fully contained within a provided structure, such as a string or list. Useful for partial matches. * **Parameters**: * `key (str)`: Metadata key to evaluate. * `value (str, List[str])`: A master string, or a list of string, of which the key's value is a substring. Required. * **Example Usage**: ```python # Find collections where keywords are contained within "machine learning research" query_filter = { "lookup": "contain_by", "on": "collection", "key": "machine learning research", "value": "science", } ``` *** #### 4. `has_key` * **Description**: checks if a specified key exists within the metadata, without verifying its value. Useful for filtering documents or collections based on the presence of a particular metadata field. * **Parameters**: * `key (str)`: Metadata key to check for existence * Must not contain a `value` parameter * **Validation**: * `key` must be a string. * Must not contain a `value` parameter * **Example Usage**: ```python # Retrieve documents where "publication_year" field exists query_filter = { "lookup": "has_key", "on": "document", "key": "publication_year", } ``` *** #### 5. `has_keys` * **Description**: ensures that all specified keys in a list exist within the metadata. Useful for filtering documents or collections based on the presence of a list of metadata fields. * **Parameters**: * `key (List[str])`: Must be a list of strings. * Must not contain a `value` parameter * **Example Usage**: ```python # Retrieve documents with both "author" and "date" fields query_filter = { "lookup": "has_key", "on": "document", "key": ["author", "date"], } ``` *** #### 6. `has_any_keys` * **Description**: checks if at least one of the keys in a specified list exists in the metadata * **Parameters**: * `key (List[str])`: Must be a list of strings. * Must not contain a `value` parameter * **Example Usage**: ```python # Retrieve documents with either "author" or "date" fields query_filter = { "lookup": "has_any_keys", "on": "document", "key": ["author", "date"], } ``` # API Reference You can get a token from [here](https://colivara.com) and try out our API. {% openapi src="" path="/v1/health/" method="get" %} {% endopenapi %} {% openapi src="" path="/v1/search/" method="post" %} {% endopenapi %} {% openapi src="" path="/v1/search-image/" method="post" %} {% endopenapi %} {% openapi src="" path="/v1/documents/upsert-document/" method="post" %} {% endopenapi %} {% openapi src="" path="/v1/collections/" method="get" %} {% endopenapi %} {% openapi src="" path="/v1/collections/{collection\_name}/" method="get" %} {% endopenapi %} {% openapi src="" path="/v1/collections/{collection\_name}/" method="patch" %} {% endopenapi %} {% openapi src="" path="/v1/collections/{collection\_name}/" method="delete" %} {% endopenapi %} {% openapi src="" path="/v1/documents/{document\_name}/" method="get" %} {% endopenapi %} {% openapi src="" path="/v1/documents/{document\_name}/" method="patch" %} {% endopenapi %} {% openapi src="" path="/v1/documents/" method="get" %} {% endopenapi %} {% openapi src="" path="/v1/documents/delete-document/{document\_name}/" method="delete" %} {% endopenapi %} {% openapi src="" path="/v1/helpers/file-to-imgbase64/" method="post" %} {% endopenapi %} {% openapi src="" path="/v1/helpers/file-to-base64/" method="post" %} {% endopenapi %} {% openapi src="" path="/v1/embeddings/" method="post" %} {% endopenapi %} # SDKs We have Python and Typescript SDKS that support our API - with convenient methods. ## Using the SDK The SDK can be installed: ```bash pip install colivara_py ``` ```bash npm install colivara-ts ``` Then import into your project: ```python from colivara_py import ColiVara ``` ```typescript import { ColiVara } from 'colivara-ts'; ``` Lastly - initialize the client. ```python client = ColiVara(api_key=os.environ.get("COLIVARA_API_KEY")) ``` ```typescript const client = new ColiVara('your-api-key'); ``` ## Essential Methods
upsert: Adds or updates a document. This operation also supports adding metadata and providing document content through a URL, a base64-encoded string, or a file path. #### **Parameters** * `name` (`str`): The name of the document to be added or updated. This value cannot be null. * `metadata` (`Dict[str, Any]`, optional): Additional metadata for the document, such as tags or descriptive information. * `collection_name` (`str`, optional): The collection to add the document to. Defaults to `"default collection"`. * `document_url` (`str`, optional): The URL of the document if it’s available online. * `document_base64` (`str`, optional): The document content encoded in base64. * `document_path` (`str`, optional): The file path to the document, which will be read and converted to base64. * `wait` (`bool`, optional): If `True`, the method will be synchronous, which mean it will wait for the document processing to complete before returning, making . The default for this value is `False`, making asynchronous the default behavior . #### Returns * `DocumentOut`: An object containing details of the created or updated document. This is returned for synchronous processing * `GenericMessage`: A message object returned if the document is accepted for processing .This is returned for asynchronous processing #### Exceptions * `ValueError`: Raised if no valid document source (URL, base64, or file path) is provided, or if there is an issue with the file path. * `FileNotFoundError`: Raised if the specified file path does not exist. * `PermissionError`: Raised if there is no read permission for the specified file. #### Example Python ```python # This code synchronously adds/updates an "AI Research Paper" document in the "AI_Papers" collection document = client.upsert_document( name="AI_Research_Paper", metadata={ "category": "Machine Learning", "year": "2024", "author": "Dr. AI Researcher" }, collection_name="AI_Papers", document_path="/path/to/AI_Research_Paper.pdf", wait=True ) ``` Javascript/TypeScript {% code lineNumbers="true" %} ```typescript # This code synchronously adds/updates an "AI Research Paper" document in the "AI_Papers" collection const document = await client.upsertDocument({ name: 'AI_Research_Paper', // optional - add metadata metadata={ "category": "Machine Learning", "year": "2024", "author": "Dr. AI Researcher" }, // optional - specify a collection collection_name: 'AI_Papers', // You can use a file path, base64 encoded file, or a URL document_path: '/path/to/AI_Research_Paper.pdf', // optional - wait for the document to index. Webhooks are also supported. wait: true }); ``` {% endcode %}
search: Sends a query to the server. The query can specifies which collection to search within, the number of top results to return, and optional filters to refine the search. It returns the most relevant results based on the given parameters. #### **Parameters** * `query` (`str`): The search query string. This value cannot be null or empty. * `collection_name` (`str`, optional): The name of the collection to search within. Defaults to `"all"`, which searches across all collections. * `top_k` (`int`, optional): Specifies the maximum number of results to return. Defaults to `3`. * `query_filter` (`Dict[str, Any]`, optional): An optional filter to narrow down search results. Read more about[ **Advance Filtering**](/guide/filtering) options here. * `on` (`str`): Specifies whether the filter applies to `"document"` or `"collection"`. * `key` (`str`, `List[str]]`): A single key or a list of keys to match. * `value` (`str`, `int`, `float`, `bool`, `List[str, int, float, bool]`): The value(s) to match for the specified key(s). * `lookup` (`str`): Defines the matching condition. Options include `"key_lookup"`, `"contains"`, `"contained_by"`, `"has_key"`, `"has_keys"`, and `"has_any_keys"`. #### Returns * `QueryOut`: An object that includes the search query and a list of relevant pages based on the specified parameters. #### Exceptions * `ValueError`: Raised if the `query` is empty, the specified `collection_name` does not exist, or the `query_filter` is improperly configured. #### Example Python ```python # searches for pages within the "my_collection" collection that contains # content related to "What's is RAG?" and are categorized under "AI". # returns 5 tops results results = client.search( query="What's is RAG?", collection_name="my_collection", top_k=5, query_filter={ "on": "document", "key": "category", "value": "AI", "lookup": "contains" } ) ``` JavaScript/TypeScript {% code lineNumbers="true" %} ```typescript // searches for pages within the "my_collection" collection that contains // content related to "What's is RAG?" and are categorized under "AI". // returns 5 tops result const results = await client.search({ query: "What's is RAG?", // optional - specify a collection collection_name: 'my_collection', // default is 3 top_k: 5, // optional - add a filter to limit search query_filter:{ "on": "document", "key": "category", "value": "AI", "lookup": "contains" } }); ``` {% endcode %}
filter: Retrieves documents or collections that match specific criteria based This allows you to apply flexible criteria to retrieve specific documents or collections. It supports advanced lookups like filtering by key presence or matching values. This method is useful for narrowing for getting documents or collections without performing a semantic search. For example, give me all the documents that has a metaddata where value is "permissions", and key is "admin". #### **Parameters** * `query_filter` (`Dict[str, Any]`): A dictionary specifying the filter criteria. The dictionary must contain: * `on` (`str`): Specifies the target, either `"document"` or `"collection"`. * `lookup` (`str`): The type of lookup. Options include `"key_lookup"`, `"contains"`, `"contained_by"`, `"has_key"`, `"has_keys"`, and `"has_any_keys"`. Read more about[ **Advance Filtering**](/guide/filtering) options here. * `key` (`str` or `List[str]`): The key(s) to filter by. * `value` (`str, int, float, bool,` optional): The value(s) the key(s) should match (optional, depending on the filter option used). * `expand` (`str`, optional): * Currently, only `"pages"` is supported, which when used to filter documnets will include their pages in the returned value. #### Returns * `List[DocumentOut]`, or `List[CollectionOut]`: A list of objects representing either documents or collections that match the filter criteria. #### Exceptions * `ValueError`: Raised if the filter is invalid. #### Example Python ```python # Filters documents with category containing "AI" and # includes document pages in the output data results = client.filter( query_filter= { "on": "document", "key": "category", "value": "AI", "lookup": "contains" }, expand="pages", ) ``` Typescript/Javascript ```typescript const filterParams= { query_filter: { on: 'document' as 'collection', key: 'topic', value: 'AI', lookup: 'contains' as 'contains' } } const result = await client.filter(filterParams); ```
create_embedding: Generates embeddings on the processed images or text Embeddings are vector representations of the data. This method can generate vectors on either image data, or on a text query. After generation, these vectors comparison is processed to generate query result. #### Parameters * `input_data` (`str`, `List[str]`): A single string or a list of strings representing the data for which embeddings need to be generated. This could be text (for a query) or paths to image files. * `task` (`str,`, optional): Specifies the type of embedding task. * Acceptable values are `"query"` (default) for text queries or `"image"` for images. #### Returns * `EmbeddingsOut`: An object containing the generated embeddings, along with information about the model and usage data. #### Exceptions * `ValueError`: Raised if an invalid task type is provided (i.e., not `"query"` or `"image"`) or if the input data is improperly formatted. #### Example Python ```python # Create embeddings for a text query text_embeddings = client.create_embedding( input_data="What is artificial intelligence?", task="query") # Create embeddings for a list of image paths image_embeddings = client.create_embedding( input_data=["image1.jpg", "image2.jpg"], task="image") ``` Javascript/Typescript ```javascript // Create embeddings for text const embeddings = await client.createEmbedding({ input_data: 'What is artificial intelligence?', task: 'query' }); // Create embeddings for images const imageEmbeddings = await client.createEmbedding({ input_data: ['./path/to/image1.jpg', './path/to/image2.jpg'], task: 'image' }); ```
create_collection:Creates a new collection A collection is a storage within ColiVara server that your documents could be uploaded into for search purposes. A collection is created with a specified name and optional metadata. #### Parameters * `name` (`str`): The name of the collection to be created. This value cannot be null or empty. * `metadata` (`Dict[str, Any]`, optional): A dictionary containing metadata for the new collection. This is optional and can include any relevant key-value pairs. #### Returns * `CollectionOut`: An object representing the newly created collection, including details such as its name and metadata. #### Exceptions * `Exception`with the message `Conflict error`: Raised if there’s a conflict, such as when a collection with the same name already exists. #### Example Python ```python results = client.create_collection( name = 'research', metadata = { topic: 'AI' } ) ``` Javascipt/Typescript ```typescript const collection = await client.createCollection({ name: 'research', metadata: { topic: 'AI' } }); ```
## Collection Manipulation Methods
get_collection: Retrieves a collection #### Parameters * `collection_name` (`str`): The name of the collection to retrieve. This value cannot be null or empty and must match an existing collection. #### Returns * `CollectionOut`: An object representing the retrieved collection, including details such as its name and metadata. #### Exceptions * `Exception`with message `Collection not found` : Raised if the specified collection does not exist. #### Example Python ```python # retrieves the "research" collection collection = client.get_collection(collection_name="research") ``` Javascript/Typescript ```javascript // Get a specific collection const collection = await client.getCollection({ collection_name: 'research' }); ```
list_collections:Retrieves a list of all collections available to the user #### **Parameters** #### Returns * `List[CollectionOut]`: A list of `CollectionOut` objects, each representing a collection with details such as name and metadata. #### Exceptions * `ValueError`: Raised if the server response format is unexpected (e.g., not a list). #### Example Python ```python # Retrieve the list of all collections collections = client.list_collections() ``` Javascript/Typescript ```typescript // List all collections const collections = await client.listCollections(); ```
partial_update_collection: Partially updates a collection's metadata. Only the fields provided in the parameters will be updated. Metadata already exists for the collection but was not provided will not be removed. #### Parameters * `collection_name` (`str`): The name of the collection to update. This value cannot be null. * `name` (`str`, optional): A new name for the collection, if you wish to rename it. * `metadata` (`Dict[str, Any]`, optional): New metadata for the collection. This replaces or adds to the existing metadata. #### Returns * `CollectionOut`: An object representing the updated collection, including details such as its name and metadata. #### Exceptions * `Exception`with message `Collection not found` : Raised if the specified collection does not exist. #### Example Python ```python # updates the "AI_Projects" collection name to "AI_Research_Projects" and adds more metadata updated_collection = client.partial_update_collection( collection_name="AI_Projects", name="AI_Research_Projects", metadata={ "updated_by": "admin", "status": "active"} ) ``` Javascript/Typescript ```typescript const updatedCollection = client.partialUpdateCollection({ collection_name:"AI_Projects", name:"AI_Research_Projects", metadata:{ "updated_by": "admin", "status": "active" } } ) ```
delete_collection:Removes a collection The collection will be deleted from ColiVara server. This action is permanent and final. #### **Parameters** * `collection_name` (`str`): The name of the collection to delete. This must match an existing collection. #### Returns #### Exceptions * `Exception`with message `Collection not found` : Raised if the specified collection does not exist. #### Example Python ```python # Delete the specified collection client.delete_collection(collection_name="obsolete_collection") ``` Javascript/Typescript ```javascript client.deleteCollection({ collection_name: "obsolete_collection" }) ```
## Document Manipulation Methods
get_document: Retrieves a document #### Parameters * `document_name` (`str`): The name of the document to retrieve. * `collection_name` (`str`, optional): The name of the collection containing the document. Defaults to `"default collection"`. * `expand` (`str`, optional): if the value is `"pages"`,the method will include the document’s pages. #### Returns * `DocumentOut`: An object containing details of the retrieved document, including details such as its name and metadata. #### Exceptions * `ValueError` with message `Document not found` : Raised if the document is not found because the document name does not exist #### Example Python ```python # retrieves a document with its pages included document = client.get_document( document_name="research_paper", collection_name="AI_research", expand="pages" ) ``` Javascript/Typescript ```typescript const document = client.getDocument( { document_name: "research_paper, collection_name: "AI_research", expand: 'pages' } ) ```
list_documents:Retrieves a list of all documents in a collection or in all collections available to the user #### **Parameters** * `collection_name` (`str`, optional): The name of the collection to fetch documents from. Defaults to `"default collection"`. * Use `"all"` to fetch documents from all collections. * `expand` (`str`, optional): if the value is `"pages"`,the method will include all the documents’ pages. #### Returns * `List[DocumentOut]`: A list of `DocumentOut` objects, each representing a document with its details. #### Exceptions #### Example Python ```python # Retrieves all documents in the "AI_Research" collection, including pages for each document documents = client.list_documents( collection_name="AI_research", expand="pages") ``` Javascript/Typescript ```typescript const documents = client.listDocuments({ collection_name="AI_Research", expand="pages" } ) ```
partial_update_document: Partially updates a document This method can update either or both the document's content or metadata. Only the fields provided in the parameters will be updated. Metadata already exists for the document but was not provided will not be removed. #### Parameters * `document_name` (`str`): The name of the document to be updated. * `name` (`str`, optional): A new name for the document, if renaming. * `metadata` (`Dict[str, Any]`, optional): Updated metadata for the document. * `collection_name` (`str`, optional): The new collection name if you wish to move the document to a different collection. * `document_url` (`str`, optional): A new URL for the document content, if changing. * `document_base64` (`str`, optional): A new base64-encoded string of the document content, if changing. #### Returns * `DocumentOut`: An object containing details of the retrieved document, including details such as its name and metadata. #### Exceptions * `ValueError` with message `Update failed`: Raised if the document is not found or if there is an issue with the update #### Example Python ```python # Update a document's name from "Research_Paper" to "Updated_Research_Paper" # Also updates its metadata, and URL updated_document = client.partial_update_document( document_name="Research_Paper", name="Updated_Research_Paper", metadata={ "author": "Dr. AI Researcher", "year": "2024" }, document_url="https://example.com/updated_paper.pdf" ) ``` Typescript/Javascript ```typescript const updatedDocument = await client.partialUpdateDocument({ document_name: "Research_Paper", name: "Updated_Research_Paper", metadata: { author: "Dr. AI Researcher", year: "2024"}, document_url: "https://example.com/updated_paper.pdf" }); ```
delete_document:Removes a document The document to be deleted can be identified from a specific collection, or from all collections. The document will be deleted from ColiVara server. This action is permanent and final. #### **Parameters** * `document_name` (`str`): The name of the document to delete. * `collection_name` (`str`, optional): The name of the collection containing the document. * Defaults to `"default collection"`. * Use `"all"` to access documents across all collections belonging to the user. #### Returns #### Exceptions * `ValueError`: Raised if the document does not exist or if there is an issue with the deletion #### Example Python ```python # Deletes the "Old_Report" document from the "Archived_Documents" collection client.delete_document( document_name="Old_Report", collection_name="Archived_Documents" ) ``` Typescript/Javascript {% code overflow="wrap" %} ```typescript client.deleteDocument({ document_name: "Old_Report", collection_name: "Archived_Documents" }); ``` {% endcode %}
## Other Methods
file_to_imgbase64:Convert a file into a list of base64 strings, each represents a page from the document This method is useful to convert a document's content into clean pages of base64 images. For example, you can send a test.pdf with 50 pages, and you will get back 50 images of the same pdf. #### **Parameters** * `file_path` (`str`): The path to the file you want to convert. #### Returns * `List[FileOut]`: A list of `FileOut` objects, each containing a base64-encoded string of an image, and the page number within the document. #### Exceptions * `Exception`: Raised if there’s an error during the file reading or encoding process. #### Example Python ```python # Converts the contents of multi_page_document.pdf to a list of base64 string base64_images = client.file_to_imgbase64("/path/to/multi_page_document.pdf") ``` Typescript/Javascript ```typescript // Converts the contents of multi_page_document.pdf to a list of base64 string const base64Images = await client.fileToImgbase64("/path/to/multi_page_document.pdf"); ```
file_to_base64: Converts file content to a base64-encoded string This method is useful to update a document's content - it converting the file content into a base64 string, and returns that. This is useful if you have documents where you prefer to send us a base64 instead of a url (if the URL is protected via authentication or anti-scrapping measures) or a local path. #### **Parameters** * `file_path` (`str`): The path to the file you want to convert. #### Returns * `str`: A base64-encoded string representing the file's content #### Exceptions * `Exception`: Raised if there’s an error during the file reading or encoding process. #### Example Python ```python # Converts the contents of document.pdf to a base64 string. base64_string = client.file_to_base64("/path/to/document.pdf") ``` Javascript/Typescript ```typescript // Converts the contents of multi_page_document.pdf to a base64 string const base64File = await client.fileToBase64("/path/to/multi_page_document.pdf"); ```
# Webhooks **ColiVara** supports Webhookw for asynchronous document upsertion. If not set up, you will receive an email on document upsertion **failures only**. If setup, you will receive events for success and failure. You can create and verify the Webhook using our provided SDK, or via the User Interface at [**www.colivara.com**](https://docs.colivara.com/guide/www.colivara.com). Webhooks can be useful to manage downstream event handling and knowing when the documents are ready to query. They are especially useful when upserting documents more than 100 pages in length or when waiting for upsertion is not efficient. ## Setting up a webhook via the User Interface (Recommended) {% stepper %} {% step %} ### Visit your Account Dashboard Via {% endstep %} {% step %} ### Enter your webhook URL Make sure that your URL enpoint is *Publicly Available*
{% endstep %} {% step %} ### Store your Webhook Secret
{% endstep %} {% endstepper %} ## Setting up a web hook via the SDK (Alternative) The Python SDK library provides the method to register a webhook URL **`add_webhook`** **Parameters:** * `url(str)`: The URL of the webhook endpoint to register. This must be a valid, publicly accessible URL. **Returns:** * `WebhookOut`: An object containing details about the registered webhook, including the webhook ID, associated app ID, and *webhook secret*. **Exceptions:** * `ValueError`: Raised if the provided URL is invalid or the request is rejected due to a bad request (e.g., missing or incorrect parameters). * `requests.HTTPError`: Raised for other HTTP errors, such as server-side issues. **Example:** ```python # Register a webhook to handle document upsertion events webhook = client.add_webhook( url="https://your-domain.com/webhook-handler" ) ``` ## Verifying Webhook from inside your application Now that ColiVara have noted your app URL to send notification to, you will start receiving HTTP request containing information regarding your document process status. However, prior to ingesting the data from these HTTP requests, you *should* validate the webhook to verify that it truly comes from ColiVara. Afterall, any malicious actor can send an HTTP request to a publicly available URL. Your webhook secret is a unique key used to verify that incoming webhook requests are authentic. This protects your application from unauthorized or fake notifications. {% hint style="warning" %} Keep your webhook secret safe and never share it publicly. If exposed, attackers could forge requests and compromise your system. {% endhint %} The SDK provides a method that you can use to validate the webhook from inside your Python application **`validate_webhook`:** **Parameters:** * `webhook_secret (str)`: The secret key provided when the webhook was registered. Used for validation. * `payload (str)`: The raw body of the incoming webhook request. * `headers (Dict[str, Any])`: The headers from the incoming webhook request, which include the signature used for validation. **Returns:** * `bool`: Returns `True` if the webhook is valid; otherwise, returns `False`. **Example:** ```python # Validate an incoming webhook request is_valid = client.validate_webhook( webhook_secret="your_webhook_secret", payload=raw_payload, headers=request_headers ) if is_valid: print("Webhook is valid!") else: print("Invalid webhook!") ``` # Integrations {% hint style="info" %} We are adding more integrations regularly. If you are looking for a specific integration, you can request one via our [Github issues](https://github.com/tjmlabs/ColiVara/issues). {% endhint %} In this section, we’ll guide you through integrating ColiVara with LLM frameworks. This integration will allow you to seamlessly connect to other moving parts in your RAG project. Currently, we are supporting integrating with * [**LlamaIndex**](https://www.llamaindex.ai) * [**LangChain**](https://www.langchain.com/) * [**OpenRouter**](https://openrouter.ai/) *** ## Overview 1. [Install dependencies](#id-1.-install-dependencies) 2. [Set up **ColiVara** RAG Client and Process your documents](#id-2.-set-up-colivara-rag-client-and-process-your-documents) 3. [Create the Context](#id-3.-create-the-context) 4. [Create a Client with your integrated LLM Framework provider of choice](#create-a-client-with-your-integrated-llm-framework) 5. [Run your RAG query](#id-5.-run-your-rag-query-and-generate-the-answer) *** ## 1. Install Dependencies
LlamaIndex ```bash pip install colivara-py -q -U pip install llama-index-llms-openai -q pip install llama-index-multi-modal-llms-openai -q -U pip install llama-index-core -q -U pip install --force-reinstall llama-index==0.11.22 ```
Langchain ```bash pip install colivara-py --q pip install langchain --q pip install langchain-core --q pip install langchain-openai --q ```
OpenRouter ```bash pip install colivara-py -q -U ```
*** ## 2. Set up **ColiVara** RAG client and Process your documents {% stepper %} {% step %} ### Initialize your Colivara RAG client ```python import base64 # for converting docs binaries to base64 from pathlib import Path import os import requests from colivara_py import ColiVara rag_client = ColiVara( base_url="https://api.colivara.com", api_key=YOUR_COLIVARA_KEY ) ``` {% endstep %} {% step %} ### (Optional) Download your documents You can skip this step if your documents are already downloaded into a folder. Here we are downloading documents using their URLs, and save them in the `docs/`folder ```python # List of filenames and their URLS files = [ { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/Work-From-Home%20Guidance.pdf", "filename": "docs/Work-From-Home-Guidance.pdf", }, { "url": "https://github.com/tjmlabs/colivara-demo/raw/main/docs/StaffVendorPolicy-Jan2019.pdf", "filename": "docs/StaffVendorPolicy-Jan2019.pdf", }, ] # Downloading the files into the docs/ folder def download_file(url, local_filename): response = requests.get(url) if response.status_code == 200: os.makedirs("docs", exist_ok=True) with open(local_filename, "wb") as f: f.write(response.content) print(f"Successfully downloaded: {local_filename}") else: print(f"Failed to download: {url}") for file in files: download_file(file["url"], file["filename"]) ``` {% endstep %} {% step %} ### Upload and Process the documents ```python def sync_documents(): documents_dir = Path("docs") files = list(documents_dir.glob("**/*")) for file in files: with open(file, "rb") as f: file_content = f.read() encoded_content = base64.b64encode(file_content).decode("utf-8") rag_client.upsert_document( name=file.name, document_base64=encoded_content, collection_name="openrouter-demo", wait=True, ) print(f"Upserted: {file.name}") sync_documents() ``` {% endstep %} {% endstepper %} *** ## 3. Create the Context {% hint style="info" %} For simplicity, we are skipping *Query transformation* step. In RAG, query transformation converts user queries into the format the model needs. Instead, we are using the query directly. {% endhint %} ```python def get_context(query): results = rag_client.search(query=query, collection_name="openrouter-demo", top_k=3) results = results.results context = [] for result in results: base64 = result.img_base64 if "data:image" not in base64: base64 = f"data:image/png;base64,{base64}" context.append(base64) return context query = "What is the work from home policy?" context = get_context(query) ``` *** ## 4. Create a Client with your integrated LLM Framework
Llamaindex ```python from llama_index.multi_modal_llms.openai import OpenAIMultiModal from llama_index.core.schema import ImageNode openai_api_key = YOUR_OPENAI_API_KEY gpt_4o = OpenAIMultiModal(model="gpt-4o", max_new_tokens=500, api_key=openai_api_key, temperature=0) ```
Langchain ```python from langchain_core.prompts import ChatPromptTemplate from langchain_openai import ChatOpenAI model = ChatOpenAI(model="gpt-4o", temperature=0) ```
OpenRouter No need to create a Client as we will be calling OpenRouter via an HTTP request
*** ## 5. Run your RAG query and Generate the answer
LlamaIndex ```python complete_response = gpt_4o.complete(query, context) print(complete_response) ```
Langchain ```python prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a helpful assistant that answers questions based on the provided images/docs.", ), ( "user", [ { "type": "text", "text": "Here are some images for context:", }, *[ { "type": "image_url", "image_url": {"url": image_data["base64"]}, } for image_data in context ], {"type": "text", "text": "Now, please answer the following question:"}, { "type": "text", "text": "{query}", }, ], ), ] ) chain = prompt | model response = chain.invoke({"query": query}) print(response.content) ```
OpenRouter ```python url = "https://openrouter.ai/api/v1/chat/completions" headers = { "Authorization": f"Bearer {os.environ['OPEN_ROUTER_API_KEY']}", "Content-Type": "application/json", } payload = { "model": "openai/gpt-4o", "messages": [ { "role": "system", "content": "You are a helpful assistant that answers questions based on the provided images/docs.", }, { "role": "user", "content": [ { "type": "text", "text": "Here are some images for context:", }, *[ { "type": "image_url", "image_url": { "url": path, }, } for path in context ], {"type": "text", "text": query}, ], }, ], "max_tokens": 500, "temperature": 0.0, } response = requests.post(url, headers=headers, json=payload) print(response.json()["choices"][0]["message"]["content"]) ```
# Advanced usage ### Overview The `use_proxy` parameter in the Colivara API is a boolean flag that allows users to route their requests through a proxy. This can be useful when accessing certain sites that may block direct requests. However, enabling this feature incurs an additional cost of **10 extra credits per request**. ### Default Behavior By default, `use_proxy` is set to `False`, meaning requests are sent directly without using a proxy. ### Usage To enable proxy usage, simply set `use_proxy` to `True` in your request payload. Below is an example of how to use the parameter in a request: ```python import requests response = requests.post( "https://api.colivara.com/v1/documents/upsert-document/", headers={"Authorization":"Bearer ","Content-Type":"application/json"}, json={"name":"text", "metadata":{}, "collection_name":"default_collection", "url": "string", "wait": False, "use_proxy": True } ) data = response.json() ``` ### Considerations * **Additional Cost**: Enabling `use_proxy` incurs an additional charge of **10 extra credits per request**. * **When to Use**: Use this option when dealing with sites that restrict direct access. Proxies are helpful for accessing some sites that may block us when not using a proxy.