feat: Qdrant Vector Search Support (#684)

* feat: Qdrant vector search support

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* chore: Review updates

Signed-off-by: Anush008 <anushshetty90@gmail.com>

---------

Signed-off-by: Anush008 <anushshetty90@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
Anush
2025-11-11 17:05:00 +05:30
committed by GitHub
parent 70dbd21bdf
commit aa027faf95
13 changed files with 1010 additions and 30 deletions

View File

@@ -78,6 +78,17 @@ TAVILY_API_KEY=tvly-xxx
# MILVUS_EMBEDDING_API_KEY=
# MILVUS_AUTO_LOAD_EXAMPLES=true
# RAG_PROVIDER: qdrant (using qdrant cloud or self-hosted: https://qdrant.tech/documentation/quick-start/)
# RAG_PROVIDER=qdrant
# QDRANT_LOCATION=https://xyz-example.eu-central.aws.cloud.qdrant.io:6333
# QDRANT_API_KEY=<your_qdrant_api_key> # Optional, only for cloud/authenticated instances
# QDRANT_COLLECTION=documents
# QDRANT_EMBEDDING_PROVIDER=openai # support openai,dashscope
# QDRANT_EMBEDDING_BASE_URL=
# QDRANT_EMBEDDING_MODEL=text-embedding-ada-002
# QDRANT_EMBEDDING_API_KEY=
# QDRANT_AUTO_LOAD_EXAMPLES=true
# Optional, volcengine TTS for generating podcast
VOLCENGINE_TTS_APPID=xxx
VOLCENGINE_TTS_ACCESS_TOKEN=xxx

View File

@@ -183,10 +183,10 @@ SEARCH_API=tavily
### Private Knowledgebase
DeerFlow support private knowledgebase such as ragflow and vikingdb, so that you can use your private documents to answer questions.
DeerFlow supports private knowledgebase such as RAGFlow, Qdrant, Milvus, and VikingDB, so that you can use your private documents to answer questions.
- **[RAGFlow](https://ragflow.io/docs/dev/)**open source RAG engine
```
- **[RAGFlow](https://ragflow.io/docs/dev/)**: open source RAG engine
```bash
# examples in .env.example
RAG_PROVIDER=ragflow
RAGFLOW_API_URL="http://localhost:9388"
@@ -195,6 +195,19 @@ DeerFlow support private knowledgebase such as ragflow and vikingdb, so that you
RAGFLOW_CROSS_LANGUAGES=English,Chinese,Spanish,French,German,Japanese,Korean
```
- **[Qdrant](https://qdrant.tech/)**: open source vector database
```bash
# Using Qdrant Cloud or self-hosted
RAG_PROVIDER=qdrant
QDRANT_LOCATION=https://xyz-example.eu-central.aws.cloud.qdrant.io:6333
QDRANT_API_KEY=your_qdrant_api_key
QDRANT_COLLECTION=documents
QDRANT_EMBEDDING_PROVIDER=openai
QDRANT_EMBEDDING_MODEL=text-embedding-ada-002
QDRANT_EMBEDDING_API_KEY=your_openai_api_key
QDRANT_AUTO_LOAD_EXAMPLES=true
```
## Features
### Core Capabilities
@@ -215,7 +228,9 @@ DeerFlow support private knowledgebase such as ragflow and vikingdb, so that you
- 📃 **RAG Integration**
- Supports mentioning files from [RAGFlow](https://github.com/infiniflow/ragflow) within the input box. [Start up RAGFlow server](https://ragflow.io/docs/dev/).
- Supports multiple vector databases: [Qdrant](https://qdrant.tech/), [Milvus](https://milvus.io/), [RAGFlow](https://github.com/infiniflow/ragflow), VikingDB, MOI, and Dify
- Supports mentioning files from RAG providers within the input box
- Easy switching between different vector databases through configuration
- 🔗 **MCP Seamless Integration**
- Expand capabilities for private domain access, knowledge graph, web browsing and more

View File

@@ -263,6 +263,26 @@ DeerFlow supports multiple RAG providers for document retrieval. Configure the R
- **RAGFlow**: Document retrieval using RAGFlow API
- **VikingDB Knowledge Base**: ByteDance's VikingDB knowledge base service
- **Milvus**: Open-source vector database for similarity search
- **Qdrant**: Open-source vector search engine with cloud and self-hosted options
- **MOI**: Hybrid database for enterprise users
- **Dify**: AI application platform with RAG capabilities
### Qdrant Configuration
To use Qdrant as your RAG provider, set the following environment variables:
```bash
# RAG_PROVIDER: qdrant (using Qdrant Cloud or self-hosted)
RAG_PROVIDER=qdrant
QDRANT_LOCATION=https://xyz-example.eu-central.aws.cloud.qdrant.io:6333
QDRANT_API_KEY=<your_qdrant_api_key>
QDRANT_COLLECTION=documents
QDRANT_EMBEDDING_PROVIDER=openai # support openai, dashscope
QDRANT_EMBEDDING_BASE_URL=
QDRANT_EMBEDDING_MODEL=text-embedding-ada-002
QDRANT_EMBEDDING_API_KEY=<your_embedding_api_key>
QDRANT_AUTO_LOAD_EXAMPLES=true # automatically load example markdown files
```
### Milvus Configuration

View File

@@ -41,6 +41,8 @@ dependencies = [
"pymilvus>=2.3.0",
"langchain-milvus>=0.2.1",
"psycopg[binary]>=3.2.9",
"qdrant-client>=1.15.1",
"langchain-qdrant>=0.2.0,<1.0.0",
]
[project.optional-dependencies]

View File

@@ -28,6 +28,7 @@ class RAGProvider(enum.Enum):
VIKINGDB_KNOWLEDGE_BASE = "vikingdb_knowledge_base"
MOI = "moi"
MILVUS = "milvus"
QDRANT = "qdrant"
SELECTED_RAG_PROVIDER = os.getenv("RAG_PROVIDER")

View File

@@ -3,7 +3,9 @@
from .builder import build_retriever
from .dify import DifyProvider
from .milvus import MilvusProvider
from .moi import MOIProvider
from .qdrant import QdrantProvider
from .ragflow import RAGFlowProvider
from .retriever import Chunk, Document, Resource, Retriever
from .vikingdb_knowledge_base import VikingDBKnowledgeBaseProvider
@@ -15,6 +17,8 @@ __all__ = [
DifyProvider,
RAGFlowProvider,
MOIProvider,
MilvusProvider,
QdrantProvider,
VikingDBKnowledgeBaseProvider,
Chunk,
build_retriever,

View File

@@ -5,6 +5,7 @@ from src.config.tools import SELECTED_RAG_PROVIDER, RAGProvider
from src.rag.dify import DifyProvider
from src.rag.milvus import MilvusProvider
from src.rag.moi import MOIProvider
from src.rag.qdrant import QdrantProvider
from src.rag.ragflow import RAGFlowProvider
from src.rag.retriever import Retriever
from src.rag.vikingdb_knowledge_base import VikingDBKnowledgeBaseProvider
@@ -21,6 +22,8 @@ def build_retriever() -> Retriever | None:
return VikingDBKnowledgeBaseProvider()
elif SELECTED_RAG_PROVIDER == RAGProvider.MILVUS.value:
return MilvusProvider()
elif SELECTED_RAG_PROVIDER == RAGProvider.QDRANT.value:
return QdrantProvider()
elif SELECTED_RAG_PROVIDER:
raise ValueError(f"Unsupported RAG provider: {SELECTED_RAG_PROVIDER}")
return None

View File

@@ -120,7 +120,7 @@ class MilvusRetriever(Retriever):
else:
raise ValueError(
f"Unsupported embedding provider: {self.embedding_provider}. "
"Supported providers: openai,dashscope"
"Supported providers: openai, dashscope"
)
def _get_embedding_dimension(self, model_name: str) -> int:

505
src/rag/qdrant.py Normal file
View File

@@ -0,0 +1,505 @@
# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
# SPDX-License-Identifier: MIT
import hashlib
import logging
import uuid
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Set
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client import grpc
from qdrant_client.models import (
Distance,
FieldCondition,
Filter,
MatchValue,
PointStruct,
VectorParams,
)
from src.config.loader import get_bool_env, get_int_env, get_str_env
from src.rag.retriever import Chunk, Document, Resource, Retriever
logger = logging.getLogger(__name__)
SCROLL_SIZE = 64
class DashscopeEmbeddings:
def __init__(self, **kwargs: Any) -> None:
self._client: OpenAI = OpenAI(
api_key=kwargs.get("api_key", ""), base_url=kwargs.get("base_url", "")
)
self._model: str = kwargs.get("model", "")
self._encoding_format: str = kwargs.get("encoding_format", "float")
def _embed(self, texts: Sequence[str]) -> List[List[float]]:
clean_texts = [t if isinstance(t, str) else str(t) for t in texts]
if not clean_texts:
return []
resp = self._client.embeddings.create(
model=self._model,
input=clean_texts,
encoding_format=self._encoding_format,
)
return [d.embedding for d in resp.data]
def embed_query(self, text: str) -> List[float]:
embeddings = self._embed([text])
return embeddings[0] if embeddings else []
def embed_documents(self, texts: List[str]) -> List[List[float]]:
return self._embed(texts)
class QdrantProvider(Retriever):
def __init__(self) -> None:
self.location: str = get_str_env("QDRANT_LOCATION", ":memory:")
self.api_key: str = get_str_env("QDRANT_API_KEY", "")
self.collection_name: str = get_str_env("QDRANT_COLLECTION", "documents")
top_k_raw = get_str_env("QDRANT_TOP_K", "10")
self.top_k: int = int(top_k_raw) if top_k_raw.isdigit() else 10
self.embedding_model_name = get_str_env("QDRANT_EMBEDDING_MODEL")
self.embedding_api_key = get_str_env("QDRANT_EMBEDDING_API_KEY")
self.embedding_base_url = get_str_env("QDRANT_EMBEDDING_BASE_URL")
self.embedding_dim: int = self._get_embedding_dimension(
self.embedding_model_name
)
self.embedding_provider = get_str_env("QDRANT_EMBEDDING_PROVIDER", "openai")
self.auto_load_examples: bool = get_bool_env("QDRANT_AUTO_LOAD_EXAMPLES", True)
self.examples_dir: str = get_str_env("QDRANT_EXAMPLES_DIR", "examples")
self.chunk_size: int = get_int_env("QDRANT_CHUNK_SIZE", 4000)
self._init_embedding_model()
self.client: Any = None
self.vector_store: Any = None
def _init_embedding_model(self) -> None:
kwargs = {
"api_key": self.embedding_api_key,
"model": self.embedding_model_name,
"base_url": self.embedding_base_url,
"encoding_format": "float",
"dimensions": self.embedding_dim,
}
if self.embedding_provider.lower() == "openai":
self.embedding_model = OpenAIEmbeddings(**kwargs)
elif self.embedding_provider.lower() == "dashscope":
self.embedding_model = DashscopeEmbeddings(**kwargs)
else:
raise ValueError(
f"Unsupported embedding provider: {self.embedding_provider}. "
"Supported providers: openai, dashscope"
)
def _get_embedding_dimension(self, model_name: str) -> int:
embedding_dims = {
"text-embedding-ada-002": 1536,
"text-embedding-v4": 2048,
}
explicit_dim = get_int_env("QDRANT_EMBEDDING_DIM", 0)
if explicit_dim > 0:
return explicit_dim
return embedding_dims.get(model_name, 1536)
def _ensure_collection_exists(self) -> None:
if not self.client.collection_exists(self.collection_name):
self.client.create_collection(
collection_name=self.collection_name,
vectors_config=VectorParams(
size=self.embedding_dim, distance=Distance.COSINE
),
)
logger.info("Created Qdrant collection: %s", self.collection_name)
def _load_example_files(self) -> None:
current_file = Path(__file__)
project_root = current_file.parent.parent.parent
examples_path = project_root / self.examples_dir
if not examples_path.exists():
logger.info("Examples directory not found: %s", examples_path)
return
logger.info("Loading example files from: %s", examples_path)
md_files = list(examples_path.glob("*.md"))
if not md_files:
logger.info("No markdown files found in examples directory")
return
existing_docs = self._get_existing_document_ids()
loaded_count = 0
for md_file in md_files:
doc_id = self._generate_doc_id(md_file)
if doc_id in existing_docs:
continue
try:
content = md_file.read_text(encoding="utf-8")
title = self._extract_title_from_markdown(content, md_file.name)
chunks = self._split_content(content)
for i, chunk in enumerate(chunks):
chunk_id = f"{doc_id}_chunk_{i}" if len(chunks) > 1 else doc_id
self._insert_document_chunk(
doc_id=chunk_id,
content=chunk,
title=title,
url=f"qdrant://{self.collection_name}/{md_file.name}",
metadata={"source": "examples", "file": md_file.name},
)
loaded_count += 1
logger.debug("Loaded example markdown: %s", md_file.name)
except Exception as e:
logger.warning("Error loading %s: %s", md_file.name, e)
logger.info("Successfully loaded %d example files into Qdrant", loaded_count)
def _generate_doc_id(self, file_path: Path) -> str:
file_stat = file_path.stat()
content_hash = hashlib.md5(
f"{file_path.name}_{file_stat.st_size}_{file_stat.st_mtime}".encode()
).hexdigest()[:8]
return f"example_{file_path.stem}_{content_hash}"
def _extract_title_from_markdown(self, content: str, filename: str) -> str:
lines = content.split("\n")
for line in lines:
line = line.strip()
if line.startswith("# "):
return line[2:].strip()
return filename.replace(".md", "").replace("_", " ").title()
def _split_content(self, content: str) -> List[str]:
if len(content) <= self.chunk_size:
return [content]
chunks = []
paragraphs = content.split("\n\n")
current_chunk = ""
for paragraph in paragraphs:
if len(current_chunk) + len(paragraph) <= self.chunk_size:
current_chunk += paragraph + "\n\n"
else:
if current_chunk:
chunks.append(current_chunk.strip())
current_chunk = paragraph + "\n\n"
if current_chunk:
chunks.append(current_chunk.strip())
return chunks
def _string_to_uuid(self, text: str) -> str:
namespace = uuid.NAMESPACE_DNS
return str(uuid.uuid5(namespace, text))
def _scroll_all_points(
self,
scroll_filter: Optional[Filter] = None,
with_payload: bool = True,
with_vectors: bool = False,
) -> List[Any]:
results = []
next_offset = None
stop_scrolling = False
while not stop_scrolling:
points, next_offset = self.client.scroll(
collection_name=self.collection_name,
scroll_filter=scroll_filter,
limit=SCROLL_SIZE,
offset=next_offset,
with_payload=with_payload,
with_vectors=with_vectors,
)
stop_scrolling = next_offset is None or (
isinstance(next_offset, grpc.PointId)
and getattr(next_offset, "num", 0) == 0
and getattr(next_offset, "uuid", "") == ""
)
results.extend(points)
return results
def _get_existing_document_ids(self) -> Set[str]:
try:
points = self._scroll_all_points(with_payload=True, with_vectors=False)
return {
point.payload.get("doc_id", str(point.id))
for point in points
if point.payload
}
except Exception:
return set()
def _insert_document_chunk(
self, doc_id: str, content: str, title: str, url: str, metadata: Dict[str, Any]
) -> None:
embedding = self._get_embedding(content)
payload = {
"doc_id": doc_id,
"content": content,
"title": title,
"url": url,
**metadata,
}
point_id = self._string_to_uuid(doc_id)
point = PointStruct(id=point_id, vector=embedding, payload=payload)
self.client.upsert(
collection_name=self.collection_name, points=[point], wait=True
)
def _connect(self) -> None:
client_kwargs = {"location": self.location}
if self.api_key:
client_kwargs["api_key"] = self.api_key
self.client = QdrantClient(**client_kwargs)
self._ensure_collection_exists()
try:
self.vector_store = QdrantVectorStore(
client=self.client,
collection_name=self.collection_name,
embedding=self.embedding_model,
)
except Exception:
self.vector_store = None
def _get_embedding(self, text: str) -> List[float]:
return self.embedding_model.embed_query(text=text.strip())
def list_resources(self, query: Optional[str] = None) -> List[Resource]:
resources: List[Resource] = []
if not self.client:
try:
self._connect()
except Exception:
return self._list_local_markdown_resources()
try:
if query and self.vector_store:
docs = self.vector_store.similarity_search(
query, k=100, filter={"source": "examples"}
)
for d in docs:
meta = d.metadata or {}
uri = meta.get("url", "") or f"qdrant://{meta.get('id', '')}"
if any(r.uri == uri for r in resources):
continue
resources.append(
Resource(
uri=uri,
title=meta.get("title", "") or meta.get("id", "Unnamed"),
description="Stored Qdrant document",
)
)
else:
all_points = self._scroll_all_points(
scroll_filter=Filter(
must=[
FieldCondition(
key="source", match=MatchValue(value="examples")
)
]
),
with_payload=True,
with_vectors=False,
)
for point in all_points:
payload = point.payload or {}
doc_id = payload.get("doc_id", str(point.id))
uri = payload.get("url", "") or f"qdrant://{doc_id}"
resources.append(
Resource(
uri=uri,
title=payload.get("title", "") or doc_id,
description="Stored Qdrant document",
)
)
logger.info(
"Successfully listed %d resources from Qdrant collection: %s",
len(resources),
self.collection_name,
)
except Exception:
logger.warning(
"Failed to query Qdrant for resources, falling back to local examples."
)
return self._list_local_markdown_resources()
return resources
def _list_local_markdown_resources(self) -> List[Resource]:
current_file = Path(__file__)
project_root = current_file.parent.parent.parent
examples_path = project_root / self.examples_dir
if not examples_path.exists():
return []
md_files = list(examples_path.glob("*.md"))
resources: list[Resource] = []
for md_file in md_files:
try:
content = md_file.read_text(encoding="utf-8", errors="ignore")
title = self._extract_title_from_markdown(content, md_file.name)
uri = f"qdrant://{self.collection_name}/{md_file.name}"
resources.append(
Resource(
uri=uri,
title=title,
description="Local markdown example (not yet ingested)",
)
)
except Exception:
continue
return resources
def query_relevant_documents(
self, query: str, resources: Optional[List[Resource]] = None
) -> List[Document]:
resources = resources or []
if not self.client:
self._connect()
query_embedding = self._get_embedding(query)
search_results = self.client.query_points(
collection_name=self.collection_name,
query=query_embedding,
limit=self.top_k,
with_payload=True,
).points
documents = {}
for result in search_results:
payload = result.payload or {}
doc_id = payload.get("doc_id", str(result.id))
content = payload.get("content", "")
title = payload.get("title", "")
url = payload.get("url", "")
score = result.score
if resources:
doc_in_resources = False
for resource in resources:
if (url and url in resource.uri) or doc_id in resource.uri:
doc_in_resources = True
break
if not doc_in_resources:
continue
if doc_id not in documents:
documents[doc_id] = Document(id=doc_id, url=url, title=title, chunks=[])
chunk = Chunk(content=content, similarity=score)
documents[doc_id].chunks.append(chunk)
return list(documents.values())
def create_collection(self) -> None:
if not self.client:
self._connect()
else:
self._ensure_collection_exists()
def load_examples(self, force_reload: bool = False) -> None:
if not self.client:
self._connect()
if force_reload:
self._clear_example_documents()
self._load_example_files()
def _clear_example_documents(self) -> None:
try:
all_points = self._scroll_all_points(
scroll_filter=Filter(
must=[
FieldCondition(key="source", match=MatchValue(value="examples"))
]
),
with_payload=False,
with_vectors=False,
)
if all_points:
point_ids = [str(point.id) for point in all_points]
self.client.delete(
collection_name=self.collection_name, points_selector=point_ids
)
logger.info("Cleared %d existing example documents", len(point_ids))
except Exception as e:
logger.warning("Could not clear existing examples: %s", e)
def get_loaded_examples(self) -> List[Dict[str, str]]:
if not self.client:
self._connect()
all_points = self._scroll_all_points(
scroll_filter=Filter(
must=[FieldCondition(key="source", match=MatchValue(value="examples"))]
),
with_payload=True,
with_vectors=False,
)
examples = []
for point in all_points:
payload = point.payload or {}
examples.append(
{
"id": payload.get("doc_id", str(point.id)),
"title": payload.get("title", ""),
"file": payload.get("file", ""),
"url": payload.get("url", ""),
}
)
return examples
def close(self) -> None:
if hasattr(self, "client") and self.client:
try:
if hasattr(self.client, "close"):
self.client.close()
self.client = None
self.vector_store = None
except Exception as e:
logger.warning("Exception occurred while closing QdrantProvider: %s", e)
def __del__(self) -> None:
self.close()
def load_examples() -> None:
auto_load_examples = get_bool_env("QDRANT_AUTO_LOAD_EXAMPLES", False)
rag_provider = get_str_env("RAG_PROVIDER", "")
if rag_provider == "qdrant" and auto_load_examples:
provider = QdrantProvider()
provider.load_examples()

View File

@@ -35,7 +35,8 @@ from src.ppt.graph.builder import build_graph as build_ppt_graph
from src.prompt_enhancer.graph.builder import build_graph as build_prompt_enhancer_graph
from src.prose.graph.builder import build_graph as build_prose_graph
from src.rag.builder import build_retriever
from src.rag.milvus import load_examples
from src.rag.milvus import load_examples as load_milvus_examples
from src.rag.qdrant import load_examples as load_qdrant_examples
from src.rag.retriever import Resource
from src.server.chat_request import (
ChatRequest,
@@ -93,9 +94,9 @@ app.add_middleware(
allow_methods=["GET", "POST", "OPTIONS"], # Use the configured list of methods
allow_headers=["*"], # Now allow all headers, but can be restricted further
)
# Load examples into Milvus if configured
load_examples()
# Load examples into RAG providers if configured
load_milvus_examples()
load_qdrant_examples()
in_memory_store = InMemoryStore()
graph = build_graph_with_memory()

View File

@@ -12,6 +12,7 @@ Tests that the duplicate locale assignment issue is resolved:
"""
import pytest
from src.graph.nodes import preserve_state_meta_fields
from src.graph.types import State
from src.prompts.planner_model import Plan

View File

@@ -0,0 +1,333 @@
# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
# SPDX-License-Identifier: MIT
from __future__ import annotations
import shutil
from pathlib import Path
from uuid import uuid4
import pytest
import src.rag.qdrant as qdrant_mod
from src.rag.qdrant import QdrantProvider
class DummyEmbedding:
def __init__(self, **kwargs):
self.kwargs = kwargs
def embed_query(self, text: str):
return [0.1] * 1536
def embed_documents(self, texts):
return [[0.1] * 1536 for _ in texts]
@pytest.fixture(autouse=True)
def patch_embeddings(monkeypatch):
monkeypatch.setenv("QDRANT_EMBEDDING_PROVIDER", "openai")
monkeypatch.setenv("QDRANT_EMBEDDING_MODEL", "text-embedding-ada-002")
monkeypatch.setenv("QDRANT_COLLECTION", "documents")
monkeypatch.setenv("QDRANT_LOCATION", ":memory:")
monkeypatch.setattr(qdrant_mod, "OpenAIEmbeddings", DummyEmbedding)
monkeypatch.setattr(qdrant_mod, "DashscopeEmbeddings", DummyEmbedding)
yield
@pytest.fixture
def project_root():
return Path(qdrant_mod.__file__).parent.parent.parent
@pytest.fixture
def temp_examples_dir(project_root):
temp_dir_name = f"examples_test_{uuid4().hex}"
temp_dir_path = project_root / temp_dir_name
temp_dir_path.mkdir(parents=True, exist_ok=True)
yield temp_dir_path
if temp_dir_path.exists():
shutil.rmtree(temp_dir_path)
@pytest.fixture
def temp_error_examples_dir(project_root):
temp_dir_name = f"examples_error_{uuid4().hex}"
temp_dir_path = project_root / temp_dir_name
temp_dir_path.mkdir(parents=True, exist_ok=True)
yield temp_dir_path
if temp_dir_path.exists():
shutil.rmtree(temp_dir_path)
@pytest.fixture
def temp_load_skip_examples_dir(project_root):
temp_dir_name = f"examples_load_skip_{uuid4().hex}"
temp_dir_path = project_root / temp_dir_name
temp_dir_path.mkdir(parents=True, exist_ok=True)
yield temp_dir_path
if temp_dir_path.exists():
shutil.rmtree(temp_dir_path)
def test_init_openai_provider(monkeypatch):
monkeypatch.setenv("QDRANT_EMBEDDING_PROVIDER", "openai")
provider = QdrantProvider()
assert provider.embedding_provider == "openai"
assert isinstance(provider.embedding_model, DummyEmbedding)
def test_init_dashscope_provider(monkeypatch):
monkeypatch.setenv("QDRANT_EMBEDDING_PROVIDER", "dashscope")
provider = QdrantProvider()
assert provider.embedding_provider == "dashscope"
assert isinstance(provider.embedding_model, DummyEmbedding)
def test_init_invalid_provider(monkeypatch):
monkeypatch.setenv("QDRANT_EMBEDDING_PROVIDER", "invalid_provider")
with pytest.raises(ValueError, match="Unsupported embedding provider"):
QdrantProvider()
def test_get_embedding_dimension_explicit(monkeypatch):
monkeypatch.setenv("QDRANT_EMBEDDING_DIM", "2048")
provider = QdrantProvider()
assert provider.embedding_dim == 2048
def test_get_embedding_dimension_default(monkeypatch):
monkeypatch.delenv("QDRANT_EMBEDDING_DIM", raising=False)
monkeypatch.setenv("QDRANT_EMBEDDING_MODEL", "text-embedding-ada-002")
provider = QdrantProvider()
assert provider.embedding_dim == 1536
def test_get_embedding_dimension_unknown_model(monkeypatch):
monkeypatch.delenv("QDRANT_EMBEDDING_DIM", raising=False)
monkeypatch.setenv("QDRANT_EMBEDDING_MODEL", "unknown-model")
provider = QdrantProvider()
assert provider.embedding_dim == 1536
def test_connect_memory_mode(monkeypatch):
monkeypatch.setenv("QDRANT_LOCATION", ":memory:")
provider = QdrantProvider()
provider._connect()
assert provider.client is not None
def test_create_collection(monkeypatch):
provider = QdrantProvider()
provider.create_collection()
assert provider.client is not None
def test_extract_title_from_markdown():
provider = QdrantProvider()
content = "# Test Title\n\nSome content"
title = provider._extract_title_from_markdown(content, "test.md")
assert title == "Test Title"
def test_extract_title_fallback():
provider = QdrantProvider()
content = "No title here"
title = provider._extract_title_from_markdown(content, "test_file.md")
assert title == "Test File"
def test_split_content_short():
provider = QdrantProvider()
content = "Short content"
chunks = provider._split_content(content)
assert len(chunks) == 1
assert chunks[0] == content
def test_split_content_long(monkeypatch):
monkeypatch.setenv("QDRANT_CHUNK_SIZE", "20")
provider = QdrantProvider()
content = "Paragraph one here\n\nParagraph two here\n\nParagraph three here\n\nParagraph four here"
chunks = provider._split_content(content)
assert len(chunks) > 1
def test_string_to_uuid():
provider = QdrantProvider()
uuid1 = provider._string_to_uuid("test")
uuid2 = provider._string_to_uuid("test")
assert uuid1 == uuid2
def test_get_embedding():
provider = QdrantProvider()
embedding = provider._get_embedding("test text")
assert len(embedding) == 1536
assert all(isinstance(x, float) for x in embedding)
def test_load_examples_no_directory(monkeypatch, project_root):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", "nonexistent_dir")
provider = QdrantProvider()
provider.load_examples()
def test_load_examples_empty_directory(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
provider = QdrantProvider()
provider.load_examples()
def test_load_examples_with_files(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
loaded = provider.get_loaded_examples()
assert len(loaded) == 1
assert loaded[0]["title"] == "Test"
def test_load_examples_skip_existing(monkeypatch, temp_load_skip_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_load_skip_examples_dir.name)
md_file = temp_load_skip_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
provider.load_examples()
loaded = provider.get_loaded_examples()
assert len(loaded) == 1
def test_load_examples_force_reload(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
provider.load_examples(force_reload=True)
loaded = provider.get_loaded_examples()
assert len(loaded) == 1
def test_load_examples_error_handling(monkeypatch, temp_error_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_error_examples_dir.name)
good_file = temp_error_examples_dir / "good.md"
good_file.write_text("# Good\n\nContent", encoding="utf-8")
bad_file = temp_error_examples_dir / "bad.md"
bad_file.write_text("# Bad\n\n", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
loaded = provider.get_loaded_examples()
assert len(loaded) >= 1
def test_list_resources_no_query(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
resources = provider.list_resources()
assert len(resources) >= 1
def test_list_resources_with_query(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
resources = provider.list_resources(query="test")
assert isinstance(resources, list)
def test_query_relevant_documents(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent about testing", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
documents = provider.query_relevant_documents("testing")
assert isinstance(documents, list)
def test_query_relevant_documents_with_resources(monkeypatch, temp_examples_dir):
monkeypatch.setenv("QDRANT_EXAMPLES_DIR", temp_examples_dir.name)
md_file = temp_examples_dir / "test.md"
md_file.write_text("# Test\n\nContent", encoding="utf-8")
provider = QdrantProvider()
provider.load_examples()
resources = provider.list_resources()
documents = provider.query_relevant_documents("test", resources=resources)
assert isinstance(documents, list)
def test_close():
provider = QdrantProvider()
provider._connect()
provider.close()
assert provider.client is None
def test_del():
provider = QdrantProvider()
provider._connect()
del provider
def test_top_k_configuration(monkeypatch):
monkeypatch.setenv("QDRANT_TOP_K", "20")
provider = QdrantProvider()
assert provider.top_k == 20
def test_top_k_invalid(monkeypatch):
monkeypatch.setenv("QDRANT_TOP_K", "invalid")
provider = QdrantProvider()
assert provider.top_k == 10
def test_chunk_size_configuration(monkeypatch):
monkeypatch.setenv("QDRANT_CHUNK_SIZE", "5000")
provider = QdrantProvider()
assert provider.chunk_size == 5000
def test_collection_name_configuration(monkeypatch):
monkeypatch.setenv("QDRANT_COLLECTION", "custom_collection")
provider = QdrantProvider()
assert provider.collection_name == "custom_collection"
def test_auto_load_examples_configuration(monkeypatch):
monkeypatch.setenv("QDRANT_AUTO_LOAD_EXAMPLES", "false")
provider = QdrantProvider()
assert provider.auto_load_examples is False

126
uv.lock generated
View File

@@ -409,6 +409,7 @@ dependencies = [
{ name = "langchain-mcp-adapters" },
{ name = "langchain-milvus" },
{ name = "langchain-openai" },
{ name = "langchain-qdrant" },
{ name = "langchain-tavily" },
{ name = "langgraph" },
{ name = "langgraph-checkpoint-mongodb" },
@@ -421,6 +422,7 @@ dependencies = [
{ name = "psycopg", extra = ["binary"] },
{ name = "pymilvus" },
{ name = "python-dotenv" },
{ name = "qdrant-client" },
{ name = "readabilipy" },
{ name = "socksio" },
{ name = "sse-starlette" },
@@ -460,6 +462,7 @@ requires-dist = [
{ name = "langchain-mcp-adapters", specifier = ">=0.0.9" },
{ name = "langchain-milvus", specifier = ">=0.2.1" },
{ name = "langchain-openai", specifier = ">=0.3.8" },
{ name = "langchain-qdrant", specifier = ">=0.2.0,<1.0.0" },
{ name = "langchain-tavily", specifier = "<0.3" },
{ name = "langgraph", specifier = ">=0.3.5" },
{ name = "langgraph-checkpoint-mongodb", specifier = ">=0.1.4" },
@@ -479,6 +482,7 @@ requires-dist = [
{ name = "pytest-cov", marker = "extra == 'test'", specifier = ">=6.0.0" },
{ name = "pytest-postgresql", marker = "extra == 'test'", specifier = ">=7.0.2" },
{ name = "python-dotenv", specifier = ">=1.0.1" },
{ name = "qdrant-client", specifier = ">=1.15.1" },
{ name = "readabilipy", specifier = ">=0.3.0" },
{ name = "ruff", marker = "extra == 'dev'" },
{ name = "socksio", specifier = ">=1.0.0" },
@@ -777,6 +781,28 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl", hash = "sha256:e3fe4ac4b851c468cc8363d500db52c2ead036020723024a109d37346efaa761", size = 58259, upload-time = "2022-09-25T15:39:59.68Z" },
]
[[package]]
name = "h2"
version = "4.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "hpack" },
{ name = "hyperframe" },
]
sdist = { url = "https://files.pythonhosted.org/packages/1d/17/afa56379f94ad0fe8defd37d6eb3f89a25404ffc71d4d848893d270325fc/h2-4.3.0.tar.gz", hash = "sha256:6c59efe4323fa18b47a632221a1888bd7fde6249819beda254aeca909f221bf1", size = 2152026, upload-time = "2025-08-23T18:12:19.778Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/69/b2/119f6e6dcbd96f9069ce9a2665e0146588dc9f88f29549711853645e736a/h2-4.3.0-py3-none-any.whl", hash = "sha256:c438f029a25f7945c69e0ccf0fb951dc3f73a5f6412981daee861431b70e2bdd", size = 61779, upload-time = "2025-08-23T18:12:17.779Z" },
]
[[package]]
name = "hpack"
version = "4.1.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/2c/48/71de9ed269fdae9c8057e5a4c0aa7402e8bb16f2c6e90b3aa53327b113f8/hpack-4.1.0.tar.gz", hash = "sha256:ec5eca154f7056aa06f196a557655c5b009b382873ac8d1e66e79e87535f1dca", size = 51276, upload-time = "2025-01-22T21:44:58.347Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/07/c6/80c95b1b2b94682a72cbdbfb85b81ae2daffa4291fbfa1b1464502ede10d/hpack-4.1.0-py3-none-any.whl", hash = "sha256:157ac792668d995c657d93111f46b4535ed114f0c9c8d672271bbec7eae1b496", size = 34357, upload-time = "2025-01-22T21:44:56.92Z" },
]
[[package]]
name = "html5lib"
version = "1.1"
@@ -818,6 +844,11 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
]
[package.optional-dependencies]
http2 = [
{ name = "h2" },
]
[[package]]
name = "httpx-sse"
version = "0.4.0"
@@ -845,6 +876,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/40/0c/37d380846a2e5c9a3c6a73d26ffbcfdcad5fc3eacf42fdf7cff56f2af634/huggingface_hub-0.29.3-py3-none-any.whl", hash = "sha256:0b25710932ac649c08cdbefa6c6ccb8e88eef82927cacdb048efb726429453aa", size = 468997, upload-time = "2025-03-11T10:49:38.674Z" },
]
[[package]]
name = "hyperframe"
version = "6.1.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/02/e7/94f8232d4a74cc99514c13a9f995811485a6903d48e5d952771ef6322e30/hyperframe-6.1.0.tar.gz", hash = "sha256:f630908a00854a7adeabd6382b43923a4c4cd4b821fcb527e6ab9e15382a3b08", size = 26566, upload-time = "2025-01-22T21:41:49.302Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/48/30/47d0bf6072f7252e6521f3447ccfa40b421b6824517f82854703d0f5a98b/hyperframe-6.1.0-py3-none-any.whl", hash = "sha256:b03380493a519fce58ea5af42e4a42317bf9bd425596f7a0835ffce80f1a42e5", size = 13007, upload-time = "2025-01-22T21:41:47.295Z" },
]
[[package]]
name = "idna"
version = "3.10"
@@ -1172,6 +1212,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ec/dd/effc847fd55b808d04f7e453d1e4bd3dc813708113ee283055e77be6d651/langchain_openai-0.3.22-py3-none-any.whl", hash = "sha256:945d3b18f2293504d0b81971a9017fc1294571cce4204c18aba3cfbfc43d24c6", size = 65295, upload-time = "2025-06-10T19:56:00.609Z" },
]
[[package]]
name = "langchain-qdrant"
version = "0.2.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "langchain-core" },
{ name = "pydantic" },
{ name = "qdrant-client" },
]
sdist = { url = "https://files.pythonhosted.org/packages/a2/f0/a5624a77f6f69f9b8c1533460e92b7afe7f4574d79a4ba6415da8a8098c6/langchain_qdrant-0.2.1.tar.gz", hash = "sha256:19d8cce3e305e87c32f3be6fdcca6b5acb595297695a4f373f233e7fda6a2b7c", size = 34823, upload-time = "2025-09-10T18:07:06.555Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e1/a7/61365d11afa5e20fa194ad54cc299b4b0b6708b96975ed121a7fffa6f669/langchain_qdrant-0.2.1-py3-none-any.whl", hash = "sha256:d82637eae4828ca67ac806d722fc21b660617fdd5d7eef07b99249d0e7976c3b", size = 24335, upload-time = "2025-09-10T18:07:05.669Z" },
]
[[package]]
name = "langchain-tavily"
version = "0.2.11"
@@ -1836,6 +1890,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9c/a2/579dcefbb0285b31f8d65b537f8a9932ed51319e0a3694e01b5bbc271f92/port_for-0.7.4-py3-none-any.whl", hash = "sha256:08404aa072651a53dcefe8d7a598ee8a1dca320d9ac44ac464da16ccf2a02c4a", size = 21369, upload-time = "2024-10-09T12:28:37.853Z" },
]
[[package]]
name = "portalocker"
version = "3.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pywin32", marker = "sys_platform == 'win32'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/5e/77/65b857a69ed876e1951e88aaba60f5ce6120c33703f7cb61a3c894b8c1b6/portalocker-3.2.0.tar.gz", hash = "sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac", size = 95644, upload-time = "2025-06-14T13:20:40.03Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/4b/a6/38c8e2f318bf67d338f4d629e93b0b4b9af331f455f0390ea8ce4a099b26/portalocker-3.2.0-py3-none-any.whl", hash = "sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968", size = 22424, upload-time = "2025-06-14T13:20:38.083Z" },
]
[[package]]
name = "primp"
version = "0.14.0"
@@ -1947,27 +2013,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/97/b7/15cc7d93443d6c6a84626ae3258a91f4c6ac8c0edd5df35ea7658f71b79c/protobuf-6.32.1-py3-none-any.whl", hash = "sha256:2601b779fc7d32a866c6b4404f9d42a3f67c5b9f3f15b4db3cccabe06b95c346", size = 169289, upload-time = "2025-09-11T21:38:41.234Z" },
]
[[package]]
name = "pyasn1"
version = "0.6.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/ba/e9/01f1a64245b89f039897cb0130016d79f77d52669aae6ee7b159a6c4c018/pyasn1-0.6.1.tar.gz", hash = "sha256:6f580d2bdd84365380830acf45550f2511469f673cb4a5ae3857a3170128b034", size = 145322, upload-time = "2024-09-10T22:41:42.55Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c8/f1/d6a797abb14f6283c0ddff96bbdd46937f64122b8c925cab503dd37f8214/pyasn1-0.6.1-py3-none-any.whl", hash = "sha256:0d632f46f2ba09143da3a8afe9e33fb6f92fa2320ab7e886e2d0f7672af84629", size = 83135, upload-time = "2024-09-11T16:00:36.122Z" },
]
[[package]]
name = "pyasn1-modules"
version = "0.4.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pyasn1" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" },
]
[[package]]
name = "psutil"
version = "7.0.0"
@@ -2042,6 +2087,27 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/47/fd/4feb52a55c1a4bd748f2acaed1903ab54a723c47f6d0242780f4d97104d4/psycopg_pool-3.2.6-py3-none-any.whl", hash = "sha256:5887318a9f6af906d041a0b1dc1c60f8f0dda8340c2572b74e10907b51ed5da7", size = 38252, upload-time = "2025-02-26T12:03:45.073Z" },
]
[[package]]
name = "pyasn1"
version = "0.6.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/ba/e9/01f1a64245b89f039897cb0130016d79f77d52669aae6ee7b159a6c4c018/pyasn1-0.6.1.tar.gz", hash = "sha256:6f580d2bdd84365380830acf45550f2511469f673cb4a5ae3857a3170128b034", size = 145322, upload-time = "2024-09-10T22:41:42.55Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c8/f1/d6a797abb14f6283c0ddff96bbdd46937f64122b8c925cab503dd37f8214/pyasn1-0.6.1-py3-none-any.whl", hash = "sha256:0d632f46f2ba09143da3a8afe9e33fb6f92fa2320ab7e886e2d0f7672af84629", size = 83135, upload-time = "2024-09-11T16:00:36.122Z" },
]
[[package]]
name = "pyasn1-modules"
version = "0.4.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pyasn1" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" },
]
[[package]]
name = "pycparser"
version = "2.22"
@@ -2319,6 +2385,24 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/de/02b54f42487e3d3c6efb3f89428677074ca7bf43aae402517bc7cca949f3/PyYAML-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:8388ee1976c416731879ac16da0aff3f63b286ffdd57cdeb95f3f2e085687563", size = 156446, upload-time = "2024-08-06T20:33:04.33Z" },
]
[[package]]
name = "qdrant-client"
version = "1.15.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "grpcio" },
{ name = "httpx", extra = ["http2"] },
{ name = "numpy" },
{ name = "portalocker" },
{ name = "protobuf" },
{ name = "pydantic" },
{ name = "urllib3" },
]
sdist = { url = "https://files.pythonhosted.org/packages/79/8b/76c7d325e11d97cb8eb5e261c3759e9ed6664735afbf32fdded5b580690c/qdrant_client-1.15.1.tar.gz", hash = "sha256:631f1f3caebfad0fd0c1fba98f41be81d9962b7bf3ca653bed3b727c0e0cbe0e", size = 295297, upload-time = "2025-07-31T19:35:19.627Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ef/33/d8df6a2b214ffbe4138db9a1efe3248f67dc3c671f82308bea1582ecbbb7/qdrant_client-1.15.1-py3-none-any.whl", hash = "sha256:2b975099b378382f6ca1cfb43f0d59e541be6e16a5892f282a4b8de7eff5cb63", size = 337331, upload-time = "2025-07-31T19:35:17.539Z" },
]
[[package]]
name = "readabilipy"
version = "0.3.0"