mirror of
https://gitee.com/wanwujie/deer-flow
synced 2026-04-02 22:02:13 +08:00
refactor: split backend into harness (deerflow.*) and app (app.*) (#1131)
* refactor: extract shared utils to break harness→app cross-layer imports Move _validate_skill_frontmatter to src/skills/validation.py and CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py. This eliminates the two reverse dependencies from client.py (harness layer) into gateway/routers/ (app layer), preparing for the harness/app package split. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split backend/src into harness (deerflow.*) and app (app.*) Physically split the monolithic backend/src/ package into two layers: - **Harness** (`packages/harness/deerflow/`): publishable agent framework package with import prefix `deerflow.*`. Contains agents, sandbox, tools, models, MCP, skills, config, and all core infrastructure. - **App** (`app/`): unpublished application code with import prefix `app.*`. Contains gateway (FastAPI REST API) and channels (IM integrations). Key changes: - Move 13 harness modules to packages/harness/deerflow/ via git mv - Move gateway + channels to app/ via git mv - Rename all imports: src.* → deerflow.* (harness) / app.* (app layer) - Set up uv workspace with deerflow-harness as workspace member - Update langgraph.json, config.example.yaml, all scripts, Docker files - Add build-system (hatchling) to harness pyproject.toml - Add PYTHONPATH=. to gateway startup commands for app.* resolution - Update ruff.toml with known-first-party for import sorting - Update all documentation to reflect new directory structure Boundary rule enforced: harness code never imports from app. All 429 tests pass. Lint clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add harness→app boundary check test and update docs Add test_harness_boundary.py that scans all Python files in packages/harness/deerflow/ and fails if any `from app.*` or `import app.*` statement is found. This enforces the architectural rule that the harness layer never depends on the app layer. Update CLAUDE.md to document the harness/app split architecture, import conventions, and the boundary enforcement test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add config versioning with auto-upgrade on startup When config.example.yaml schema changes, developers' local config.yaml files can silently become outdated. This adds a config_version field and auto-upgrade mechanism so breaking changes (like src.* → deerflow.* renames) are applied automatically before services start. - Add config_version: 1 to config.example.yaml - Add startup version check warning in AppConfig.from_file() - Add scripts/config-upgrade.sh with migration registry for value replacements - Add `make config-upgrade` target - Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services - Add config error hints in service failure messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix comments * fix: update src.* import in test_sandbox_tools_security to deerflow.* Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: handle empty config and search parent dirs for config.example.yaml Address Copilot review comments on PR #1131: - Guard against yaml.safe_load() returning None for empty config files - Search parent directories for config.example.yaml instead of only looking next to config.yaml, fixing detection in common setups Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: correct skills root path depth and config_version type coercion - loader.py: fix get_skills_root_path() to use 5 parent levels (was 3) after harness split, file lives at packages/harness/deerflow/skills/ so parent×3 resolved to backend/packages/harness/ instead of backend/ - app_config.py: coerce config_version to int() before comparison in _check_config_version() to prevent TypeError when YAML stores value as string (e.g. config_version: "1") - tests: add regression tests for both fixes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update test imports from src.* to deerflow.*/app.* after harness refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
12
.github/copilot-instructions.md
vendored
12
.github/copilot-instructions.md
vendored
@@ -143,12 +143,12 @@ Root-level orchestration and config:
|
||||
|
||||
Backend core:
|
||||
|
||||
- `backend/src/agents/` - lead agent, middleware chain, memory
|
||||
- `backend/src/gateway/` - FastAPI gateway API
|
||||
- `backend/src/sandbox/` - sandbox provider + tool wrappers
|
||||
- `backend/src/subagents/` - subagent registry/execution
|
||||
- `backend/src/mcp/` - MCP integration
|
||||
- `backend/langgraph.json` - graph entrypoint (`src.agents:make_lead_agent`)
|
||||
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
|
||||
- `backend/app/gateway/` - FastAPI gateway API
|
||||
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
|
||||
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
|
||||
- `backend/packages/harness/deerflow/mcp/` - MCP integration
|
||||
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
|
||||
- `backend/pyproject.toml` - Python deps and `requires-python`
|
||||
- `backend/ruff.toml` - lint/format policy
|
||||
- `backend/tests/` - backend unit and integration-like tests
|
||||
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -51,3 +51,4 @@ web/
|
||||
|
||||
# Deployment artifacts
|
||||
backend/Dockerfile.langgraph
|
||||
config.yaml.bak
|
||||
|
||||
8
Makefile
8
Makefile
@@ -1,12 +1,13 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
|
||||
.PHONY: help config check install dev dev-daemon start stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
.PHONY: help config config-upgrade check install dev dev-daemon start stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
|
||||
PYTHON ?= python
|
||||
|
||||
help:
|
||||
@echo "DeerFlow Development Commands:"
|
||||
@echo " make config - Generate local config files (aborts if config already exists)"
|
||||
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
|
||||
@echo " make check - Check if all required tools are installed"
|
||||
@echo " make install - Install all dependencies (frontend + backend)"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@@ -31,6 +32,9 @@ help:
|
||||
config:
|
||||
@$(PYTHON) ./scripts/configure.py
|
||||
|
||||
config-upgrade:
|
||||
@./scripts/config-upgrade.sh
|
||||
|
||||
# Check required tools
|
||||
check:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@@ -96,7 +100,7 @@ dev-daemon:
|
||||
stop:
|
||||
@echo "Stopping all services..."
|
||||
@-pkill -f "langgraph dev" 2>/dev/null || true
|
||||
@-pkill -f "uvicorn src.gateway.app:app" 2>/dev/null || true
|
||||
@-pkill -f "uvicorn app.gateway.app:app" 2>/dev/null || true
|
||||
@-pkill -f "next dev" 2>/dev/null || true
|
||||
@-pkill -f "next start" 2>/dev/null || true
|
||||
@-pkill -f "next-server" 2>/dev/null || true
|
||||
|
||||
@@ -145,7 +145,7 @@ make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` starts `provisioner` only when `config.yaml` uses provisioner mode (`sandbox.use: src.community.aio_sandbox:AioSandboxProvider` with `provisioner_url`).
|
||||
`make docker-start` starts `provisioner` only when `config.yaml` uses provisioner mode (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` with `provisioner_url`).
|
||||
|
||||
**Production** (builds images locally, mounts runtime config and data):
|
||||
|
||||
@@ -437,7 +437,7 @@ DeerFlow is model-agnostic — it works with any LLM that implements the OpenAI-
|
||||
DeerFlow can be used as an embedded Python library without running the full HTTP services. The `DeerFlowClient` provides direct in-process access to all agent and Gateway capabilities, returning the same response schemas as the HTTP Gateway API:
|
||||
|
||||
```python
|
||||
from src.client import DeerFlowClient
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
@@ -456,7 +456,7 @@ client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/src/client.py` for full API documentation.
|
||||
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/packages/harness/deerflow/client.py` for full API documentation.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
@@ -22,33 +22,38 @@ deer-flow/
|
||||
├── backend/ # Backend application (this directory)
|
||||
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
|
||||
│ ├── langgraph.json # LangGraph server configuration
|
||||
│ ├── src/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # 10 middleware components
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ ├── packages/
|
||||
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── deerflow/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # 10 middleware components
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ ├── app/ # Application layer (import: app.*)
|
||||
│ │ ├── gateway/ # FastAPI Gateway API
|
||||
│ │ │ ├── app.py # FastAPI application
|
||||
│ │ │ └── routers/ # 6 route modules
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ │ └── channels/ # IM platform integrations
|
||||
│ ├── tests/ # Test suite
|
||||
│ └── docs/ # Documentation
|
||||
├── frontend/ # Next.js frontend application
|
||||
@@ -92,19 +97,48 @@ Regression tests related to Docker/provisioner behavior:
|
||||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||||
|
||||
Boundary check (harness → app import firewall):
|
||||
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
|
||||
|
||||
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Harness / App Split
|
||||
|
||||
The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram).
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
**Import conventions**:
|
||||
```python
|
||||
# Harness internal
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
# App internal
|
||||
from app.gateway.app import app
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# App → Harness (allowed)
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
|
||||
# from app.gateway.routers.uploads import ... # ← will fail CI
|
||||
```
|
||||
|
||||
### Agent System
|
||||
|
||||
**Lead Agent** (`src/agents/lead_agent/agent.py`):
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||||
|
||||
**ThreadState** (`src/agents/thread_state.py`):
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||||
|
||||
@@ -116,7 +150,7 @@ CI runs these regression tests for every pull request via [.github/workflows/bac
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Middlewares execute in strict order in `src/agents/lead_agent/agent.py`:
|
||||
Middlewares execute in strict order in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories (`backend/.deer-flow/threads/{thread_id}/user-data/{workspace,uploads,outputs}`)
|
||||
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
|
||||
@@ -136,6 +170,8 @@ Middlewares execute in strict order in `src/agents/lead_agent/agent.py`:
|
||||
|
||||
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
|
||||
|
||||
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
@@ -154,7 +190,7 @@ Configuration priority:
|
||||
3. `extensions_config.json` in current directory (backend/)
|
||||
4. `extensions_config.json` in parent directory (project root - **recommended location**)
|
||||
|
||||
### Gateway API (`src/gateway/`)
|
||||
### Gateway API (`app/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`.
|
||||
|
||||
@@ -172,13 +208,13 @@ FastAPI application on port 8001 with health check at `GET /health`.
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
### Sandbox System (`src/sandbox/`)
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
|
||||
- `AioSandboxProvider` (`src/community/`) - Docker-based isolation
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
@@ -186,14 +222,14 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||||
|
||||
**Sandbox Tools** (in `src/sandbox/tools.py`):
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `write_file` - Write/append to files, creates directories
|
||||
- `str_replace` - Substring replacement (single or all occurrences)
|
||||
|
||||
### Subagent System (`src/subagents/`)
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
|
||||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||||
@@ -201,7 +237,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result
|
||||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||||
|
||||
### Tool System (`src/tools/`)
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
|
||||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||||
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
|
||||
@@ -213,13 +249,13 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
4. **Subagent tool** (if enabled):
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type, max_turns)
|
||||
|
||||
**Community tools** (`src/community/`):
|
||||
**Community tools** (`packages/harness/deerflow/community/`):
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||||
- `firecrawl/` - Web scraping via Firecrawl API
|
||||
- `image_search/` - Image search via DuckDuckGo
|
||||
|
||||
### MCP System (`src/mcp/`)
|
||||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||||
|
||||
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
|
||||
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
|
||||
@@ -228,7 +264,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
|
||||
|
||||
### Skills System (`src/skills/`)
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
|
||||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
|
||||
@@ -236,7 +272,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
|
||||
### Model Factory (`src/models/factory.py`)
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||||
@@ -244,7 +280,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||||
|
||||
### IM Channels System (`src/channels/`)
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow agent via the LangGraph Server.
|
||||
|
||||
@@ -273,7 +309,7 @@ Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow a
|
||||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token)
|
||||
|
||||
### Memory System (`src/agents/memory/`)
|
||||
### Memory System (`packages/harness/deerflow/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `updater.py` - LLM-based memory updates with fact extraction and atomic file I/O
|
||||
@@ -300,7 +336,7 @@ Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow a
|
||||
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
|
||||
- `max_injection_tokens` - Token limit for prompt injection (2000)
|
||||
|
||||
### Reflection System (`src/reflection/`)
|
||||
### Reflection System (`packages/harness/deerflow/reflection/`)
|
||||
|
||||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||||
@@ -324,11 +360,11 @@ Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow a
|
||||
|
||||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||||
|
||||
### Embedded Client (`src/client.py`)
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
|
||||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||||
|
||||
**Architecture**: Imports the same `src/` modules that LangGraph Server and Gateway API use. Shares the same config files and data directories. No FastAPI dependency.
|
||||
**Architecture**: Imports the same `deerflow` modules that LangGraph Server and Gateway API use. Shares the same config files and data directories. No FastAPI dependency.
|
||||
|
||||
**Agent Conversation** (replaces LangGraph Server):
|
||||
- `chat(message, thread_id)` — synchronous, returns final text
|
||||
@@ -367,7 +403,7 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
- Run the full suite before and after your change: `make test`
|
||||
- Tests must pass before a feature is considered complete
|
||||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `src.subagents.executor`)
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
|
||||
@@ -227,7 +227,7 @@ Example test:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from src.models.factory import create_chat_model
|
||||
from deerflow.models.factory import create_chat_model
|
||||
|
||||
def test_create_chat_model_with_valid_name():
|
||||
"""Test that a valid model name creates a model instance."""
|
||||
@@ -269,10 +269,10 @@ Include in your PR description:
|
||||
|
||||
### Adding New Tools
|
||||
|
||||
1. Create tool in `src/tools/builtins/` or `src/community/`:
|
||||
1. Create tool in `packages/harness/deerflow/tools/builtins/` or `packages/harness/deerflow/community/`:
|
||||
|
||||
```python
|
||||
# src/tools/builtins/my_tool.py
|
||||
# packages/harness/deerflow/tools/builtins/my_tool.py
|
||||
from langchain_core.tools import tool
|
||||
|
||||
@tool
|
||||
@@ -294,15 +294,15 @@ def my_tool(param: str) -> str:
|
||||
tools:
|
||||
- name: my_tool
|
||||
group: my_group
|
||||
use: src.tools.builtins.my_tool:my_tool
|
||||
use: deerflow.tools.builtins.my_tool:my_tool
|
||||
```
|
||||
|
||||
### Adding New Middleware
|
||||
|
||||
1. Create middleware in `src/agents/middlewares/`:
|
||||
1. Create middleware in `packages/harness/deerflow/agents/middlewares/`:
|
||||
|
||||
```python
|
||||
# src/agents/middlewares/my_middleware.py
|
||||
# packages/harness/deerflow/agents/middlewares/my_middleware.py
|
||||
from langchain.agents.middleware import BaseMiddleware
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
@@ -315,7 +315,7 @@ class MyMiddleware(BaseMiddleware):
|
||||
return state
|
||||
```
|
||||
|
||||
2. Register in `src/agents/lead_agent/agent.py`:
|
||||
2. Register in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
|
||||
```python
|
||||
middlewares = [
|
||||
@@ -329,10 +329,10 @@ middlewares = [
|
||||
|
||||
### Adding New API Endpoints
|
||||
|
||||
1. Create router in `src/gateway/routers/`:
|
||||
1. Create router in `app/gateway/routers/`:
|
||||
|
||||
```python
|
||||
# src/gateway/routers/my_router.py
|
||||
# app/gateway/routers/my_router.py
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/my-endpoint", tags=["my-endpoint"])
|
||||
@@ -348,10 +348,10 @@ async def create_item(data: dict):
|
||||
return {"created": data}
|
||||
```
|
||||
|
||||
2. Register in `src/gateway/app.py`:
|
||||
2. Register in `app/gateway/app.py`:
|
||||
|
||||
```python
|
||||
from src.gateway.routers import my_router
|
||||
from app.gateway.routers import my_router
|
||||
|
||||
app.include_router(my_router.router)
|
||||
```
|
||||
@@ -360,7 +360,7 @@ app.include_router(my_router.router)
|
||||
|
||||
When adding new configuration options:
|
||||
|
||||
1. Update `src/config/app_config.py` with new fields
|
||||
1. Update `packages/harness/deerflow/config/app_config.py` with new fields
|
||||
2. Add default values in `config.example.yaml`
|
||||
3. Document in `docs/CONFIGURATION.md`
|
||||
|
||||
|
||||
@@ -36,4 +36,4 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
EXPOSE 8001 2024
|
||||
|
||||
# Default command (can be overridden in docker-compose)
|
||||
CMD ["sh", "-c", "uv run uvicorn src.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
@@ -5,7 +5,7 @@ dev:
|
||||
uv run langgraph dev --no-browser --allow-blocking --no-reload
|
||||
|
||||
gateway:
|
||||
uv run uvicorn src.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. uv run pytest tests/ -v
|
||||
|
||||
@@ -5,8 +5,8 @@ Provides a pluggable channel system that connects external messaging platforms
|
||||
which uses ``langgraph-sdk`` to communicate with the underlying LangGraph Server.
|
||||
"""
|
||||
|
||||
from src.channels.base import Channel
|
||||
from src.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
|
||||
|
||||
__all__ = [
|
||||
"Channel",
|
||||
@@ -6,7 +6,7 @@ import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
from src.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -8,8 +8,8 @@ import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from src.channels.base import Channel
|
||||
from src.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -9,8 +9,8 @@ import time
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from src.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from src.channels.store import ChannelStore
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -242,7 +242,7 @@ def _resolve_attachments(thread_id: str, artifacts: list[str]) -> list[ResolvedA
|
||||
Skips artifacts that cannot be resolved (missing files, invalid paths)
|
||||
and logs warnings for them.
|
||||
"""
|
||||
from src.config.paths import get_paths
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
attachments: list[ResolvedAttachment] = []
|
||||
paths = get_paths()
|
||||
@@ -266,14 +266,16 @@ def _resolve_attachments(thread_id: str, artifacts: list[str]) -> list[ResolvedA
|
||||
continue
|
||||
mime, _ = mimetypes.guess_type(str(actual))
|
||||
mime = mime or "application/octet-stream"
|
||||
attachments.append(ResolvedAttachment(
|
||||
virtual_path=virtual_path,
|
||||
actual_path=actual,
|
||||
filename=actual.name,
|
||||
mime_type=mime,
|
||||
size=actual.stat().st_size,
|
||||
is_image=mime.startswith("image/"),
|
||||
))
|
||||
attachments.append(
|
||||
ResolvedAttachment(
|
||||
virtual_path=virtual_path,
|
||||
actual_path=actual,
|
||||
filename=actual.name,
|
||||
mime_type=mime,
|
||||
size=actual.stat().st_size,
|
||||
is_image=mime.startswith("image/"),
|
||||
)
|
||||
)
|
||||
except (ValueError, OSError) as exc:
|
||||
logger.warning("[Manager] failed to resolve artifact %s: %s", virtual_path, exc)
|
||||
return attachments
|
||||
@@ -348,12 +350,7 @@ class ChannelManager:
|
||||
def _resolve_run_params(self, msg: InboundMessage, thread_id: str) -> tuple[str, dict[str, Any], dict[str, Any]]:
|
||||
channel_layer, user_layer = self._resolve_session_layer(msg)
|
||||
|
||||
assistant_id = (
|
||||
user_layer.get("assistant_id")
|
||||
or channel_layer.get("assistant_id")
|
||||
or self._default_session.get("assistant_id")
|
||||
or self._assistant_id
|
||||
)
|
||||
assistant_id = user_layer.get("assistant_id") or channel_layer.get("assistant_id") or self._default_session.get("assistant_id") or self._assistant_id
|
||||
if not isinstance(assistant_id, str) or not assistant_id.strip():
|
||||
assistant_id = self._assistant_id
|
||||
|
||||
@@ -5,17 +5,17 @@ from __future__ import annotations
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from src.channels.manager import ChannelManager
|
||||
from src.channels.message_bus import MessageBus
|
||||
from src.channels.store import ChannelStore
|
||||
from app.channels.manager import ChannelManager
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Channel name → import path for lazy loading
|
||||
_CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"feishu": "src.channels.feishu:FeishuChannel",
|
||||
"slack": "src.channels.slack:SlackChannel",
|
||||
"telegram": "src.channels.telegram:TelegramChannel",
|
||||
"feishu": "app.channels.feishu:FeishuChannel",
|
||||
"slack": "app.channels.slack:SlackChannel",
|
||||
"telegram": "app.channels.telegram:TelegramChannel",
|
||||
}
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ class ChannelService:
|
||||
@classmethod
|
||||
def from_app_config(cls) -> ChannelService:
|
||||
"""Create a ChannelService from the application config."""
|
||||
from src.config.app_config import get_app_config
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
config = get_app_config()
|
||||
channels_config = {}
|
||||
@@ -116,7 +116,7 @@ class ChannelService:
|
||||
return False
|
||||
|
||||
try:
|
||||
from src.reflection import resolve_class
|
||||
from deerflow.reflection import resolve_class
|
||||
|
||||
channel_cls = resolve_class(import_path, base_class=None)
|
||||
except Exception:
|
||||
@@ -8,8 +8,8 @@ from typing import Any
|
||||
|
||||
from markdown_to_mrkdwn import SlackMarkdownConverter
|
||||
|
||||
from src.channels.base import Channel
|
||||
from src.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -35,7 +35,7 @@ class ChannelStore:
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
if path is None:
|
||||
from src.config.paths import get_paths
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
path = Path(get_paths().base_dir) / "channels" / "store.json"
|
||||
self._path = Path(path)
|
||||
@@ -7,8 +7,8 @@ import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from src.channels.base import Channel
|
||||
from src.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -4,9 +4,8 @@ from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from src.config.app_config import get_app_config
|
||||
from src.gateway.config import get_gateway_config
|
||||
from src.gateway.routers import (
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.routers import (
|
||||
agents,
|
||||
artifacts,
|
||||
channels,
|
||||
@@ -17,6 +16,7 @@ from src.gateway.routers import (
|
||||
suggestions,
|
||||
uploads,
|
||||
)
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
@@ -50,7 +50,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
# Start IM channel service if any channels are configured
|
||||
try:
|
||||
from src.channels.service import start_channel_service
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
channel_service = await start_channel_service()
|
||||
logger.info("Channel service started: %s", channel_service.get_status())
|
||||
@@ -61,7 +61,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
|
||||
# Stop channel service on shutdown
|
||||
try:
|
||||
from src.channels.service import stop_channel_service
|
||||
from app.channels.service import stop_channel_service
|
||||
|
||||
await stop_channel_service()
|
||||
except Exception:
|
||||
@@ -4,7 +4,7 @@ from pathlib import Path
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from src.config.paths import get_paths
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
|
||||
def resolve_thread_virtual_path(thread_id: str, virtual_path: str) -> Path:
|
||||
@@ -8,8 +8,8 @@ import yaml
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.config.agents_config import AgentConfig, list_custom_agents, load_agent_config, load_agent_soul
|
||||
from src.config.paths import get_paths
|
||||
from deerflow.config.agents_config import AgentConfig, list_custom_agents, load_agent_config, load_agent_soul
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["agents"])
|
||||
@@ -7,7 +7,7 @@ from urllib.parse import quote
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import FileResponse, HTMLResponse, PlainTextResponse, Response
|
||||
|
||||
from src.gateway.path_utils import resolve_thread_virtual_path
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -25,7 +25,7 @@ class ChannelRestartResponse(BaseModel):
|
||||
@router.get("/", response_model=ChannelStatusResponse)
|
||||
async def get_channels_status() -> ChannelStatusResponse:
|
||||
"""Get the status of all IM channels."""
|
||||
from src.channels.service import get_channel_service
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
@@ -37,7 +37,7 @@ async def get_channels_status() -> ChannelStatusResponse:
|
||||
@router.post("/{name}/restart", response_model=ChannelRestartResponse)
|
||||
async def restart_channel(name: str) -> ChannelRestartResponse:
|
||||
"""Restart a specific IM channel."""
|
||||
from src.channels.service import get_channel_service
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
@@ -6,7 +6,7 @@ from typing import Literal
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.config.extensions_config import ExtensionsConfig, get_extensions_config, reload_extensions_config
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, get_extensions_config, reload_extensions_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["mcp"])
|
||||
@@ -3,8 +3,8 @@
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.agents.memory.updater import get_memory_data, reload_memory_data
|
||||
from src.config.memory_config import get_memory_config
|
||||
from deerflow.agents.memory.updater import get_memory_data, reload_memory_data
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["memory"])
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.config import get_app_config
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["models"])
|
||||
|
||||
@@ -1,22 +1,19 @@
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import tempfile
|
||||
import zipfile
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
import yaml
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from src.gateway.path_utils import resolve_thread_virtual_path
|
||||
from src.skills import Skill, load_skills
|
||||
from src.skills.loader import get_skills_root_path
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from deerflow.skills import Skill, load_skills
|
||||
from deerflow.skills.loader import get_skills_root_path
|
||||
from deerflow.skills.validation import _validate_skill_frontmatter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -129,23 +126,6 @@ class SkillInstallResponse(BaseModel):
|
||||
message: str = Field(..., description="Installation result message")
|
||||
|
||||
|
||||
# Allowed properties in SKILL.md frontmatter
|
||||
ALLOWED_FRONTMATTER_PROPERTIES = {
|
||||
"name",
|
||||
"description",
|
||||
"license",
|
||||
"allowed-tools",
|
||||
"metadata",
|
||||
"compatibility",
|
||||
"version",
|
||||
"author",
|
||||
}
|
||||
|
||||
|
||||
def _safe_load_frontmatter(frontmatter_text: str) -> object:
|
||||
return cast(object, yaml.safe_load(frontmatter_text))
|
||||
|
||||
|
||||
def _should_ignore_archive_entry(path: Path) -> bool:
|
||||
return path.name.startswith(".") or path.name == "__MACOSX"
|
||||
|
||||
@@ -159,81 +139,6 @@ def _resolve_skill_dir_from_archive_root(temp_path: Path) -> Path:
|
||||
return temp_path
|
||||
|
||||
|
||||
def _validate_skill_frontmatter(skill_dir: Path) -> tuple[bool, str, str | None]:
|
||||
"""Validate a skill directory's SKILL.md frontmatter.
|
||||
|
||||
Args:
|
||||
skill_dir: Path to the skill directory containing SKILL.md.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message, skill_name).
|
||||
"""
|
||||
skill_md = skill_dir / "SKILL.md"
|
||||
if not skill_md.exists():
|
||||
return False, "SKILL.md not found", None
|
||||
|
||||
content = skill_md.read_text()
|
||||
if not content.startswith("---"):
|
||||
return False, "No YAML frontmatter found", None
|
||||
|
||||
# Extract frontmatter
|
||||
match = re.match(r"^---\n(.*?)\n---", content, re.DOTALL)
|
||||
if not match:
|
||||
return False, "Invalid frontmatter format", None
|
||||
|
||||
frontmatter_text = match.group(1)
|
||||
|
||||
# Parse YAML frontmatter
|
||||
try:
|
||||
parsed_frontmatter = _safe_load_frontmatter(frontmatter_text)
|
||||
if not isinstance(parsed_frontmatter, Mapping):
|
||||
return False, "Frontmatter must be a YAML dictionary", None
|
||||
parsed_frontmatter = cast(Mapping[object, object], parsed_frontmatter)
|
||||
frontmatter: dict[str, object] = {str(key): value for key, value in parsed_frontmatter.items()}
|
||||
except yaml.YAMLError as e:
|
||||
return False, f"Invalid YAML in frontmatter: {e}", None
|
||||
|
||||
# Check for unexpected properties
|
||||
unexpected_keys = set(frontmatter.keys()) - ALLOWED_FRONTMATTER_PROPERTIES
|
||||
if unexpected_keys:
|
||||
return False, f"Unexpected key(s) in SKILL.md frontmatter: {', '.join(sorted(unexpected_keys))}", None
|
||||
|
||||
# Check required fields
|
||||
if "name" not in frontmatter:
|
||||
return False, "Missing 'name' in frontmatter", None
|
||||
if "description" not in frontmatter:
|
||||
return False, "Missing 'description' in frontmatter", None
|
||||
|
||||
# Validate name
|
||||
name = frontmatter.get("name", "")
|
||||
if not isinstance(name, str):
|
||||
return False, f"Name must be a string, got {type(name).__name__}", None
|
||||
name = name.strip()
|
||||
if not name:
|
||||
return False, "Name cannot be empty", None
|
||||
|
||||
# Check naming convention (hyphen-case: lowercase with hyphens)
|
||||
if not re.match(r"^[a-z0-9-]+$", name):
|
||||
return False, f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)", None
|
||||
if name.startswith("-") or name.endswith("-") or "--" in name:
|
||||
return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens", None
|
||||
if len(name) > 64:
|
||||
return False, f"Name is too long ({len(name)} characters). Maximum is 64 characters.", None
|
||||
|
||||
# Validate description
|
||||
description = frontmatter.get("description", "")
|
||||
if not isinstance(description, str):
|
||||
return False, f"Description must be a string, got {type(description).__name__}", None
|
||||
description = description.strip()
|
||||
if description:
|
||||
if "<" in description or ">" in description:
|
||||
return False, "Description cannot contain angle brackets (< or >)", None
|
||||
if len(description) > 1024:
|
||||
return False, f"Description is too long ({len(description)} characters). Maximum is 1024 characters.", None
|
||||
|
||||
return True, "Skill is valid!", name
|
||||
|
||||
|
||||
def _skill_to_response(skill: Skill) -> SkillResponse:
|
||||
"""Convert a Skill object to a SkillResponse."""
|
||||
return SkillResponse(
|
||||
@@ -4,7 +4,7 @@ import logging
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from src.models import create_chat_model
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -6,24 +6,14 @@ from pathlib import Path
|
||||
from fastapi import APIRouter, File, HTTPException, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from src.config.paths import VIRTUAL_PATH_PREFIX, get_paths
|
||||
from src.sandbox.sandbox_provider import get_sandbox_provider
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/threads/{thread_id}/uploads", tags=["uploads"])
|
||||
|
||||
# File extensions that should be converted to markdown
|
||||
CONVERTIBLE_EXTENSIONS = {
|
||||
".pdf",
|
||||
".ppt",
|
||||
".pptx",
|
||||
".xls",
|
||||
".xlsx",
|
||||
".doc",
|
||||
".docx",
|
||||
}
|
||||
|
||||
|
||||
class UploadResponse(BaseModel):
|
||||
"""Response model for file upload."""
|
||||
@@ -47,32 +37,6 @@ def get_uploads_dir(thread_id: str) -> Path:
|
||||
return base_dir
|
||||
|
||||
|
||||
async def convert_file_to_markdown(file_path: Path) -> Path | None:
|
||||
"""Convert a file to markdown using markitdown.
|
||||
|
||||
Args:
|
||||
file_path: Path to the file to convert.
|
||||
|
||||
Returns:
|
||||
Path to the markdown file if conversion was successful, None otherwise.
|
||||
"""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
|
||||
md = MarkItDown()
|
||||
result = md.convert(str(file_path))
|
||||
|
||||
# Save as .md file with same name
|
||||
md_path = file_path.with_suffix(".md")
|
||||
md_path.write_text(result.text_content, encoding="utf-8")
|
||||
|
||||
logger.info(f"Converted {file_path.name} to markdown: {md_path.name}")
|
||||
return md_path
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to convert {file_path.name} to markdown: {e}")
|
||||
return None
|
||||
|
||||
|
||||
@router.post("", response_model=UploadResponse)
|
||||
async def upload_files(
|
||||
thread_id: str,
|
||||
@@ -3,6 +3,12 @@
|
||||
Debug script for lead_agent.
|
||||
Run this file directly in VS Code with breakpoints.
|
||||
|
||||
Requirements:
|
||||
Run with `uv run` from the backend/ directory so that the uv workspace
|
||||
resolves deerflow-harness and app packages correctly:
|
||||
|
||||
cd backend && PYTHONPATH=. uv run python debug.py
|
||||
|
||||
Usage:
|
||||
1. Set breakpoints in agent.py or other files
|
||||
2. Press F5 or use "Run and Debug" panel
|
||||
@@ -11,21 +17,14 @@ Usage:
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Ensure we can import from src
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Load environment variables
|
||||
from dotenv import load_dotenv
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from src.agents import make_lead_agent
|
||||
from deerflow.agents import make_lead_agent
|
||||
|
||||
load_dotenv()
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
@@ -36,7 +35,7 @@ logging.basicConfig(
|
||||
async def main():
|
||||
# Initialize MCP tools at startup
|
||||
try:
|
||||
from src.mcp import initialize_mcp_tools
|
||||
from deerflow.mcp import initialize_mcp_tools
|
||||
|
||||
await initialize_mcp_tools()
|
||||
except Exception as e:
|
||||
|
||||
@@ -80,7 +80,7 @@ docker stop <id> # Auto-removes due to --rm
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The implementation is in `backend/src/community/aio_sandbox/aio_sandbox_provider.py`:
|
||||
The implementation is in `backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py`:
|
||||
|
||||
- `_detect_container_runtime()`: Detects available runtime at startup
|
||||
- `_start_container()`: Uses detected runtime, skips Docker-specific options for Apple Container
|
||||
@@ -93,14 +93,14 @@ No configuration changes are needed! The system works automatically.
|
||||
However, you can verify the runtime in use by checking the logs:
|
||||
|
||||
```
|
||||
INFO:src.community.aio_sandbox.aio_sandbox_provider:Detected Apple Container: container version 0.1.0
|
||||
INFO:src.community.aio_sandbox.aio_sandbox_provider:Starting sandbox container using container: ...
|
||||
INFO:deerflow.community.aio_sandbox.aio_sandbox_provider:Detected Apple Container: container version 0.1.0
|
||||
INFO:deerflow.community.aio_sandbox.aio_sandbox_provider:Starting sandbox container using container: ...
|
||||
```
|
||||
|
||||
Or for Docker:
|
||||
```
|
||||
INFO:src.community.aio_sandbox.aio_sandbox_provider:Apple Container not available, falling back to Docker
|
||||
INFO:src.community.aio_sandbox.aio_sandbox_provider:Starting sandbox container using docker: ...
|
||||
INFO:deerflow.community.aio_sandbox.aio_sandbox_provider:Apple Container not available, falling back to Docker
|
||||
INFO:deerflow.community.aio_sandbox.aio_sandbox_provider:Starting sandbox container using docker: ...
|
||||
```
|
||||
|
||||
## Container Images
|
||||
@@ -109,7 +109,7 @@ Both runtimes use OCI-compatible images. The default image works with both:
|
||||
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider
|
||||
image: enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest # Default image
|
||||
```
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ This document provides a comprehensive overview of the DeerFlow backend architec
|
||||
|
||||
The LangGraph server is the core agent runtime, built on LangGraph for robust multi-agent workflow orchestration.
|
||||
|
||||
**Entry Point**: `src/agents/lead_agent/agent.py:make_lead_agent`
|
||||
**Entry Point**: `packages/harness/deerflow/agents/lead_agent/agent.py:make_lead_agent`
|
||||
|
||||
**Key Responsibilities**:
|
||||
- Agent creation and configuration
|
||||
@@ -70,7 +70,7 @@ The LangGraph server is the core agent runtime, built on LangGraph for robust mu
|
||||
{
|
||||
"agent": {
|
||||
"type": "agent",
|
||||
"path": "src.agents:make_lead_agent"
|
||||
"path": "deerflow.agents:make_lead_agent"
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -79,7 +79,7 @@ The LangGraph server is the core agent runtime, built on LangGraph for robust mu
|
||||
|
||||
FastAPI application providing REST endpoints for non-agent operations.
|
||||
|
||||
**Entry Point**: `src/gateway/app.py`
|
||||
**Entry Point**: `app/gateway/app.py`
|
||||
|
||||
**Routers**:
|
||||
- `models.py` - `/api/models` - Model listing and details
|
||||
@@ -158,7 +158,7 @@ class ThreadState(AgentState):
|
||||
▼ ▼
|
||||
┌─────────────────────────┐ ┌─────────────────────────┐
|
||||
│ LocalSandboxProvider │ │ AioSandboxProvider │
|
||||
│ (src/sandbox/local.py) │ │ (src/community/) │
|
||||
│ (packages/harness/deerflow/sandbox/local.py) │ │ (packages/harness/deerflow/community/) │
|
||||
│ │ │ │
|
||||
│ - Singleton instance │ │ - Docker-based │
|
||||
│ - Direct execution │ │ - Isolated containers │
|
||||
@@ -192,7 +192,7 @@ class ThreadState(AgentState):
|
||||
|
||||
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
|
||||
│ Built-in Tools │ │ Configured Tools │ │ MCP Tools │
|
||||
│ (src/tools/) │ │ (config.yaml) │ │ (extensions.json) │
|
||||
│ (packages/harness/deerflow/tools/) │ │ (config.yaml) │ │ (extensions.json) │
|
||||
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
|
||||
│ - present_file │ │ - web_search │ │ - github │
|
||||
│ - ask_clarification │ │ - web_fetch │ │ - filesystem │
|
||||
@@ -208,7 +208,7 @@ class ThreadState(AgentState):
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ get_available_tools() │
|
||||
│ (src/tools/__init__) │
|
||||
│ (packages/harness/deerflow/tools/__init__) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -217,7 +217,7 @@ class ThreadState(AgentState):
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ Model Factory │
|
||||
│ (src/models/factory.py) │
|
||||
│ (packages/harness/deerflow/models/factory.py) │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
config.yaml:
|
||||
@@ -264,7 +264,7 @@ config.yaml:
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ MCP Integration │
|
||||
│ (src/mcp/manager.py) │
|
||||
│ (packages/harness/deerflow/mcp/manager.py) │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
extensions_config.json:
|
||||
@@ -302,7 +302,7 @@ extensions_config.json:
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ Skills System │
|
||||
│ (src/skills/loader.py) │
|
||||
│ (packages/harness/deerflow/skills/loader.py) │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Directory Structure:
|
||||
|
||||
@@ -50,7 +50,7 @@ checkpointer = PostgresSaver.from_conn_string(
|
||||
```json
|
||||
{
|
||||
"graphs": {
|
||||
"lead_agent": "src.agents:lead_agent"
|
||||
"lead_agent": "deerflow.agents:lead_agent"
|
||||
},
|
||||
"checkpointer": "checkpointer:checkpointer"
|
||||
}
|
||||
@@ -71,7 +71,7 @@ title:
|
||||
或在代码中配置:
|
||||
|
||||
```python
|
||||
from src.config.title_config import TitleConfig, set_title_config
|
||||
from deerflow.config.title_config import TitleConfig, set_title_config
|
||||
|
||||
set_title_config(TitleConfig(
|
||||
enabled=True,
|
||||
@@ -185,7 +185,7 @@ sequenceDiagram
|
||||
```python
|
||||
# 测试 title 生成
|
||||
import pytest
|
||||
from src.agents.title_middleware import TitleMiddleware
|
||||
from deerflow.agents.title_middleware import TitleMiddleware
|
||||
|
||||
def test_title_generation():
|
||||
# TODO: 添加单元测试
|
||||
@@ -243,11 +243,11 @@ def after_agent(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | N
|
||||
|
||||
## 相关文件
|
||||
|
||||
- [`src/agents/thread_state.py`](../src/agents/thread_state.py) - ThreadState 定义
|
||||
- [`src/agents/title_middleware.py`](../src/agents/title_middleware.py) - TitleMiddleware 实现
|
||||
- [`src/config/title_config.py`](../src/config/title_config.py) - 配置管理
|
||||
- [`packages/harness/deerflow/agents/thread_state.py`](../packages/harness/deerflow/agents/thread_state.py) - ThreadState 定义
|
||||
- [`packages/harness/deerflow/agents/title_middleware.py`](../packages/harness/deerflow/agents/title_middleware.py) - TitleMiddleware 实现
|
||||
- [`packages/harness/deerflow/config/title_config.py`](../packages/harness/deerflow/config/title_config.py) - 配置管理
|
||||
- [`config.yaml`](../config.yaml) - 配置文件
|
||||
- [`src/agents/lead_agent/agent.py`](../src/agents/lead_agent/agent.py) - Middleware 注册
|
||||
- [`packages/harness/deerflow/agents/lead_agent/agent.py`](../packages/harness/deerflow/agents/lead_agent/agent.py) - Middleware 注册
|
||||
|
||||
## 参考资料
|
||||
|
||||
|
||||
@@ -2,6 +2,19 @@
|
||||
|
||||
This guide explains how to configure DeerFlow for your environment.
|
||||
|
||||
## Config Versioning
|
||||
|
||||
`config.example.yaml` contains a `config_version` field that tracks schema changes. When the example version is higher than your local `config.yaml`, the application emits a startup warning:
|
||||
|
||||
```
|
||||
WARNING - Your config.yaml (version 0) is outdated — the latest version is 1.
|
||||
Run `make config-upgrade` to merge new fields into your config.
|
||||
```
|
||||
|
||||
- **Missing `config_version`** in your config is treated as version 0.
|
||||
- Run `make config-upgrade` to auto-merge missing fields (your existing values are preserved, a `.bak` backup is created).
|
||||
- When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||||
|
||||
## Configuration Sections
|
||||
|
||||
### Models
|
||||
@@ -103,7 +116,7 @@ Configure specific tools available to the agent:
|
||||
tools:
|
||||
- name: web_search
|
||||
group: web
|
||||
use: src.community.tavily.tools:web_search_tool
|
||||
use: deerflow.community.tavily.tools:web_search_tool
|
||||
max_results: 5
|
||||
# api_key: $TAVILY_API_KEY # Optional
|
||||
```
|
||||
@@ -124,13 +137,13 @@ DeerFlow supports multiple sandbox execution modes. Configure your preferred mod
|
||||
**Local Execution** (runs sandbox code directly on the host machine):
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.sandbox.local:LocalSandboxProvider # Local execution
|
||||
use: deerflow.sandbox.local:LocalSandboxProvider # Local execution
|
||||
```
|
||||
|
||||
**Docker Execution** (runs sandbox code in isolated Docker containers):
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider # Docker-based sandbox
|
||||
```
|
||||
|
||||
**Docker Execution with Kubernetes** (runs sandbox code in Kubernetes pods via provisioner service):
|
||||
@@ -139,7 +152,7 @@ This mode runs each sandbox in an isolated Kubernetes Pod on your **host machine
|
||||
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider
|
||||
provisioner_url: http://provisioner:8002
|
||||
```
|
||||
|
||||
@@ -152,13 +165,13 @@ Choose between local execution or Docker-based isolation:
|
||||
**Option 1: Local Sandbox** (default, simpler setup):
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.sandbox.local:LocalSandboxProvider
|
||||
use: deerflow.sandbox.local:LocalSandboxProvider
|
||||
```
|
||||
|
||||
**Option 2: Docker Sandbox** (isolated, more secure):
|
||||
```yaml
|
||||
sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider
|
||||
port: 8080
|
||||
auto_start: true
|
||||
container_prefix: deer-flow-sandbox
|
||||
|
||||
@@ -212,11 +212,11 @@ backend/.deer-flow/threads/
|
||||
|
||||
### 组件
|
||||
|
||||
1. **Upload Router** (`src/gateway/routers/uploads.py`)
|
||||
1. **Upload Router** (`app/gateway/routers/uploads.py`)
|
||||
- 处理文件上传、列表、删除请求
|
||||
- 使用 markitdown 转换文档
|
||||
|
||||
2. **Uploads Middleware** (`src/agents/middlewares/uploads_middleware.py`)
|
||||
2. **Uploads Middleware** (`packages/harness/deerflow/agents/middlewares/uploads_middleware.py`)
|
||||
- 在每次 Agent 请求前注入文件列表
|
||||
- 自动生成格式化的文件列表消息
|
||||
|
||||
|
||||
343
backend/docs/HARNESS_APP_SPLIT.md
Normal file
343
backend/docs/HARNESS_APP_SPLIT.md
Normal file
@@ -0,0 +1,343 @@
|
||||
# DeerFlow 后端拆分设计文档:Harness + App
|
||||
|
||||
> 状态:Draft
|
||||
> 作者:DeerFlow Team
|
||||
> 日期:2026-03-13
|
||||
|
||||
## 1. 背景与动机
|
||||
|
||||
DeerFlow 后端当前是一个单一 Python 包(`src.*`),包含了从底层 agent 编排到上层用户产品的所有代码。随着项目发展,这种结构带来了几个问题:
|
||||
|
||||
- **复用困难**:其他产品(CLI 工具、Slack bot、第三方集成)想用 agent 能力,必须依赖整个后端,包括 FastAPI、IM SDK 等不需要的依赖
|
||||
- **职责模糊**:agent 编排逻辑和用户产品逻辑混在同一个 `src/` 下,边界不清晰
|
||||
- **依赖膨胀**:LangGraph Server 运行时不需要 FastAPI/uvicorn/Slack SDK,但当前必须安装全部依赖
|
||||
|
||||
本文档提出将后端拆分为两部分:**deerflow-harness**(可发布的 agent 框架包)和 **app**(不打包的用户产品代码)。
|
||||
|
||||
## 2. 核心概念
|
||||
|
||||
### 2.1 Harness(线束/框架层)
|
||||
|
||||
Harness 是 agent 的构建与编排框架,回答 **"如何构建和运行 agent"** 的问题:
|
||||
|
||||
- Agent 工厂与生命周期管理
|
||||
- Middleware pipeline
|
||||
- 工具系统(内置工具 + MCP + 社区工具)
|
||||
- 沙箱执行环境
|
||||
- 子 agent 委派
|
||||
- 记忆系统
|
||||
- 技能加载与注入
|
||||
- 模型工厂
|
||||
- 配置系统
|
||||
|
||||
**Harness 是一个可发布的 Python 包**(`deerflow-harness`),可以独立安装和使用。
|
||||
|
||||
**Harness 的设计原则**:对上层应用完全无感知。它不知道也不关心谁在调用它——可以是 Web App、CLI、Slack Bot、或者一个单元测试。
|
||||
|
||||
### 2.2 App(应用层)
|
||||
|
||||
App 是面向用户的产品代码,回答 **"如何将 agent 呈现给用户"** 的问题:
|
||||
|
||||
- Gateway API(FastAPI REST 接口)
|
||||
- IM Channels(飞书、Slack、Telegram 集成)
|
||||
- Custom Agent 的 CRUD 管理
|
||||
- 文件上传/下载的 HTTP 接口
|
||||
|
||||
**App 不打包、不发布**,它是 DeerFlow 项目内部的应用代码,直接运行。
|
||||
|
||||
**App 依赖 Harness,但 Harness 不依赖 App。**
|
||||
|
||||
### 2.3 边界划分
|
||||
|
||||
| 模块 | 归属 | 说明 |
|
||||
|------|------|------|
|
||||
| `config/` | Harness | 配置系统是基础设施 |
|
||||
| `reflection/` | Harness | 动态模块加载工具 |
|
||||
| `utils/` | Harness | 通用工具函数 |
|
||||
| `agents/` | Harness | Agent 工厂、middleware、state、memory |
|
||||
| `subagents/` | Harness | 子 agent 委派系统 |
|
||||
| `sandbox/` | Harness | 沙箱执行环境 |
|
||||
| `tools/` | Harness | 工具注册与发现 |
|
||||
| `mcp/` | Harness | MCP 协议集成 |
|
||||
| `skills/` | Harness | 技能加载、解析、定义 schema |
|
||||
| `models/` | Harness | LLM 模型工厂 |
|
||||
| `community/` | Harness | 社区工具(tavily、jina 等) |
|
||||
| `client.py` | Harness | 嵌入式 Python 客户端 |
|
||||
| `gateway/` | App | FastAPI REST API |
|
||||
| `channels/` | App | IM 平台集成 |
|
||||
|
||||
**关于 Custom Agents**:agent 定义格式(`config.yaml` + `SOUL.md` schema)由 Harness 层的 `config/agents_config.py` 定义,但文件的存储、CRUD、发现机制由 App 层的 `gateway/routers/agents.py` 负责。
|
||||
|
||||
## 3. 目标架构
|
||||
|
||||
### 3.1 目录结构
|
||||
|
||||
```
|
||||
backend/
|
||||
├── packages/
|
||||
│ └── harness/
|
||||
│ ├── pyproject.toml # deerflow-harness 包定义
|
||||
│ └── deerflow/ # Python 包根(import 前缀: deerflow.*)
|
||||
│ ├── __init__.py
|
||||
│ ├── config/
|
||||
│ ├── reflection/
|
||||
│ ├── utils/
|
||||
│ ├── agents/
|
||||
│ │ ├── lead_agent/
|
||||
│ │ ├── middlewares/
|
||||
│ │ ├── memory/
|
||||
│ │ ├── checkpointer/
|
||||
│ │ └── thread_state.py
|
||||
│ ├── subagents/
|
||||
│ ├── sandbox/
|
||||
│ ├── tools/
|
||||
│ ├── mcp/
|
||||
│ ├── skills/
|
||||
│ ├── models/
|
||||
│ ├── community/
|
||||
│ └── client.py
|
||||
├── app/ # 不打包(import 前缀: app.*)
|
||||
│ ├── __init__.py
|
||||
│ ├── gateway/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── app.py
|
||||
│ │ ├── config.py
|
||||
│ │ ├── path_utils.py
|
||||
│ │ └── routers/
|
||||
│ └── channels/
|
||||
│ ├── __init__.py
|
||||
│ ├── base.py
|
||||
│ ├── manager.py
|
||||
│ ├── service.py
|
||||
│ ├── store.py
|
||||
│ ├── message_bus.py
|
||||
│ ├── feishu.py
|
||||
│ ├── slack.py
|
||||
│ └── telegram.py
|
||||
├── pyproject.toml # uv workspace root
|
||||
├── langgraph.json
|
||||
├── tests/
|
||||
├── docs/
|
||||
└── Makefile
|
||||
```
|
||||
|
||||
### 3.2 Import 规则
|
||||
|
||||
两个层使用不同的 import 前缀,职责边界一目了然:
|
||||
|
||||
```python
|
||||
# ---------------------------------------------------------------
|
||||
# Harness 内部互相引用(deerflow.* 前缀)
|
||||
# ---------------------------------------------------------------
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.tools import get_available_tools
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# App 内部互相引用(app.* 前缀)
|
||||
# ---------------------------------------------------------------
|
||||
from app.gateway.app import app
|
||||
from app.gateway.routers.uploads import upload_files
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# App 调用 Harness(单向依赖,Harness 永远不 import app)
|
||||
# ---------------------------------------------------------------
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.skills import load_skills
|
||||
from deerflow.config.extensions_config import get_extensions_config
|
||||
```
|
||||
|
||||
**App 调用 Harness 示例 — Gateway 中启动 agent**:
|
||||
|
||||
```python
|
||||
# app/gateway/routers/chat.py
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
async def create_chat_session(thread_id: str, model_name: str):
|
||||
config = get_app_config()
|
||||
model = create_chat_model(name=model_name)
|
||||
agent = make_lead_agent(config=...)
|
||||
# ... 使用 agent 处理用户消息
|
||||
```
|
||||
|
||||
**App 调用 Harness 示例 — Channel 中查询 skills**:
|
||||
|
||||
```python
|
||||
# app/channels/manager.py
|
||||
from deerflow.skills import load_skills
|
||||
from deerflow.agents.memory.updater import get_memory_data
|
||||
|
||||
def handle_status_command():
|
||||
skills = load_skills(enabled_only=True)
|
||||
memory = get_memory_data()
|
||||
return f"Skills: {len(skills)}, Memory facts: {len(memory.get('facts', []))}"
|
||||
```
|
||||
|
||||
**禁止方向**:Harness 代码中绝不能出现 `from app.` 或 `import app.`。
|
||||
|
||||
### 3.3 为什么 App 不打包
|
||||
|
||||
| 方面 | 打包(放 packages/ 下) | 不打包(放 backend/app/) |
|
||||
|------|------------------------|--------------------------|
|
||||
| 命名空间 | 需要 pkgutil `extend_path` 合并,或独立前缀 | 天然独立,`app.*` vs `deerflow.*` |
|
||||
| 发布需求 | 没有——App 是项目内部代码 | 不需要 pyproject.toml |
|
||||
| 复杂度 | 需要管理两个包的构建、版本、依赖声明 | 直接运行,零额外配置 |
|
||||
| 运行方式 | `pip install deerflow-app` | `PYTHONPATH=. uvicorn app.gateway.app:app` |
|
||||
|
||||
App 的唯一消费者是 DeerFlow 项目自身,没有独立发布的需求。放在 `backend/app/` 下作为普通 Python 包,通过 `PYTHONPATH` 或 editable install 让 Python 找到即可。
|
||||
|
||||
### 3.4 依赖关系
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ app/ (不打包,直接运行) │
|
||||
│ ├── fastapi, uvicorn │
|
||||
│ ├── slack-sdk, lark-oapi, ... │
|
||||
│ └── import deerflow.* │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ deerflow-harness (可发布的包) │
|
||||
│ ├── langgraph, langchain │
|
||||
│ ├── markitdown, pydantic, ... │
|
||||
│ └── 零 app 依赖 │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**依赖分类**:
|
||||
|
||||
| 分类 | 依赖包 |
|
||||
|------|--------|
|
||||
| Harness only | agent-sandbox, langchain*, langgraph*, markdownify, markitdown, pydantic, pyyaml, readabilipy, tavily-python, firecrawl-py, tiktoken, ddgs, duckdb, httpx, kubernetes, dotenv |
|
||||
| App only | fastapi, uvicorn, sse-starlette, python-multipart, lark-oapi, slack-sdk, python-telegram-bot, markdown-to-mrkdwn |
|
||||
| Shared | langgraph-sdk(channels 用 HTTP client), pydantic, httpx |
|
||||
|
||||
### 3.5 Workspace 配置
|
||||
|
||||
`backend/pyproject.toml`(workspace root):
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "deer-flow"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["deerflow-harness"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pytest>=8.0.0", "ruff>=0.14.11"]
|
||||
# App 的额外依赖(fastapi 等)也声明在 workspace root,因为 app 不打包
|
||||
app = ["fastapi", "uvicorn", "sse-starlette", "python-multipart"]
|
||||
channels = ["lark-oapi", "slack-sdk", "python-telegram-bot"]
|
||||
|
||||
[tool.uv.workspace]
|
||||
members = ["packages/harness"]
|
||||
|
||||
[tool.uv.sources]
|
||||
deerflow-harness = { workspace = true }
|
||||
```
|
||||
|
||||
## 4. 当前的跨层依赖问题
|
||||
|
||||
在拆分之前,需要先解决 `client.py` 中两处从 harness 到 app 的反向依赖:
|
||||
|
||||
### 4.1 `_validate_skill_frontmatter`
|
||||
|
||||
```python
|
||||
# client.py — harness 导入了 app 层代码
|
||||
from src.gateway.routers.skills import _validate_skill_frontmatter
|
||||
```
|
||||
|
||||
**解决方案**:将该函数提取到 `deerflow/skills/validation.py`。这是一个纯逻辑函数(解析 YAML frontmatter、校验字段),与 FastAPI 无关。
|
||||
|
||||
### 4.2 `CONVERTIBLE_EXTENSIONS` + `convert_file_to_markdown`
|
||||
|
||||
```python
|
||||
# client.py — harness 导入了 app 层代码
|
||||
from src.gateway.routers.uploads import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
```
|
||||
|
||||
**解决方案**:将它们提取到 `deerflow/utils/file_conversion.py`。仅依赖 `markitdown` + `pathlib`,是通用工具函数。
|
||||
|
||||
## 5. 基础设施变更
|
||||
|
||||
### 5.1 LangGraph Server
|
||||
|
||||
LangGraph Server 只需要 harness 包。`langgraph.json` 更新:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": ["./packages/harness"],
|
||||
"graphs": {
|
||||
"lead_agent": "deerflow.agents:make_lead_agent"
|
||||
},
|
||||
"checkpointer": {
|
||||
"path": "./packages/harness/deerflow/agents/checkpointer/async_provider.py:make_checkpointer"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Gateway API
|
||||
|
||||
```bash
|
||||
# serve.sh / Makefile
|
||||
# PYTHONPATH 包含 backend/ 根目录,使 app.* 和 deerflow.* 都能被找到
|
||||
PYTHONPATH=. uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
```
|
||||
|
||||
### 5.3 Nginx
|
||||
|
||||
无需变更(只做 URL 路由,不涉及 Python 模块路径)。
|
||||
|
||||
### 5.4 Docker
|
||||
|
||||
Dockerfile 中的 module 引用从 `src.` 改为 `deerflow.` / `app.`,`COPY` 命令需覆盖 `packages/` 和 `app/` 目录。
|
||||
|
||||
## 6. 实施计划
|
||||
|
||||
分 3 个 PR 递进执行:
|
||||
|
||||
### PR 1:提取共享工具函数(Low Risk)
|
||||
|
||||
1. 创建 `src/skills/validation.py`,从 `gateway/routers/skills.py` 提取 `_validate_skill_frontmatter`
|
||||
2. 创建 `src/utils/file_conversion.py`,从 `gateway/routers/uploads.py` 提取文件转换逻辑
|
||||
3. 更新 `client.py`、`gateway/routers/skills.py`、`gateway/routers/uploads.py` 的 import
|
||||
4. 运行全部测试确认无回归
|
||||
|
||||
### PR 2:Rename + 物理拆分(High Risk,原子操作)
|
||||
|
||||
1. 创建 `packages/harness/` 目录,创建 `pyproject.toml`
|
||||
2. `git mv` 将 harness 相关模块从 `src/` 移入 `packages/harness/deerflow/`
|
||||
3. `git mv` 将 app 相关模块从 `src/` 移入 `app/`
|
||||
4. 全局替换 import:
|
||||
- harness 模块:`src.*` → `deerflow.*`(所有 `.py` 文件、`langgraph.json`、测试、文档)
|
||||
- app 模块:`src.gateway.*` → `app.gateway.*`、`src.channels.*` → `app.channels.*`
|
||||
5. 更新 workspace root `pyproject.toml`
|
||||
6. 更新 `langgraph.json`、`Makefile`、`Dockerfile`
|
||||
7. `uv sync` + 全部测试 + 手动验证服务启动
|
||||
|
||||
### PR 3:边界检查 + 文档(Low Risk)
|
||||
|
||||
1. 添加 lint 规则:检查 harness 不 import app 模块
|
||||
2. 更新 `CLAUDE.md`、`README.md`
|
||||
|
||||
## 7. 风险与缓解
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| 全局 rename 误伤 | 字符串中的 `src` 被错误替换 | 正则精确匹配 `\bsrc\.`,review diff |
|
||||
| LangGraph Server 找不到模块 | 服务启动失败 | `langgraph.json` 的 `dependencies` 指向正确的 harness 包路径 |
|
||||
| App 的 `PYTHONPATH` 缺失 | Gateway/Channel 启动 import 报错 | Makefile/Docker 统一设置 `PYTHONPATH=.` |
|
||||
| `config.yaml` 中的 `use` 字段引用旧路径 | 运行时模块解析失败 | `config.yaml` 中的 `use` 字段同步更新为 `deerflow.*` |
|
||||
| 测试中 `sys.path` 混乱 | 测试失败 | 用 editable install(`uv sync`)确保 deerflow 可导入,`conftest.py` 中添加 `app/` 到 `sys.path` |
|
||||
|
||||
## 8. 未来演进
|
||||
|
||||
- **独立发布**:harness 可以发布到内部 PyPI,让其他项目直接 `pip install deerflow-harness`
|
||||
- **插件化 App**:不同的 app(web、CLI、bot)可以各自独立,都依赖同一个 harness
|
||||
- **更细粒度拆分**:如果 harness 内部模块继续增长,可以进一步拆分(如 `deerflow-sandbox`、`deerflow-mcp`)
|
||||
@@ -33,6 +33,6 @@ No `current_context` argument is currently available in `main`.
|
||||
|
||||
## Verification Pointers
|
||||
|
||||
- Implementation: `backend/src/agents/memory/prompt.py`
|
||||
- Prompt assembly: `backend/src/agents/lead_agent/prompt.py`
|
||||
- Implementation: `packages/harness/deerflow/agents/memory/prompt.py`
|
||||
- Prompt assembly: `packages/harness/deerflow/agents/lead_agent/prompt.py`
|
||||
- Regression tests: `backend/tests/test_memory_prompt_injection.py`
|
||||
|
||||
@@ -144,7 +144,7 @@ async function uploadAndProcess(threadId: string, file: File) {
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from src.agents.middlewares.thread_data_middleware import THREAD_DATA_BASE_DIR
|
||||
from deerflow.agents.middlewares.thread_data_middleware import THREAD_DATA_BASE_DIR
|
||||
|
||||
def process_uploaded_file(thread_id: str, filename: str):
|
||||
# 使用实际路径
|
||||
|
||||
@@ -30,7 +30,7 @@ DeerFlow uses a YAML configuration file that should be placed in the **project r
|
||||
4. **Verify configuration**:
|
||||
```bash
|
||||
cd backend
|
||||
python -c "from src.config import get_app_config; print('✓ Config loaded:', get_app_config().models[0].name)"
|
||||
python -c "from deerflow.config import get_app_config; print('✓ Config loaded:', get_app_config().models[0].name)"
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
@@ -51,7 +51,7 @@ The backend searches for `config.yaml` in this order:
|
||||
|
||||
## Sandbox Setup (Optional but Recommended)
|
||||
|
||||
If you plan to use Docker/Container-based sandbox (configured in `config.yaml` under `sandbox.use: src.community.aio_sandbox:AioSandboxProvider`), it's highly recommended to pre-pull the container image:
|
||||
If you plan to use Docker/Container-based sandbox (configured in `config.yaml` under `sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider`), it's highly recommended to pre-pull the container image:
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
@@ -72,7 +72,7 @@ If you skip this step, the image will be automatically pulled on first agent exe
|
||||
```bash
|
||||
# Check where the backend is looking
|
||||
cd deer-flow/backend
|
||||
python -c "from src.config.app_config import AppConfig; print(AppConfig.resolve_config_path())"
|
||||
python -c "from deerflow.config.app_config import AppConfig; print(AppConfig.resolve_config_path())"
|
||||
```
|
||||
|
||||
If it can't find the config:
|
||||
|
||||
@@ -4,27 +4,27 @@
|
||||
|
||||
### 1. 核心实现文件
|
||||
|
||||
#### [`src/agents/thread_state.py`](../src/agents/thread_state.py)
|
||||
#### [`packages/harness/deerflow/agents/thread_state.py`](../packages/harness/deerflow/agents/thread_state.py)
|
||||
- ✅ 添加 `title: str | None = None` 字段到 `ThreadState`
|
||||
|
||||
#### [`src/config/title_config.py`](../src/config/title_config.py) (新建)
|
||||
#### [`packages/harness/deerflow/config/title_config.py`](../packages/harness/deerflow/config/title_config.py) (新建)
|
||||
- ✅ 创建 `TitleConfig` 配置类
|
||||
- ✅ 支持配置:enabled, max_words, max_chars, model_name, prompt_template
|
||||
- ✅ 提供 `get_title_config()` 和 `set_title_config()` 函数
|
||||
- ✅ 提供 `load_title_config_from_dict()` 从配置文件加载
|
||||
|
||||
#### [`src/agents/title_middleware.py`](../src/agents/title_middleware.py) (新建)
|
||||
#### [`packages/harness/deerflow/agents/title_middleware.py`](../packages/harness/deerflow/agents/title_middleware.py) (新建)
|
||||
- ✅ 创建 `TitleMiddleware` 类
|
||||
- ✅ 实现 `_should_generate_title()` 检查是否需要生成
|
||||
- ✅ 实现 `_generate_title()` 调用 LLM 生成标题
|
||||
- ✅ 实现 `after_agent()` 钩子,在首次对话后自动触发
|
||||
- ✅ 包含 fallback 策略(LLM 失败时使用用户消息前几个词)
|
||||
|
||||
#### [`src/config/app_config.py`](../src/config/app_config.py)
|
||||
#### [`packages/harness/deerflow/config/app_config.py`](../packages/harness/deerflow/config/app_config.py)
|
||||
- ✅ 导入 `load_title_config_from_dict`
|
||||
- ✅ 在 `from_file()` 中加载 title 配置
|
||||
|
||||
#### [`src/agents/lead_agent/agent.py`](../src/agents/lead_agent/agent.py)
|
||||
#### [`packages/harness/deerflow/agents/lead_agent/agent.py`](../packages/harness/deerflow/agents/lead_agent/agent.py)
|
||||
- ✅ 导入 `TitleMiddleware`
|
||||
- ✅ 注册到 `middleware` 列表:`[SandboxMiddleware(), TitleMiddleware()]`
|
||||
|
||||
@@ -131,7 +131,7 @@ checkpointer = SqliteSaver.from_conn_string("checkpoints.db")
|
||||
// langgraph.json
|
||||
{
|
||||
"graphs": {
|
||||
"lead_agent": "src.agents:lead_agent"
|
||||
"lead_agent": "deerflow.agents:lead_agent"
|
||||
},
|
||||
"checkpointer": "checkpointer:checkpointer"
|
||||
}
|
||||
|
||||
@@ -21,8 +21,8 @@
|
||||
- [ ] Support for more document formats in upload
|
||||
- [ ] Skill marketplace / remote skill installation
|
||||
- [ ] Optimize async concurrency in agent hot path (IM channels multi-task scenario)
|
||||
- Replace `time.sleep(5)` with `asyncio.sleep()` in `src/tools/builtins/task_tool.py` (subagent polling)
|
||||
- Replace `subprocess.run()` with `asyncio.create_subprocess_shell()` in `src/sandbox/local/local_sandbox.py`
|
||||
- Replace `time.sleep(5)` with `asyncio.sleep()` in `packages/harness/deerflow/tools/builtins/task_tool.py` (subagent polling)
|
||||
- Replace `subprocess.run()` with `asyncio.create_subprocess_shell()` in `packages/harness/deerflow/sandbox/local/local_sandbox.py`
|
||||
- Replace sync `requests` with `httpx.AsyncClient` in community tools (tavily, jina_ai, firecrawl, infoquest, image_search)
|
||||
- Replace sync `model.invoke()` with async `model.ainvoke()` in title_middleware and memory updater
|
||||
- Consider `asyncio.to_thread()` wrapper for remaining blocking file I/O
|
||||
|
||||
@@ -19,7 +19,7 @@ Plan mode is controlled via **runtime configuration** through the `is_plan_mode`
|
||||
|
||||
```python
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from src.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
|
||||
# Enable plan mode via runtime configuration
|
||||
config = RunnableConfig(
|
||||
@@ -72,7 +72,7 @@ The agent will skip using the todo list for:
|
||||
|
||||
```python
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from src.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
|
||||
# Create agent with plan mode ENABLED
|
||||
config_with_plan_mode = RunnableConfig(
|
||||
@@ -101,7 +101,7 @@ You can enable/disable plan mode dynamically for different conversations or task
|
||||
|
||||
```python
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from src.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
|
||||
def create_agent_for_task(task_complexity: str):
|
||||
"""Create agent with plan mode based on task complexity."""
|
||||
@@ -154,7 +154,7 @@ make_lead_agent(config)
|
||||
## Implementation Details
|
||||
|
||||
### Agent Module
|
||||
- **Location**: `src/agents/lead_agent/agent.py`
|
||||
- **Location**: `packages/harness/deerflow/agents/lead_agent/agent.py`
|
||||
- **Function**: `_create_todo_list_middleware(is_plan_mode: bool)` - Creates TodoListMiddleware if plan mode is enabled
|
||||
- **Function**: `_build_middlewares(config: RunnableConfig)` - Builds middleware chain based on runtime config
|
||||
- **Function**: `make_lead_agent(config: RunnableConfig)` - Creates agent with appropriate middlewares
|
||||
@@ -194,7 +194,7 @@ DeerFlow uses custom `system_prompt` and `tool_description` for the TodoListMidd
|
||||
- Comprehensive best practices section
|
||||
- Task completion requirements to prevent premature marking
|
||||
|
||||
The custom prompts are defined in `_create_todo_list_middleware()` in `/Users/hetao/workspace/deer-flow/backend/src/agents/lead_agent/agent.py:57`.
|
||||
The custom prompts are defined in `_create_todo_list_middleware()` in `/Users/hetao/workspace/deer-flow/backend/packages/harness/deerflow/agents/lead_agent/agent.py:57`.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -269,8 +269,8 @@ The middleware intelligently preserves message context:
|
||||
|
||||
### Code Structure
|
||||
|
||||
- **Configuration**: `src/config/summarization_config.py`
|
||||
- **Integration**: `src/agents/lead_agent/agent.py`
|
||||
- **Configuration**: `packages/harness/deerflow/config/summarization_config.py`
|
||||
- **Integration**: `packages/harness/deerflow/agents/lead_agent/agent.py`
|
||||
- **Middleware**: Uses `langchain.agents.middleware.SummarizationMiddleware`
|
||||
|
||||
### Middleware Order
|
||||
|
||||
@@ -65,7 +65,7 @@ The `task_status_tool` is no longer exposed to the LLM. It's kept in the codebas
|
||||
|
||||
### Polling Logic
|
||||
|
||||
Located in `src/tools/builtins/task_tool.py`:
|
||||
Located in `packages/harness/deerflow/tools/builtins/task_tool.py`:
|
||||
|
||||
```python
|
||||
# Start background execution
|
||||
@@ -93,7 +93,7 @@ while True:
|
||||
|
||||
In addition to polling timeout, subagent execution now has a built-in timeout mechanism:
|
||||
|
||||
**Configuration** (`src/subagents/config.py`):
|
||||
**Configuration** (`packages/harness/deerflow/subagents/config.py`):
|
||||
```python
|
||||
@dataclass
|
||||
class SubagentConfig:
|
||||
|
||||
@@ -6,9 +6,9 @@
|
||||
],
|
||||
"env": ".env",
|
||||
"graphs": {
|
||||
"lead_agent": "src.agents:make_lead_agent"
|
||||
"lead_agent": "deerflow.agents:make_lead_agent"
|
||||
},
|
||||
"checkpointer": {
|
||||
"path": "./src/agents/checkpointer/async_provider.py:make_checkpointer"
|
||||
"path": "./packages/harness/deerflow/agents/checkpointer/async_provider.py:make_checkpointer"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
0
backend/packages/harness/deerflow/__init__.py
Normal file
0
backend/packages/harness/deerflow/__init__.py
Normal file
@@ -7,12 +7,12 @@ Supported backends: memory, sqlite, postgres.
|
||||
|
||||
Usage (e.g. FastAPI lifespan)::
|
||||
|
||||
from src.agents.checkpointer.async_provider import make_checkpointer
|
||||
from deerflow.agents.checkpointer.async_provider import make_checkpointer
|
||||
|
||||
async with make_checkpointer() as checkpointer:
|
||||
app.state.checkpointer = checkpointer # InMemorySaver if not configured
|
||||
|
||||
For sync usage see :mod:`src.agents.checkpointer.provider`.
|
||||
For sync usage see :mod:`deerflow.agents.checkpointer.provider`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -23,13 +23,13 @@ from collections.abc import AsyncIterator
|
||||
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from src.agents.checkpointer.provider import (
|
||||
from deerflow.agents.checkpointer.provider import (
|
||||
POSTGRES_CONN_REQUIRED,
|
||||
POSTGRES_INSTALL,
|
||||
SQLITE_INSTALL,
|
||||
_resolve_sqlite_conn_str,
|
||||
)
|
||||
from src.config.app_config import get_app_config
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -7,7 +7,7 @@ Supported backends: memory, sqlite, postgres.
|
||||
|
||||
Usage::
|
||||
|
||||
from src.agents.checkpointer.provider import get_checkpointer, checkpointer_context
|
||||
from deerflow.agents.checkpointer.provider import get_checkpointer, checkpointer_context
|
||||
|
||||
# Singleton — reused across calls, closed on process exit
|
||||
cp = get_checkpointer()
|
||||
@@ -25,9 +25,9 @@ from collections.abc import Iterator
|
||||
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from src.config.app_config import get_app_config
|
||||
from src.config.checkpointer_config import CheckpointerConfig
|
||||
from src.config.paths import resolve_path
|
||||
from deerflow.config.app_config import get_app_config
|
||||
from deerflow.config.checkpointer_config import CheckpointerConfig
|
||||
from deerflow.config.paths import resolve_path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -128,8 +128,8 @@ def get_checkpointer() -> Checkpointer:
|
||||
# Ensure app config is loaded before checking checkpointer config
|
||||
# This prevents returning InMemorySaver when config.yaml actually has a checkpointer section
|
||||
# but hasn't been loaded yet
|
||||
from src.config.app_config import _app_config
|
||||
from src.config.checkpointer_config import get_checkpointer_config
|
||||
from deerflow.config.app_config import _app_config
|
||||
from deerflow.config.checkpointer_config import get_checkpointer_config
|
||||
|
||||
if _app_config is None:
|
||||
# Only load config if it hasn't been initialized yet
|
||||
@@ -4,20 +4,20 @@ from langchain.agents import create_agent
|
||||
from langchain.agents.middleware import SummarizationMiddleware
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
from src.agents.lead_agent.prompt import apply_prompt_template
|
||||
from src.agents.middlewares.clarification_middleware import ClarificationMiddleware
|
||||
from src.agents.middlewares.loop_detection_middleware import LoopDetectionMiddleware
|
||||
from src.agents.middlewares.memory_middleware import MemoryMiddleware
|
||||
from src.agents.middlewares.subagent_limit_middleware import SubagentLimitMiddleware
|
||||
from src.agents.middlewares.title_middleware import TitleMiddleware
|
||||
from src.agents.middlewares.todo_middleware import TodoMiddleware
|
||||
from src.agents.middlewares.tool_error_handling_middleware import build_lead_runtime_middlewares
|
||||
from src.agents.middlewares.view_image_middleware import ViewImageMiddleware
|
||||
from src.agents.thread_state import ThreadState
|
||||
from src.config.agents_config import load_agent_config
|
||||
from src.config.app_config import get_app_config
|
||||
from src.config.summarization_config import get_summarization_config
|
||||
from src.models import create_chat_model
|
||||
from deerflow.agents.lead_agent.prompt import apply_prompt_template
|
||||
from deerflow.agents.middlewares.clarification_middleware import ClarificationMiddleware
|
||||
from deerflow.agents.middlewares.loop_detection_middleware import LoopDetectionMiddleware
|
||||
from deerflow.agents.middlewares.memory_middleware import MemoryMiddleware
|
||||
from deerflow.agents.middlewares.subagent_limit_middleware import SubagentLimitMiddleware
|
||||
from deerflow.agents.middlewares.title_middleware import TitleMiddleware
|
||||
from deerflow.agents.middlewares.todo_middleware import TodoMiddleware
|
||||
from deerflow.agents.middlewares.tool_error_handling_middleware import build_lead_runtime_middlewares
|
||||
from deerflow.agents.middlewares.view_image_middleware import ViewImageMiddleware
|
||||
from deerflow.agents.thread_state import ThreadState
|
||||
from deerflow.config.agents_config import load_agent_config
|
||||
from deerflow.config.app_config import get_app_config
|
||||
from deerflow.config.summarization_config import get_summarization_config
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -256,8 +256,8 @@ def _build_middlewares(config: RunnableConfig, model_name: str | None, agent_nam
|
||||
|
||||
def make_lead_agent(config: RunnableConfig):
|
||||
# Lazy import to avoid circular dependency
|
||||
from src.tools import get_available_tools
|
||||
from src.tools.builtins import setup_agent
|
||||
from deerflow.tools import get_available_tools
|
||||
from deerflow.tools.builtins import setup_agent
|
||||
|
||||
cfg = config.get("configurable", {})
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import datetime
|
||||
|
||||
from src.config.agents_config import load_agent_soul
|
||||
from src.skills import load_skills
|
||||
from deerflow.config.agents_config import load_agent_soul
|
||||
from deerflow.skills import load_skills
|
||||
|
||||
|
||||
def _build_subagent_section(max_concurrent: int) -> str:
|
||||
@@ -292,8 +292,8 @@ def _get_memory_context(agent_name: str | None = None) -> str:
|
||||
Formatted memory context string wrapped in XML tags, or empty string if disabled.
|
||||
"""
|
||||
try:
|
||||
from src.agents.memory import format_memory_for_injection, get_memory_data
|
||||
from src.config.memory_config import get_memory_config
|
||||
from deerflow.agents.memory import format_memory_for_injection, get_memory_data
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
|
||||
config = get_memory_config()
|
||||
if not config.enabled or not config.injection_enabled:
|
||||
@@ -323,7 +323,7 @@ def get_skills_prompt_section(available_skills: set[str] | None = None) -> str:
|
||||
skills = load_skills(enabled_only=True)
|
||||
|
||||
try:
|
||||
from src.config import get_app_config
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
config = get_app_config()
|
||||
container_base_path = config.skills.container_path
|
||||
@@ -6,19 +6,19 @@ This module provides a global memory mechanism that:
|
||||
- Injects relevant memory into system prompts for personalized responses
|
||||
"""
|
||||
|
||||
from src.agents.memory.prompt import (
|
||||
from deerflow.agents.memory.prompt import (
|
||||
FACT_EXTRACTION_PROMPT,
|
||||
MEMORY_UPDATE_PROMPT,
|
||||
format_conversation_for_update,
|
||||
format_memory_for_injection,
|
||||
)
|
||||
from src.agents.memory.queue import (
|
||||
from deerflow.agents.memory.queue import (
|
||||
ConversationContext,
|
||||
MemoryUpdateQueue,
|
||||
get_memory_queue,
|
||||
reset_memory_queue,
|
||||
)
|
||||
from src.agents.memory.updater import (
|
||||
from deerflow.agents.memory.updater import (
|
||||
MemoryUpdater,
|
||||
get_memory_data,
|
||||
reload_memory_data,
|
||||
@@ -6,7 +6,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from src.config.memory_config import get_memory_config
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -84,7 +84,7 @@ class MemoryUpdateQueue:
|
||||
def _process_queue(self) -> None:
|
||||
"""Process all queued conversation contexts."""
|
||||
# Import here to avoid circular dependency
|
||||
from src.agents.memory.updater import MemoryUpdater
|
||||
from deerflow.agents.memory.updater import MemoryUpdater
|
||||
|
||||
with self._lock:
|
||||
if self._processing:
|
||||
@@ -7,13 +7,13 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from src.agents.memory.prompt import (
|
||||
from deerflow.agents.memory.prompt import (
|
||||
MEMORY_UPDATE_PROMPT,
|
||||
format_conversation_for_update,
|
||||
)
|
||||
from src.config.memory_config import get_memory_config
|
||||
from src.config.paths import get_paths
|
||||
from src.models import create_chat_model
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
|
||||
def _get_memory_file_path(agent_name: str | None = None) -> Path:
|
||||
@@ -7,8 +7,8 @@ from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.agents.memory.queue import get_memory_queue
|
||||
from src.config.memory_config import get_memory_config
|
||||
from deerflow.agents.memory.queue import get_memory_queue
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
|
||||
|
||||
class MemoryMiddlewareState(AgentState):
|
||||
@@ -7,7 +7,7 @@ from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.subagents.executor import MAX_CONCURRENT_SUBAGENTS
|
||||
from deerflow.subagents.executor import MAX_CONCURRENT_SUBAGENTS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -4,8 +4,8 @@ from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.agents.thread_state import ThreadDataState
|
||||
from src.config.paths import Paths, get_paths
|
||||
from deerflow.agents.thread_state import ThreadDataState
|
||||
from deerflow.config.paths import Paths, get_paths
|
||||
|
||||
|
||||
class ThreadDataMiddlewareState(AgentState):
|
||||
@@ -6,8 +6,8 @@ from langchain.agents import AgentState
|
||||
from langchain.agents.middleware import AgentMiddleware
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.config.title_config import get_title_config
|
||||
from src.models import create_chat_model
|
||||
from deerflow.config.title_config import get_title_config
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
|
||||
class TitleMiddlewareState(AgentState):
|
||||
@@ -26,10 +26,7 @@ class ToolErrorHandlingMiddleware(AgentMiddleware[AgentState]):
|
||||
if len(detail) > 500:
|
||||
detail = detail[:497] + "..."
|
||||
|
||||
content = (
|
||||
f"Error: Tool '{tool_name}' failed with {exc.__class__.__name__}: {detail}. "
|
||||
"Continue with available context, or choose an alternative tool."
|
||||
)
|
||||
content = f"Error: Tool '{tool_name}' failed with {exc.__class__.__name__}: {detail}. Continue with available context, or choose an alternative tool."
|
||||
return ToolMessage(
|
||||
content=content,
|
||||
tool_call_id=tool_call_id,
|
||||
@@ -75,8 +72,8 @@ def _build_runtime_middlewares(
|
||||
lazy_init: bool = True,
|
||||
) -> list[AgentMiddleware]:
|
||||
"""Build shared base middlewares for agent execution."""
|
||||
from src.agents.middlewares.thread_data_middleware import ThreadDataMiddleware
|
||||
from src.sandbox.middleware import SandboxMiddleware
|
||||
from deerflow.agents.middlewares.thread_data_middleware import ThreadDataMiddleware
|
||||
from deerflow.sandbox.middleware import SandboxMiddleware
|
||||
|
||||
middlewares: list[AgentMiddleware] = [
|
||||
ThreadDataMiddleware(lazy_init=lazy_init),
|
||||
@@ -84,12 +81,12 @@ def _build_runtime_middlewares(
|
||||
]
|
||||
|
||||
if include_uploads:
|
||||
from src.agents.middlewares.uploads_middleware import UploadsMiddleware
|
||||
from deerflow.agents.middlewares.uploads_middleware import UploadsMiddleware
|
||||
|
||||
middlewares.insert(1, UploadsMiddleware())
|
||||
|
||||
if include_dangling_tool_call_patch:
|
||||
from src.agents.middlewares.dangling_tool_call_middleware import DanglingToolCallMiddleware
|
||||
from deerflow.agents.middlewares.dangling_tool_call_middleware import DanglingToolCallMiddleware
|
||||
|
||||
middlewares.append(DanglingToolCallMiddleware())
|
||||
|
||||
@@ -9,7 +9,7 @@ from langchain.agents.middleware import AgentMiddleware
|
||||
from langchain_core.messages import HumanMessage
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.config.paths import Paths, get_paths
|
||||
from deerflow.config.paths import Paths, get_paths
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -7,7 +7,7 @@ from langchain.agents.middleware import AgentMiddleware
|
||||
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from src.agents.thread_state import ViewedImageData
|
||||
from deerflow.agents.thread_state import ViewedImageData
|
||||
|
||||
|
||||
class ViewImageMiddlewareState(AgentState):
|
||||
@@ -4,7 +4,7 @@ Provides direct programmatic access to DeerFlow's agent capabilities
|
||||
without requiring LangGraph Server or Gateway API processes.
|
||||
|
||||
Usage:
|
||||
from src.client import DeerFlowClient
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
@@ -34,13 +34,13 @@ from langchain.agents import create_agent
|
||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
from src.agents.lead_agent.agent import _build_middlewares
|
||||
from src.agents.lead_agent.prompt import apply_prompt_template
|
||||
from src.agents.thread_state import ThreadState
|
||||
from src.config.app_config import get_app_config, reload_app_config
|
||||
from src.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from src.config.paths import get_paths
|
||||
from src.models import create_chat_model
|
||||
from deerflow.agents.lead_agent.agent import _build_middlewares
|
||||
from deerflow.agents.lead_agent.prompt import apply_prompt_template
|
||||
from deerflow.agents.thread_state import ThreadState
|
||||
from deerflow.config.app_config import get_app_config, reload_app_config
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -81,7 +81,7 @@ class DeerFlowClient:
|
||||
|
||||
Example::
|
||||
|
||||
from src.client import DeerFlowClient
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
@@ -211,7 +211,7 @@ class DeerFlowClient:
|
||||
}
|
||||
checkpointer = self._checkpointer
|
||||
if checkpointer is None:
|
||||
from src.agents.checkpointer import get_checkpointer
|
||||
from deerflow.agents.checkpointer import get_checkpointer
|
||||
|
||||
checkpointer = get_checkpointer()
|
||||
if checkpointer is not None:
|
||||
@@ -224,7 +224,7 @@ class DeerFlowClient:
|
||||
@staticmethod
|
||||
def _get_tools(*, model_name: str | None, subagent_enabled: bool):
|
||||
"""Lazy import to avoid circular dependency at module level."""
|
||||
from src.tools import get_available_tools
|
||||
from deerflow.tools import get_available_tools
|
||||
|
||||
return get_available_tools(model_name=model_name, subagent_enabled=subagent_enabled)
|
||||
|
||||
@@ -422,7 +422,7 @@ class DeerFlowClient:
|
||||
Dict with "skills" key containing list of skill info dicts,
|
||||
matching the Gateway API ``SkillsListResponse`` schema.
|
||||
"""
|
||||
from src.skills.loader import load_skills
|
||||
from deerflow.skills.loader import load_skills
|
||||
|
||||
return {
|
||||
"skills": [
|
||||
@@ -443,7 +443,7 @@ class DeerFlowClient:
|
||||
Returns:
|
||||
Memory data dict (see src/agents/memory/updater.py for structure).
|
||||
"""
|
||||
from src.agents.memory.updater import get_memory_data
|
||||
from deerflow.agents.memory.updater import get_memory_data
|
||||
|
||||
return get_memory_data()
|
||||
|
||||
@@ -528,7 +528,7 @@ class DeerFlowClient:
|
||||
Returns:
|
||||
Skill info dict, or None if not found.
|
||||
"""
|
||||
from src.skills.loader import load_skills
|
||||
from deerflow.skills.loader import load_skills
|
||||
|
||||
skill = next((s for s in load_skills(enabled_only=False) if s.name == name), None)
|
||||
if skill is None:
|
||||
@@ -555,7 +555,7 @@ class DeerFlowClient:
|
||||
ValueError: If the skill is not found.
|
||||
OSError: If the config file cannot be written.
|
||||
"""
|
||||
from src.skills.loader import load_skills
|
||||
from deerflow.skills.loader import load_skills
|
||||
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == name), None)
|
||||
@@ -603,8 +603,8 @@ class DeerFlowClient:
|
||||
FileNotFoundError: If the file does not exist.
|
||||
ValueError: If the file is invalid.
|
||||
"""
|
||||
from src.gateway.routers.skills import _validate_skill_frontmatter
|
||||
from src.skills.loader import get_skills_root_path
|
||||
from deerflow.skills.loader import get_skills_root_path
|
||||
from deerflow.skills.validation import _validate_skill_frontmatter
|
||||
|
||||
path = Path(skill_path)
|
||||
if not path.exists():
|
||||
@@ -664,7 +664,7 @@ class DeerFlowClient:
|
||||
Returns:
|
||||
The reloaded memory data dict.
|
||||
"""
|
||||
from src.agents.memory.updater import reload_memory_data
|
||||
from deerflow.agents.memory.updater import reload_memory_data
|
||||
|
||||
return reload_memory_data()
|
||||
|
||||
@@ -674,7 +674,7 @@ class DeerFlowClient:
|
||||
Returns:
|
||||
Memory config dict.
|
||||
"""
|
||||
from src.config.memory_config import get_memory_config
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
|
||||
config = get_memory_config()
|
||||
return {
|
||||
@@ -726,7 +726,7 @@ class DeerFlowClient:
|
||||
FileNotFoundError: If any file does not exist.
|
||||
ValueError: If any supplied path exists but is not a regular file.
|
||||
"""
|
||||
from src.gateway.routers.uploads import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
|
||||
# Validate all files upfront to avoid partial uploads.
|
||||
resolved_files = []
|
||||
@@ -3,7 +3,7 @@ import logging
|
||||
|
||||
from agent_sandbox import Sandbox as AioSandboxClient
|
||||
|
||||
from src.sandbox.sandbox import Sandbox
|
||||
from deerflow.sandbox.sandbox import Sandbox
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -20,10 +20,10 @@ import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from src.config import get_app_config
|
||||
from src.config.paths import VIRTUAL_PATH_PREFIX, Paths, get_paths
|
||||
from src.sandbox.sandbox import Sandbox
|
||||
from src.sandbox.sandbox_provider import SandboxProvider
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, Paths, get_paths
|
||||
from deerflow.sandbox.sandbox import Sandbox
|
||||
from deerflow.sandbox.sandbox_provider import SandboxProvider
|
||||
|
||||
from .aio_sandbox import AioSandbox
|
||||
from .backend import SandboxBackend, wait_for_sandbox_ready
|
||||
@@ -51,7 +51,7 @@ class AioSandboxProvider(SandboxProvider):
|
||||
- Remote/K8s mode (connect to pre-existing sandbox URL)
|
||||
|
||||
Configuration options in config.yaml under sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider
|
||||
image: <container image>
|
||||
port: 8080 # Base port for local containers
|
||||
container_prefix: deer-flow-sandbox
|
||||
@@ -10,7 +10,7 @@ import logging
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
from src.utils.network import get_free_port, release_port
|
||||
from deerflow.utils.network import get_free_port, release_port
|
||||
|
||||
from .backend import SandboxBackend, wait_for_sandbox_ready
|
||||
from .sandbox_info import SandboxInfo
|
||||
@@ -36,7 +36,7 @@ class RemoteSandboxBackend(SandboxBackend):
|
||||
Typical config.yaml::
|
||||
|
||||
sandbox:
|
||||
use: src.community.aio_sandbox:AioSandboxProvider
|
||||
use: deerflow.community.aio_sandbox:AioSandboxProvider
|
||||
provisioner_url: http://provisioner:8002
|
||||
"""
|
||||
|
||||
@@ -3,7 +3,7 @@ import json
|
||||
from firecrawl import FirecrawlApp
|
||||
from langchain.tools import tool
|
||||
|
||||
from src.config import get_app_config
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
|
||||
def _get_firecrawl_client() -> FirecrawlApp:
|
||||
@@ -7,7 +7,7 @@ import logging
|
||||
|
||||
from langchain.tools import tool
|
||||
|
||||
from src.config import get_app_config
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from langchain.tools import tool
|
||||
|
||||
from src.config import get_app_config
|
||||
from src.utils.readability import ReadabilityExtractor
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.utils.readability import ReadabilityExtractor
|
||||
|
||||
from .infoquest_client import InfoQuestClient
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
from langchain.tools import tool
|
||||
|
||||
from src.community.jina_ai.jina_client import JinaClient
|
||||
from src.config import get_app_config
|
||||
from src.utils.readability import ReadabilityExtractor
|
||||
from deerflow.community.jina_ai.jina_client import JinaClient
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.utils.readability import ReadabilityExtractor
|
||||
|
||||
readability_extractor = ReadabilityExtractor()
|
||||
|
||||
@@ -3,7 +3,7 @@ import json
|
||||
from langchain.tools import tool
|
||||
from tavily import TavilyClient
|
||||
|
||||
from src.config import get_app_config
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
|
||||
def _get_tavily_client() -> TavilyClient:
|
||||
@@ -7,7 +7,7 @@ from typing import Any
|
||||
import yaml
|
||||
from pydantic import BaseModel
|
||||
|
||||
from src.config.paths import get_paths
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Self
|
||||
@@ -6,19 +7,21 @@ import yaml
|
||||
from dotenv import load_dotenv
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from src.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
|
||||
from src.config.extensions_config import ExtensionsConfig
|
||||
from src.config.memory_config import load_memory_config_from_dict
|
||||
from src.config.model_config import ModelConfig
|
||||
from src.config.sandbox_config import SandboxConfig
|
||||
from src.config.skills_config import SkillsConfig
|
||||
from src.config.subagents_config import load_subagents_config_from_dict
|
||||
from src.config.summarization_config import load_summarization_config_from_dict
|
||||
from src.config.title_config import load_title_config_from_dict
|
||||
from src.config.tool_config import ToolConfig, ToolGroupConfig
|
||||
from deerflow.config.checkpointer_config import CheckpointerConfig, load_checkpointer_config_from_dict
|
||||
from deerflow.config.extensions_config import ExtensionsConfig
|
||||
from deerflow.config.memory_config import load_memory_config_from_dict
|
||||
from deerflow.config.model_config import ModelConfig
|
||||
from deerflow.config.sandbox_config import SandboxConfig
|
||||
from deerflow.config.skills_config import SkillsConfig
|
||||
from deerflow.config.subagents_config import load_subagents_config_from_dict
|
||||
from deerflow.config.summarization_config import load_summarization_config_from_dict
|
||||
from deerflow.config.title_config import load_title_config_from_dict
|
||||
from deerflow.config.tool_config import ToolConfig, ToolGroupConfig
|
||||
|
||||
load_dotenv()
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AppConfig(BaseModel):
|
||||
"""Config for the DeerFlow application"""
|
||||
@@ -75,7 +78,11 @@ class AppConfig(BaseModel):
|
||||
"""
|
||||
resolved_path = cls.resolve_config_path(config_path)
|
||||
with open(resolved_path, encoding="utf-8") as f:
|
||||
config_data = yaml.safe_load(f)
|
||||
config_data = yaml.safe_load(f) or {}
|
||||
|
||||
# Check config version before processing
|
||||
cls._check_config_version(config_data, resolved_path)
|
||||
|
||||
config_data = cls.resolve_env_variables(config_data)
|
||||
|
||||
# Load title config if present
|
||||
@@ -105,6 +112,52 @@ class AppConfig(BaseModel):
|
||||
result = cls.model_validate(config_data)
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def _check_config_version(cls, config_data: dict, config_path: Path) -> None:
|
||||
"""Check if the user's config.yaml is outdated compared to config.example.yaml.
|
||||
|
||||
Emits a warning if the user's config_version is lower than the example's.
|
||||
Missing config_version is treated as version 0 (pre-versioning).
|
||||
"""
|
||||
try:
|
||||
user_version = int(config_data.get("config_version", 0))
|
||||
except (TypeError, ValueError):
|
||||
user_version = 0
|
||||
|
||||
# Find config.example.yaml by searching config.yaml's directory and its parents
|
||||
example_path = None
|
||||
search_dir = config_path.parent
|
||||
for _ in range(5): # search up to 5 levels
|
||||
candidate = search_dir / "config.example.yaml"
|
||||
if candidate.exists():
|
||||
example_path = candidate
|
||||
break
|
||||
parent = search_dir.parent
|
||||
if parent == search_dir:
|
||||
break
|
||||
search_dir = parent
|
||||
if example_path is None:
|
||||
return
|
||||
|
||||
try:
|
||||
with open(example_path, encoding="utf-8") as f:
|
||||
example_data = yaml.safe_load(f)
|
||||
raw = example_data.get("config_version", 0) if example_data else 0
|
||||
try:
|
||||
example_version = int(raw)
|
||||
except (TypeError, ValueError):
|
||||
example_version = 0
|
||||
except Exception:
|
||||
return
|
||||
|
||||
if user_version < example_version:
|
||||
logger.warning(
|
||||
"Your config.yaml (version %d) is outdated — the latest version is %d. "
|
||||
"Run `make config-upgrade` to merge new fields into your config.",
|
||||
user_version,
|
||||
example_version,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def resolve_env_variables(cls, config: Any) -> Any:
|
||||
"""Recursively resolve environment variables in the config.
|
||||
@@ -27,7 +27,7 @@ class SandboxConfig(BaseModel):
|
||||
|
||||
use: str = Field(
|
||||
...,
|
||||
description="Class path of the sandbox provider (e.g. src.sandbox.local:LocalSandboxProvider)",
|
||||
description="Class path of the sandbox provider (e.g. deerflow.sandbox.local:LocalSandboxProvider)",
|
||||
)
|
||||
image: str | None = Field(
|
||||
default=None,
|
||||
@@ -31,7 +31,7 @@ class SkillsConfig(BaseModel):
|
||||
return path.resolve()
|
||||
else:
|
||||
# Default: ../skills relative to backend directory
|
||||
from src.skills.loader import get_skills_root_path
|
||||
from deerflow.skills.loader import get_skills_root_path
|
||||
|
||||
return get_skills_root_path()
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user