Files
deer-flow/docker/docker-compose-dev.yaml
DanielWalnut 76803b826f refactor: split backend into harness (deerflow.*) and app (app.*) (#1131)
* refactor: extract shared utils to break harness→app cross-layer imports

Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: split backend/src into harness (deerflow.*) and app (app.*)

Physically split the monolithic backend/src/ package into two layers:

- **Harness** (`packages/harness/deerflow/`): publishable agent framework
  package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
  models, MCP, skills, config, and all core infrastructure.

- **App** (`app/`): unpublished application code with import prefix `app.*`.
  Contains gateway (FastAPI REST API) and channels (IM integrations).

Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure

Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: add harness→app boundary check test and update docs

Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.

Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add config versioning with auto-upgrade on startup

When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.

- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix comments

* fix: update src.* import in test_sandbox_tools_security to deerflow.*

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle empty config and search parent dirs for config.example.yaml

Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
  looking next to config.yaml, fixing detection in common setups

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: correct skills root path depth and config_version type coercion

- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
  after harness split, file lives at packages/harness/deerflow/skills/
  so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
  _check_config_version() to prevent TypeError when YAML stores value
  as string (e.g. config_version: "1")
- tests: add regression tests for both fixes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update test imports from src.* to deerflow.*/app.* after harness refactor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 22:55:52 +08:00

190 lines
7.0 KiB
YAML

# DeerFlow Development Environment
# Usage: docker-compose -f docker-compose-dev.yaml up --build
#
# Services:
# - nginx: Reverse proxy (port 2026)
# - frontend: Frontend Next.js dev server (port 3000)
# - gateway: Backend Gateway API (port 8001)
# - langgraph: LangGraph server (port 2024)
# - provisioner (optional): Sandbox provisioner (creates Pods in host Kubernetes)
#
# Prerequisites:
# - Kubernetes cluster + kubeconfig are only required when using provisioner mode.
#
# Access: http://localhost:2026
services:
# ── Sandbox Provisioner ────────────────────────────────────────────────
# Manages per-sandbox Pod + Service lifecycle in the host Kubernetes
# cluster via the K8s API.
# Backend accesses sandboxes directly via host.docker.internal:{NodePort}.
provisioner:
profiles:
- provisioner
build:
context: ./provisioner
dockerfile: Dockerfile
container_name: deer-flow-provisioner
volumes:
- ~/.kube/config:/root/.kube/config:ro
environment:
- K8S_NAMESPACE=deer-flow
- SANDBOX_IMAGE=enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest
# Host paths for K8s HostPath volumes (must be absolute paths accessible by K8s node)
# On Docker Desktop/OrbStack, use your actual host paths like /Users/username/...
# Set these in your shell before running docker-compose:
# export DEER_FLOW_ROOT=/absolute/path/to/deer-flow
- SKILLS_HOST_PATH=${DEER_FLOW_ROOT}/skills
- THREADS_HOST_PATH=${DEER_FLOW_ROOT}/backend/.deer-flow/threads
- KUBECONFIG_PATH=/root/.kube/config
- NODE_HOST=host.docker.internal
# Override K8S API server URL since kubeconfig uses 127.0.0.1
# which is unreachable from inside the container
- K8S_API_SERVER=https://host.docker.internal:26443
env_file:
- ../.env
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- deer-flow-dev
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8002/health"]
interval: 10s
timeout: 5s
retries: 6
start_period: 15s
# ── Reverse Proxy ──────────────────────────────────────────────────────
# Routes API traffic to gateway/langgraph and (optionally) provisioner.
# Select nginx config via NGINX_CONF:
# - nginx.local.conf (default): no provisioner route (local/aio modes)
# - nginx.conf: includes provisioner route (provisioner mode)
nginx:
image: nginx:alpine
container_name: deer-flow-nginx
ports:
- "2026:2026"
volumes:
- ./nginx/${NGINX_CONF:-nginx.conf}:/etc/nginx/nginx.conf:ro
depends_on:
- frontend
- gateway
- langgraph
networks:
- deer-flow-dev
restart: unless-stopped
# Frontend - Next.js Development Server
frontend:
build:
context: ../
dockerfile: frontend/Dockerfile
target: dev
args:
PNPM_STORE_PATH: ${PNPM_STORE_PATH:-/root/.local/share/pnpm/store}
container_name: deer-flow-frontend
command: sh -c "cd frontend && pnpm run dev > /app/logs/frontend.log 2>&1"
volumes:
- ../frontend/src:/app/frontend/src
- ../frontend/public:/app/frontend/public
- ../frontend/next.config.js:/app/frontend/next.config.js:ro
- ../logs:/app/logs
# Mount pnpm store for caching
- ${PNPM_STORE_PATH:-~/.local/share/pnpm/store}:/root/.local/share/pnpm/store
working_dir: /app
environment:
- NODE_ENV=development
- WATCHPACK_POLLING=true
- CI=true
env_file:
- ../frontend/.env
networks:
- deer-flow-dev
restart: unless-stopped
# Backend - Gateway API
gateway:
build:
context: ../
dockerfile: backend/Dockerfile
# cache_from disabled - requires manual setup: mkdir -p /tmp/docker-cache-gateway
container_name: deer-flow-gateway
command: sh -c "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload --reload-include='*.yaml .env' > /app/logs/gateway.log 2>&1"
volumes:
- ../backend/:/app/backend/
# Preserve the .venv built during Docker image build — mounting the full backend/
# directory above would otherwise shadow it with the (empty) host directory.
- gateway-venv:/app/backend/.venv
- ../config.yaml:/app/config.yaml
- ../extensions_config.json:/app/extensions_config.json
- ../skills:/app/skills
- ../logs:/app/logs
# Mount uv cache for faster dependency installation
- ~/.cache/uv:/root/.cache/uv
# DooD: same as gateway — AioSandboxProvider runs inside LangGraph process.
- /var/run/docker.sock:/var/run/docker.sock
working_dir: /app
environment:
- CI=true
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_ROOT}/skills
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
env_file:
- ../.env
extra_hosts:
# For Linux: map host.docker.internal to host gateway
- "host.docker.internal:host-gateway"
networks:
- deer-flow-dev
restart: unless-stopped
# Backend - LangGraph Server
langgraph:
build:
context: ../
dockerfile: backend/Dockerfile
# cache_from disabled - requires manual setup: mkdir -p /tmp/docker-cache-langgraph
container_name: deer-flow-langgraph
command: sh -c "cd backend && uv run langgraph dev --no-browser --allow-blocking --host 0.0.0.0 --port 2024 > /app/logs/langgraph.log 2>&1"
volumes:
- ../backend/:/app/backend/
# Preserve the .venv built during Docker image build — mounting the full backend/
# directory above would otherwise shadow it with the (empty) host directory.
- langgraph-venv:/app/backend/.venv
- ../config.yaml:/app/config.yaml
- ../extensions_config.json:/app/extensions_config.json
- ../skills:/app/skills
- ../logs:/app/logs
# Mount uv cache for faster dependency installation
- ~/.cache/uv:/root/.cache/uv
# DooD: same as gateway — AioSandboxProvider runs inside LangGraph process.
- /var/run/docker.sock:/var/run/docker.sock
working_dir: /app
environment:
- CI=true
- DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_ROOT}/backend/.deer-flow
- DEER_FLOW_HOST_SKILLS_PATH=${DEER_FLOW_ROOT}/skills
- DEER_FLOW_SANDBOX_HOST=host.docker.internal
env_file:
- ../.env
extra_hosts:
# For Linux: map host.docker.internal to host gateway
- "host.docker.internal:host-gateway"
networks:
- deer-flow-dev
restart: unless-stopped
volumes:
# Persist .venv across container restarts so dependencies installed during
# image build are not shadowed by the host backend/ directory mount.
gateway-venv:
langgraph-venv:
networks:
deer-flow-dev:
driver: bridge
ipam:
config:
- subnet: 192.168.200.0/24