mirror of
https://gitee.com/wanwujie/deer-flow
synced 2026-04-03 06:12:14 +08:00
76803b826f
* refactor: extract shared utils to break harness→app cross-layer imports Move _validate_skill_frontmatter to src/skills/validation.py and CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py. This eliminates the two reverse dependencies from client.py (harness layer) into gateway/routers/ (app layer), preparing for the harness/app package split. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split backend/src into harness (deerflow.*) and app (app.*) Physically split the monolithic backend/src/ package into two layers: - **Harness** (`packages/harness/deerflow/`): publishable agent framework package with import prefix `deerflow.*`. Contains agents, sandbox, tools, models, MCP, skills, config, and all core infrastructure. - **App** (`app/`): unpublished application code with import prefix `app.*`. Contains gateway (FastAPI REST API) and channels (IM integrations). Key changes: - Move 13 harness modules to packages/harness/deerflow/ via git mv - Move gateway + channels to app/ via git mv - Rename all imports: src.* → deerflow.* (harness) / app.* (app layer) - Set up uv workspace with deerflow-harness as workspace member - Update langgraph.json, config.example.yaml, all scripts, Docker files - Add build-system (hatchling) to harness pyproject.toml - Add PYTHONPATH=. to gateway startup commands for app.* resolution - Update ruff.toml with known-first-party for import sorting - Update all documentation to reflect new directory structure Boundary rule enforced: harness code never imports from app. All 429 tests pass. Lint clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add harness→app boundary check test and update docs Add test_harness_boundary.py that scans all Python files in packages/harness/deerflow/ and fails if any `from app.*` or `import app.*` statement is found. This enforces the architectural rule that the harness layer never depends on the app layer. Update CLAUDE.md to document the harness/app split architecture, import conventions, and the boundary enforcement test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add config versioning with auto-upgrade on startup When config.example.yaml schema changes, developers' local config.yaml files can silently become outdated. This adds a config_version field and auto-upgrade mechanism so breaking changes (like src.* → deerflow.* renames) are applied automatically before services start. - Add config_version: 1 to config.example.yaml - Add startup version check warning in AppConfig.from_file() - Add scripts/config-upgrade.sh with migration registry for value replacements - Add `make config-upgrade` target - Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services - Add config error hints in service failure messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix comments * fix: update src.* import in test_sandbox_tools_security to deerflow.* Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: handle empty config and search parent dirs for config.example.yaml Address Copilot review comments on PR #1131: - Guard against yaml.safe_load() returning None for empty config files - Search parent directories for config.example.yaml instead of only looking next to config.yaml, fixing detection in common setups Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: correct skills root path depth and config_version type coercion - loader.py: fix get_skills_root_path() to use 5 parent levels (was 3) after harness split, file lives at packages/harness/deerflow/skills/ so parent×3 resolved to backend/packages/harness/ instead of backend/ - app_config.py: coerce config_version to int() before comparison in _check_config_version() to prevent TypeError when YAML stores value as string (e.g. config_version: "1") - tests: add regression tests for both fixes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update test imports from src.* to deerflow.*/app.* after harness refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
214 lines
6.7 KiB
Markdown
214 lines
6.7 KiB
Markdown
# Copilot Onboarding Instructions for DeerFlow
|
|
|
|
Use this file as the default operating guide for this repository. Follow it first, and only search the codebase when this file is incomplete or incorrect.
|
|
|
|
## 1) Repository Summary
|
|
|
|
DeerFlow is a full-stack "super agent harness".
|
|
|
|
- Backend: Python 3.12, LangGraph + FastAPI gateway, sandbox/tool system, memory, MCP integration.
|
|
- Frontend: Next.js 16 + React 19 + TypeScript + pnpm.
|
|
- Local dev entrypoint: root `Makefile` starts backend + frontend + nginx on `http://localhost:2026`.
|
|
- Docker dev entrypoint: `make docker-*` (mode-aware provisioner startup from `config.yaml`).
|
|
|
|
Current repo footprint is medium-large (backend service, frontend app, docker stack, skills library, docs).
|
|
|
|
## 2) Runtime and Toolchain Requirements
|
|
|
|
Validated in this repo on macOS:
|
|
|
|
- Node.js `>=22` (validated with Node `23.11.0`)
|
|
- pnpm (repo expects lockfile generated by pnpm 10; validated with pnpm `10.26.2` and `10.15.0`)
|
|
- Python `>=3.12` (CI uses `3.12`)
|
|
- `uv` (validated with `0.7.20`)
|
|
- `nginx` (required for `make dev` unified local endpoint)
|
|
|
|
Always run from repo root unless a command explicitly says otherwise.
|
|
|
|
## 3) Build/Test/Lint/Run - Verified Command Sequences
|
|
|
|
These were executed and validated in this repository.
|
|
|
|
### A. Bootstrap and install
|
|
|
|
1. Check prerequisites:
|
|
|
|
```bash
|
|
make check
|
|
```
|
|
|
|
Observed: passes when required tools are installed.
|
|
|
|
2. Install dependencies (recommended order: backend then frontend, as implemented by `make install`):
|
|
|
|
```bash
|
|
make install
|
|
```
|
|
|
|
### B. Backend CI-equivalent validation
|
|
|
|
Run from `backend/`:
|
|
|
|
```bash
|
|
make lint
|
|
make test
|
|
```
|
|
|
|
Validated results:
|
|
|
|
- `make lint`: pass (`ruff check .`)
|
|
- `make test`: pass (`277 passed, 15 warnings in ~76.6s`)
|
|
|
|
CI parity:
|
|
|
|
- `.github/workflows/backend-unit-tests.yml` runs on pull requests.
|
|
- CI executes `uv sync --group dev`, then `make lint`, then `make test` in `backend/`.
|
|
|
|
### C. Frontend validation
|
|
|
|
Run from `frontend/`.
|
|
|
|
Recommended reliable sequence:
|
|
|
|
```bash
|
|
pnpm lint
|
|
pnpm typecheck
|
|
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
|
```
|
|
|
|
Observed failure modes and workarounds:
|
|
|
|
- `pnpm build` fails without `BETTER_AUTH_SECRET` in production-mode env validation.
|
|
- Workaround: set `BETTER_AUTH_SECRET` (best) or set `SKIP_ENV_VALIDATION=1`.
|
|
- Even with `SKIP_ENV_VALIDATION=1`, Better Auth can still warn/error in logs about default secret; prefer setting a real non-default secret.
|
|
- `pnpm check` currently fails (`next lint` invocation is incompatible here and resolves to an invalid directory). Do not rely on `pnpm check`; run `pnpm lint` and `pnpm typecheck` explicitly.
|
|
|
|
### D. Run locally (all services)
|
|
|
|
From root:
|
|
|
|
```bash
|
|
make dev
|
|
```
|
|
|
|
Behavior:
|
|
|
|
- Stops existing local services first.
|
|
- Starts LangGraph (`2024`), Gateway (`8001`), Frontend (`3000`), nginx (`2026`).
|
|
- Unified app endpoint: `http://localhost:2026`.
|
|
- Logs: `logs/langgraph.log`, `logs/gateway.log`, `logs/frontend.log`, `logs/nginx.log`.
|
|
|
|
Stop services:
|
|
|
|
```bash
|
|
make stop
|
|
```
|
|
|
|
If tool sessions/timeouts interrupt `make dev`, run `make stop` again to ensure cleanup.
|
|
|
|
### E. Config bootstrap
|
|
|
|
From root:
|
|
|
|
```bash
|
|
make config
|
|
```
|
|
|
|
Important behavior:
|
|
|
|
- This intentionally aborts if `config.yaml` (or `config.yml`/`configure.yml`) already exists.
|
|
- Use `make config` only for first-time setup in a clean clone.
|
|
|
|
## 4) Command Order That Minimizes Failures
|
|
|
|
Use this exact order for local code changes:
|
|
|
|
1. `make check`
|
|
2. `make install` (if frontend fails with proxy errors, rerun frontend install with proxy vars unset)
|
|
3. Backend checks: `cd backend && make lint && make test`
|
|
4. Frontend checks: `cd frontend && pnpm lint && pnpm typecheck`
|
|
5. Frontend build (if UI changes or release-sensitive changes): `BETTER_AUTH_SECRET=... pnpm build`
|
|
|
|
Always run backend lint/tests before opening PRs because that is what CI enforces.
|
|
|
|
## 5) Project Layout and Architecture (High-Value Paths)
|
|
|
|
Root-level orchestration and config:
|
|
|
|
- `Makefile` - main local/dev/docker command entrypoints
|
|
- `config.example.yaml` - primary app config template
|
|
- `config.yaml` - local active config (gitignored)
|
|
- `docker/docker-compose-dev.yaml` - Docker dev topology
|
|
- `.github/workflows/backend-unit-tests.yml` - PR validation workflow
|
|
|
|
Backend core:
|
|
|
|
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
|
|
- `backend/app/gateway/` - FastAPI gateway API
|
|
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
|
|
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
|
|
- `backend/packages/harness/deerflow/mcp/` - MCP integration
|
|
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
|
|
- `backend/pyproject.toml` - Python deps and `requires-python`
|
|
- `backend/ruff.toml` - lint/format policy
|
|
- `backend/tests/` - backend unit and integration-like tests
|
|
|
|
Frontend core:
|
|
|
|
- `frontend/src/app/` - Next.js routes/pages
|
|
- `frontend/src/components/` - UI components
|
|
- `frontend/src/core/` - app logic (threads, tools, API, models)
|
|
- `frontend/src/env.js` - env schema/validation (critical for build behavior)
|
|
- `frontend/package.json` - scripts/deps
|
|
- `frontend/eslint.config.js` - lint rules
|
|
- `frontend/tsconfig.json` - TS config
|
|
|
|
Skills and assets:
|
|
|
|
- `skills/public/` - built-in skill packs loaded by agent runtime
|
|
|
|
## 6) Pre-Checkin / Validation Expectations
|
|
|
|
Before submitting changes, run at minimum:
|
|
|
|
- Backend: `cd backend && make lint && make test`
|
|
- Frontend (if touched): `cd frontend && pnpm lint && pnpm typecheck`
|
|
- Frontend build when changing env/auth/routing/build-sensitive files: `BETTER_AUTH_SECRET=... pnpm build`
|
|
|
|
If touching orchestration/config (`Makefile`, `docker/*`, `config*.yaml`), also run `make dev` and verify the four services start.
|
|
|
|
## 7) Non-Obvious Dependencies and Gotchas
|
|
|
|
- Proxy env vars can silently break frontend network operations (`pnpm install`/registry access).
|
|
- `BETTER_AUTH_SECRET` is effectively required for reliable frontend production build validation.
|
|
- Next.js may warn about multiple lockfiles and workspace root inference; this is currently a warning, not a build blocker.
|
|
- `make config` is non-idempotent by design when config already exists.
|
|
- `make dev` includes process cleanup and can emit shutdown logs/noise if interrupted; this is expected.
|
|
|
|
## 8) Root Inventory (quick reference)
|
|
|
|
Important root entries:
|
|
|
|
- `.github/`
|
|
- `backend/`
|
|
- `frontend/`
|
|
- `docker/`
|
|
- `skills/`
|
|
- `scripts/`
|
|
- `docs/`
|
|
- `README.md`
|
|
- `CONTRIBUTING.md`
|
|
- `Makefile`
|
|
- `config.example.yaml`
|
|
- `extensions_config.example.json`
|
|
|
|
## 9) Instruction Priority
|
|
|
|
Trust this onboarding guide first.
|
|
|
|
Only do broad repo searches (`grep/find/code search`) when:
|
|
|
|
- you need file-level implementation details not listed here,
|
|
- a command here fails and you need updated replacement behavior,
|
|
- or CI/workflow definitions have changed since this file was written.
|