Commit Graph

66 Commits

Author SHA1 Message Date
Willem Jiang
6b73a53999 fix(config): Add support for MCP server configuration parameters (#812)
* fix(config): Add support for MCP server configuration parameters

* refact: rename the sse_readtimeout to sse_read_timeout

* update the code with review comments

* update the MCP document for the latest change
2026-01-10 15:59:49 +08:00
Willem Jiang
a376b0cb4e fix: Add runtime parameter to compress_messages method(#803)
* fix: Add runtime parameter to compress_messages method(#803)

    The compress_messages method was being called by PreModelHookMiddleware
    with both state and runtime parameters, but only accepted state parameter.
    This caused a TypeError when the middleware executed the pre_model_hook.

    Added optional runtime parameter to compress_messages signature to match
    the expected interface while maintaining backward compatibility.

* Update the code with the review comments
2026-01-07 20:36:15 +08:00
Willem Jiang
d4ab77de5c fix: migrate from deprecated create_react_agent to langchain.agents.create_agent (#802)
* fix: migrate from deprecated create_react_agent to langchain.agents.create_agent

Fixes #799

- Replace deprecated langgraph.prebuilt.create_react_agent with
  langchain.agents.create_agent (LangGraph 1.0 migration)
- Add DynamicPromptMiddleware to handle dynamic prompt templates
  (replaces the 'prompt' callable parameter)
- Add PreModelHookMiddleware to handle pre-model hooks
  (replaces the 'pre_model_hook' parameter)
- Update AgentState import from langchain.agents in template.py
- Update tests to use the new API

* fix:update the code with review comments
2026-01-07 09:06:16 +08:00
Willem Jiang
bab60e6e3d fix(podcast): add fallback for models without json_object support (#747) (#785)
* fix(podcast): add fallback for models without json_object support (#747)

Models like Kimi K2 don't support response_format.type: json_object.
Add try-except to fall back to regular prompting with JSON parsing
when BadRequestError mentions json_object not supported.

- Add fallback to prompting + repair_json_output parsing
- Re-raise other BadRequestError types
- Add unit tests for script_writer_node with 100% coverage

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fixes: the unit test error of test_script_writer_node.py

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-26 23:04:20 +08:00
Willem Jiang
8d9d767051 feat(eval): add report quality evaluation module and UI integration (#776)
* feat(eval): add report quality evaluation module

Addresses issue #773 - How to evaluate generated report quality objectively.

This module provides two evaluation approaches:
1. Automated metrics (no LLM required):
   - Citation count and source diversity
   - Word count compliance per report style
   - Section structure validation
   - Image inclusion tracking

2. LLM-as-Judge evaluation:
   - Factual accuracy scoring
   - Completeness assessment
   - Coherence evaluation
   - Relevance and citation quality checks

The combined evaluator provides a final score (1-10) and letter grade (A+ to F).

Files added:
- src/eval/__init__.py
- src/eval/metrics.py
- src/eval/llm_judge.py
- src/eval/evaluator.py
- tests/unit/eval/test_metrics.py
- tests/unit/eval/test_evaluator.py

* feat(eval): integrate report evaluation with web UI

This commit adds the web UI integration for the evaluation module:

Backend:
- Add EvaluateReportRequest/Response models in src/server/eval_request.py
- Add /api/report/evaluate endpoint to src/server/app.py

Frontend:
- Add evaluateReport API function in web/src/core/api/evaluate.ts
- Create EvaluationDialog component with grade badge, metrics display,
  and optional LLM deep evaluation
- Add evaluation button (graduation cap icon) to research-block.tsx toolbar
- Add i18n translations for English and Chinese

The evaluation UI allows users to:
1. View quick metrics-only evaluation (instant)
2. Optionally run deep LLM-based evaluation for detailed analysis
3. See grade (A+ to F), score (1-10), and metric breakdown

* feat(eval): improve evaluation reliability and add LLM judge tests

- Extract MAX_REPORT_LENGTH constant in llm_judge.py for maintainability
- Add comprehensive unit tests for LLMJudge class (parse_response,
  calculate_weighted_score, evaluate with mocked LLM)
- Pass reportStyle prop to EvaluationDialog for accurate evaluation criteria
- Add researchQueries store map to reliably associate queries with research
- Add getResearchQuery helper to retrieve query by researchId
- Remove unused imports in test_metrics.py

* fix(eval): use resolveServiceURL for evaluate API endpoint

The evaluateReport function was using a relative URL '/api/report/evaluate'
which sent requests to the Next.js server instead of the FastAPI backend.
Changed to use resolveServiceURL() consistent with other API functions.

* fix: improve type accuracy and React hooks in evaluation components

- Fix get_word_count_target return type from Optional[Dict] to Dict since it always returns a value via default fallback
- Fix useEffect dependency issue in EvaluationDialog using useRef to prevent unwanted re-evaluations
- Add aria-label to GradeBadge for screen reader accessibility
2025-12-25 21:55:48 +08:00
Willem Jiang
fb319aaa44 test: add unit tests for global connection pool (Issue #778) (#780)
* test: add unit tests for global connection pool (Issue #778)

- Add TestLifespanFunction class with 9 tests for lifespan management:
  - PostgreSQL/MongoDB pool initialization success/failure
  - Cleanup on shutdown
  - Skip initialization when not configured

- Add TestGlobalConnectionPoolUsage class with 4 tests:
  - Using global pools when available
  - Fallback to per-request connections

- Fix missing dict_row import in app.py (bug from PR #757)

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-23 23:06:39 +08:00
Willem Jiang
04296cdf5a feat: add resource upload support for RAG (#768)
* feat: add resource upload support for RAG

- Backend: Added ingest_file method to Retriever and MilvusRetriever
- Backend: Added /api/rag/upload endpoint
- Frontend: Added RAGTab in settings for uploading resources
- Frontend: Updated translations and settings registration

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestions from code review

* Apply suggestions from code review of src/rag/milvus.py

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-19 09:55:34 +08:00
Willem Jiang
2a97170b6c feat: add Serper search engine support (#762)
* feat: add Serper search engine support

* docs: update configuration guide and env example for Serper

* test: add test case for Serper with missing API key
2025-12-15 23:04:26 +08:00
Jiahe Wu
93d81d450d feat: add enable_web_search config to disable web search (#681) (#760)
* feat: add enable_web_search config to disable web search (#681)

* fix: skip enforce_researcher_search validation when web search is disabled

- Return json.dumps([]) instead of empty string for consistency in background_investigation_node

- Add enable_web_search check to skip validation warning when user intentionally disabled web search

- Add warning log when researcher has no tools available

- Update tests to include new enable_web_search parameter

* fix: address Copilot review feedback

- Coordinate enforce_web_search with enable_web_search in validate_and_fix_plan

- Fix misleading comment in background_investigation_node

* docs: add warning about local RAG setup when disabling web search

* docs: add web search toggle section to configuration guide
2025-12-15 19:17:24 +08:00
Willem Jiang
ec99338c9a fix(agents): patch _run in ToolInterceptor to ensure interrupt triggering (#753)
Fixes #752

* fix(agents): patch _run in ToolInterceptor to ensure interrupt triggering

* Update the code with review comments
2025-12-10 22:15:08 +08:00
Willem Jiang
84c449cf79 fix(checkpoint): clear in-memory store after successful persistence (#751)
* fix(checkpoint): clear in-memory store after successful persistence

* test(checkpoint): add unit test for memory leak check

* Update tests/unit/checkpoint/test_memory_leak.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-09 23:32:13 +08:00
Willem Jiang
c36ab393f1 fix: update Interrupt object attribute access for LangGraph 1.0+ (#730) (#731)
* Update uv.lock to sync with pyproject.toml

* fix: update Interrupt object attribute access for LangGraph 1.0+ (#730)

The Interrupt class in LangGraph 1.0 no longer has the 'ns' attribute.
This change updates _create_interrupt_event() to use the new 'id'
attribute instead, with a fallback to thread_id for compatibility.

Changes:
- Replace event_data["__interrupt__"][0].ns[0] with interrupt.id
- Use getattr() with fallback for backward compatibility
- Update debug log message from 'ns=' to 'id='
- Add unit tests for _create_interrupt_event function

* fix the unit test error and address review comment

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-12-02 11:16:00 +08:00
infoquest-byteplus
7ec9e45702 feat: support infoquest (#708)
* support infoquest

* support html checker

* support html checker

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* change line break format

* Fix several critical issues in the codebase
- Resolve crawler panic by improving error handling
- Fix plan validation to prevent invalid configurations
- Correct InfoQuest crawler JSON conversion logic

* add test for infoquest

* add test for infoquest

* Add InfoQuest introduction to the README

* add test for infoquest

* fix readme for infoquest

* fix readme for infoquest

* resolve the conflict

* resolve the conflict

* resolve the conflict

* Fix formatting of INFOQUEST in SearchEngine enum

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-02 08:16:35 +08:00
Willem Jiang
4a78cfe12a fix(llm): filter unexpected config keys to prevent LangChain warnings (#411) (#726)
* fix(llm): filter unexpected config keys to prevent LangChain warnings (#411)

Add allowlist validation for LLM configuration keys to prevent unexpected
parameters like SEARCH_ENGINE from being passed to LLM constructors.

Changes:
- Add ALLOWED_LLM_CONFIG_KEYS set with valid LLM configuration parameters
- Filter out unexpected keys before creating LLM instances
- Log clear warning messages when unexpected keys are removed
- Add unit test for configuration key filtering

This fixes the confusing LangChain warning "WARNING! SEARCH_ENGINE is not
default parameter. SEARCH_ENGINE was transferred to model_kwargs" that
occurred when users accidentally placed configuration keys in wrong sections
of conf.yaml.

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-29 16:13:05 +08:00
Willem Jiang
2e010a4619 feat: add analysis step type for non-code reasoning tasks (#677) (#723)
Add a new "analysis" step type to handle reasoning and synthesis tasks
that don't require code execution, addressing the concern that routing
all non-search tasks to the coder agent was inappropriate.

Changes:
- Add ANALYSIS enum value to StepType in planner_model.py
- Create analyst_node for pure LLM reasoning without tools
- Update graph routing to route analysis steps to analyst agent
- Add analyst agent to AGENT_LLM_MAP configuration
- Create analyst prompts (English and Chinese)
- Update planner prompts with guidance on choosing between
  analysis (reasoning/synthesis) and processing (code execution)
- Change default step_type inference from "processing" to "analysis"
  when need_search=false

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-11-29 09:46:55 +08:00
Willem Jiang
170c4eb33c Upgrade langchain version to 1.x (#720)
* fix: revert the part of patch of issue-710 to extract the content from the plan

* Upgrade the ddgs for the new compatible version

* Upgraded langchain to 1.1.0
updated langchain related package to the new compatable version

* Update pyproject.toml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-28 22:09:13 +08:00
Willem Jiang
4559197505 fix: revert the part of patch of issue-710 to extract the content from the plan (#718) 2025-11-27 23:59:31 +08:00
Willem Jiang
667916959b fix: the exception of plan validation (#714)
* fix: Missing Required Fields in Plan Validation

* fix: the exception of plan validation

* Fixed the test errors

* Addressed the comments of the PR reviews
2025-11-27 19:39:25 +08:00
Willem Jiang
bec97f02ae fix: the crawling error when encountering PDF URLs (#707)
* fix: the crawling error when encountering PDF URLs

* Added the unit test for the new feature of crawl tool

* fix: address the code review problems

* fix: address the code review problems
2025-11-25 09:24:52 +08:00
Anush
aa027faf95 feat: Qdrant Vector Search Support (#684)
* feat: Qdrant vector search support

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* chore: Review updates

Signed-off-by: Anush008 <anushshetty90@gmail.com>

---------

Signed-off-by: Anush008 <anushshetty90@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-11-11 19:35:00 +08:00
Willem Jiang
6ae4bc588a fix: remove the unnessary conditional edge. (#671) 2025-10-29 10:12:32 +08:00
Willem Jiang
0415f622da fix: presever the local setting between frontend and backend (#670)
* fix: presever the local setting between frontend and backend

* Added unit test for the state preservation

* fix: passing the locale to the agent call

* fix: apply the fix after code review
2025-10-28 21:45:29 +08:00
Willem Jiang
b4c09aa4b1 security: add log injection attack prevention with input sanitization (#667)
* security: add log injection attack prevention with input sanitization

- Created src/utils/log_sanitizer.py to sanitize user-controlled input before logging
- Prevents log injection attacks using newlines, tabs, carriage returns, etc.
- Escapes dangerous characters: \n, \r, \t, \0, \x1b
- Provides specialized functions for different input types:
  - sanitize_log_input: general purpose sanitization
  - sanitize_thread_id: for user-provided thread IDs
  - sanitize_user_content: for user messages (more aggressive truncation)
  - sanitize_agent_name: for agent identifiers
  - sanitize_tool_name: for tool names
  - sanitize_feedback: for user interrupt feedback
  - create_safe_log_message: template-based safe message creation

- Updated src/server/app.py to sanitize all user input in logging:
  - Thread IDs from request parameter
  - Message content from user
  - Agent names and node information
  - Tool names and feedback

- Updated src/agents/tool_interceptor.py to sanitize:
  - Tool names during execution
  - User feedback during interrupt handling
  - Tool input data

- Added 29 comprehensive unit tests covering:
  - Classic newline injection attacks
  - Carriage return injection
  - Tab and null character injection
  - HTML/ANSI escape sequence injection
  - Combined multi-character attacks
  - Truncation and length limits

Fixes potential log forgery vulnerability where malicious users could inject
fake log entries via unsanitized input containing control characters.
2025-10-27 20:57:23 +08:00
Willem Jiang
bcc403ecd3 feat: implement tool-specific interrupts for create_react_agent (#572) (#659)
* feat: implement tool-specific interrupts for create_react_agent (#572)

          Add selective tool interrupt capability allowing interrupts before specific tools
          rather than all tools. Users can now configure which tools trigger interrupts via
          the interrupt_before_tools parameter.

          Changes:
          - Create ToolInterceptor class to handle tool-specific interrupt logic
          - Add interrupt_before_tools parameter to create_agent() function
          - Extend Configuration with interrupt_before_tools field
          - Add interrupt_before_tools to ChatRequest API
          - Update nodes.py to pass interrupt configuration to agents
          - Update app.py workflow to support tool interrupt configuration
          - Add comprehensive unit tests for tool interceptor

          Features:
          - Selective tool interrupts: interrupt only specific tools by name
          - Approval keywords: recognize user approval (approved, proceed, accept, etc.)
          - Backward compatible: optional parameter, existing code unaffected
          - Flexible: works with default tools and MCP-powered tools
          - Works with existing resume mechanism for seamless workflow

          Example usage:
            request = ChatRequest(
              messages=[...],
              interrupt_before_tools=['db_tool', 'sensitive_api']
            )

* test: add comprehensive integration tests for tool-specific interrupts (#572)

Add 24 integration tests covering all aspects of the tool interceptor feature:

Test Coverage:
- Agent creation with tool interrupts
- Configuration support (with/without interrupts)
- ChatRequest API integration
- Multiple tools with selective interrupts
- User approval/rejection flows
- Tool wrapping and functionality preservation
- Error handling and edge cases
- Approval keyword recognition
- Complex tool inputs
- Logging and monitoring

All tests pass with 100% coverage of tool interceptor functionality.

Tests verify:
✓ Selective tool interrupts work correctly
✓ Only specified tools trigger interrupts
✓ Non-matching tools execute normally
✓ User feedback is properly parsed
✓ Tool functionality is preserved after wrapping
✓ Error handling works as expected
✓ Configuration options are properly respected
✓ Logging provides useful debugging info

* fix: mock get_llm_by_type in agent creation test

Fix test_agent_creation_with_tool_interrupts which was failing because
get_llm_by_type() was being called before create_react_agent was mocked.

Changes:
- Add mock for get_llm_by_type in test
- Use context manager composition for multiple patches
- Test now passes and validates tool wrapping correctly

All 24 integration tests now pass successfully.

* refactor: use mock assertion methods for consistent and clearer error messages

Update integration tests to use mock assertion methods instead of direct
attribute checking for consistency and clearer error messages:

Changes:
- Replace 'assert mock_interrupt.called' with 'mock_interrupt.assert_called()'
- Replace 'assert not mock_interrupt.called' with 'mock_interrupt.assert_not_called()'

Benefits:
- Consistent with pytest-mock and unittest.mock best practices
- Clearer error messages when assertions fail
- Better IDE autocompletion support
- More professional test code

All 42 tests pass with improved assertion patterns.

* refactor: use default_factory for interrupt_before_tools consistency

Improve consistency between ChatRequest and Configuration implementations:

Changes:
- ChatRequest.interrupt_before_tools: Use Field(default_factory=list) instead of Optional[None]
- Remove unnecessary 'or []' conversion in app.py line 505
- Aligns with Configuration.interrupt_before_tools implementation pattern
- No functional changes - all tests still pass

Benefits:
- Consistent field definition across codebase
- Simpler and cleaner code
- Reduced chance of None/empty list bugs
- Better alignment with Pydantic best practices

All 42 tests passing.

* refactor: improve tool input formatting in interrupt messages

Enhance tool input representation for better readability in interrupt messages:

Changes:
- Add json import for better formatting
- Create _format_tool_input() static method with JSON serialization
- Use JSON formatting for dicts, lists, tuples with indent=2
- Fall back to str() for non-serializable types
- Handle None input specially (returns 'No input')
- Improve interrupt message formatting with better spacing

Benefits:
- Complex tool inputs now display as readable JSON
- Nested structures are properly indented and visible
- Better user experience when reviewing tool inputs before approval
- Handles edge cases gracefully with fallbacks
- Improved logging output for debugging

Example improvements:
Before: {'query': 'SELECT...', 'limit': 10, 'nested': {'key': 'value'}}
After:
{
  "query": "SELECT...",
  "limit": 10,
  "nested": {
    "key": "value"
  }
}

All 42 tests still passing.

* test: add comprehensive unit tests for tool input formatting
2025-10-26 09:47:03 +08:00
Willem Jiang
c7a82b82b4 fix: parsed json with extra tokens issue (#656)
Fixes #598 

* fix: parsed json with extra tokens issue

* Added unit test for json.ts

* fix the json unit test running issue

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update the code with code review suggestion

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-26 07:24:25 +08:00
Willem Jiang
f2be4d6af1 fix: prevent tool name concatenation in consecutive tool calls to fix #523 (#654)
- Implement index-based grouping of tool call chunks in _process_tool_call_chunks()
- Add _validate_tool_call_chunks() for debug logging and validation
- Enhance _process_message_chunk() with tool call ID validation and boundary detection
- Add comprehensive unit tests (17 tests) for tool call chunk processing
- Fix issue where tool names were incorrectly concatenated (e.g., 'web_searchweb_search')
- Ensure chunks from different tool calls (different indices) remain properly separated
- Add detailed logging for debugging tool call streaming issues

* update the code with suggestions of reviewing
2025-10-24 22:26:25 +08:00
Willem Jiang
36bf5c9ccd fix: repair missing step_type fields in Plan validation (#653)
* fix: resolve issue #650 - repair missing step_type fields in Plan validation

- Add step_type repair logic to validate_and_fix_plan() to auto-infer missing step_type
- Infer as 'research' when need_search=true, 'processing' when need_search=false
- Add explicit CRITICAL REQUIREMENT section to planner.md emphasizing step_type mandatory for every step
- Include validation checklist and examples showing both research and processing steps
- Add 23 comprehensive unit tests for validate_and_fix_plan() covering all scenarios
- Add 4 integration tests specifically for Issue #650 with actual Plan validation
- Prevents Pydantic ValidationError: 'Field required' for missing step_type

* Update tests/unit/graph/test_plan_validation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/unit/graph/test_plan_validation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update the planner.zh_CN.md with recent changes of planner.md

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-24 21:26:48 +08:00
Willem Jiang
975b344ca7 fix: resolve issue #651 - crawl error with None content handling (#652)
* fix: resolve issue #651 - crawl error with None content handling
Fixed issue #651 by adding comprehensive null-safety checks and error handling to the crawl system.
The fix prevents the ‘TypeError: Incoming markup is of an invalid type: None’ crash by:
1. Validating HTTP responses from Jina API
2. Handling None/empty content at extraction stage
3. Adding fallback handling in Article markdown/message conversion
4. Improving error diagnostics with detailed logging
5. Adding 16 new tests with 100% coverage for critical paths

* Update src/crawler/readability_extractor.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/crawler/article.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-24 17:06:54 +08:00
jimmyuconn1982
2001a7c223 Fix: clarification bugs - max rounds, locale passing, and over-clarification (#647)
Fixes: Max rounds bug, locale passing bug, over-clarification issue

* reslove Copilot spelling comments

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-10-24 16:43:39 +08:00
Willem Jiang
052490b116 fix: resolve issue #467 - message content validation and Tavily search error handling (#645)
* fix: resolve issue #467 - message content validation and Tavily search error handling

This commit implements a comprehensive fix for issue #467 where the application
crashed with 'Field required: input.messages.3.content' error when generating reports.

## Root Cause Analysis
The issue had multiple interconnected causes:
1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON
2. background_investigation_node didn't handle error cases properly, returning None
3. Missing message content validation before LLM calls
4. Insufficient error diagnostics for content-related errors

## Changes Made

### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py)
- Modified _run() and _arun() methods to return JSON strings instead of mixed types
- Error responses now return JSON: {"error": repr(e)}
- Successful responses return JSON string: json.dumps(cleaned_results)
- Ensures tool results always have valid string content for ToolMessages

### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py)
- Initialize background_investigation_results to empty list instead of None
- Added proper JSON parsing for string responses from Tavily tool
- Handle error responses with explicit error logging
- Always return valid JSON (empty list if error) instead of None

### Part 3: Add Message Content Validation (utils/context_manager.py)
- New validate_message_content() function validates all messages before LLM calls
- Ensures all messages have content attribute and valid string content
- Converts complex types (lists, dicts) to JSON strings
- Provides graceful fallback for messages with issues

### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py)
- Call message validation before agent invocation
- Add detailed logging for content-related errors
- Log message types, content types, and lengths when validation fails
- Helps with future debugging of similar issues

## Testing
- All unit tests pass (395 tests)
- Python syntax verified for all modified files
- No breaking changes to existing functionality

* test: update tests for issue #467 fixes

Update test expectations to match the new implementation:
- Tavily search tool now returns JSON strings instead of mixed types
- background_investigation_node returns empty list [] for errors instead of None
- All tests updated to verify the new behavior
- All 391 tests pass successfully

* Update src/graph/nodes.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
Willem Jiang
9ece3fd9c3 fix: support additional Tavily search parameters via configuration to fix #548 (#643)
* fix: support additional Tavily search parameters via configuration to fix #548

- Add include_answer, search_depth, include_raw_content, include_images, include_image_descriptions to SEARCH_ENGINE config
- Update get_web_search_tool() to load these parameters from configuration with sensible defaults
- Parameters are now properly passed to TavilySearchWithImages during initialization
- This fixes 'got an unexpected keyword argument' errors when using web_search tool
- Update tests to verify new parameters are correctly set

* test: add comprehensive unit tests for web search configuration loading

- Add test for custom configuration values (include_answer, search_depth, etc.)
- Add test for empty configuration (all defaults)
- Add test for image_descriptions logic when include_images is false
- Add test for partial configuration
- Add test for missing config file
- Add test for multiple domains in include/exclude lists

All 7 new tests pass and provide comprehensive coverage of configuration loading
and parameter handling for Tavily search tool initialization.

* test: verify all Tavily configuration parameters are optional

Add 8 comprehensive tests to verify that all Tavily engine configuration
parameters are truly optional:

- test_tavily_with_no_search_engine_section: SEARCH_ENGINE section missing
- test_tavily_with_completely_empty_config: Entire config missing
- test_tavily_with_only_include_answer_param: Single param, rest default
- test_tavily_with_only_search_depth_param: Single param, rest default
- test_tavily_with_only_include_domains_param: Domain param, rest default
- test_tavily_with_explicit_false_boolean_values: False values work correctly
- test_tavily_with_empty_domain_lists: Empty lists handled correctly
- test_tavily_all_parameters_optional_mix: Multiple missing params work

These tests verify:
- Tool creation never fails regardless of missing configuration
- All parameters have sensible defaults
- Boolean parameters can be explicitly set to False
- Any combination of optional parameters works
- Domain lists can be empty or omitted

All 15 Tavily configuration tests pass successfully.
2025-10-22 22:56:02 +08:00
jimmyuconn1982
003f081a7b fix: Refine clarification workflow state handling (#641)
* fix: support local models by making thought field optional in Plan model

- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements

Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.

* feat: Add intelligent clarification feature for research queries

- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide

Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide

* fix: Improve clarification conversation continuity

- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds

Fixes issue where LLM would restart clarification instead of building upon previous exchanges.

* fix: Add conversation history to clarification context

* fix: resolve clarification feature message to planer, prompt, test issues

- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py

* fix: Add configurable max_clarification_rounds and comprehensive tests

- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
  - Recursive call failed due to missing initial_state parameter
  - Clarification exited prematurely at max rounds
  - Incorrect logging of max rounds reached

* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json

* fix: add max_clarification_rounds parameter passing from frontend to backend

- Add max_clarification_rounds parameter in store.ts sendMessage function
- Add max_clarification_rounds type definition in chat.ts
- Ensure frontend settings page clarification rounds are correctly passed to backend

* fix: refine clarification workflow state handling and coverage

- Add clarification history reconstruction
- Fix clarified topic accumulation
- Add clarified_research_topic state field
- Preserve clarification state in recursive calls
- Add comprehensive test coverage

* refactor: optimize coordinator logic and type annotations

- Simplify handoff topic logic in coordinator_node
- Update type annotations from Tuple to tuple
- Improve code readability and maintainability

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-10-22 22:49:07 +08:00
Willem Jiang
d30c4d00d3 fix: convert crawl_tool dict return to JSON string for type consistency (#636)
Keep fixing #631
This pull request updates the crawl_tool function to return its results as a JSON string instead of a dictionary, and adjusts the unit tests accordingly to handle the new return type. The changes ensure consistent serialization of output and proper validation in tests.
2025-10-21 10:00:33 +08:00
Willem Jiang
e2ff765460 fix: correct image result format for OpenAI compatibility to fix #632 (#634)
- Change image result type from 'image' to 'image_url' to match OpenAI API expectations
- Wrap image URL in dict structure: {"url": "..."} instead of plain string
- Update SearchResultPostProcessor to handle dict-based image_url during duplicate removal
- Update tests to validate new image format

This fixes the 400 error: Invalid value: 'image'. Supported values are: 'text', 'image_url'...

Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-20 23:14:09 +08:00
jimmyuconn1982
2510cc61de feat: Add intelligent clarification feature in coordinate step for research queries (#613)
* fix: support local models by making thought field optional in Plan model

- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements

Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.

* feat: Add intelligent clarification feature for research queries

- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide

Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide

* fix: Improve clarification conversation continuity

- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds

Fixes issue where LLM would restart clarification instead of building upon previous exchanges.

* fix: Add conversation history to clarification context

* fix: resolve clarification feature message to planer, prompt, test issues

- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py

* fix: Add configurable max_clarification_rounds and comprehensive tests

- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
  - Recursive call failed due to missing initial_state parameter
  - Clarification exited prematurely at max rounds
  - Incorrect logging of max rounds reached

* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json
2025-10-14 13:35:57 +08:00
Willem Jiang
81c91dda43 feature: clean up the temp file which are generated when running the unit test of milvus (#612)
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
2025-10-12 22:10:15 +08:00
Fancy-hjyp
5f4eb38fdb feat: add context compress (#590)
* feat:Add context compress

* feat: Add unit test

* feat: add unit test for context manager

* feat: add postprocessor param && code format

* feat: add configuration guide

* fix: fix the configuration_guide

* fix: fix the unit test

* fix: fix the default value

* feat: add test and log for context_manager
2025-09-27 21:42:22 +08:00
Chayton Bai
7694bb5d72 feat: support dify in rag module (#550)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-16 20:30:45 +08:00
CHANGXUBO
dd9af1eb50 feat: Implement Milvus retriver for RAG (#516)
* feat: Implement MilvusRetriever with embedding model and resource management

* chore: Update configuration and loader files for consistency

* chore: Clean up test_milvus.py for improved readability and organization

* feat: Add tests for DashscopeEmbeddings query and document embedding methods

* feat: Add tests for embedding model initialization and example file loading in MilvusProvider

* chore: Remove unused imports and clean up test_milvus.py for better readability

* chore: Clean up test_milvus.py for improved readability and organization

* chore: Clean up test_milvus.py for improved readability and organization

* fix: replace print statements with logging in recursion limit function

* Implement feature X to enhance user experience and optimize performance

* refactor: clean up unused imports and comments in AboutTab component

* Implement feature X to enhance user experience and fix bug Y in module Z

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-09-12 22:20:55 +08:00
Willem Jiang
4c17d88029 feat: creating mogodb and postgres mock instance in checkpoint test (#561)
* fix: using mongomock for the checkpoint test

* Add postgres mock setting to the unit test

* Added utils file of postgres_mock_utils

* fixed the runtime loading error of deerflow server
2025-09-09 22:49:11 +08:00
Anoyer-lzh
270d8c3712 fix: env parameters exception when configuring SSE or HTTP MCP server (#513)
* fix: _create_streamable_http_session() got an unexpected keyword argument 'env'

fix unit error

* update md

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-20 17:23:57 +08:00
zgjja
3b4e993531 feat: 1. replace black with ruff for fomatting and sort import (#489)
2. use tavily from`langchain-tavily` rather than the older one from `langchain-community`

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-17 22:57:23 +08:00
CHANGXUBO
1bfec3ad05 feat: Enhance chat streaming and tool call processing (#498)
* feat: Enhance chat streaming and tool call processing

- Added support for MongoDB checkpointer in the chat streaming workflow.
- Introduced functions to process tool call chunks and sanitize arguments.
- Improved event message creation with additional metadata.
- Enhanced error handling for JSON serialization in event messages.
- Updated the frontend to convert escaped characters in tool call arguments.
- Refactored the workflow input preparation and initial message processing.
- Added new dependencies for MongoDB integration and tool argument sanitization.

* fix: Update MongoDB checkpointer configuration to use LANGGRAPH_CHECKPOINT_DB_URL

* feat: Add support for Postgres checkpointing and update README with database recommendations

* feat: Implement checkpoint saver functionality and update MongoDB connection handling

* refactor: Improve code formatting and readability in app.py and json_utils.py

* refactor: Clean up commented code and improve formatting in server.py

* refactor: Remove unused imports and improve code organization in app.py

* refactor: Improve code organization and remove unnecessary comments in app.py

* chore: use langgraph-checkpoint-postgres==2.0.21 to avoid the JSON convert issue in the latest version, implement chat stream persistant with Postgres

* feat: add MongoDB and PostgreSQL support for LangGraph checkpointing, enhance environment variable handling

* fix: update comments for clarity on Windows event loop policy

* chore: remove empty code changes in MongoDB and PostgreSQL checkpoint tests

* chore: clean up unused imports and code in checkpoint-related files

* chore: remove empty code changes in test_checkpoint.py

* chore: remove empty code changes in test_checkpoint.py

* chore: remove empty code changes in test_checkpoint.py

* test: update status code assertions in MCP endpoint tests to allow for 403 responses

* test: update MCP endpoint tests to assert specific status codes and enable MCP server configuration

* chore: remove unnecessary environment variables from unittest workflow

* fix: invert condition for MCP server configuration check to raise 403 when disabled

* chore: remove pymongo from test dependencies in uv.lock

* chore:  optimize the _get_agent_name method

* test: enhance ChatStreamManager tests for PostgreSQL and MongoDB initialization

* test: add persistence tests for ChatStreamManager with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

* test: enhance persistence tests for ChatStreamManager with PostgreSQL and MongoDB to verify message aggregation

* test: add unit tests for ChatStreamManager with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-16 21:03:12 +08:00
CHANGXUBO
d65b8f8fcc feat: Add llms to support the latest Open Source SOTA models (#497)
* fix: update README and configuration guide for new model support and reasoning capabilities

* fix: format code for consistency in agent and node files

* fix: update test cases for environment variable handling in llm configuration

* fix: refactor message chunk conversion functions for improved clarity and maintainability

* refactor: remove enable_thinking parameter from LLM configuration functions

* chore: update agent-LLM mapping for consistency

* chore: update LLM configuration handling for improved clarity

* test: add unit tests for Dashscope message chunk conversion and LLM configuration

* test: add unit tests for message chunk conversion in Dashscope

* test: add unit tests for message chunk conversion in Dashscope

* chore: remove unused imports from test_dashscope.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-08-13 22:29:22 +08:00
Willem Jiang
9e691ecf20 fix: added configuration of python_repl (#503)
* fix: added configuration of python_repl

* fix the lint and unit test errors

* fix the lint and unit test errors

* fix:the lint check errors
2025-08-06 14:27:03 +08:00
suntp
e178483971 fix: Add streamable MCP server support (#468)
* fix: Add streamable MCP server support(#349)

* “Revert-timeout”

* fix lint and test check

* modify streamable error notify

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-29 14:04:04 +08:00
Zhonghao Liu
f92bf0ca22 Feat: Cross-Language Search for RAGFlow (#469)
* cross-language search

* test passed
2025-07-24 16:39:02 +08:00
殷逸维
660395485c remove volengine package (#464) 2025-07-23 06:06:57 +08:00
道心坚定韩道友
32d8e514e1 fix:env AGENT_RECURSION_LIMIT not work (#453)
* fix:env AGENT_RECURSION_LIMIT not work

* fix:add test

* black tests/unit/config/test_configuration.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2025-07-22 15:23:21 +08:00
DanielWalnut
6d8853b7c7 refine the research prompt (#460) 2025-07-22 14:49:04 +08:00