This PR addresses token limit issues when web_search is enabled with include_raw_content by implementing a two-pronged approach: changing the default behavior to exclude raw content and adding compression logic for when raw content is included.
* feat: add Serper search engine support
* docs: update configuration guide and env example for Serper
* test: add test case for Serper with missing API key
* support infoquest
* support html checker
* support html checker
* change line break format
* change line break format
* change line break format
* change line break format
* change line break format
* change line break format
* change line break format
* change line break format
* Fix several critical issues in the codebase
- Resolve crawler panic by improving error handling
- Fix plan validation to prevent invalid configurations
- Correct InfoQuest crawler JSON conversion logic
* add test for infoquest
* add test for infoquest
* Add InfoQuest introduction to the README
* add test for infoquest
* fix readme for infoquest
* fix readme for infoquest
* resolve the conflict
* resolve the conflict
* resolve the conflict
* Fix formatting of INFOQUEST in SearchEngine enum
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: revert the part of patch of issue-710 to extract the content from the plan
* Upgrade the ddgs for the new compatible version
* Upgraded langchain to 1.1.0
updated langchain related package to the new compatable version
* Update pyproject.toml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: the crawling error when encountering PDF URLs
* Added the unit test for the new feature of crawl tool
* fix: address the code review problems
* fix: address the code review problems
* fix: resolve issue #467 - message content validation and Tavily search error handling
This commit implements a comprehensive fix for issue #467 where the application
crashed with 'Field required: input.messages.3.content' error when generating reports.
## Root Cause Analysis
The issue had multiple interconnected causes:
1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON
2. background_investigation_node didn't handle error cases properly, returning None
3. Missing message content validation before LLM calls
4. Insufficient error diagnostics for content-related errors
## Changes Made
### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py)
- Modified _run() and _arun() methods to return JSON strings instead of mixed types
- Error responses now return JSON: {"error": repr(e)}
- Successful responses return JSON string: json.dumps(cleaned_results)
- Ensures tool results always have valid string content for ToolMessages
### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py)
- Initialize background_investigation_results to empty list instead of None
- Added proper JSON parsing for string responses from Tavily tool
- Handle error responses with explicit error logging
- Always return valid JSON (empty list if error) instead of None
### Part 3: Add Message Content Validation (utils/context_manager.py)
- New validate_message_content() function validates all messages before LLM calls
- Ensures all messages have content attribute and valid string content
- Converts complex types (lists, dicts) to JSON strings
- Provides graceful fallback for messages with issues
### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py)
- Call message validation before agent invocation
- Add detailed logging for content-related errors
- Log message types, content types, and lengths when validation fails
- Helps with future debugging of similar issues
## Testing
- All unit tests pass (395 tests)
- Python syntax verified for all modified files
- No breaking changes to existing functionality
* test: update tests for issue #467 fixes
Update test expectations to match the new implementation:
- Tavily search tool now returns JSON strings instead of mixed types
- background_investigation_node returns empty list [] for errors instead of None
- All tests updated to verify the new behavior
- All 391 tests pass successfully
* Update src/graph/nodes.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: support additional Tavily search parameters via configuration to fix#548
- Add include_answer, search_depth, include_raw_content, include_images, include_image_descriptions to SEARCH_ENGINE config
- Update get_web_search_tool() to load these parameters from configuration with sensible defaults
- Parameters are now properly passed to TavilySearchWithImages during initialization
- This fixes 'got an unexpected keyword argument' errors when using web_search tool
- Update tests to verify new parameters are correctly set
* test: add comprehensive unit tests for web search configuration loading
- Add test for custom configuration values (include_answer, search_depth, etc.)
- Add test for empty configuration (all defaults)
- Add test for image_descriptions logic when include_images is false
- Add test for partial configuration
- Add test for missing config file
- Add test for multiple domains in include/exclude lists
All 7 new tests pass and provide comprehensive coverage of configuration loading
and parameter handling for Tavily search tool initialization.
* test: verify all Tavily configuration parameters are optional
Add 8 comprehensive tests to verify that all Tavily engine configuration
parameters are truly optional:
- test_tavily_with_no_search_engine_section: SEARCH_ENGINE section missing
- test_tavily_with_completely_empty_config: Entire config missing
- test_tavily_with_only_include_answer_param: Single param, rest default
- test_tavily_with_only_search_depth_param: Single param, rest default
- test_tavily_with_only_include_domains_param: Domain param, rest default
- test_tavily_with_explicit_false_boolean_values: False values work correctly
- test_tavily_with_empty_domain_lists: Empty lists handled correctly
- test_tavily_all_parameters_optional_mix: Multiple missing params work
These tests verify:
- Tool creation never fails regardless of missing configuration
- All parameters have sensible defaults
- Boolean parameters can be explicitly set to False
- Any combination of optional parameters works
- Domain lists can be empty or omitted
All 15 Tavily configuration tests pass successfully.
* fix: support local models by making thought field optional in Plan model
- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements
Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.
* feat: Add intelligent clarification feature for research queries
- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide
Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide
* fix: Improve clarification conversation continuity
- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds
Fixes issue where LLM would restart clarification instead of building upon previous exchanges.
* fix: Add conversation history to clarification context
* fix: resolve clarification feature message to planer, prompt, test issues
- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py
* fix: Add configurable max_clarification_rounds and comprehensive tests
- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
- Recursive call failed due to missing initial_state parameter
- Clarification exited prematurely at max rounds
- Incorrect logging of max rounds reached
* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json
* fix: add max_clarification_rounds parameter passing from frontend to backend
- Add max_clarification_rounds parameter in store.ts sendMessage function
- Add max_clarification_rounds type definition in chat.ts
- Ensure frontend settings page clarification rounds are correctly passed to backend
* fix: refine clarification workflow state handling and coverage
- Add clarification history reconstruction
- Fix clarified topic accumulation
- Add clarified_research_topic state field
- Preserve clarification state in recursive calls
- Add comprehensive test coverage
* refactor: optimize coordinator logic and type annotations
- Simplify handoff topic logic in coordinator_node
- Update type annotations from Tuple to tuple
- Improve code readability and maintainability
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Keep fixing #631
This pull request updates the crawl_tool function to return its results as a JSON string instead of a dictionary, and adjusts the unit tests accordingly to handle the new return type. The changes ensure consistent serialization of output and proper validation in tests.
- Change image result type from 'image' to 'image_url' to match OpenAI API expectations
- Wrap image URL in dict structure: {"url": "..."} instead of plain string
- Update SearchResultPostProcessor to handle dict-based image_url during duplicate removal
- Update tests to validate new image format
This fixes the 400 error: Invalid value: 'image'. Supported values are: 'text', 'image_url'...
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
* fix: support local models by making thought field optional in Plan model
- Make thought field optional in Plan model to fix Pydantic validation errors with local models
- Add Ollama configuration example to conf.yaml.example
- Update documentation to include local model support
- Improve planner prompt with better JSON format requirements
Fixes local model integration issues where models like qwen3:14b would fail
due to missing thought field in JSON output.
* feat: Add intelligent clarification feature for research queries
- Add multi-turn clarification process to refine vague research questions
- Implement three-dimension clarification standard (Tech/App, Focus, Scope)
- Add clarification state management in coordinator node
- Update coordinator prompt with detailed clarification guidelines
- Add UI settings to enable/disable clarification feature (disabled by default)
- Update workflow to handle clarification rounds recursively
- Add comprehensive test coverage for clarification functionality
- Update documentation with clarification feature usage guide
Key components:
- src/graph/nodes.py: Core clarification logic and state management
- src/prompts/coordinator.md: Detailed clarification guidelines
- src/workflow.py: Recursive clarification handling
- web/: UI settings integration
- tests/: Comprehensive test coverage
- docs/: Updated configuration guide
* fix: Improve clarification conversation continuity
- Add comprehensive conversation history to clarification context
- Include previous exchanges summary in system messages
- Add explicit guidelines for continuing rounds in coordinator prompt
- Prevent LLM from starting new topics during clarification
- Ensure topic continuity across clarification rounds
Fixes issue where LLM would restart clarification instead of building upon previous exchanges.
* fix: Add conversation history to clarification context
* fix: resolve clarification feature message to planer, prompt, test issues
- Optimize coordinator.md prompt template for better clarification flow
- Simplify final message sent to planner after clarification
- Fix API key assertion issues in test_search.py
* fix: Add configurable max_clarification_rounds and comprehensive tests
- Add max_clarification_rounds parameter for external configuration
- Add comprehensive test cases for clarification feature in test_app.py
- Fixes issues found during interactive mode testing where:
- Recursive call failed due to missing initial_state parameter
- Clarification exited prematurely at max rounds
- Incorrect logging of max rounds reached
* Move clarification tests to test_nodes.py and add max_clarification_rounds to zh.json
* feat:Add context compress
* feat: Add unit test
* feat: add unit test for context manager
* feat: add postprocessor param && code format
* feat: add configuration guide
* fix: fix the configuration_guide
* fix: fix the unit test
* fix: fix the default value
* feat: add test and log for context_manager
* add searx/searxng support
* nit
* Fix indentation in search.py for readability
* Clean up imports in search.py
Removed unused imports from search.py
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat:support config tavily search results
* feat: support config tavily search results
* feat: update the default value of include_images
* fix: fix the test
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test: add unit tests in server
* test: add unit tests of app.py in server
* test: reformat the codes
* test: add more tests to cover the exception part
* test: add more tests on the server app part
* fix: don't show the detail exception to the client
* test: try to fix the CI test
* fix: keep the TTS API call without exposure information
* Fixed the unit test errors
* Fixed the lint error
* test: add more test on test_tts.py
* test: add unit test of search and retriever in tools
* test: remove the main code of search.py
* test: add the travily_search unit test
* reformate the codes
* test: add unit tests of tools
* Added the pytest-asyncio dependency
* added the license header of test_tavily_search_api_wrapper.py
* feat: local search tool call result display
* chore: add file copyright
* fix: miss edit plan interrupt feedback
* feat: disable pasting html into input box
