Files
deer-flow/tests/unit/tools/test_tavily_search_results_with_images.py

207 lines
6.9 KiB
Python
Raw Normal View History

# Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
# SPDX-License-Identifier: MIT
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
import json
from unittest.mock import AsyncMock, Mock, patch
import pytest
from src.tools.tavily_search.tavily_search_api_wrapper import (
EnhancedTavilySearchAPIWrapper,
)
from src.tools.tavily_search.tavily_search_results_with_images import (
TavilySearchWithImages,
)
class TestTavilySearchWithImages:
@pytest.fixture
def mock_api_wrapper(self):
"""Create a mock API wrapper."""
wrapper = Mock(spec=EnhancedTavilySearchAPIWrapper)
return wrapper
@pytest.fixture
def search_tool(self, mock_api_wrapper):
"""Create a TavilySearchWithImages instance with mocked dependencies."""
tool = TavilySearchWithImages(
max_results=5,
include_answer=True,
include_raw_content=True,
include_images=True,
include_image_descriptions=True,
)
tool.api_wrapper = mock_api_wrapper
return tool
@pytest.fixture
def sample_raw_results(self):
"""Sample raw results from Tavily API."""
return {
"query": "test query",
"answer": "Test answer",
"images": ["https://example.com/image1.jpg"],
"results": [
{
"title": "Test Title",
"url": "https://example.com",
"content": "Test content",
"score": 0.95,
"raw_content": "Raw test content",
}
],
"response_time": 1.5,
}
@pytest.fixture
def sample_cleaned_results(self):
"""Sample cleaned results."""
return [
{
"title": "Test Title",
"url": "https://example.com",
"content": "Test content",
}
]
def test_init_default_values(self):
"""Test initialization with default values."""
tool = TavilySearchWithImages()
assert tool.include_image_descriptions is False
assert isinstance(tool.api_wrapper, EnhancedTavilySearchAPIWrapper)
def test_init_custom_values(self):
"""Test initialization with custom values."""
tool = TavilySearchWithImages(max_results=10, include_image_descriptions=True)
assert tool.max_results == 10
assert tool.include_image_descriptions is True
def test_run_success(
self,
search_tool,
mock_api_wrapper,
sample_raw_results,
sample_cleaned_results,
):
"""Test successful synchronous run."""
mock_api_wrapper.raw_results.return_value = sample_raw_results
mock_api_wrapper.clean_results_with_images.return_value = sample_cleaned_results
result, raw = search_tool._run("test query")
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
assert result == json.dumps(sample_cleaned_results, ensure_ascii=False)
assert raw == sample_raw_results
mock_api_wrapper.raw_results.assert_called_once_with(
"test query",
search_tool.max_results,
search_tool.search_depth,
search_tool.include_domains,
search_tool.exclude_domains,
search_tool.include_answer,
search_tool.include_raw_content,
search_tool.include_images,
search_tool.include_image_descriptions,
)
mock_api_wrapper.clean_results_with_images.assert_called_once_with(
sample_raw_results
)
2025-07-22 14:49:04 +08:00
def test_run_exception(self, search_tool, mock_api_wrapper):
"""Test synchronous run with exception."""
mock_api_wrapper.raw_results.side_effect = Exception("API Error")
result, raw = search_tool._run("test query")
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
result_dict = json.loads(result)
assert "error" in result_dict
assert "API Error" in result_dict["error"]
assert raw == {}
mock_api_wrapper.clean_results_with_images.assert_not_called()
@pytest.mark.asyncio
async def test_arun_success(
self,
search_tool,
mock_api_wrapper,
sample_raw_results,
sample_cleaned_results,
):
"""Test successful asynchronous run."""
mock_api_wrapper.raw_results_async = AsyncMock(return_value=sample_raw_results)
mock_api_wrapper.clean_results_with_images.return_value = sample_cleaned_results
result, raw = await search_tool._arun("test query")
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
assert result == json.dumps(sample_cleaned_results, ensure_ascii=False)
assert raw == sample_raw_results
mock_api_wrapper.raw_results_async.assert_called_once_with(
"test query",
search_tool.max_results,
search_tool.search_depth,
search_tool.include_domains,
search_tool.exclude_domains,
search_tool.include_answer,
search_tool.include_raw_content,
search_tool.include_images,
search_tool.include_image_descriptions,
)
mock_api_wrapper.clean_results_with_images.assert_called_once_with(
sample_raw_results
)
@pytest.mark.asyncio
2025-07-22 14:49:04 +08:00
async def test_arun_exception(self, search_tool, mock_api_wrapper):
"""Test asynchronous run with exception."""
mock_api_wrapper.raw_results_async = AsyncMock(
side_effect=Exception("Async API Error")
)
result, raw = await search_tool._arun("test query")
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
result_dict = json.loads(result)
assert "error" in result_dict
assert "Async API Error" in result_dict["error"]
assert raw == {}
mock_api_wrapper.clean_results_with_images.assert_not_called()
def test_run_with_run_manager(
self,
search_tool,
mock_api_wrapper,
sample_raw_results,
sample_cleaned_results,
):
"""Test run with callback manager."""
mock_run_manager = Mock()
mock_api_wrapper.raw_results.return_value = sample_raw_results
mock_api_wrapper.clean_results_with_images.return_value = sample_cleaned_results
result, raw = search_tool._run("test query", run_manager=mock_run_manager)
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
assert result == json.dumps(sample_cleaned_results, ensure_ascii=False)
assert raw == sample_raw_results
@pytest.mark.asyncio
async def test_arun_with_run_manager(
self,
search_tool,
mock_api_wrapper,
sample_raw_results,
sample_cleaned_results,
):
"""Test async run with callback manager."""
mock_run_manager = Mock()
mock_api_wrapper.raw_results_async = AsyncMock(return_value=sample_raw_results)
mock_api_wrapper.clean_results_with_images.return_value = sample_cleaned_results
result, raw = await search_tool._arun(
"test query", run_manager=mock_run_manager
)
fix: resolve issue #467 - message content validation and Tavily search error handling (#645) * fix: resolve issue #467 - message content validation and Tavily search error handling This commit implements a comprehensive fix for issue #467 where the application crashed with 'Field required: input.messages.3.content' error when generating reports. ## Root Cause Analysis The issue had multiple interconnected causes: 1. Tavily tool returned mixed types (lists/error strings) instead of consistent JSON 2. background_investigation_node didn't handle error cases properly, returning None 3. Missing message content validation before LLM calls 4. Insufficient error diagnostics for content-related errors ## Changes Made ### Part 1: Fix Tavily Search Tool (tavily_search_results_with_images.py) - Modified _run() and _arun() methods to return JSON strings instead of mixed types - Error responses now return JSON: {"error": repr(e)} - Successful responses return JSON string: json.dumps(cleaned_results) - Ensures tool results always have valid string content for ToolMessages ### Part 2: Fix background_investigation_node Error Handling (graph/nodes.py) - Initialize background_investigation_results to empty list instead of None - Added proper JSON parsing for string responses from Tavily tool - Handle error responses with explicit error logging - Always return valid JSON (empty list if error) instead of None ### Part 3: Add Message Content Validation (utils/context_manager.py) - New validate_message_content() function validates all messages before LLM calls - Ensures all messages have content attribute and valid string content - Converts complex types (lists, dicts) to JSON strings - Provides graceful fallback for messages with issues ### Part 4: Enhanced Error Diagnostics (_execute_agent_step in graph/nodes.py) - Call message validation before agent invocation - Add detailed logging for content-related errors - Log message types, content types, and lengths when validation fails - Helps with future debugging of similar issues ## Testing - All unit tests pass (395 tests) - Python syntax verified for all modified files - No breaking changes to existing functionality * test: update tests for issue #467 fixes Update test expectations to match the new implementation: - Tavily search tool now returns JSON strings instead of mixed types - background_investigation_node returns empty list [] for errors instead of None - All tests updated to verify the new behavior - All 391 tests pass successfully * Update src/graph/nodes.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-23 22:08:14 +08:00
assert result == json.dumps(sample_cleaned_results, ensure_ascii=False)
assert raw == sample_raw_results