Advanced Testing Capabilities

Advanced testing features, performance testing, and security validation

Chapter 6

Chapter 6: Advanced Testing Capabilities

Comprehensive Test Suites

While the basic testing capabilities covered in the previous chapter provide a solid foundation, the mcp-client-cli offers advanced testing features for more thorough validation of MCP servers. Comprehensive test suites combine multiple test types into cohesive validation strategies.

Suite Structure and Organization

According to the TESTING.md documentation, comprehensive test suites are organized into a hierarchical structure:

test-suites/
├── basic/
│   ├── connectivity.json
│   ├── discovery.json
│   └── simple-tools.json
├── functional/
│   ├── all-tools.json
│   ├── workflows.json
│   └── error-handling.json
├── security/
│   ├── authentication.json
│   ├── authorization.json
│   └── input-validation.json
└── performance/
    ├── response-time.json
    ├── concurrency.json
    └── resource-usage.json

This organization allows for modular test composition, where suites can be combined and customized to match specific testing needs.

Predefined Test Suites

The mcp-client-cli includes several predefined test suites for common testing scenarios:

Basic Suite

# Run the basic test suite
mcp-test run-suite --suite-name basic --server-name your-server

The basic suite focuses on fundamental functionality:

  • Server connectivity and protocol compliance
  • Tool discovery and basic tool execution
  • Simple error handling

This suite is ideal for quick validation during development or as a smoke test before more comprehensive testing.

Comprehensive Suite

# Run the comprehensive test suite
mcp-test run-suite --suite-name comprehensive --server-name your-server

The comprehensive suite provides thorough validation across all aspects:

  • Complete functional testing of all tools
  • Security testing for common vulnerabilities
  • Basic performance testing
  • End-to-end workflow validation

This suite is recommended for pre-release validation or periodic health checks of production servers.

CI Suite

# Run the CI test suite
mcp-test run-suite --suite-name ci --server-name your-server

The CI suite is optimized for continuous integration environments:

  • Focused on critical functionality
  • Optimized for execution speed
  • Designed for clear pass/fail results
  • Includes regression tests for previously identified issues

This suite is ideal for automated testing in CI/CD pipelines.

Custom Suite Definition

For specialized testing needs, you can define custom test suites:

// custom-suite.json
{
  "name": "Custom Validation Suite",
  "description": "Specialized testing for data processing MCP server",
  "tests": [
    {
      "type": "functional",
      "include_tools": ["data_read", "data_process", "data_write"],
      "parameters": {
        "timeout": 30,
        "validation_level": "strict"
      }
    },
    {
      "type": "security",
      "include_categories": ["input_validation", "data_protection"],
      "parameters": {
        "malicious_payloads": true
      }
    },
    {
      "type": "performance",
      "include_scenarios": ["data_throughput", "concurrent_processing"],
      "parameters": {
        "duration": 60,
        "concurrent_connections": 10
      }
    },
    {
      "type": "workflow",
      "workflow_file": "workflows/data-pipeline.json"
    }
  ],
  "execution": {
    "parallel": true,
    "stop_on_failure": false,
    "retry_attempts": 2
  },
  "reporting": {
    "include_details": true,
    "generate_charts": true,
    "export_format": "html"
  }
}

To run a custom suite:

# Run a custom test suite
mcp-test run-suite --suite-file custom-suite.json --server-name your-server

Custom suites allow you to tailor testing to your specific requirements, focusing on the aspects most relevant to your MCP server implementation.

Security Testing

Security testing is critical for MCP servers, especially those that provide access to sensitive data or systems. The mcp-client-cli includes comprehensive security testing capabilities based on industry best practices.

Authentication Testing

Authentication testing verifies that the server properly validates client identity:

# Run authentication tests
mcp-test security --server-name your-server --category authentication

According to the Spring AI Reference documentation, authentication testing includes:

  1. Missing Credentials: Attempting access without authentication
  2. Invalid Credentials: Using incorrect authentication information
  3. Expired Credentials: Using outdated authentication tokens
  4. Malformed Headers: Sending improperly formatted authentication headers
  5. Brute Force Resistance: Testing protection against repeated authentication attempts

These tests help ensure that your MCP server properly enforces authentication requirements.

Authorization Testing

Authorization testing verifies that authenticated clients can only access appropriate resources:

# Run authorization tests
mcp-test security --server-name your-server --category authorization

Authorization testing includes:

  1. Access Control: Verifying that clients can only access authorized tools
  2. Privilege Escalation: Attempting to access higher-privilege functionality
  3. Horizontal Access: Attempting to access resources belonging to other users
  4. Parameter Restrictions: Verifying that parameter values are properly restricted
  5. Resource Limitations: Testing enforcement of resource usage limits

These tests help identify potential authorization bypasses or privilege escalation vulnerabilities.

Input Validation Testing

Input validation testing verifies that the server properly validates and sanitizes inputs:

# Run input validation tests
mcp-test security --server-name your-server --category input-validation

As noted in the TESTING.md documentation, input validation testing includes:

  1. SQL Injection: Attempting to inject SQL commands
  2. Command Injection: Attempting to execute system commands
  3. Cross-Site Scripting (XSS): Testing for script injection vulnerabilities
  4. Path Traversal: Attempting to access files outside allowed directories
  5. Format String Attacks: Testing for format string vulnerabilities
  6. Integer Overflow/Underflow: Testing numeric boundary conditions
  7. JSON/XML Injection: Attempting to manipulate structured data formats

These tests help identify input validation weaknesses that could lead to security vulnerabilities.

Data Protection Testing

Data protection testing verifies that sensitive information is properly safeguarded:

# Run data protection tests
mcp-test security --server-name your-server --category data-protection

Data protection testing includes:

  1. Sensitive Data Exposure: Checking for unintended disclosure of sensitive information
  2. Data Encryption: Verifying proper encryption of sensitive data
  3. Information Leakage: Testing for information disclosure in error messages
  4. Cache Security: Checking for sensitive data in caches
  5. Logging Security: Verifying that sensitive data is not logged inappropriately

These tests help ensure that your MCP server properly protects sensitive information.

Comprehensive Security Testing

For thorough security validation, you can run comprehensive security testing:

# Run comprehensive security tests
mcp-test security --server-name your-server --comprehensive

Comprehensive security testing combines all security test categories with additional advanced tests, providing thorough validation of your server’s security posture.

Security Testing Configuration

Security testing can be customized through configuration:

{
  "security": {
    "enabled": true,
    "test_authentication": true,
    "test_authorization": true,
    "test_input_validation": true,
    "test_data_protection": true,
    "malicious_payloads": true,
    "custom_security_tests": [
      "test_sql_injection",
      "test_xss_prevention",
      "test_command_injection"
    ],
    "payload_database": "security/payloads.json",
    "severity_threshold": "medium",
    "report_format": "detailed"
  }
}

This configuration allows you to enable specific security test categories, include custom tests, specify payload databases, set severity thresholds, and configure reporting options.

Performance Testing

Performance testing evaluates how well your MCP server performs under various conditions. The mcp-client-cli provides comprehensive performance testing capabilities to help you identify bottlenecks and optimize your implementation.

Response Time Testing

Response time testing measures how quickly your server responds to requests:

# Run response time tests
mcp-test performance --server-name your-server --category response-time

According to the mcp-client-cli documentation, response time testing includes:

  1. Basic Latency: Measuring time from request to response
  2. Tool-Specific Timing: Measuring response times for different tools
  3. Parameter Impact: Assessing how different parameters affect response time
  4. Consistency: Evaluating variation in response times
  5. Timeout Handling: Testing behavior when operations take too long

These tests help you understand and optimize your server’s responsiveness.

Load Testing

Load testing evaluates how your server performs under increasing request volume:

# Run load tests
mcp-test performance --server-name your-server --category load

Load testing includes:

  1. Throughput Measurement: Determining requests per second capacity
  2. Concurrency Handling: Testing with multiple simultaneous connections
  3. Saturation Point: Identifying when performance begins to degrade
  4. Recovery Time: Measuring how quickly the server recovers after high load
  5. Error Rates: Tracking error frequency under load

These tests help you understand your server’s capacity and optimize for higher throughput.

Resource Utilization Testing

Resource utilization testing monitors how efficiently your server uses system resources:

# Run resource utilization tests
mcp-test performance --server-name your-server --category resources

Resource utilization testing includes:

  1. CPU Usage: Monitoring processor utilization
  2. Memory Usage: Tracking memory allocation and deallocation
  3. Disk I/O: Measuring file system operations
  4. Network I/O: Monitoring network traffic
  5. Resource Leaks: Identifying resources that aren’t properly released

These tests help you identify inefficient resource usage and potential leaks.

Endurance Testing

Endurance testing evaluates how your server performs during extended operation:

# Run endurance tests
mcp-test performance --server-name your-server --category endurance --duration 3600

Endurance testing includes:

  1. Stability Over Time: Monitoring for degradation during extended operation
  2. Memory Leaks: Identifying gradual memory consumption
  3. Resource Exhaustion: Testing for eventual resource depletion
  4. Error Accumulation: Checking if errors increase over time
  5. Recovery Mechanisms: Evaluating self-healing capabilities

These tests help ensure that your server remains stable and performant during extended operation.

Performance Testing Configuration

Performance testing can be customized through configuration:

{
  "performance": {
    "enabled": true,
    "benchmark_tools": true,
    "concurrent_connections": 50,
    "test_duration": 300,
    "memory_monitoring": true,
    "resource_limits": {
      "max_memory_mb": 512,
      "max_cpu_percent": 80
    },
    "scenarios": [
      {
        "name": "light_load",
        "connections": 10,
        "duration": 60
      },
      {
        "name": "medium_load",
        "connections": 50,
        "duration": 120
      },
      {
        "name": "heavy_load",
        "connections": 100,
        "duration": 180
      }
    ],
    "monitoring_interval": 1,
    "report_percentiles": [50, 90, 95, 99]
  }
}

This configuration allows you to specify connection counts, test durations, resource limits, test scenarios, monitoring intervals, and reporting options.

Integration Testing

Integration testing validates how your MCP server works with other systems and in end-to-end scenarios. The mcp-client-cli provides several approaches to integration testing.

LLM Integration Testing

LLM integration testing verifies that your MCP server works correctly with language models:

# Run LLM integration tests
mcp-test integration --server-name your-server --category llm

According to Anthropic’s MCP documentation, LLM integration testing includes:

  1. Tool Selection: Verifying that LLMs can correctly identify and select tools
  2. Parameter Mapping: Testing how LLMs map natural language to tool parameters
  3. Result Interpretation: Evaluating how LLMs interpret and use tool results
  4. Error Handling: Testing LLM behavior when tools return errors
  5. Multi-Step Reasoning: Validating complex workflows involving multiple tool calls

These tests help ensure that your MCP server works effectively with language models.

Client Integration Testing

Client integration testing verifies compatibility with different MCP clients:

# Run client integration tests
mcp-test integration --server-name your-server --category clients

Client integration testing includes:

  1. Protocol Compatibility: Testing with different client implementations
  2. Version Compatibility: Verifying support for different protocol versions
  3. Edge Cases: Testing unusual client behaviors
  4. Error Handling: Validating client-server error communication
  5. Performance Characteristics: Measuring client-specific performance patterns

These tests help ensure broad compatibility with the MCP client ecosystem.

External System Integration

External system integration testing validates interaction with backend systems:

# Run external system integration tests
mcp-test integration --server-name your-server --category external

External system integration testing includes:

  1. API Connectivity: Verifying connections to external APIs
  2. Data Consistency: Validating data integrity across systems
  3. Authentication Propagation: Testing how credentials are handled
  4. Error Propagation: Verifying how external errors are communicated
  5. Performance Impact: Measuring how external systems affect performance

These tests help ensure reliable integration with the systems your MCP server depends on.

End-to-End Workflow Testing

End-to-end workflow testing validates complete user scenarios:

# Run end-to-end workflow tests
mcp-test integration --server-name your-server --category workflows

End-to-end workflow testing includes:

  1. Scenario Simulation: Testing realistic user workflows
  2. Cross-Tool Interaction: Validating data flow between tools
  3. State Management: Testing state preservation across operations
  4. Error Recovery: Verifying workflow resilience to failures
  5. Performance Characteristics: Measuring end-to-end performance

These tests help ensure that your MCP server works correctly in realistic usage scenarios.

Integration Testing Configuration

Integration testing can be customized through configuration:

{
  "integration": {
    "enabled": true,
    "llm_integration": {
      "enabled": true,
      "providers": ["openai", "anthropic"],
      "models": ["gpt-4", "claude-3-opus"],
      "scenarios": ["simple_queries", "complex_reasoning"]
    },
    "client_integration": {
      "enabled": true,
      "clients": ["python-client", "nodejs-client", "browser-client"],
      "protocol_versions": ["1.0", "2.0"]
    },
    "external_integration": {
      "enabled": true,
      "systems": ["database", "api", "file_system"],
      "mock_external": false
    },
    "workflow_integration": {
      "enabled": true,
      "workflow_files": ["workflows/data-processing.json", "workflows/content-generation.json"]
    }
  }
}

This configuration allows you to specify which integration test categories to run, which specific components to test, and how to configure each test category.

Issue Detection and Remediation

Beyond basic testing, the mcp-client-cli includes advanced capabilities for detecting and addressing issues in MCP servers.

Automated Issue Detection

The tool can automatically identify common issues:

# Run issue detection
mcp-test detect-issues --server-name your-server

According to the TESTING.md documentation, issue detection includes:

  1. Pattern Matching: Identifying known issue patterns
  2. Anomaly Detection: Flagging unusual behavior
  3. Performance Degradation: Detecting performance issues
  4. Security Vulnerabilities: Identifying potential security problems
  5. Protocol Violations: Flagging non-compliant behavior

These detection mechanisms help identify issues that might not be caught by standard tests.

Root Cause Analysis

For identified issues, the tool can help determine root causes:

# Analyze issue root causes
mcp-test analyze-issue --server-name your-server --issue-id issue-123

Root cause analysis includes:

  1. Trace Analysis: Examining request and response sequences
  2. Log Correlation: Connecting issues to log entries
  3. Dependency Mapping: Identifying related components
  4. Pattern Recognition: Matching issues to known patterns
  5. Historical Comparison: Comparing with previously identified issues

This analysis helps you understand why issues occur, facilitating effective resolution.

Automated Remediation

For some issues, the tool can suggest or apply automated fixes:

# Get remediation suggestions
mcp-test suggest-remediation --server-name your-server --issue-id issue-123

# Apply automated remediation
mcp-test apply-remediation --server-name your-server --issue-id issue-123 --remediation-id rem-456

Automated remediation includes:

  1. Configuration Adjustments: Modifying server configuration
  2. Resource Allocation: Adjusting resource limits
  3. Dependency Updates: Updating external dependencies
  4. Code Fixes: Applying patches for known issues
  5. Security Hardening: Implementing security best practices

These remediation capabilities help you quickly address identified issues.

Issue Tracking and Management

The mcp-client-cli includes features for tracking and managing issues over time:

# List all issues
mcp-test list-issues --server-name your-server

# Get issue details
mcp-test issue-details --issue-id issue-123

# Update issue status
mcp-test update-issue --issue-id issue-123 --status resolved

Issue tracking includes:

  1. Issue Database: Maintaining a record of identified issues
  2. Status Tracking: Monitoring issue resolution progress
  3. Prioritization: Ranking issues by severity and impact
  4. Trend Analysis: Identifying patterns in issue occurrence
  5. Resolution Verification: Confirming that issues have been resolved

These tracking capabilities help you manage the overall health of your MCP server.

Issue Detection Configuration

Issue detection and remediation can be customized through configuration:

{
  "issue_detection": {
    "enabled": true,
    "detection_interval": 5,
    "detection_methods": ["pattern", "anomaly", "performance", "security"],
    "sensitivity": "medium",
    "auto_remediation": {
      "enabled": true,
      "approval_required": true,
      "allowed_remediation_types": ["configuration", "resource", "dependency"],
      "excluded_remediation_types": ["code", "security"]
    },
    "notification": {
      "enabled": true,
      "channels": ["console", "email", "webhook"],
      "min_severity": "medium"
    }
  }
}

This configuration allows you to specify detection methods, sensitivity levels, auto-remediation settings, and notification preferences.

Custom Test Development

For specialized testing needs, the mcp-client-cli supports custom test development. This allows you to create tests tailored to your specific MCP server implementation.

Custom Test Structure

Custom tests are defined in Python modules with a specific structure:

# custom_tests/data_validation_test.py
from mcp_client_cli.testing import TestCase, TestResult, TestStatus

class DataValidationTest(TestCase):
    """Test data validation in the MCP server."""
    
    def __init__(self, config=None):
        super().__init__(config)
        self.name = "data_validation_test"
        self.description = "Validates data handling in MCP server"
        self.category = "functional"
    
    async def setup(self):
        """Prepare for test execution."""
        # Setup code here
        pass
    
    async def execute(self):
        """Execute the test."""
        # Test implementation
        client = await self.get_client()
        
        # Test valid data
        valid_result = await client.execute_tool("data_process", {
            "data": {"id": 123, "name": "Test Data"}
        })
        
        # Test invalid data
        invalid_result = await client.execute_tool("data_process", {
            "data": {"id": "not_a_number", "name": 12345}
        })
        
        # Validate results
        if valid_result.get("status") == "success" and invalid_result.get("status") == "error":
            return TestResult(
                status=TestStatus.PASSED,
                message="Data validation working correctly",
                confidence=95
            )
        else:
            return TestResult(
                status=TestStatus.FAILED,
                message="Data validation issues detected",
                confidence=90,
                details={
                    "valid_result": valid_result,
                    "invalid_result": invalid_result
                }
            )
    
    async def cleanup(self):
        """Clean up after test execution."""
        # Cleanup code here
        pass

This structure includes setup, execution, and cleanup phases, along with result reporting.

Registering Custom Tests

Custom tests must be registered with the testing framework:

# custom_tests/__init__.py
from mcp_client_cli.testing import register_test
from .data_validation_test import DataValidationTest

def register_custom_tests():
    register_test("data_validation", DataValidationTest)

Once registered, custom tests can be included in test suites or run directly:

# Run a custom test
mcp-test custom --server-name your-server --test-name data_validation

# Include in a test suite
mcp-test run-suite --server-name your-server --suite-file custom-suite.json

Custom Test Configuration

Custom tests can be configured through the standard configuration system:

{
  "custom_tests": {
    "data_validation": {
      "enabled": true,
      "parameters": {
        "validation_level": "strict",
        "test_data_file": "test-data/validation-samples.json"
      }
    }
  }
}

This configuration is passed to the test constructor and can be accessed during test execution.

Custom Test Utilities

The mcp-client-cli provides utilities to simplify custom test development:

from mcp_client_cli.testing.utils import (
    compare_results,
    generate_test_data,
    validate_schema,
    measure_performance,
    capture_logs
)

# Compare actual results with expected results
comparison = compare_results(actual_result, expected_result)

# Generate test data based on schema
test_data = generate_test_data(schema, count=10)

# Validate data against schema
validation_result = validate_schema(data, schema)

# Measure performance metrics
performance = measure_performance(lambda: client.execute_tool("example"))

# Capture logs during test execution
with capture_logs() as logs:
    result = client.execute_tool("example")
    log_entries = logs.get_entries()

These utilities help you create more sophisticated and effective custom tests.

Advanced Reporting and Analytics

The mcp-client-cli provides advanced reporting and analytics capabilities to help you understand test results and server performance.

Comprehensive Test Reports

For detailed analysis of test results:

# Generate a comprehensive report
mcp-test generate-report --server-name your-server --format html --output comprehensive-report.html

According to the mcp-client-cli documentation, comprehensive reports include:

  1. Executive Summary: High-level overview of test results
  2. Detailed Test Results: Complete information about each test
  3. Issue Analysis: Breakdown of identified issues
  4. Performance Metrics: Detailed performance data
  5. Trend Analysis: Comparison with previous test runs
  6. Recommendations: Suggested improvements and fixes

These reports provide a complete picture of your MCP server’s status and any issues that need attention.

Performance Analytics

For in-depth performance analysis:

# Generate performance analytics
mcp-test analyze-performance --server-name your-server --output performance-analysis.html

Performance analytics include:

  1. Response Time Distribution: Statistical analysis of response times
  2. Throughput Analysis: Requests per second under different conditions
  3. Resource Usage Patterns: CPU, memory, and I/O utilization
  4. Bottleneck Identification: Pinpointing performance limitations
  5. Optimization Suggestions: Recommendations for performance improvement

These analytics help you understand and optimize your server’s performance characteristics.

Security Analysis

For comprehensive security assessment:

# Generate security analysis
mcp-test analyze-security --server-name your-server --output security-analysis.html

Security analysis includes:

  1. Vulnerability Assessment: Identified security issues
  2. Risk Scoring: Severity and impact ratings
  3. Attack Surface Analysis: Potential entry points
  4. Mitigation Recommendations: Suggested security improvements
  5. Compliance Status: Alignment with security standards

This analysis helps you understand and address security risks in your MCP server.

Custom Analytics

For specialized analysis needs:

# Generate custom analytics
mcp-test analyze-custom --server-name your-server --analysis-file custom-analysis.json --output custom-analysis.html

Custom analytics allow you to define specific metrics and analyses tailored to your requirements:

// custom-analysis.json
{
  "name": "Data Processing Analysis",
  "metrics": [
    {
      "name": "data_throughput",
      "tool": "data_process",
      "measurement": "records_per_second"
    },
    {
      "name": "error_rate",
      "tool": "data_process",
      "measurement": "error_percentage"
    },
    {
      "name": "data_size_impact",
      "tool": "data_process",
      "measurement": "response_time_vs_size"
    }
  ],
  "visualizations": [
    {
      "type": "line_chart",
      "metric": "data_throughput",
      "x_axis": "concurrent_connections",
      "title": "Data Throughput vs. Concurrency"
    },
    {
      "type": "scatter_plot",
      "metric": "data_size_impact",
      "x_axis": "data_size_kb",
      "y_axis": "response_time_ms",
      "title": "Response Time vs. Data Size"
    },
    {
      "type": "bar_chart",
      "metric": "error_rate",
      "x_axis": "data_type",
      "title": "Error Rate by Data Type"
    }
  ],
  "comparisons": [
    {
      "name": "version_comparison",
      "baseline": "v1.0",
      "comparison": "v2.0",
      "metrics": ["data_throughput", "error_rate"]
    }
  ]
}

This flexibility allows you to create analytics that address your specific concerns and requirements.

Conclusion

This chapter has explored the advanced testing capabilities of the mcp-client-cli, including comprehensive test suites, security testing, performance testing, integration testing, issue detection and remediation, custom test development, and advanced reporting and analytics.

These advanced capabilities enable thorough validation of MCP servers across multiple dimensions, helping ensure that your implementations are functional, secure, performant, and reliable. By leveraging these capabilities, you can develop high-quality MCP servers that meet the needs of your users and integrate effectively with the broader MCP ecosystem.

In the next chapter, we’ll explore the AI-driven configuration system, which simplifies the setup and management of MCP server testing.