Tool Chaos

Inject failures into AI tool execution

Cruel provides cruelTool() and cruelTools() to simulate tool execution failures, timeouts, and delays when using AI SDK tools.

API Reference

cruelTool

function cruelTool<T extends { execute: (...args: any[]) => any }>(
  tool: T,
  options?: CruelChaosOptions
): WrappedTool<T>

tool · { execute: Function } · required

Any object with an execute function. Works with AI SDK tool() and custom tools.

options · CruelChaosOptions

Chaos options:

• toolFailure - Probability of tool execution failure (0-1)
• toolTimeout - Probability of tool timeout (0-1)
• delay - Add latency before tool execution
• onChaos - Monitor chaos events

return · WrappedTool<T>

Returns wrapped tool with async execute function.

cruelTools

function cruelTools<T extends Record<string, unknown>>(
  tools: T,
  options?: CruelChaosOptions
): WrappedTools<T>

tools · Record<string, Tool> · required

Object of tools (typically from AI SDK’s tools parameter).

options · CruelChaosOptions

Same chaos options applied to all tools.

return · WrappedTools<T>

Returns object with all tools wrapped.

Basic Usage

Single Tool

import { tool } from 'ai'
import { cruelTool } from 'cruel/ai-sdk'
import { z } from 'zod'

const weatherTool = tool({
  description: 'Get weather',
  inputSchema: z.object({ city: z.string() }),
  execute: async ({ city }) => {
    return `${city}: 22C, sunny`
  },
})

const chaosTool = cruelTool(weatherTool, {
  toolFailure: 0.2,   // 20% chance of failure
  delay: [100, 300],  // 100-300ms delay
})

// Use in generateText
import { generateText, openai } from 'ai'

const result = await generateText({
  model: openai('gpt-4'),
  tools: { weather: chaosTool },
  prompt: 'What is the weather in Berlin?',
})

Tool Chaos Options

toolFailure · number

Probability that tool execution throws an error (0-1).

cruelTool(tool, { toolFailure: 0.2 })  // 20% failure rate

Throws: Error('Tool execution failed')

toolTimeout · number

Probability that tool execution never completes (0-1).

cruelTool(tool, { toolTimeout: 0.1 })  // 10% timeout rate

Hangs indefinitely (simulates timeout)

delay · number | [number, number]

Add latency before tool execution.

// Fixed delay
delay: 500  // Always 500ms

// Random delay
delay: [100, 1000]  // 100-1000ms

onChaos · (event: ChaosEvent) => void

Callback when chaos is injected.

onChaos: (event) => {
  console.log('Tool chaos:', event.type)
}

Events: toolFailure, toolTimeout

Tool chaos options (toolFailure, toolTimeout) work with cruelTool and cruelTools.

When used with cruelModel, cruelProvider, or cruelMiddleware, these options don’t affect the model wrapper — they’re only for tool wrapping.

How It Works

// Simplified implementation
function cruelTool(tool, options) {
  return {
    ...tool,
    execute: async (...args) => {
      // Check for failure
      if (chance(options.toolFailure)) {
        options.onChaos?.({ type: 'toolFailure' })
        throw new Error('Tool execution failed')
      }
      
      // Check for timeout
      if (chance(options.toolTimeout)) {
        options.onChaos?.({ type: 'toolTimeout' })
        await new Promise(() => {})  // Never resolves
      }
      
      // Add delay
      const ms = getDelay(options.delay)
      if (ms > 0) await sleep(ms)
      
      // Execute original
      return tool.execute(...args)
    },
  }
}

See ai-sdk.ts:495-516 for implementation.

Combining with Model Chaos

You can combine tool chaos with model chaos:

import { openai } from '@ai-sdk/openai'
import { generateText, tool } from 'ai'
import { cruelModel, cruelTools } from 'cruel/ai-sdk'
import { z } from 'zod'

// Chaos at model level
const model = cruelModel(openai('gpt-4'), {
  rateLimit: 0.15,
  delay: [100, 300],
})

// Chaos at tool level
const tools = cruelTools(
  {
    weather: tool({
      description: 'Get weather',
      inputSchema: z.object({ city: z.string() }),
      execute: async ({ city }) => `${city}: 22C`,
    }),
  },
  {
    toolFailure: 0.2,
    delay: [50, 150],
  }
)

const result = await generateText({
  model,
  tools,
  prompt: 'What is the weather in Paris?',
  maxRetries: 3,
})

// Now you test:
// 1. Model failures (rate limits, etc.)
// 2. Tool failures (execution errors)
// 3. Combined scenarios

Real-World Example

import { openai } from '@ai-sdk/openai'
import { generateText, tool } from 'ai'
import { cruelModel, cruelTools, presets } from 'cruel/ai-sdk'
import { z } from 'zod'

// Simulate realistic failures
const model = cruelModel(openai('gpt-4'), presets.unstable)

const tools = cruelTools(
  {
    database: tool({
      description: 'Query database',
      inputSchema: z.object({ query: z.string() }),
      execute: async ({ query }) => {
        // Actual database call
        return await db.query(query)
      },
    }),
    
    api: tool({
      description: 'Call external API',
      inputSchema: z.object({ endpoint: z.string() }),
      execute: async ({ endpoint }) => {
        // Actual API call
        return await fetch(`https://api.example.com/${endpoint}`)
      },
    }),
  },
  {
    toolFailure: 0.15,   // 15% failure (network issues, etc.)
    toolTimeout: 0.05,   // 5% timeout
    delay: [100, 500],   // Network latency
    
    onChaos: (event) => {
      console.error(`Tool ${event.type} detected`)
      sendAlert(`Tool chaos: ${event.type}`)
    },
  }
)

try {
  const result = await generateText({
    model,
    tools,
    prompt: 'Get user data from database and enrich with API',
    maxRetries: 3,
  })
  
  console.log('Success:', result.text)
} catch (error) {
  console.error('Failed after retries:', error)
}

Monitoring Tool Failures

import { cruelTools } from 'cruel/ai-sdk'

let toolCalls = 0
let toolFailures = 0
let toolTimeouts = 0

const tools = cruelTools(myTools, {
  toolFailure: 0.2,
  toolTimeout: 0.1,
  
  onChaos: (event) => {
    if (event.type === 'toolFailure') {
      toolFailures++
    }
    if (event.type === 'toolTimeout') {
      toolTimeouts++
    }
  },
})

// Wrap tool execution to count calls
for (const [name, tool] of Object.entries(tools)) {
  const original = tool.execute
  tool.execute = async (...args) => {
    toolCalls++
    return original(...args)
  }
}

// Run tests
await runTestSuite(tools)

console.log(`Tool calls: ${toolCalls}`)
console.log(`Failures: ${toolFailures} (${(toolFailures/toolCalls*100).toFixed(1)}%)`)
console.log(`Timeouts: ${toolTimeouts} (${(toolTimeouts/toolCalls*100).toFixed(1)}%)`)

With Presets

Built-in presets include tool chaos:

import { cruelTools, presets } from 'cruel/ai-sdk'

// presets.nightmare includes:
// - toolFailure: 0.1
// - toolTimeout: 0

// presets.apocalypse includes:
// - toolFailure: 0.2
// - toolTimeout: 0.1

const tools = cruelTools(myTools, {
  ...presets.nightmare,
  onChaos: (event) => console.log(event.type),
})

See Presets for details.

Testing Tool Resilience

Use tool chaos to verify:

Error Handling

Does your app handle tool failures gracefully?

toolFailure: 0.5  // 50% failure rate

Timeout Handling

Does your app timeout tools properly?

toolTimeout: 0.3  // 30% timeout rate

Retry Logic

Do retries work for failed tools?

maxRetries: 3
toolFailure: 0.3

Fallback Strategies

Do you fallback when tools fail?

// Test fallback paths
toolFailure: 0.8

Non-Tool Objects

cruelTool works with any object that has an execute function:

import { cruelTool } from 'cruel/ai-sdk'

const customTool = {
  name: 'calculator',
  execute: async (a: number, b: number) => a + b,
}

const chaosTool = cruelTool(customTool, {
  toolFailure: 0.1,
  delay: [10, 50],
})

// Works!
const result = await chaosTool.execute(2, 3)

If the object doesn’t have execute, cruelTool returns it unchanged:

const notATool = { foo: 'bar' }
const wrapped = cruelTool(notATool, { toolFailure: 0.1 })
// wrapped === notATool (unchanged)

TypeScript

import type { CruelChaosOptions } from 'cruel/ai-sdk'
import { cruelTool, cruelTools } from 'cruel/ai-sdk'

type MyTool = {
  execute: (arg: string) => Promise<number>
}

const tool: MyTool = {
  execute: async (arg) => arg.length,
}

// Type-safe wrapping
const wrapped = cruelTool(tool, { toolFailure: 0.1 })
// typeof wrapped.execute === (arg: string) => Promise<number>

// Works with tool objects
const tools = {
  a: { execute: async () => 1 },
  b: { execute: async () => 2 },
  c: { notATool: true },  // Ignored
}

const wrappedTools = cruelTools(tools, { toolFailure: 0.1 })
// wrappedTools.a.execute: () => Promise<number>
// wrappedTools.b.execute: () => Promise<number>
// wrappedTools.c: { notATool: true }

Next Steps

Model Chaos

Learn about model-level chaos

Presets

Use built-in chaos presets

Overview

Back to AI SDK overview