cruelModel()

Wrap a language model to inject chaos and test error handling

Overview

cruelModel() wraps an AI SDK v3 language model to inject realistic failure scenarios and performance issues. Use it to test how your application handles AI errors, delays, and edge cases.

Function Signature

function cruelModel<T extends LanguageModelV3>(
  model: T,
  options?: CruelModelOptions
): T

model · LanguageModelV3 · required

The AI SDK v3 language model to wrap. This can be any model from providers like OpenAI, Anthropic, Google, etc.

options · CruelModelOptions

Chaos configuration options. See Chaos Options below.

Returns

Returns a wrapped model with the same type signature as the input model. The wrapped model intercepts doGenerate() and doStream() calls to inject chaos before and during generation.

Chaos Options

All options accept probability values between 0 and 1 (e.g., 0.1 = 10% chance).

Error Simulation

invalidApiKey · number

Probability of throwing an invalid API key error (401)

quotaExceeded · number

Probability of throwing a quota exceeded error (429)

modelUnavailable · number

Probability of throwing a model unavailable error (404)

contextLength · number

Probability of throwing a context length exceeded error (400)

contentFilter · number

Probability of throwing a content filter violation error (400)

emptyResponse · number

Probability of throwing an empty response error

fail · number

Probability of throwing a generic API failure (500)

timeout · number

Probability of hanging indefinitely (simulates network timeout)

rateLimit · number | { rate: number; retryAfter?: number }

Probability of throwing a rate limit error (429). Can specify retryAfter in seconds (default: 60)

// Simple form
rateLimit: 0.1

// With custom retry-after
rateLimit: { rate: 0.1, retryAfter: 120 }

overloaded · number

Probability of throwing a server overloaded error (503)

Performance Issues

delay · number | [number, number]

Add artificial delay before request in milliseconds. Can be a fixed value or [min, max] range

delay: 500        // Fixed 500ms delay
delay: [100, 500] // Random delay between 100-500ms

slowTokens · number | [number, number]

Add delay between each token in streaming responses (milliseconds)

Streaming Issues

streamCut · number

Probability of abruptly cutting the stream mid-response

corruptChunks · number

Probability of corrupting individual stream chunks by injecting replacement characters

Response Manipulation

partialResponse · number

Probability of truncating response text (returns 10-70% of content)

finishReason · 'stop' | 'length' | 'content-filter' | 'tool-calls' | 'error' | 'other'

Override the finish reason in the response. When set, always replaces the actual finish reason

tokenUsage · { inputTokens?: number; outputTokens?: number }

Override token usage counts in the response

tokenUsage: { inputTokens: 100, outputTokens: 50 }

Observability

onChaos · (event: ChaosEvent) => void

Callback invoked whenever chaos is injected. Useful for logging, metrics, and testing

onChaos: (event) => {
  console.log(`Chaos event: ${event.type}`, event)
}

Examples

Basic Error Testing

import { openai } from '@ai-sdk/openai'
import { cruelModel } from '@anthropic-ai/cruel'

const model = cruelModel(openai('gpt-4'), {
  rateLimit: 0.1,      // 10% chance of rate limit
  overloaded: 0.05,    // 5% chance of overload
  delay: [50, 200],    // Random 50-200ms delay
})

const result = await generateText({ model, prompt: 'Hello!' })

Testing Stream Failures

const model = cruelModel(openai('gpt-4'), {
  streamCut: 0.15,      // 15% chance stream cuts out
  slowTokens: [20, 100], // 20-100ms between tokens
  corruptChunks: 0.05,   // 5% chance of corrupt chunks
})

const result = await streamText({ model, prompt: 'Write a story' })

Production-Like Testing with Observability

const model = cruelModel(openai('gpt-4'), {
  rateLimit: 0.02,
  overloaded: 0.01,
  delay: [50, 150],
  onChaos: (event) => {
    // Track chaos events in your monitoring
    metrics.increment('chaos_event', { type: event.type })
  },
})

Simulating Token Counting Issues

const model = cruelModel(openai('gpt-4'), {
  tokenUsage: {
    inputTokens: 999999,  // Test billing logic with high counts
    outputTokens: 999999,
  },
})

Integration with AI SDK

The wrapped model works seamlessly with all AI SDK functions:

import { generateText, streamText, generateObject } from 'ai'
import { cruelModel } from '@anthropic-ai/cruel'
import { anthropic } from '@ai-sdk/anthropic'

const model = cruelModel(anthropic('claude-3-5-sonnet-20241022'), {
  rateLimit: 0.1,
})

// Works with generateText
await generateText({ model, prompt: 'Hello' })

// Works with streamText
await streamText({ model, prompt: 'Hello' })

// Works with generateObject
await generateObject({ model, schema: z.object({ name: z.string() }), prompt: 'Extract' })

Chaos Events

When onChaos is provided, it receives events with these types:

type ChaosEvent =
  | { type: 'rateLimit'; modelId: string }
  | { type: 'overloaded'; modelId: string }
  | { type: 'contextLength'; modelId: string }
  | { type: 'contentFilter'; modelId: string }
  | { type: 'modelUnavailable'; modelId: string }
  | { type: 'invalidApiKey'; modelId: string }
  | { type: 'quotaExceeded'; modelId: string }
  | { type: 'emptyResponse'; modelId: string }
  | { type: 'fail'; modelId: string }
  | { type: 'timeout'; modelId: string }
  | { type: 'delay'; modelId: string; ms: number }
  | { type: 'streamCut'; modelId: string }
  | { type: 'slowTokens'; modelId: string; ms: number }
  | { type: 'corruptChunk'; modelId: string }
  | { type: 'partialResponse'; modelId: string }

Related Functions

• cruelProvider() - Wrap an entire provider instead of individual models
• cruelMiddleware() - Apply chaos as AI SDK middleware
• presets - Pre-configured chaos scenarios