Hedge

Send multiple redundant requests and use the first successful response

Overview

The Hedge pattern (also known as “hedged requests” or “tail latency reduction”) sends multiple redundant requests to the same operation and returns the first successful response. This is useful for reducing tail latency when dealing with unpredictable response times.

When to Use

• Reducing tail latency (P99, P99.9) in distributed systems
• When you have multiple replicas that can serve the same request
• Operations where consistency between requests is guaranteed
• Systems where redundant work is acceptable for better latency

How It Works

1. Initial Request

   Send the first request immediately.

2. Wait

   Wait for a configured delay.

3. Hedge Request

   If the first request hasn’t completed, send another request.

4. First Wins

   Return the first successful response, cancel others.

API Reference

Function Signature

function withHedge<T extends AnyFn>(fn: T, options: HedgeOptions): T

Options

count · number · required

Total number of requests to send (including the initial request). Must be at least 2.

delay · number · required

Delay in milliseconds between sending each hedged request.

Examples

Basic Hedging

import { withHedge } from 'cruel'

const fetchUser = async (id: string) => {
  const response = await fetch(`https://api.example.com/users/${id}`)
  return response.json()
}

// Send up to 3 requests, 100ms apart
const hedgedFetch = withHedge(fetchUser, {
  count: 3,
  delay: 100,
})

// First successful response wins
const user = await hedgedFetch('123')

Database Query Hedging

// Query multiple read replicas
const hedgedQuery = withHedge(
  async (sql: string) => {
    const replica = selectRandomReplica()
    return replica.query(sql)
  },
  {
    count: 2,     // Query 2 replicas
    delay: 50,    // 50ms apart
  }
)

const results = await hedgedQuery('SELECT * FROM users WHERE id = 1')

Microservice Hedging

// Call the same microservice endpoint multiple times
const hedgedServiceCall = withHedge(
  async (data: RequestData) => {
    // Service has multiple instances behind load balancer
    return await serviceClient.request(data)
  },
  {
    count: 3,
    delay: 100,
  }
)

Search Query Hedging

// Search across multiple data centers
const hedgedSearch = withHedge(
  async (query: string) => {
    const datacenter = selectDatacenter()
    return datacenter.search(query)
  },
  {
    count: 2,
    delay: 200,  // Give first DC 200ms before hedging
  }
)

const results = await hedgedSearch('example query')

CDN Request Hedging

// Fetch from multiple CDN edge locations
const hedgedCDNFetch = withHedge(
  async (url: string) => {
    const edge = selectRandomEdge()
    return fetch(`${edge.url}${url}`)
  },
  {
    count: 2,
    delay: 100,
  }
)

const asset = await hedgedCDNFetch('/images/logo.png')

Combining with Other Patterns

Hedge + Timeout

import { withHedge, withTimeout } from 'cruel'

// Each hedged request has its own timeout
const resilientAPI = withHedge(
  withTimeout(apiCall, { ms: 5000 }),
  {
    count: 3,
    delay: 100,
  }
)

Hedge + Retry

import { withHedge, withRetry } from 'cruel'

// Retry each hedged request individually
const resilientAPI = withHedge(
  withRetry(apiCall, {
    attempts: 2,
    delay: 500,
  }),
  {
    count: 2,
    delay: 200,
  }
)

Hedge + Circuit Breaker

import { withHedge, createCircuitBreaker } from 'cruel'

// Circuit breaker applies to all hedged requests
const resilientAPI = withHedge(
  createCircuitBreaker(apiCall, {
    threshold: 5,
    timeout: 30000,
  }),
  {
    count: 2,
    delay: 100,
  }
)

With Compose

import { cruel } from 'cruel'

const resilientAPI = cruel.compose(apiCall, {
  hedge: {
    count: 3,
    delay: 100,
  },
  timeoutMs: 5000,
  retry: {
    attempts: 2,
    backoff: 'exponential',
  },
})

Advanced Examples

Hedging with Target Selection

interface Target {
  url: string
  latency: number  // Historical average latency
}

function createSmartHedge(
  targets: Target[],
  hedgeCount: number = 2
) {
  return withHedge(
    async (endpoint: string) => {
      // Select target with lowest historical latency
      const sorted = [...targets].sort((a, b) => a.latency - b.latency)
      const target = sorted[Math.floor(Math.random() * hedgeCount)]
      
      const start = Date.now()
      const response = await fetch(`${target.url}${endpoint}`)
      
      // Update latency stats
      target.latency = (target.latency * 0.9) + ((Date.now() - start) * 0.1)
      
      return response.json()
    },
    {
      count: hedgeCount,
      delay: 50,
    }
  )
}

const smartAPI = createSmartHedge([
  { url: 'https://us-east.api.example.com', latency: 100 },
  { url: 'https://us-west.api.example.com', latency: 150 },
  { url: 'https://eu.api.example.com', latency: 200 },
], 2)

Adaptive Hedging

class AdaptiveHedge {
  private p95Latency: number = 100
  private successRate: number = 1

  createHedge<T extends AnyFn>(fn: T) {
    // Adjust hedging based on observed performance
    const shouldHedge = this.successRate < 0.95 || this.p95Latency > 200
    
    if (!shouldHedge) {
      return fn  // Don't hedge if performance is good
    }

    const hedgeDelay = Math.max(50, this.p95Latency * 0.5)
    const hedgeCount = this.successRate < 0.8 ? 3 : 2

    return withHedge(fn, {
      count: hedgeCount,
      delay: hedgeDelay,
    })
  }

  recordLatency(latency: number) {
    this.p95Latency = (this.p95Latency * 0.95) + (latency * 0.05)
  }

  recordSuccess(success: boolean) {
    this.successRate = (this.successRate * 0.95) + (success ? 0.05 : 0)
  }
}

const adaptive = new AdaptiveHedge()
const apiCall = adaptive.createHedge(fetchData)

Hedging with Cancellation

import { withHedge } from 'cruel'

function createCancelableHedge<T extends AnyFn>(
  fn: T,
  hedgeOptions: HedgeOptions
) {
  const controllers: AbortController[] = []

  const wrapped = withHedge(
    async (...args: Parameters<T>) => {
      const controller = new AbortController()
      controllers.push(controller)

      try {
        return await fn(...args, { signal: controller.signal })
      } finally {
        const idx = controllers.indexOf(controller)
        if (idx > -1) controllers.splice(idx, 1)
      }
    },
    hedgeOptions
  )

  // Cancel all in-flight requests
  const cancelAll = () => {
    controllers.forEach(c => c.abort())
    controllers.length = 0
  }

  return Object.assign(wrapped, { cancelAll })
}

const hedgedAPI = createCancelableHedge(apiCall, {
  count: 3,
  delay: 100,
})

// Later...
hedgedAPI.cancelAll()

Hedging with Cost Tracking

interface HedgeMetrics {
  totalRequests: number
  hedgedRequests: number
  wastedRequests: number
  latencyImprovement: number
}

function createTrackedHedge<T extends AnyFn>(
  fn: T,
  options: HedgeOptions
): T & { getMetrics: () => HedgeMetrics } {
  const metrics: HedgeMetrics = {
    totalRequests: 0,
    hedgedRequests: 0,
    wastedRequests: 0,
    latencyImprovement: 0,
  }

  const wrapped = async (...args: Parameters<T>): Promise<ReturnType<T>> => {
    metrics.totalRequests++
    let completedCount = 0
    let firstResponseTime = 0

    const tracked = async () => {
      const start = Date.now()
      try {
        const result = await fn(...args)
        const duration = Date.now() - start
        
        completedCount++
        if (firstResponseTime === 0) {
          firstResponseTime = duration
        }
        
        return result
      } catch (error) {
        completedCount++
        throw error
      }
    }

    try {
      const result = await withHedge(tracked, options)()
      
      const wasted = options.count - 1
      metrics.hedgedRequests += options.count
      metrics.wastedRequests += wasted
      
      return result
    } catch (error) {
      throw error
    }
  }

  return Object.assign(wrapped as T, {
    getMetrics: () => ({ ...metrics }),
  })
}

const trackedAPI = createTrackedHedge(apiCall, {
  count: 3,
  delay: 100,
})

setInterval(() => {
  const metrics = trackedAPI.getMetrics()
  console.log('Hedge metrics:', metrics)
}, 60000)

Error Handling

If all hedged requests fail, the error from the first request is thrown:

import { CruelError } from 'cruel'

try {
  await hedgedAPI()
} catch (error) {
  if (error.code === 'CRUEL_HEDGE_FAILED') {
    console.log('All hedged requests failed')
  }
}

Best Practices

• Use for read operations: Hedging should only be used for idempotent operations
• Don’t hedge writes: Avoid hedging for operations that modify state
• Choose appropriate delays: Too short wastes resources, too long doesn’t help latency
• Monitor wasted work: Track how many hedged requests are wasted
• Consider costs: Hedging increases load and costs (compute, network, API calls)
• Use with timeouts: Prevent slow requests from holding resources
• Tune hedge count: More hedges improve latency but increase cost
• Combine with retries carefully: Hedging + retries can create many requests

When NOT to Use Hedging

• Write operations: Can cause duplicate writes
• Non-idempotent operations: May have side effects
• Rate-limited APIs: Wastes quota
• Expensive operations: High compute/cost impact
• Low latency variance: If P50 ≈ P99, hedging won’t help
• Resource-constrained systems: Increases load

Configuration Guidelines

Scenario	Count	Delay	Reasoning
Database reads	2	50-100ms	Balance latency vs load
Search queries	2-3	100-200ms	High tail latency
CDN requests	2	50-100ms	Geographic variance
Microservices	2	100-200ms	Instance variance
Cache lookups	2	10-50ms	Very low latency

Performance Considerations

Latency Improvement

Hedging is most effective when:

• P99 latency >> P50 latency (high tail latency)
• Variance is high
• Cost of extra work < cost of slow responses

Resource Cost

With hedge count N:

• Best case: 1 request completes
• Worst case: N requests complete
• Average: ~1.5-2 requests complete

Example Metrics

// Without hedging
P50: 100ms, P99: 2000ms

// With hedging (count: 2, delay: 100ms)
P50: 100ms, P99: 500ms  // 75% improvement in P99
Cost: 1.5x requests on average