Webhook Retry Policy

The Audian API automatically retries failed webhook deliveries to ensure your application receives all events. This document explains the retry behavior and how to handle retries in your application.

Overview

When we send a webhook to your endpoint and don't receive a successful response (2xx status code), we automatically retry the delivery. This ensures that transient failures don't cause you to miss important events.

Retry Behavior

Initial Delivery + Retries

Attempt 1: Immediately
Attempt 2: After 1 minute (if failed)
Attempt 3: After 5 minutes (if failed)
Attempt 4: After 15 minutes (if failed)
Attempt 5: After 30 minutes (if failed)

Backoff Strategy

Audian uses exponential backoff with jitter to retry failed deliveries:

Backoff = base_delay * (2 ^ attempt_number) + random(0, jitter)

• Base delay: 1 minute
• Max attempts: 5 total attempts (initial + 4 retries)
• Max time: Up to 50 minutes from initial attempt
• Jitter: Random 0-60 seconds added to prevent thundering herd

Timeline Example

Attempt	Time	Status
1	0:00	Failed (timeout)
2	1:00 - 1:60	Retrying
3	6:00 - 6:60	Retrying
4	21:00 - 21:60	Retrying
5	51:00 - 51:60	Final attempt

If all 5 attempts fail, the webhook delivery is abandoned and an alert is sent to your account.

Response Status Codes

Success (2xx)

The webhook delivery is considered successful and won't be retried:

200 OK
201 Created
202 Accepted
204 No Content

Your endpoint should return 200 OK for all webhooks:

app.post('/webhooks', (req, res) => {
  handleWebhookAsync(req.body);
  res.status(200).json({ received: true });
});

Client Errors (4xx)

Client errors are NOT retried, as they typically indicate a problem with the request:

400 Bad Request      - Invalid webhook format
401 Unauthorized     - Invalid signature
403 Forbidden        - Access denied
404 Not Found        - Endpoint doesn't exist
410 Gone             - Endpoint has been removed
422 Unprocessable    - Malformed data

Server Errors (5xx)

Server errors ARE retried, as they may be transient:

500 Internal Server Error
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout

Network Issues

These are treated as errors and ARE retried:

• Connection refused
• Timeout (> 30 seconds)
• DNS resolution failure
• SSL/TLS errors

Idempotence and Deduplication

Since webhooks may be delivered multiple times, you must handle retries idempotently. Use the event id field for deduplication:

// Store processed event IDs in your database
const processedEvents = new Map();

app.post('/webhooks', (req, res) => {
  const event = req.body;
  const eventId = event.id;

  // Check if we've already processed this event
  if (processedEvents.has(eventId)) {
    console.log(`Event ${eventId} already processed`);
    return res.status(200).json({ received: true });
  }

  // Mark as processing
  processedEvents.set(eventId, { timestamp: Date.now(), status: 'processing' });

  // Process the event asynchronously
  handleEvent(event)
    .then(() => {
      processedEvents.set(eventId, { timestamp: Date.now(), status: 'completed' });
    })
    .catch((err) => {
      console.error(`Error processing event ${eventId}:`, err);
      // Return 5xx to trigger retry, or handle the error appropriately
    });

  // Return success immediately
  res.status(200).json({ received: true });
});

Database-Based Deduplication

For production systems, store processed event IDs in your database:

const pool = require('./db-pool');

app.post('/webhooks', async (req, res) => {
  const event = req.body;
  const eventId = event.id;

  try {
    // Check if processed
    const result = await pool.query(
      'SELECT id FROM webhook_events WHERE event_id = $1',
      [eventId]
    );

    if (result.rows.length > 0) {
      return res.status(200).json({ received: true });
    }

    // Process event
    await processEvent(event);

    // Mark as processed
    await pool.query(
      'INSERT INTO webhook_events (event_id, event_type, processed_at) VALUES ($1, $2, NOW())',
      [eventId, event.event]
    );

    res.status(200).json({ received: true });
  } catch (error) {
    console.error('Error processing webhook:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

Handling Retries Properly

DO: Process Asynchronously

Return HTTP 200 immediately, process the webhook in the background:

// Correct: Return quickly, process async
app.post('/webhooks', (req, res) => {
  const event = req.body;

  // Return immediately
  res.status(200).json({ received: true });

  // Process in the background
  setImmediate(() => {
    processEvent(event).catch(err => {
      console.error('Error processing event:', err);
    });
  });
});

DON'T: Block on Processing

Don't wait for processing to complete before responding:

// Wrong: Blocks until processing completes
app.post('/webhooks', async (req, res) => {
  const event = req.body;
  await processEvent(event); // Can timeout!
  res.status(200).json({ received: true });
});

DO: Set Appropriate Timeouts

Set generous timeouts for any external API calls:

async function processEvent(event) {
  try {
    // 30 second timeout for external calls
    const result = await Promise.race([
      callExternalAPI(event),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('Timeout')), 30000)
      )
    ]);
    return result;
  } catch (error) {
    console.error('Processing error:', error);
    throw error;
  }
}

DO: Return 5xx on Processing Errors

Return a 5xx status code if you want to retry:

app.post('/webhooks', async (req, res) => {
  try {
    // Process the webhook
    await processEvent(req.body);
    res.status(200).json({ received: true });
  } catch (error) {
    // This will trigger a retry
    console.error('Error processing webhook:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

Monitoring Retries

Check Webhook Delivery Logs

curl https://api.audian.com:8443/v2/webhooks/wh_1234567890abcdef/logs?status=failed \
  -H "X-Auth-Token: YOUR_API_KEY"

Response:

{
  "data": [
    {
      "id": "evt_1234567890abcdef",
      "webhook_id": "wh_1234567890abcdef",
      "event": "call.completed",
      "timestamp": "2024-01-15T10:30:00Z",
      "status": "failed",
      "response_status": 503,
      "response_body": "Service temporarily unavailable",
      "attempts": 5,
      "error": "All retry attempts exhausted",
      "last_attempt_at": "2024-01-15T10:50:00Z"
    }
  ]
}

Available Log Filters

Filter	Values	Description
status	delivered, failed, retrying	Delivery status
event	Event type	Filter by event type
since	ISO 8601 timestamp	Events after this time
until	ISO 8601 timestamp	Events before this time

Get Delivery Details

curl https://api.audian.com:8443/v2/webhooks/wh_1234567890abcdef/logs/evt_1234567890abcdef \
  -H "X-Auth-Token: YOUR_API_KEY"

Response:

{
  "id": "evt_1234567890abcdef",
  "webhook_id": "wh_1234567890abcdef",
  "event": "call.completed",
  "payload": {
    "id": "evt_1234567890abcdef",
    "timestamp": "2024-01-15T10:30:00Z",
    "event": "call.completed",
    "data": { /* ... */ }
  },
  "attempts": [
    {
      "number": 1,
      "timestamp": "2024-01-15T10:30:05Z",
      "status": 503,
      "duration_ms": 5000,
      "error": "Service unavailable"
    },
    {
      "number": 2,
      "timestamp": "2024-01-15T10:31:10Z",
      "status": 503,
      "duration_ms": 5000,
      "error": "Service unavailable"
    },
    // ... more attempts
  ]
}

Alerts and Notifications

Audian sends alerts when:

1. All retries exhausted: A webhook fails after 5 delivery attempts
2. High failure rate: > 5% of webhooks to an endpoint fail
3. Repeated failures: Same webhook fails multiple times

These alerts are sent to:

• Account email address
• Configured webhook alert endpoints
• Audian Dashboard notifications

Retry Limits

Per Event

• Maximum attempts: 5 (1 initial + 4 retries)
• Maximum time window: ~50 minutes

Per Webhook Endpoint

• Rate limit: Retries are rate-limited to prevent overwhelming slow endpoints
• Concurrent: Max 10 concurrent delivery attempts per webhook

Best Practices

1. Design for Idempotence: Use event IDs to deduplicate
2. Return Quickly: Send 200 within 2-5 seconds
3. Process Asynchronously: Handle events in background workers
4. Log Everything: Record all webhook events and processing
5. Monitor Delivery: Watch webhook logs for failures
6. Handle Duplicates: Always expect to see duplicate events
7. Set Timeouts: Add timeouts to any external API calls
8. Error Handling: Gracefully handle errors and partial failures

Troubleshooting

Webhook Not Being Retried

Check:

• Is the endpoint returning a 2xx status code?
• Is the HTTP response within 30 seconds?
• Is the endpoint publicly accessible?

Receiving Duplicate Events

This is expected behavior. Use event IDs to deduplicate in your code.

Too Many Retries

Check:

• Is your endpoint returning 5xx errors?
• Does it have a timeout? (should be > 30 seconds)
• Are you processing events too slowly?

Missing Events

Check:

• Is the webhook still active?
• Does the subscription include this event type?
• Check delivery logs for failures
• Verify the endpoint URL is correct

Next Steps

• Set up your webhook endpoint
• View all webhook events
• Verify webhook signatures