Handling Long-Running API Requests with Webhooks in Node.js

While working on a data migration API for MongoDB collections, we encountered issues with large datasets. Processing over 10,000 records took more than 4 minutes — longer than our API gateway's 3-minute-45-second timeout. The task was completed on the server, but the client saw an error. Sound familiar?

Quick Answer: Use webhooks to return an immediate response, process the task, and push the result when ready.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks. They allow one application (the source) to automatically send data to another application (the receiver) via a POST request when a specific event occurs. Think of them as event-driven callbacks — instead of the client requesting updates, your server pushes them.

Like getting a text message when your Uber arrives, instead of constantly checking the app.

Why Use Webhooks for Long-Running Tasks?

The timeout issue isn't rare — most API gateways (AWS, Nginx, Cloudflare, etc.) enforce limits of around 30 seconds to 5 minutes to prevent hanging connections.

Without webhooks, your options are limited:

- Shorten the task → Often impossible for large data processing.

- Client-side polling → Client repeatedly calls a "/status" endpoint → wastes bandwidth, battery, server resources, and delays updates.

- Background queues + status API → Still requires polling.

Webhooks flip the model:

- Your API returns success immediately (e.g., 202 Accepted). - The heavy work continues on the server. - When finished (success or failure), your server pushes the result directly to a URL the client provided.

Benefits: - No more timeout errors for clients. - Real-time updates without polling overhead. - Better scalability (fewer status checks hitting your servers). - Improved UX — clients get notified exactly when ready.

This pattern is used by Stripe (payment events), GitHub (push events), and many SaaS tools for async jobs like file conversions or data imports.

How It Works: Simple Webhook Notification Pattern in Node.js

This example demonstrates how easy it is to add webhook support to any API endpoint.

How it works:

1. The client includes optional webhook details in the request body.

2. Your API performs its main task (synchronously or asynchronously).

3. Upon completion (whether successful or not), if a webhook was provided, we send a notification with the result.

4. We include a safe retry mechanism in case the client's endpoint is temporarily unavailable.

• 

Implementation: Node.js + Express (Step-by-Step)

1. Request Format (Client sends webhook details)

{
  "sourceCollection": "old_users",
  "targetCollection": "new_users",
  "webhook": {
    "url": "https://client.com/api/webhook-handler",
    "secret": "super-secret-token-123"
  }
}

2. Core Processing (Immediate response + async webhook)

processRequest(requestData.body).catch((error) => {
     console.error('Unexpected error in background task runner:', error);                 
});

const axios = require('axios');

async function processRequest(requestBody) {
  const { webhook } = requestBody;

  // === Immediate response to client ===
  const immediateResponse = {
    status: 'accepted',
    message: 'Task started. You will be notified via webhook when complete.',
    timestamp: new Date().toISOString()
  };

  // === Fire-and-forget: Run long task in background ===
  (async () => {
    try {
      // Replace this with your actual operation
      await runBackgroundTask(
        requestBody,
        async (data) => {
          // This is your generic long-running work
          // Example: migrateMongoData(data), processFile(data), generateReport(data), etc.
          return await migrateMongoData(data); // Or any other async function
        },
        webhook
      );
    } catch (err) {
      console.error('Unexpected error in background task runner:', err);
    }
  })();

  return immediateResponse;
}

Why this works: Client gets 202 Accepted synchronously. Webhook fires asynchronously with retries.

Note: In production environments (e.g., containerised or serverless deployments), it’s safer to offload long-running tasks to a background job queue (e.g., BullMQ, SQS, RabbitMQ) instead of running them in-process. The above example is simplified for clarity.

3. Reliable Webhook Delivery (Exponential Backoff)

•   async function notifyWebhookWithRetry(url, secret, payload) {
      const maxAttempts = 4;
      let attempt = 1;
    
      while (attempt <= maxAttempts) {
        try {
          await axios.post(url, payload, {
            timeout: 8000,
            headers: {
              'Content-Type': 'application/json',
              'X-Webhook-Secret': secret
            }
          });
          console.log(`✅ Webhook delivered (attempt ${attempt})`);
          return;
        } catch (err) {
          if (attempt === maxAttempts) {
            console.error('❌ All retries failed. Log to dead-letter queue');
            return;
          }
          
          // Exponential backoff: 1s → 2s → 4s
          const delay = Math.pow(2, attempt - 1) * 1000;
          await new Promise(r => setTimeout(r, delay));
          attempt++;
        }
      }
    }
  

  Important Best Practices (Don't Skip These!)

  To make this pattern robust in production:

  1. Make your webhook handler idempotent.

     Since we retry on failure, the client might receive the same notification multiple times.

     → Always design your webhook endpoint to be idempotent: use a unique ID (e.g., from your task) to check if the event was already processed.

     Example: Before inserting a record, check if (await Record .exists({ taskId })) → skip if exists. This prevents duplicates during retries.

  2. Only return 2xx after successful processing.

     Your webhook endpoint should not return 200 OK immediately just to acknowledge receipt.

     → Return 2xx only after you have fully processed and persisted the data.

     If you return 2xx too early and then crash, we won't retry — your server assumes success.

     Best: Respond quickly, but process synchronously before sending 2xx (or queue and track).

  3. Use HTTPS for webhook URLs and validate the secret header.

  4. Log failed deliveries after all retries (e.g., to the database or monitoring tool).

  Conclusion

  This webhook pattern helped us resolve a frustrating timeout issue with minimal additional complexity. By returning an immediate response and allowing the server to notify the client when the long task finishes, we avoided gateway errors and provided users with a much smoother experience.

  It’s a simple yet powerful approach that works great for any operation that takes more than a few seconds — data migrations, file processing, report generation, or anything similar.

  I’m still learning about building reliable APIs myself, but this solution felt straightforward and effective in our project. If you’re facing similar timeout problems, give it a try and see how it works for you.