Design a Resumable Parallel File Uploader

Uploading small images or documents is straightforward. However, uploading large files (e.g., a 10GB video or a zip archive containing thousands of assets) introduces significant client-side architectural challenges. A naive upload will fail due to network drops, freeze the browser during file processing, exceed browser memory capacity, or overload server connection limits.

An interviewer asking this question wants to see if you can engineer a resilient, high-throughput uploading system using client-side APIs, background processes, and custom concurrency schedulers.

1. What the Interviewer Is Testing

When an interviewer asks you to design a file uploader, they are checking if you know how to build a robust system that can withstand volatile environments. Specifically, they evaluate:

• Blob Manipulation & Chunking: Do you know how the HTML5 File API works, and how to slice files without loading gigabytes of data into browser memory?
• Performance & Thread Management: Can you offload heavy computations (like calculating file hashes for duplicate prevention/resumption) to background threads (Web Workers) to maintain a smooth 60 FPS UI?
• Concurrency Control: How do you prevent browser connection limits from throttling your requests? Can you implement a custom scheduling queue to upload chunks in parallel?
• State Persistence & Offline Resilience: If the user's Wi-Fi drops mid-upload, how does the app resume automatically? If the user closes the tab or the browser crashes, how do we resume from the exact chunk we left off without re-uploading?
• Edge Cases: How do you handle directory uploads (recursive file parsing), duplicate file drops, and cleanup of aborted sessions?

2. How to Open Your Answer (The First 2 Minutes)

Clarify the scope and constraints of the file uploader before suggesting any concrete designs. Ask questions that define the scale and performance requirements:

Say something like:

*"Before I dive into the architecture, I'd like to clarify a few key assumptions.

First, what is the maximum file size we expect to support? I will assume files can be as large as 10GB to force a chunked architecture.

Second, are we uploading multiple files simultaneously, and do we need to support entire folder uploads (with nested directory structures)? I'll design the system to handle folder drops and parse the file tree recursively.

Third, how should we handle network interruptions? If the connection drops for 30 seconds and comes back, should the upload pause and auto-resume? What if the user closes the tab and reopens it the next day? I will design the system to persist chunk progress on the client (using IndexedDB) so uploads can resume even after a tab closure or browser crash.

Finally, are there any server-side upload limitations we must respect (like maximum HTTP request sizes)? I will design the client to split files into configurable chunks (e.g., 5MB each) to ensure compatibility with API gateways and cloud storage endpoints."*

3. The 6-Step Answer Framework

Step 1: Requirements Gathering (Functional & Non-Functional)

• Functional:
  • Support drag-and-drop and manual file selector inputs.
  • Render a list of active uploads showing individual progress bars, speeds, and estimated time of arrival (ETA).
  • Allow users to pause, resume, or cancel any upload.
  • Handle folder selection and nested folder drops.
• Non-Functional:
  • Resumability: Recover and resume uploads from the last successful chunk after network disconnects, tab refreshes, or browser crashes.
  • Zero Main-Thread Blocking: Keep the UI responsive during file processing. Hashing must not lock the screen.
  • Concurrency Limits: Upload chunks in parallel, capped at a maximum of $N$ concurrent connections (typically 3–6) to respect browser-domain limits.
  • Memory Constraints: Operate with $O(1)$ memory space overhead relative to file size.

Step 2: Component & UI Architecture

The file uploader interface must be modular to isolate state rendering:

DOM & File API Integration:

1. Drop Zone: A drop container listening to dragover, dragenter, dragleave, and drop events.
2. Directory Traversal: When folders are dropped, we access item.webkitGetAsEntry() from the DataTransferItem list. We write a recursive helper to traverse directory trees using FileSystemDirectoryReader to collect all File objects.
3. Local UI State:
   • uploads: A dictionary map tracking each file's metadata:

     interface FileUploadState {
       id: string; // Unique hash/fingerprint of the file
       file: File;
       progress: number; // 0 to 100
       status:
         | "IDLE"
         | "HASHING"
         | "UPLOADING"
         | "PAUSED"
         | "SUCCESS"
         | "ERROR";
       uploadedBytes: number;
       speedBytesPerSecond: number;
       etaSeconds: number;
       error?: string;
     }

Step 3: State Management & Resumption Flow

To support robust upload resumption, client-side metadata must persist locally. We use IndexedDB instead of localStorage because:

• localStorage is synchronous and blocks the main thread.
• localStorage has a strict storage limit (~5MB), which can easily be exhausted if tracking thousands of chunks.
• IndexedDB provides transactional object stores capable of storing megabytes of metadata asynchronously.

IndexedDB Storage Schema:

We establish an object store uploads_store keyed by the file's unique hash/fingerprint:

{
  "fileHash": "c20ad4188b4382cc0f749b5c2d33454b",
  "fileName": "large_presentation.mp4",
  "fileSize": 104857600,
  "chunkSize": 5242880,
  "uploadedChunks": [0, 1, 2, 4], // Indexes of chunks successfully uploaded
  "uploadSessionId": "sess_89a71b2d"
}

Step 4: Data Fetching & API Contract

We define a 3-stage REST API contract:

1. Initiate Upload Session

• Endpoint: POST /api/upload/initiate
• Request Body:

  {
    "fileName": "screencast.mov",
    "fileSize": 209715200,
    "fileHash": "ab83c129e928cf61c8b749d28ba111ab"
  }

• Response Body:

  {
    "sessionId": "sess_f89b210a",
    "chunkSize": 5242880, // Server-enforced chunk size (5MB)
    "uploadedChunks": [0, 1, 2] // List of chunks the server already has
  }

2. Chunk Upload

• Endpoint: POST /api/upload/chunk
• Headers:
  • Content-Type: application/octet-stream
  • X-Session-Id: sess_f89b210a
  • X-Chunk-Index: 3
  • X-Chunk-MD5: 9a0b12a84d...
• Request Body: Raw binary slice buffer.
• Response Body:

  { "success": true, "chunkIndex": 3 }

3. Finalize Upload

• Endpoint: POST /api/upload/finalize
• Request Body:

  {
    "sessionId": "sess_f89b210a",
    "fileHash": "ab83c129e928cf61c8b749d28ba111ab"
  }

• Response:

  {
    "success": true,
    "fileUrl": "https://cdn.example.com/uploads/screencast.mov"
  }

Step 5: Performance Considerations

1. $O(1)$ Browser Memory Control

When chunking, never read the entire file into memory (e.g., using reader.readAsArrayBuffer(file)). This will crash the browser tab for multi-gigabyte files. Instead, use file.slice(start, end).

• The slice method returns a Blob object, which is merely a reference to a slice of the file on disk.
• Only read the slice into memory right before uploading it (using fetch(blob) or XMLHttpRequest.send(blob)). The browser will garbage collect the uploaded chunk's memory immediately after the request completes.

2. Web Workers for Hashing

To calculate the MD5 file fingerprint, we slice the file into small chunks and feed them to an incremental hashing library (like spark-md5). Doing this on the main thread blocks user interactions (causing UI freeze/jank).

• Create a hash.worker.ts Web Worker.
• Post the File reference to the worker.
• The worker reads chunks sequentially using FileReaderSync (which blocks the worker's thread safely without touching the main thread), updates the hash block, and posts the final hexadecimal string back.

3. Concurrency Queuing

Browsers limit the number of parallel connections to a single domain (HTTP/1.1 limits to 6 connections; HTTP/2 allows multiplexing but server-side stream limits still apply).

• If you launch 50 chunk uploads concurrently, the browser will queue them internally, which makes network request cancellations, progress calculations, and timeouts hard to control.
• Implement a client-side worker pool pattern with a custom queue. Manage active request counters and pull from a buffer task list.

Step 6: Accessibility & Edge Cases

• Keyboard & Screen Reader Access:
  • Wrap input selectors in a <label> styled as a button with clear focus indicators.
  • Announce progress changes to screen reader users using an aria-live container. Instead of announcing every percent (which is annoying), announce progress in 10% increments using a throttled screen reader state updates.
• Graceful Retries with Exponential Backoff:
  • If a chunk request fails (network timeout/socket hangup), do not fail the entire file upload. Put the chunk back into the queue and retry.
  • Implement exponential backoff: retry after 1s, then 2s, 4s, 8s, up to a maximum limit (e.g., 5 attempts).
• Network Detection:
  • Listen to window.addEventListener('online', resumePendingUploads) and 'offline' events.
  • If the user goes offline, pause the queue automatically, switch the files to an ERROR or PAUSED status state, and alert the user. Once online, resume the queue.

4. What Candidates Miss

Stand out by raising these senior-level considerations:

1. Dynamic Chunk Sizing: Using a static 5MB chunk size is suboptimal. On a 1Gbps fiber connection, 5MB chunks cause excessive HTTP request overhead. On a slow 3G connection, 5MB chunks take too long and are prone to network timeouts.
   • Solution: Implement a dynamic chunk resizing algorithm. Measure the network speed of completed chunks. If a chunk uploads in under 1 second, double the next chunk's size (up to a 20MB ceiling). If it takes over 5 seconds, halve the chunk size (down to a 1MB floor).
2. Handling Duplicate File Drops: If a user drops the same 2GB file twice under different names, calculating the MD5 hash of both takes several minutes.
   • Solution: Generate a quick, "shallow" fingerprint first. Combine the file's name, size, type, and last modified timestamp into a string hash. Use this shallow hash to initiate the check. Calculate the full MD5 fingerprint as a background verify step only if a naming collision occurs.
3. Server-Side Zombie Chunk Cleanup: If a user pauses an upload and never returns, the server is left with hundreds of orphaned chunks occupying disk storage.
   • Solution: The server must implement a garbage collection cron job. Clean up any upload session where no chunk has been uploaded for more than 7 days. Provide a client endpoint to explicitly cancel/abort uploads to purge chunks immediately.

5. Code Implementation

Here are the two core client-side engines required to make a resumable file uploader function correctly: the background hashing worker controller and the concurrency queue scheduler.

1. File Hash Utility (Main Thread Worker Wrapper)

This class acts as the interface to the Web Worker, ensuring file fingerprinting is handled entirely in the background.

// file-hasher.ts
export class FileHasher {
  private worker: Worker | null = null;
 
  calculateHash(
    file: File,
    chunkSize: number = 5 * 1024 * 1024,
  ): Promise<string> {
    return new Promise((resolve, reject) => {
      // Inline worker creation or reference to external script
      const workerCode = `
        importScripts('https://cdnjs.cloudflare.com/ajax/libs/spark-md5/3.0.2/spark-md5.min.js');
        
        self.onmessage = function(e) {
          const { file, chunkSize } = e.data;
          const spark = new SparkMD5.ArrayBuffer();
          const reader = new FileReaderSync();
          
          const totalChunks = Math.ceil(file.size / chunkSize);
          let currentChunk = 0;
          
          try {
            while (currentChunk < totalChunks) {
              const start = currentChunk * chunkSize;
              const end = Math.min(start + chunkSize, file.size);
              const blob = file.slice(start, end);
              
              // Synchronously read blob inside worker thread
              const arrayBuffer = reader.readAsArrayBuffer(blob);
              spark.append(arrayBuffer);
              
              currentChunk++;
              
              // Post progress progress back to main thread
              self.postMessage({ 
                type: 'PROGRESS', 
                progress: Math.round((currentChunk / totalChunks) * 100) 
              });
            }
            
            const hash = spark.end();
            self.postMessage({ type: 'COMPLETE', hash });
          } catch (err) {
            self.postMessage({ type: 'ERROR', error: err.message });
          }
        };
      `;
 
      const blob = new Blob([workerCode], { type: "application/javascript" });
      this.worker = new Worker(URL.createObjectURL(blob));
 
      this.worker.postMessage({ file, chunkSize });
 
      this.worker.onmessage = (e) => {
        const { type, hash, progress, error } = e.data;
        if (type === "COMPLETE") {
          resolve(hash);
          this.terminate();
        } else if (type === "ERROR") {
          reject(new Error(error));
          this.terminate();
        } else if (type === "PROGRESS") {
          console.log(`Hashing progress: ${progress}%`);
        }
      };
 
      this.worker.onerror = (err) => {
        reject(err);
        this.terminate();
      };
    });
  }
 
  private terminate() {
    if (this.worker) {
      this.worker.terminate();
      this.worker = null;
    }
  }
}

2. The Concurrency-Limited Queue

This engine manages parallel requests, handles retries, and updates progress metrics for active upload tasks.

// upload-queue.ts
interface UploadTask {
  chunkIndex: number;
  chunkBlob: Blob;
  sessionId: string;
  chunkMD5: string;
  retryCount: number;
}
 
export class UploadQueue {
  private queue: UploadTask[] = [];
  private activeConnections = 0;
  private isPaused = false;
 
  constructor(
    private concurrencyLimit: number,
    private maxRetries: number,
    private onProgress: (chunkIndex: number, uploadedBytes: number) => void,
    private onComplete: () => void,
    private onError: (err: Error) => void,
  ) {}
 
  addTasks(tasks: UploadTask[]) {
    this.queue.push(...tasks);
    this.processNext();
  }
 
  pause() {
    this.isPaused = true;
  }
 
  resume() {
    this.isPaused = false;
    this.processNext();
  }
 
  private processNext() {
    if (
      this.isPaused ||
      this.activeConnections >= this.concurrencyLimit ||
      this.queue.length === 0
    ) {
      return;
    }
 
    // Pull next chunk task
    const task = this.queue.shift()!;
    this.activeConnections++;
    this.uploadChunk(task);
 
    // Keep filling connections up to the limit
    this.processNext();
  }
 
  private async uploadChunk(task: UploadTask) {
    const controller = new AbortController();
    const signal = controller.signal;
 
    try {
      const response = await fetch("/api/upload/chunk", {
        method: "POST",
        headers: {
          "X-Session-Id": task.sessionId,
          "X-Chunk-Index": task.chunkIndex.toString(),
          "X-Chunk-MD5": task.chunkMD5,
        },
        body: task.chunkBlob,
        signal,
      });
 
      if (!response.ok) {
        throw new Error(`Upload server responded with: ${response.status}`);
      }
 
      this.activeConnections--;
      this.onProgress(task.chunkIndex, task.chunkBlob.size);
 
      // Process next in queue
      this.processNext();
 
      // Trigger completion check if no queue remains
      if (this.activeConnections === 0 && this.queue.length === 0) {
        this.onComplete();
      }
    } catch (error: any) {
      this.activeConnections--;
 
      if (task.retryCount < this.maxRetries && !this.isPaused) {
        task.retryCount++;
        // Exponential backoff retry: 1s, 2s, 4s...
        const delay = Math.pow(2, task.retryCount) * 1000;
        console.warn(`Retrying chunk ${task.chunkIndex} in ${delay}ms...`);
 
        setTimeout(() => {
          this.queue.push(task); // Re-queue chunk task
          this.processNext();
        }, delay);
      } else {
        this.onError(
          new Error(
            `Failed to upload chunk ${task.chunkIndex} after maximum retries. Error: ${error.message}`,
          ),
        );
      }
    }
  }
}

6. Summary & Key Takeaways

• Blob Chunking preserves RAM: Never read whole files into JavaScript buffers. Slice files to get lightweight pointers (Blobs) and upload them segment by segment.
• Web Workers prevent UI freeze: Perform heavy, blocking file hash operations inside worker background scripts using synchronous FileReaders to keep the main screen responsive.
• Custom connection limits: Restrict active connections on the client-side to ensure stable uploading sequences, dynamic progress updates, and clean retry pathways.
• Persist chunk indexes: Track upload state in IndexedDB to survive page reloads and browser failures, enabling resumption from the same chunk index.