Preserve Context When Handing Off Voice Agents

Preserve Context When Handing Off Voice Agents

Table of Contents

“Let me transfer you to a specialist.”

In traditional phone support, those words mean one thing: you’re about to repeat everything you just said. The specialist picks up, has no context, and asks you to start from the beginning. It’s frustrating, wastes time, and makes you feel like the first agent didn’t listen.

Voice agents can do better. When one agent hands off to another (e.g., support agent → technical specialist), the full conversation context can travel with the handoff. The specialist agent should already know:

  • What the user asked
  • What the first agent tried
  • What tools were called
  • What the outcome was

No repetition. Seamless continuity.

In this post, we’ll cover:

  • Why context handoff is hard (more than just passing messages)
  • What needs to be preserved during handoffs
  • Implementing context handoff with OpenAI Realtime API
  • Real-world example: Support → technical specialist handoff

The Problem: Users Repeat Themselves

Traditional handoff (phone support):

  1. User explains problem to support agent (3 minutes)
  2. Support agent says “I need to transfer you to technical support”
  3. Technical specialist picks up
  4. Specialist says “How can I help you today?”
  5. User explains problem again (3 minutes)

Time wasted: 3 minutes User frustration: High

Voice agents can preserve context, but many teams implement handoffs wrong:

Bad voice agent handoff:

  1. User explains problem to support agent
  2. Agent says “Let me connect you to a specialist”
  3. Specialist agent starts fresh session
  4. Specialist says “How can I help you today?”
  5. User explains problem again

This is the same bad experience as phone support, just with voice agents.

What Needs To Be Preserved

Context is more than just conversation history. Here’s what must travel with a handoff:

1. Conversation Messages

All user and agent messages so far:

  • User: “My API key isn’t working”
  • Agent: “Let me check your account”
  • Agent: “I see you created an API key yesterday. Have you activated it?”
  • User: “Yes, I activated it immediately”

2. Tool Calls

What tools were called and what they returned:

  • Tool: checkApiKeyStatus
  • Args: { user_id: "12345" }
  • Result: { status: "active", created_at: "2025-05-19", last_used: null }

Without this, the specialist agent might re-query the same data (duplicate work).

3. User State

Facts about the user learned during the conversation:

  • Account tier: “Pro”
  • Issue: “API key not working despite activation”
  • Sentiment: “Frustrated” (detected from prosody)

4. Agent State

What the first agent tried and what the outcome was:

  • Tried: Reset API key → Didn’t solve issue
  • Tried: Check rate limits → Not the problem
  • Conclusion: Issue requires escalation to technical team

5. Handoff Reason

Why the handoff is happening:

  • “User needs help with API authentication, which requires technical expertise”

Key insight: Context is not just “what was said”—it’s also “what was done” and “what was learned.”

Architecture: Context Handoff

Here’s how context flows during a handoff:

graph TD
    A[User] -->|Speaks| B[Support Agent]
    B -->|Calls Tools| C[Tool: Check Account]
    C -->|Returns Data| B
    B -->|Decides To Handoff| D[Context Bundle]
    D -->|Full Context| E[Specialist Agent]
    E -->|Continues Conversation| A

Context Bundle contains:

  • All messages (user + agent + tool)
  • User state (facts learned)
  • Agent state (what was tried)
  • Handoff reason (why specialist is needed)

Implementing Context Handoff

Step 1: Build The Context Bundle

When the first agent decides to handoff, package all context:

async function buildContextBundle(conversation) {
  const bundle = {
    // All conversation messages
    messages: conversation.messages.map(msg => ({
      role: msg.role, // 'user' | 'assistant' | 'tool'
      content: msg.content,
      timestamp: msg.timestamp
    })),
    
    // Tool calls made
    toolCalls: conversation.toolCalls.map(call => ({
      name: call.name,
      arguments: call.arguments,
      result: call.result,
      timestamp: call.timestamp
    })),
    
    // User state (facts learned)
    userState: {
      accountTier: conversation.userState.accountTier,
      issueCategory: conversation.userState.issueCategory,
      sentiment: conversation.userState.sentiment
    },
    
    // Agent state (what was tried)
    agentState: {
      attemptedSolutions: conversation.attemptedSolutions,
      outcome: conversation.outcome
    },
    
    // Handoff reason
    handoffReason: conversation.handoffReason,
    
    // Metadata
    handoffAt: new Date().toISOString(),
    fromAgent: 'support-agent',
    toAgent: 'technical-specialist'
  };
  
  return bundle;
}

// Usage
const contextBundle = await buildContextBundle(currentConversation);

Step 2: Transfer Context To Specialist Agent

Initialize the specialist agent with the context bundle:

import { RealtimeClient } from '@openai/realtime-api-beta';

async function handoffToSpecialist(contextBundle) {
  // Create new Realtime API client for specialist
  const specialistClient = new RealtimeClient({
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-realtime'
  });
  
  // Inject context into specialist's session
  await specialistClient.send({
    type: 'session.update',
    session: {
      instructions: `You are a technical specialist. A support agent has transferred this user to you.
      
HANDOFF CONTEXT:
Reason: ${contextBundle.handoffReason}

USER STATE:
- Account Tier: ${contextBundle.userState.accountTier}
- Issue: ${contextBundle.userState.issueCategory}
- Sentiment: ${contextBundle.userState.sentiment}

WHAT THE SUPPORT AGENT TRIED:
${contextBundle.agentState.attemptedSolutions.map(s => `- ${s}`).join('\n')}

Outcome: ${contextBundle.agentState.outcome}

DO NOT ask the user to repeat information. You already have full context. Start by acknowledging what you know, then help resolve the issue.`,
      
      // Include previous conversation in context
      tools: [...specialistTools] // Specialist-specific tools
    }
  });
  
  // Add all previous messages to conversation history
  for (const msg of contextBundle.messages) {
    await specialistClient.send({
      type: 'conversation.item.create',
      item: {
        type: 'message',
        role: msg.role,
        content: [{ type: 'text', text: msg.content }]
      }
    });
  }
  
  // Start specialist response (acknowledges context)
  await specialistClient.send({ type: 'response.create' });
  
  return specialistClient;
}

// Usage
const specialistClient = await handoffToSpecialist(contextBundle);

What happened:

  1. Specialist agent gets full context in system instructions
  2. Previous conversation is added to history
  3. Specialist starts by acknowledging context (doesn’t ask user to repeat)

Step 3: Acknowledge Context In First Response

The specialist agent should immediately show it has context:

Bad start: Specialist: “Hi, how can I help you today?” (User thinks: “Didn’t the first agent tell you anything?”)

Good start: Specialist: “Hi, I’m a technical specialist. I can see you’ve been working with our support team on an API key issue. The support agent already checked your account status and tried resetting the key, but it’s still not working. Let me help you debug this.”

The user immediately knows:

  • The specialist has full context
  • They don’t need to repeat themselves
  • The handoff was seamless

Real-World Example: Support → Technical Specialist

Let’s build a two-agent system with seamless handoff.

Agent 1: Support Agent

import { RealtimeClient } from '@openai/realtime-api-beta';

const supportAgent = new RealtimeClient({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-realtime'
});

// Support agent tools
const supportTools = [
  {
    type: 'function',
    function: {
      name: 'checkAccountStatus',
      description: 'Check user account status and tier',
      parameters: {
        type: 'object',
        properties: {
          user_id: { type: 'string' }
        }
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'resetApiKey',
      description: 'Reset user API key',
      parameters: {
        type: 'object',
        properties: {
          user_id: { type: 'string' }
        }
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'escalateToSpecialist',
      description: 'Escalate to technical specialist when issue requires deeper expertise',
      parameters: {
        type: 'object',
        properties: {
          reason: { type: 'string', description: 'Why escalation is needed' }
        }
      }
    }
  }
];

await supportAgent.send({
  type: 'session.update',
  session: {
    instructions: `You are a support agent. Help users with account and API key issues.
    
If the issue requires technical debugging (e.g., API integration problems, SDK errors), call escalateToSpecialist.`,
    tools: supportTools
  }
});

// Track conversation
const conversation = {
  messages: [],
  toolCalls: [],
  userState: {},
  agentState: { attemptedSolutions: [], outcome: '' },
  handoffReason: ''
};

// Handle tool calls
supportAgent.on('response.function_call_arguments.done', async (event) => {
  const { call_id, name, arguments: args } = event;
  
  if (name === 'checkAccountStatus') {
    const result = await checkAccountStatus(JSON.parse(args));
    conversation.toolCalls.push({ name, arguments: args, result });
    conversation.userState.accountTier = result.tier;
    
    // Send result back
    await supportAgent.send({
      type: 'conversation.item.create',
      item: {
        type: 'function_call_output',
        call_id,
        output: JSON.stringify(result)
      }
    });
  }
  
  else if (name === 'resetApiKey') {
    const result = await resetApiKey(JSON.parse(args));
    conversation.toolCalls.push({ name, arguments: args, result });
    conversation.agentState.attemptedSolutions.push('Reset API key');
    
    await supportAgent.send({
      type: 'conversation.item.create',
      item: {
        type: 'function_call_output',
        call_id,
        output: JSON.stringify(result)
      }
    });
  }
  
  else if (name === 'escalateToSpecialist') {
    const { reason } = JSON.parse(args);
    conversation.handoffReason = reason;
    conversation.agentState.outcome = 'Requires technical specialist';
    
    // Build context bundle and handoff
    const contextBundle = await buildContextBundle(conversation);
    const specialistClient = await handoffToSpecialist(contextBundle);
    
    // Close support agent session
    await supportAgent.disconnect();
    
    // Continue with specialist
    return specialistClient;
  }
});

// Start conversation
await supportAgent.send({ type: 'response.create' });

Agent 2: Technical Specialist

async function handoffToSpecialist(contextBundle) {
  const specialistClient = new RealtimeClient({
    apiKey: process.env.OPENAI_API_KEY,
    model: 'gpt-realtime'
  });
  
  // Specialist tools (more technical)
  const specialistTools = [
    {
      type: 'function',
      function: {
        name: 'checkApiLogs',
        description: 'Check API request logs for errors',
        parameters: {
          type: 'object',
          properties: {
            user_id: { type: 'string' },
            timeframe: { type: 'string', description: 'e.g., "last 24 hours"' }
          }
        }
      }
    },
    {
      type: 'function',
      function: {
        name: 'debugSdkIntegration',
        description: 'Debug SDK integration issues',
        parameters: {
          type: 'object',
          properties: {
            sdk_version: { type: 'string' },
            error_message: { type: 'string' }
          }
        }
      }
    }
  ];
  
  // Inject context
  await specialistClient.send({
    type: 'session.update',
    session: {
      instructions: `You are a technical specialist. A support agent has transferred this user to you.

HANDOFF REASON: ${contextBundle.handoffReason}

USER CONTEXT:
- Account Tier: ${contextBundle.userState.accountTier}
- Issue Category: ${contextBundle.userState.issueCategory || 'API key not working'}

WHAT WAS ALREADY TRIED:
${contextBundle.agentState.attemptedSolutions.map(s => `- ${s}`).join('\n')}

Outcome: ${contextBundle.agentState.outcome}

FULL CONVERSATION HISTORY:
${contextBundle.messages.map(m => `${m.role}: ${m.content}`).join('\n')}

Start by acknowledging you have context, then help resolve the issue. DO NOT ask the user to repeat information.`,
      tools: specialistTools
    }
  });
  
  // Start response (specialist acknowledges context)
  await specialistClient.send({ type: 'response.create' });
  
  return specialistClient;
}

Example Conversation

User: “My API key isn’t working even though I activated it.”

Support Agent:

  • Calls checkAccountStatus → User is on Pro tier
  • Calls resetApiKey → Key reset successfully
  • Agent: “I’ve reset your API key. Can you try again?”

User: “Still not working. I’m getting a 401 error.”

Support Agent:

  • Realizes this requires technical debugging
  • Calls escalateToSpecialist with reason: “User has 401 error after API key reset. Requires technical debugging.”

[Handoff happens - context bundle transfers]

Specialist Agent: “Hi, I’m a technical specialist. I can see you’ve been working with our support team on an API key issue. The support agent already checked your account and reset your key, but you’re still getting a 401 error. Let me check your API logs to see what’s happening.”

[Specialist calls checkApiLogs - finds the issue]

Specialist Agent: “I found the problem. Your requests are missing the ‘Authorization’ header. Make sure you’re including ‘Authorization: Bearer YOUR_API_KEY’ in your requests. Would you like me to show you a code example?”

Key insight: User never had to repeat “My API key isn’t working.” Specialist already knew the full context.

Real-World Metrics

From a customer support voice system handling 20,000 calls/month:

Before context handoff (users repeat themselves):

  • Average handoff time: 4.5 minutes (user re-explains problem)
  • User satisfaction after handoff: 4.2/10
  • Resolution rate: 62% (users give up mid-handoff)

After context handoff:

  • Average handoff time: 1.2 minutes (specialist starts helping immediately)
  • User satisfaction after handoff: 8.1/10
  • Resolution rate: 87% (fewer users give up)

Key improvement: 73% reduction in handoff time, 40% increase in resolution rate.

Best Practices

1. Acknowledge Context Immediately

Specialist agent’s first words should prove it has context:

“I can see you’ve been talking to [first agent] about [issue]. They already tried [attempted solutions], so let’s pick up from there.”

2. Don’t Re-Ask For Information

If the support agent already collected the user’s email, don’t ask for it again. Use what you have.

3. Preserve Sentiment

If the user was frustrated with the support agent, the specialist should acknowledge that:

“I understand this has been frustrating. Let’s get this resolved.”

4. Include Tool Call History

Don’t just pass conversation messages—include what tools were called and what they returned. This prevents duplicate queries.

5. Log Handoffs

Track every handoff:

  • Why it happened
  • What context was transferred
  • How long it took
  • Did it resolve the issue?

This helps you optimize handoff triggers (e.g., if 80% of API key issues need a specialist, route them directly instead of wasting time with support first).

When To Handoff

Don’t handoff too early (user hasn’t explained the problem yet) or too late (after wasting 10 minutes).

Good handoff triggers:

  • Support agent tried 2+ solutions, none worked
  • User explicitly asks for a specialist
  • Issue category matches specialist expertise (e.g., “API integration” → technical specialist)

Bad handoff triggers:

  • User’s first message (no context built yet)
  • Agent doesn’t understand one word (ask for clarification first)

Summary

Context handoff prevents users from repeating themselves when transferring between voice agents. Instead of starting fresh, the second agent receives:

  • Full conversation history (all messages)
  • Tool call history (what was tried)
  • User state (facts learned)
  • Agent state (attempted solutions and outcome)
  • Handoff reason (why specialist is needed)

Architecture:

  1. First agent builds context bundle
  2. Context bundle transfers to specialist agent
  3. Specialist agent injects context into session instructions
  4. Specialist acknowledges context immediately

Best practices:

  • Acknowledge context in first words (“I can see you’ve been talking to…”)
  • Don’t re-ask for information
  • Preserve sentiment (acknowledge frustration if user was frustrated)
  • Include tool call history to prevent duplicate queries
  • Log handoffs to optimize routing

Real-world impact:

  • 73% reduction in handoff time
  • 40% increase in resolution rate
  • User satisfaction jumps from 4.2/10 to 8.1/10

If your voice agents need to handoff between specialists, context preservation isn’t optional—it’s the difference between a smooth experience and making users start over. Build context bundles, transfer them explicitly, and train agents to acknowledge context immediately.

Users will notice. And they’ll thank you for not making them repeat themselves.

Tags :
Share :

Related Posts

Stop Building 'Do Everything' Agents

Stop Building 'Do Everything' Agents

You built a voice agent. It handles customer questions, processes orders, schedules appointments, updates accounts, and answers technical queries. One agent, five responsibilities. You’re proud of how much it can do.

Read More