Building a Conversational AI App with Next.js

In this tutorial, we will guide you through the process of creating a web application that facilitates real-time voice interactions with AI agents using Next.js and ElevenLabs. This application will allow users to engage in natural voice conversations with an AI agent that can listen, understand, and respond using voice synthesis.

Prerequisites

To get started, ensure you have the following:

An ElevenLabs agent set up as per the official guide.
npm installed on your local machine.
Basic knowledge of Typescript or Javascript.

Setup

Step 1: Create a New Next.js Project

Open your terminal and run the following command to create a new Next.js project:

npm create next-app my-conversational-agent

Follow the default setup instructions provided by the command.

Step 2: Navigate to the Project Directory

Change your directory to the newly created project:

cd my-conversational-agent

Step 3: Install ElevenLabs Dependency

Install the ElevenLabs React package:

npm install @11labs/react

Step 4: Test the Setup

Start the development server and open the provided URL in your browser:

npm run dev

Implementing Conversational AI

Step 1: Create the Conversation Component

Create a new file app/components/conversation.tsx and add the following code:

'use client';
 
import { useConversation } from '@11labs/react';
import { useCallback } from 'react';
 
export function Conversation() {
  const conversation = useConversation({
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
    onMessage: (message) => console.log('Message:', message),
    onError: (error) => console.error('Error:', error),
  });
 
  const startConversation = useCallback(async () => {
    try {
      await navigator.mediaDevices.getUserMedia({ audio: true });
      await conversation.startSession({
        agentId: 'YOUR_AGENT_ID',
      });
    } catch (error) {
      console.error('Failed to start conversation:', error);
    }
  }, [conversation]);
 
  const stopConversation = useCallback(async () => {
    await conversation.endSession();
  }, [conversation]);
 
  return (
    <div className="flex flex-col items-center gap-4">
      <div className="flex gap-2">
        <button
          onClick={startConversation}
          disabled={conversation.status === 'connected'}
          className="px-4 py-2 bg-blue-500 text-white rounded disabled:bg-gray-300"
        >
          Start Conversation
        </button>
        <button
          onClick={stopConversation}
          disabled={conversation.status !== 'connected'}
          className="px-4 py-2 bg-red-500 text-white rounded disabled:bg-gray-300"
        >
          Stop Conversation
        </button>
      </div>
 
      <div className="flex flex-col items-center">
        <p>Status: {conversation.status}</p>
        <p>Agent is {conversation.isSpeaking ? 'speaking' : 'listening'}</p>
      </div>
    </div>
  );
}

Step 2: Update the Main Page

Replace the contents of app/page.tsx with:

import { Conversation } from './components/conversation';
 
export default function Home() {
  return (
    <main className="flex min-h-screen flex-col items-center justify-between p-24">
      <div className="z-10 max-w-5xl w-full items-center justify-between font-mono text-sm">
        <h1 className="text-4xl font-bold mb-8 text-center">
          ElevenLabs Conversational AI
        </h1>
        <Conversation />
      </div>
    </main>
  );
}

Next Steps

With the basic implementation in place, consider enhancing your application by:

Adding visual feedback for voice activity.
Implementing error handling and retry logic.
Displaying chat history.
Customizing the UI to align with your brand.

For more advanced features, explore the @11labs/react package.

Reference: This tutorial is based on the official ElevenLabs documentation. For more details, visit ElevenLabs. Author: ElevenLabs Team.