Skip to content

Connecting to AI Agents

Overview

A call on Aculab Cloud can be connected to a conversational AI agent such as Gemini Live or OpenAI. This is easily achieved in a REST Application by creating a websocket server, using the Connect action to connect the call to it, and bridging the audio between that socket and a connection (usually another websocket) to the chosen AI agent interface.

Typically the same application may provide function calling features to the AI engine which allow the AI agent to respond to certain requests from the caller (e.g. looking up flight details, booking an appointment, transfering to a live operator, making a recording) by delegating these operations back to the application.

What does the bridging application need to do

To bridge an Aculab Cloud call to a conversational AI agent, the application needs to:

  1. Run a secure WebSocket (WSS) server that Aculab Cloud can connect to from a REST Connect action, presenting a TLS certificate that Aculab Cloud trusts.
  2. Negotiate the subprotocol by echoing either v1.ws.cloud.aculab.com or v2.ws.cloud.aculab.com in the Sec-WebSocket-Protocol response header. Use v2 if you want DTMF delivered as separate DTMF event messages rather than in-band audio.
  3. Handle the Aculab Cloud control messages sent as JSON text frames:
    • call start — the caller has arrived; capture the caller's identity.
    • audio start — record the announced format, sample_rate and channels so audio can be decoded correctly. Audio is delivered as 16bit_PCM on calls to WebSockets.
    • audio end and call end — tear the session down cleanly.
    • DTMF event (v2 only) — forward digits to the AI or to your business logic as required.
  4. Open a session to the AI agent (for example a Gemini Live or OpenAI Realtime WebSocket) when the call arrives, using whatever authentication the AI provider requires. Configure the model, voice and system prompt for the conversation.
  5. Forward caller audio to the AI, decoding each binary frame from Aculab Cloud and resampling from the caller's rate (typically 8 kHz) to the rate the AI expects (for example 16 kHz PCM for Gemini Live).
  6. Forward AI audio back to the caller by:
    • Sending an audio play start control message declaring the format, sample rate and channels of the audio that will follow, together with an id so the play can be tracked.
    • Resampling the AI's output audio down to the caller's rate and sending it as binary frames. Each frame must be no larger than 1600 bytes and should contain a whole number of samples (10–20 ms of audio is recommended).
    • Sending audio play end when the AI's turn completes.
    • Sending audio play abort if the AI is interrupted (barge-in) so the caller hears the new response immediately.
  7. Respect Aculab Cloud limits. Frames larger than 1600 bytes will cause the connection to close. Aculab Cloud queues up to 250 audio frames and up to 4 concurrent plays per call; pace playback to real time to avoid an audio overflow warning.
  8. Handle function / tool calls from the AI, if used. When the AI requests a tool, execute the corresponding business logic (for example look up flight details or book an appointment) and return the result to the AI session so it can continue the conversation.
  9. End the call cleanly by sending a call hangup message when the AI or your application decides the conversation is over, or by closing the WebSocket in response to a call end from Aculab Cloud.

Reusable Python mixin and complete samples

Aculab provides the reusable aculab.websocket Python package. Its AculabCloudWebsocketMixin implements the Aculab Cloud WSS media-streaming protocol (subprotocol negotiation, JSON control framing, playback chunking, per-connection lifecycle and a built-in echo handler), so AI-specific bridges can concentrate on driving the AI session. Install the package from the repository or a package index and import the mixin with:

from aculab.websocket import AculabCloudWebsocketMixin

The mixin and each complete sample are documented on their own pages:

  • Python Websocket Mixin — a detailed reference for aculab.websocket and AculabCloudWebsocketMixin. Start here if you want to write your own bridge against a different AI backend.
  • Gemini Live Samples — sample applications that subclass AculabCloudWebsocketMixin to bridge Aculab Cloud calls to Google Gemini Live sessions, that include function-calling that illustrate integrating with other business systems.
  • OpenAI Realtime Samples — sample applications that subclass AculabCloudWebsocketMixin to bridge Aculab Cloud calls to OpenAI Realtime sessions, that include function-calling that illustrate integrating with other business systems.