Skip to content

Audio Configurations

The Audio Configurations page in the PinionAI Studio allows you to manage Speech-to-Text (transcription) and Text-to-Speech (synthesis) engines. By configuring audio interfaces, you can give your AI agents the power of hearing and speaking—connecting them to state-of-the-art cloud providers or locally hosted open-source models.

Permissions & Prerequisites

Access to the Audio Configurations page and its underlying operations is controlled by user permissions and system context:

  1. Account & Agent Context: You must first select an active Account from the sidebar. You can also filter configurations or associate new ones with a selected Agent.
  2. Required Roles: Users must have one of the following permissions:
    • owner
    • admin
    • editor
    • audios (explicit page-level permission)
    • watchover (super-admin / global monitor)

Page Overview

The page is split into two primary areas: - Audio Builder / AI Architect: For adding new configurations manually or with the help of the AI configuration assistant. - Audio Configuration List: An expandable list of all current audio configurations associated with the selected account or agent, with inline actions to edit, duplicate, or delete them.

AI Audio Architect :material/auto_awesome:

Configuring endpoints, API keys, models, and protocols can be complex. To streamline this process, PinionAI Studio includes an AI Audio Architect assistant.

  1. Click the :material/auto_awesome: AI Suggest Configuration button under the New Audio section.
  2. A large dialog will open asking you to describe your goals (e.g., "I want to configure ElevenLabs for high-quality English text-to-speech" or "Set up a local faster-whisper server on my local network").
  3. Click Generate Suggested Configuration. The AI will automatically design a complete configuration containing recommended providers, models, endpoints, port configurations, and protocols.
  4. Review the suggested fields and click Apply Suggested Configuration to automatically populate the creation form, or click Cancel to discard.

Adding a New Audio Configuration

Expand the New Audio section to create a configuration.

1. General & Core Settings

  • Direction: A selectbox determining the operational flow of the configuration:
  • speech-to-text (STT / transcription)
  • text-to-speech (TTS / voice synthesis)
  • Configuration Name: A unique, descriptive identifier for this voice profile (e.g., Corporate Voice Synthesizer).
  • Active: A toggle to enable or disable the configuration.
  • Provider: Choose from supported engines:
  • whisper-server (best for local/self-hosted STT)
  • openai (for cloud-based Whisper)
  • google (for Google Cloud STT/TTS)
  • elevenlabs (for ultra-realistic synthesis)
  • gtts (free, lightweight Google Text-to-Speech)
  • Connector: Select an optional Connector that securely manages API keys, Service Account JSONs, or authentication headers. This ensures no sensitive secrets are hardcoded.
  • Agent(s): Multi-select of agents permitted to use this configuration. Select All to make it globally available within the account.

2. Service Endpoint & Connection

  • Endpoint URL: The URL pointing to the engine (e.g., localhost or whisper-service.local). Leave blank for default cloud-hosted provider endpoints.
  • Port (optional): The specific port number of the service (must be an integer between 1 and 65535), if not already specified inside the endpoint URL.
  • Protocol: The communication protocol used:
  • http / https (for typical REST/HTTP architectures)
  • grpc (recommended for low-latency streaming)
  • websocket (for continuous interactive bi-directional streams)
  • Model Name (optional): The exact model identifier required by the provider (e.g., whisper-1, tts-1, eleven_monolingual_v1).

3. Text-to-Speech (TTS) Specific Parameters

These fields are displayed conditionally only when the direction is set to text-to-speech: - Voice Name: The specific voice or speaker ID to use (e.g., OpenAI's alloy, ElevenLabs' voice UUID like 21m00Tcm4TlvDq8ikWAM, or Google Cloud voice names like en-US-Standard-C). - Language Code: The locale identifier (e.g., en-US, es-ES, fr-FR). - Speaking Speed / Rate: A numeric multiplier controlling speaking rate (defaults to 1.0; supports values between 0.25x and 4.0x).

Service Configuration Guide & Examples

Speech to Text (STT)

1. OpenAI Whisper

Utilize OpenAI's highly accurate cloud-hosted model for transcribing voice. * Provider: openai * Connector: Select an OpenAI API Key Connector. * Model: whisper-1 * Protocol: https * Endpoint: Leave blank (defaults to standard OpenAI endpoints).

2. Local Whisper Server

Great for on-premise, secure, or free offline transcription using engines like faster-whisper-server or whisper.cpp. * Provider: whisper-server * Connector: Leave blank * Model: E.g., base, small, or medium * Endpoint: localhost or 127.0.0.1 (or local server IP) * Port: 8000 (or your local container's listening port) * Protocol: http or grpc

3. Google Cloud Speech-to-Text

Enterprise-grade engine with high accuracy and low-latency streaming capabilities. * Provider: google * Connector: Select your Google Cloud Service Account Connector. * Model: E.g., chirp-2, latest_long, telephony, or default * Protocol: grpc (recommended for streaming) or https * Endpoint: Leave blank or speech.googleapis.com

Text to Speech (TTS)

1. ElevenLabs (Ultra-Realistic)

Known for top-tier vocal realism and cloning capabilities. 

{
  "name": "My ElevenLabs TTS",
  "provider": "elevenlabs",
  "voice": "21m00Tcm4TlvDq8ikWAM",  // Custom or standard Voice ID
  "model": "eleven_monolingual_v1",
  "direction": "text-to-speech",
  "connector": "My ElevenLabs Connector"  // Securely holds 'xi-api-key'
}

2. Free gTTS (Google Text-to-Speech)

A lightweight, free, and quick TTS engine that does not require API keys or cloud billing setup. 

{
  "name": "My Free gTTS",
  "provider": "gtts",
  "language_code": "en-US",
  "direction": "text-to-speech"
}

3. Google Cloud Text-to-Speech (using Gemini model)

Google's advanced multimodal speech synthesis engine. 

{
  "name": "My Google Cloud TTS",
  "provider": "google",
  "voice": "Aoede",  // Prebuilt voice name (e.g. Aoede, Kore, Puck, Charon)
  "model": "gemini-3.1-flash-tts-preview",  // Gemini TTS model
  "direction": "text-to-speech",
  "connector": "My Google Cloud Connector"  // GCP service account JSON connector or Gemini API key
}

Managing Existing Audio Configurations

The Existing Audio Configurations panel lists all configured audio engines for the selected account, filtered to the active agent if one has been chosen in the sidebar.

Each item displays: - The configuration name, direction, provider, active status, and associated agents. - Expand to View: Clicking any row expands to show full configuration parameters, including endpoints, protocols, models, languages, and descriptions.

Available Actions:

  • Edit (:material/edit:): Launches the editor panel to adjust the configuration properties. Clicking Update Configuration commits changes to the database.
  • Duplicate (:material/content_copy:): Instantly clones the selected configuration with copy appended to the name, allowing fast prototyping of similar setups.
  • Delete (:material/delete:): Permanently removes the audio configuration from the database.
  • Include Agent Form: If a specific agent is selected and not yet associated with an audio configuration, an inclusion form is displayed to easily register the configuration to that agent.