Skip to main content

Voice-to-Action with OpenAI Whisper

Learning Objectives​

After completing this chapter, you will be able to:

  • Integrate OpenAI Whisper for speech recognition in ROS 2
  • Design voice command vocabularies for humanoid robot control
  • Implement robust voice command processing pipelines
  • Handle speech recognition errors and uncertainties

Introduction​

Voice-to-action systems bridge the gap between natural human language and robot actions, enabling intuitive human-robot interaction. In Physical AI applications with humanoid robots, voice interfaces provide a natural way for humans to communicate tasks and commands to robots. OpenAI's Whisper model offers state-of-the-art speech recognition capabilities, making it an excellent choice for implementing voice-controlled robotic systems.

This chapter explores how to integrate Whisper with ROS 2 to create robust voice-to-action systems for humanoid robots. We'll cover the integration process, command vocabulary design, and error handling strategies essential for reliable voice-controlled robot operation.

Core Concepts​

Implementing voice-to-action systems involves several key components that must work together to provide reliable and responsive interaction. The system must capture audio, process it through a speech recognition model, interpret the recognized text, and convert it to appropriate robot actions.

Speech Recognition Pipeline​

The pipeline typically involves:

  1. Audio capture from microphones
  2. Preprocessing of audio signals
  3. Speech-to-text conversion using Whisper
  4. Natural language processing of the recognized text
  5. Mapping to specific robot commands
  6. Execution of robot actions

Command Vocabulary Design​

Designing an effective command vocabulary is crucial for system usability. Commands should be:

  • Unambiguous and distinct from each other
  • Easy to pronounce and remember
  • Appropriate for the robot's capabilities
  • Robust to variations in pronunciation

Error Handling and Confidence​

Speech recognition systems must handle uncertainty and errors gracefully. This includes recognizing when the system is not confident about a recognition, handling unrecognized commands, and providing feedback to the user.

Hands-on Examples​

Let's implement a voice-to-action system with OpenAI Whisper:

#!/usr/bin/env python3

"""
Voice command recognition node using OpenAI Whisper
"""

import rclpy
from rclpy.node import Node
from rclpy.qos import QoSProfile
from std_msgs.msg import String
from geometry_msgs.msg import Twist
import numpy as np
import pyaudio
import wave
import threading
import time
import queue
import openai
import os
import json

# Configuration constants
CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000  # Whisper works well with 16kHz
RECORD_SECONDS = 3
WAKE_WORD = "hey robot"

# Command vocabulary mapping recognized text to robot actions
COMMAND_VOCABULARY = {
    "move forward": "forward",
    "go forward": "forward",
    "move back": "backward", 
    "go back": "backward",
    "move backward": "backward",
    "turn left": "left",
    "rotate left": "left",
    "turn right": "right",
    "rotate right": "right",
    "stop": "stop",
    "halt": "stop",
    "look around": "look_around",
    "raise your hand": "raise_hand",
    "lower your hand": "lower_hand",
    "wave": "wave",
    "dance": "dance"
}


class VoiceCommandNode(Node):

    def __init__(self):
        super().__init__('voice_command_node')
        
        # Initialize audio interface
        self.audio = pyaudio.PyAudio()
        self.is_listening = False
        self.audio_queue = queue.Queue()
        
        # Publishers
        self.voice_text_publisher = self.create_publisher(
            String,
            'voice_text',
            QoSProfile(depth=10)
        )
        
        self.cmd_vel_publisher = self.create_publisher(
            Twist,
            'cmd_vel',
            QoSProfile(depth=10)
        )
        
        # Timer for audio processing
        self.audio_timer = self.create_timer(0.1, self.process_audio)
        
        # Start audio capture thread
        self.audio_thread = threading.Thread(target=self.capture_audio)
        self.audio_thread.daemon = True
        self.audio_thread.start()
        
        # Initialize Whisper API key (in a real system, this would come from parameters)
        # openai.api_key = os.getenv("OPENAI_API_KEY")
        
        self.get_logger().info('Voice command node initialized')

    def capture_audio(self):
        """Capture audio from microphone and add to queue"""
        stream = self.audio.open(
            format=FORMAT,
            channels=CHANNELS,
            rate=RATE,
            input=True,
            frames_per_buffer=CHUNK
        )
        
        self.get_logger().info('Audio capture started')
        
        while rclpy.ok():
            try:
                # Read audio data
                data = stream.read(CHUNK, exception_on_overflow=False)
                self.audio_queue.put(data)
            except Exception as e:
                self.get_logger().error(f'Audio capture error: {e}')
                break
        
        stream.stop_stream()
        stream.close()

    def process_audio(self):
        """Process audio data and recognize speech"""
        # In a real implementation, we would collect audio chunks,
        # send them to Whisper, and process the recognized text
        # For this example, we'll simulate the process
        
        if not self.audio_queue.empty():
            # Collect audio data from queue
            audio_frames = []
            while not self.audio_queue.empty():
                audio_frames.append(self.audio_queue.get())
            
            if len(audio_frames) > 0:
                # In a real system, we would send audio_frames to Whisper
                # For simulation, we'll generate recognized text based on volume
                # (in practice, this would come from Whisper API)
                
                # Simulate recognition of a command
                if np.random.random() > 0.95:  # Occasionally recognize something
                    possible_commands = list(COMMAND_VOCABULARY.keys())
                    recognized_text = np.random.choice(possible_commands)
                    
                    # Publish recognized text
                    text_msg = String()
                    text_msg.data = recognized_text
                    self.voice_text_publisher.publish(text_msg)
                    
                    self.get_logger().info(f'Recognized: "{recognized_text}"')
                    
                    # Process the recognized command
                    self.process_recognized_command(recognized_text)

    def process_recognized_command(self, text):
        """Process recognized text and generate robot commands"""
        # Convert to lowercase and clean
        clean_text = text.lower().strip()
        
        # Check if command exists in vocabulary
        if clean_text in COMMAND_VOCABULARY:
            command = COMMAND_VOCABULARY[clean_text]
            self.execute_robot_command(command)
        else:
            # Check for partial matches or handle unrecognized command
            self.get_logger().info(f'Unrecognized command: "{clean_text}"')
            # Could implement fuzzy matching here

    def execute_robot_command(self, command):
        """Execute the corresponding robot action"""
        self.get_logger().info(f'Executing command: {command}')
        
        twist_msg = Twist()
        
        if command == "forward":
            twist_msg.linear.x = 0.5  # Move forward at 0.5 m/s
        elif command == "backward":
            twist_msg.linear.x = -0.5  # Move backward at 0.5 m/s
        elif command == "left":
            twist_msg.angular.z = 0.5  # Turn counter-clockwise
        elif command == "right":
            twist_msg.angular.z = -0.5  # Turn clockwise  
        elif command == "stop":
            twist_msg.linear.x = 0.0
            twist_msg.angular.z = 0.0
        # Additional commands would be implemented here
        
        # Publish the command
        self.cmd_vel_publisher.publish(twist_msg)

    def destroy_node(self):
        """Clean up resources"""
        if hasattr(self, 'audio'):
            self.audio.terminate()
        super().destroy_node()


def main(args=None):
    rclpy.init(args=args)

    voice_command_node = VoiceCommandNode()

    try:
        rclpy.spin(voice_command_node)
    except KeyboardInterrupt:
        voice_command_node.get_logger().info('Voice command node stopped by user')
    finally:
        voice_command_node.destroy_node()
        rclpy.shutdown()


if __name__ == '__main__':
    main()

Expected Output:

[INFO] [1678882844.123456789] [voice_command_node]: Voice command node initialized
[INFO] [1678882844.123456789] [voice_command_interpreter]: Voice command interpreter initialized
[INFO] [1678882845.123456789] [voice_command_node]: Recognized: "move forward"
[INFO] [1678882845.123456789] [voice_command_node]: Executing command: forward
[INFO] [1678882845.123456789] [voice_command_interpreter]: Received text: "move forward"
[INFO] [1678882845.123456789] [voice_command_interpreter]: Executing directional move: forward for 1.0m

Exercises​

Complete the following exercises to reinforce your understanding:

  1. Command Vocabulary Extension: Add more commands to the vocabulary

    • Implement commands for arm gestures (raise arms, point, etc.)
    • Add navigation commands to specific locations
    • Include safety commands (emergency stop, pause, etc.)
    • Test recognition accuracy with new commands

  2. Robustness Improvements: Enhance the system's reliability

    • Add confidence thresholding for Whisper recognition
    • Implement confirmation prompts for critical commands
    • Add timeout for command execution
    • Create a feedback mechanism to confirm command execution

Common Pitfalls and Solutions​

  • Pitfall 1: Network dependencies - Whisper API requires internet and can fail
    • Solution: Implement offline fallback or local speech recognition
  • Pitfall 2: Recognition errors - Whisper may misinterpret commands
    • Solution: Add confirmation steps and confidence thresholds
  • Pitfall 3: Audio quality issues - Background noise affects recognition
    • Solution: Implement audio preprocessing and noise reduction
  • Pitfall 4: Command ambiguity - Similar-sounding commands causing confusion
    • Solution: Design clear command vocabularies with distinct terms

Summary​

  • Voice-to-action systems bridge natural language and robot control
  • OpenAI Whisper provides high-quality speech recognition capabilities
  • Command interpretation requires careful vocabulary design
  • Robust error handling is essential for reliable operation
  • Voice interfaces enhance intuitive human-robot interaction

Further Reading​