Balloon Pop

Overview

Balloon Pop is an interactive projection-based game developed to demonstrate the integration of computer vision and real-time machine learning.

Players physically throw a ball at a projected screen, and the system detects the impact to "pop" virtual balloons. By leveraging a Raspberry Pi 4 and the Coral Edge TPU, the game maintains high frame rates while performing complex hit-detection logic and visual state updates.

Media Showcase

Parts Used

How it Works

Step 1: Image Capture

The system captures high-frequency frames via the Raspberry Pi V2 Camera, providing a constant visual stream of the projection area.

Step 2: Dynamic Cropping

The image analysis code identifies the exact coordinates of every "floating" balloon on screen. It then crops individual sub-images of these balloons from the main camera feed to isolate them for processing.

Step 3: ML Prediction

These cropped images are passed into a machine learning model. The model analyzes the crop to detect the presence of a black ball. If the ball is detected within the balloon's boundaries, the model returns a "hit" prediction.

Step 4: Screen Update

Based on the prediction, the game engine updates the visual display: it either advances the balloon one frame upward or triggers the "pop" sequence and removes the balloon from play.

Software Setup

Follow the official Raspberry Pi documentation to install Raspberry Pi OS on your Raspberry Pi 4. Make sure to install Raspberry Pi OS Bullseye (2024-10-22) (not Bookworm or Buster). After installation, complete the standard setup process.

Required Software Versions

• OpenCV: 4.11.0.86
• NumPy: 1.19.5
• PygGame: 1.9.6
• Screeninfo: 0.8.1

Installing Packages

Begin in the home directory and ensure you are in the base environment. Run the following commands exactly as shown:

# 1. Add the Coral Repo and Key
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

# 2. Update and install system dependencies
sudo apt-get update
sudo apt-get install python3-pycoral

# 3. Install Python dependencies 
pip3 install opencv-contrib-python==4.11.0.86
pip3 install numpy==1.19.5
pip3 install pygame==1.9.6
pip3 install screeninfo==0.8.1

Note: You may see errors regarding certain “coral cloud” packages; ignore them as long as the install completes. If you run into additional issues, consult the Teachable Sorter | Coral documentation.

Verify Installation

To check that your library versions are correctly installed, run the following commands in the terminal:

python3 -c "import cv2, numpy, pygame; print(f'OpenCV: {cv2.__version__}\nNumPy: {numpy.__version__}\nPygame: {pygame.version.ver}')" 
pip3 show screeninfo

Note:If your downloads still fail, ensure you're connected and that your time and date settings are up to date.

Code Implementation

The following abstracted code snippet demonstrates the core logic for hit detection and score updating within the Pygame loop. The complete source code can be downloaded from GitHub. To start the game, run the main.py file.

while not Game.quit_game:
    Game.step()  # Advance game state (balloon movement)

    # 1. Capture and Perspective Correction
    ret, frame = videoCapture.read()
    img = calibrate_frame(frame)

    # 2. Object Detection
    contours = FindBalloons.find_contours(img)
    cropped_image, is_balloon = FindBalloons.crop_objects(img, contours)

    # 3. Logic & Interaction
    if is_balloon:
        # ML model decides if the detected object is a valid "pop" action
        result = model.assess(cropped_image)
        
        # Trigger pop if model is positive and balloon is in valid Y-range
        if result and (result != prev_result) and (-140 < Game.y_pos < 900):
            Game.pop()
        
        prev_result = result

    if Game.restart_game:
        game_init(Game)