https://medium.com/@tmaurodot/creating-a-virtual-piano-with-ai-composed-music-in...

Creating a Virtual Piano with AI-Composed Music in Unity: A Deep Dive into the Project

Imagine pressing a button and hearing a piano play an original, AI-composed melody that evokes moonlight on waves or a sprint through sunlit fields. This is the essence of a Unity 2D virtual piano project that combines interactive gameplay, precise audio playback, and a suite of six unique songs crafted by artificial intelligence. Designed for a 1920x1080 resolution, this project features a 44-key piano (A2 to C6, 25 white keys, 19 black keys) and a user-friendly interface with auto-play buttons, making it both a musical playground and a showcase of AI creativity. In this article, we’ll explore how this project works, from its core components to the integration of AI-composed music, offering a clear and engaging look at its mechanics and charm.

Project Overview

The virtual piano project is built in Unity, a popular game development engine, and centers on a 2D piano interface where players can:

  • Play manually: Click piano keys or use keyboard inputs (Q, W, ..., Comma for white keys; 1, 2, ..., RightArrow for black keys) to produce notes.
  • Auto-play songs: Click one of six buttons to play AI-composed melodies like “Moonlit Waves” or “Cosmic Voyage.”
  • Interact with tracks: Play background tracks (via a TrackManager) that pause when auto-play songs start.

>span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-1sw30gj r-z2wwpe r-mtzec0 r-1471scf r-1aiqnjv r-1hq4qhi r-16dba41 r-ilng1c r-trst2h r-1noe1sz r-njp1lv">C4.wav, F#4.wav) for realistic sound. The interface includes:

  • Piano keys: Visualized as white and black rectangles at the bottom of the screen.
  • Six auto-play buttons: Positioned at X: 180, 340, 500, 660, 820, 980, Y: 400, each triggering a unique song.
  • TuneLabel: A text display at X: 0, Y: 480 showing the current song (e.g., “Starry Path” or “No Tune”).
  • TrackLabel: Shows the status of background tracks.

The project’s standout feature is its music: six original songs—“Moonlit Waves,” “Sunlit Sprint,” “Ember Glow,” “Cosmic Voyage,” “Whispering Breeze,” and “Starry Path”—composed by an AI (Grok 3, built by xAI). These replace traditional tunes like “Für Elise” or “Happy Birthday,” offering fresh, diverse melodies in keys like C minor, D major, and E minor, with black keys (e.g., F#4, C#5) adding harmonic depth.

Core Components

The project is driven by three main scripts, each handling a specific aspect of functionality:

1. PianoKey.cs

This script is attached to each piano key (white and black) and manages note playback and visual feedback.

  • Key Features:
    • Audio Playback: Each key has an AudioSource component with a clip (e.g., White_C4 uses C4.wav). The PlayNote method stops any current sound, sets volume (default 0.8), and plays the clip.
    • Visual Feedback: When played, keys scale down to 90% (X, Y) and change color (white keys to yellow, black to cyan). A FadeOutNote coroutine fades the volume over 0.5 seconds, resetting the key’s appearance.
    • Variables:
      • audioSource: Plays the note’s clip.
      • keyImage: The key’s UI Image for color changes.
      • noteDuration: 0.5 seconds, controlling playback length.
      • originalColor: Stores the key’s default color (white or black).
  • Example: Clicking White_E4 plays E4.wav, scales the key, changes it to yellow, and fades out over 0.5 seconds.

2. PianoController.cs

>span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-1sw30gj r-z2wwpe r-mtzec0 r-1471scf r-1aiqnjv r-1hq4qhi r-16dba41 r-ilng1c r-trst2h r-1noe1sz r-njp1lv">PianoController GameObject, maps keyboard inputs to piano keys, enabling manual play.

  • Key Features:
    • Input Mapping:
      • White keys (A2 to C6): Q, W, E, ..., Comma (25 keys).
      • Black keys (A#2 to F#6): 1, 2, 3, ..., RightArrow (19 keys).
    • Key Detection: In Update, it checks for Input.GetKeyDown events, calling PlayNote on the corresponding PianoKey.
    • Variables:
      • whiteKeys, blackKeys: Arrays of PianoKey components.
      • whiteKeyInputs, blackKeyInputs: Lists of KeyCode for input mapping.
  • Example: Pressing W (mapped to White_D4) triggers D4.wav and its visual feedback.

3. AutoPlayManager.cs

Attached to PianoKeys, this script manages the auto-play feature, playing AI-composed songs when buttons are clicked.

  • Key Features:
    • Song Data: Stores six songs in a tunes array, each with:
      • notes: Array of note names (e.g., ["C4", "D#4", ...]).
      • durations: Array of note lengths (e.g., [0.4f, 0.4f, ...]).
      • name: Song title (e.g., “Moonlit Waves”).
    • Playback:
      • >span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-1sw30gj r-z2wwpe r-mtzec0 r-1471scf r-1aiqnjv r-1hq4qhi r-16dba41 r-ilng1c r-trst2h r-1noe1sz r-njp1lv">PlayTune coroutine iterates through a song’s notes, matching each (e.g., F#4) to a key in whiteKeys or blackKeys, calling PlayNote, and waiting for the specified duration (e.g., 0.4 seconds).
      • Songs stop if the button is clicked again or another song/track starts.
    • Button Logic:
      • Six buttons (moonlitButton, sunlitButton, etc.) call methods like ToggleMoonlit, which start/stop songs and update TuneLabel.
      • Buttons change color (e.g., blue for moonlitButton when inactive, gray when active).
    • Track Integration: Stops background tracks (via TrackManager) when a song plays.
    • Variables:
      • whiteKeys, blackKeys: Key arrays.
      • trackManager: Links to TrackManager for track control.
      • moonlitButton, etc.: UI buttons for each song.
      • tuneLabel: Displays song names.
      • isPlayingTune, currentTune, tuneCoroutine: Track playback state.
  • Example: Clicking emberButton (X: 500, Y: 400) plays “Ember Glow” (A major, 60 notes), showing “Ember Glow” on TuneLabel and turning the button gray.

4. TrackManager.cs

>span class="css-1jxf684 r-bcqeeo r-1ttztb7 r-qvutc0 r-1sw30gj r-z2wwpe r-mtzec0 r-1471scf r-1aiqnjv r-1hq4qhi r-16dba41 r-ilng1c r-trst2h r-1noe1sz r-njp1lv">TrackPlayer GameObject, manages background tracks (not detailed here but integrated).

  • Role: Provides StopAllTracks to pause tracks when auto-play songs start, updating TrackLabel to “No Track.”

The AI-Composed Songs

The heart of the project is its six original songs, composed by Grok 3, designed to showcase variety and creativity:

  1. Moonlit Waves (C minor, 52 notes, ~40 seconds)
    • Mood: Mysterious, like waves under moonlight.
    • Structure: Intro (C4, G3, A#3), Theme (C4, D#4, G4, ...), Coda.
    • Keys: A#3 to D#5, with black keys A#3, D#4 for a haunting Cm sound.
    • Button: X: 180, Y: 400 (blue when inactive).
  2. Sunlit Sprint (D major, 56 notes, ~42 seconds)
    • Mood: Energetic, like a joyful run.
    • Structure: Verse (D4, A3, F#4, ...), Bridge (A4, C#5, ...), Verse.
    • Keys: A3 to C#5, with black keys F#4, C#5 for brightness.
    • Button: X: 340, Y: 400 (green).
  3. Ember Glow (A major, 60 notes, ~45 seconds)
    • Mood: Warm, like a glowing campfire.
    • Structure: Theme (A3, E4, F#4, ...), Refrain (C#5, F#5, ...), Ending.
    • Keys: A3 to F#5, with black keys C#4, F#4, G#4 for richness.
    • Button: X: 500, Y: 400 (purple).
  4. Cosmic Voyage (E minor, 80 notes, ~60 seconds)
    • Mood: Epic, like a sci-fi journey.
    • Structure: Intro (E3, B3, ...), A (E4, G4, B4, ...), B (faster, G4, B4, ...), A, Coda.
    • Keys: E3 to E5, no black keys for a bold, pure sound.
    • Button: X: 660, Y: 400 (red).
  5. Whispering Breeze (F major, 48 notes, ~36 seconds)
    • Mood: Gentle, like a spring wind.
    • Structure: Verse (F3, C4, F4, ...), Chorus (A4, C5, ...), Verse, Ending.
    • Keys: F3 to C5, with black key A#4 for subtle color.
    • Button: X: 820, Y: 400 (orange).
  6. Starry Path (G major, 56 notes, ~40 seconds)
    • Mood: Whimsical, uplifting.
    • Structure: Verse (G4, D4, E4, ...), Chorus (B4, D5, F#5, ...), Verse, Chorus, Ending.
    • Keys: G3 to F#5, with black keys F#4, F#5.
    • Button: X: 980, Y: 400 (cyan).

These songs use a mix of quarter notes (0.4 seconds) and eighth notes (0.2 seconds for parts of “Cosmic Voyage”), with tempos around 120 BPM. Black keys in five songs add harmonic depth, while “Cosmic Voyage” uses only white keys for contrast. All fit the piano’s A2-C6 range, leveraging existing audio clips.

How It Works: A Step-by-Step Flow

  1. Setup:
    • Unity Scene: A Canvas (1920x1080) contains:
      • 44 key GameObjects (White_A2 to White_C6, Black_A#2 to Black_F#6).
      • Six buttons (MoonlitButton, etc.).
      • TuneLabel and TrackLabel (TextMeshProUGUI).
      • PianoKeys (with AutoPlayManager) and PianoController.
    • Audio Clips: Stored in Assets/Audio (e.g., C3.wav, F#4.wav), assigned to each key’s AudioSource.
  2. Initialization:
    • On start, AutoPlayManager sets TuneLabel to “No Tune” and initializes button colors (e.g., MoonlitButton blue).
    • PianoKey scripts store original colors and load audio clips.
    • PianoController maps keyboard inputs.
  3. Manual Play:
    • Click a key (e.g., White_G4): PianoKey.PlayNote plays G4.wav, scales the key, and changes color.
    • Press E (mapped to White_F4): PianoController triggers F4.wav via PlayNote.
  4. Auto-Play:
    • Click CosmicButton (X: 660, Y: 400):
      • ToggleCosmic checks if “Cosmic Voyage” is playing:
        • If yes, stops it, resets TuneLabel to “No Tune,” turns button red.
        • If no, stops any track (via TrackManager), stops other songs, starts PlayTune(3).
      • PlayTune:
        • Iterates through “Cosmic Voyage” notes (E3, B3, E4, ...).
        • Matches each note to a key (e.g., White_E4 for E4).
        • Calls PlayNote, waits 0.4 seconds (or 0.2 for B section).
        • Updates TuneLabel to “Cosmic Voyage,” sets button to gray.
      • On finish, resets to “No Tune,” button turns red.
  5. Interactions:
    • Play “Sunlit Sprint,” click WhisperButton: “Sunlit” stops, “Whispering Breeze” plays.
    • Play a track, click EmberButton: Track stops, “Ember Glow” plays, TrackLabel shows “No Track.”
    • During “Moonlit Waves,” press Q (A2): A2.wav plays over the song, with visual feedback.

Technical Details

  • Audio:
    • Clips use PCM format, Decompress On Load, 44100 Hz for clarity.
    • Each key’s AudioSource ensures one note plays at a time, avoiding overlaps.
    • Note duration (0.5 seconds in PianoKey) allows smooth playback, even for fast sequences.
  • Performance:
    • Coroutines (PlayTune, FadeOutNote) manage timing without blocking the main thread.
    • Arrays (whiteKeys, blackKeys) enable fast note lookups (O(n) per note, n=44).
  • UI:
    • Buttons use Unity’s UI system, with Rect Transforms for precise positioning.
    • Colors (blue, green, etc.) provide clear visual cues, with gray indicating active songs.
  • Error Handling:
    • Checks for null references (e.g., tuneLabel, trackManager) prevent crashes.
    • Debug logs can be added to PlayTune to trace note playback issues.

Challenges and Solutions

  1. Wrong Notes:
    • Issue: Early versions had mismatched clips (e.g., D4.wav on Black_D#4).
    • Solution: Ensured each key’s AudioSource matches its name (e.g., Black_F#4 uses F#4.wav). Users must verify clip assignments in the Inspector.
  2. Tuning:
    • Issue: Detuned clips could occur if audio files were misgenerated.
    • Solution: Recommended re-importing clips or regenerating in Audacity (e.g., pitch-shift F4.wav +1 semitone for F#4.wav).
  3. Timing:
    • Issue: Fast songs could clip notes if durations were too short.
    • Solution: Set PianoKey.noteDuration to 0.5 seconds, adjustable to 0.6 if needed. Song durations (0.4f, 0.2f) balance tempo and clarity.
  4. Song Variety:
    • Challenge: Creating six distinct AI songs that fit A2-C6 and feel unique.
    • Solution: Varied keys (C minor to G major), structures (verse-chorus to intro-A-B-A), and black key usage (e.g., C#4, G#4 in “Ember Glow”). “Cosmic Voyage” is longer (~60 seconds) for an epic feel.

Customization Options

The project is flexible, with easy tweaks to enhance or personalize it:

  • Tempo:
    • Slow “Moonlit Waves” to 0.5f durations (~100 BPM) for a dreamier feel:
More information
Updated 6 days ago
Published 8 days ago
StatusReleased
PlatformsHTML5
AuthorMauroDot
Made withUnity

Leave a comment

Log in with itch.io to leave a comment.