On-AI-R #1: Camille - Complex AI-Driven Musical Performance [4-Track EP Release + Open-Sourcing Kinect-Controlled-Instrument for TouchDesigner and Ableton Live]

A complex AI live-style performance introducing with Camille. In her performance, gestures control harmony; AI lip/hand transfer aligns the avatar to the music. I recorded the performance from multiple angles and mapped lips + hand cues in an attempt to push “AI musical avatars” beyond just lip-sync into performance control.

Tools: TouchDesigner + Ableton Live + Antares Harmony Engine → UDIO (remix) → Ableton again | Midjourney → Kling → Runway Act-Two (lip/gesture transfer) → Adobe (Premiere/AE/PS). Also used Hailou + Nano-Banana.

Not even remotely perfect, I know, but I really wanted to test how far this pipeline would allow me to go in this particular niche. WAN 2.2 Animate just dropped and seems a bit better for gesture control, looking forward testing it in the near-future. Character consistency with this amount of movement in Act-Two is the hardest pain-in-the-ass I’ve ever experienced in AI usage so far. [As, unfortunately, you may have already noticed.]

On the other hand, If you have a Kinect lying around: the Kinect-Controlled-Instrument System is freely available. Kinect → TouchDesigner turns gestures into MIDI in real-time, so Ableton can treat your hands like a controller; trigger notes, move filters, or drive Harmony Engine for stacked vocals (as in this piece). You can access it through this very post or full tutorial at: https://www.youtube.com/watch?v=vHtUXvb6XMM

Also: 4-track silly EP (including this piece) available, also through this post.

Hope you enjoy this, just as much as I suffered making it :)

On-AI-R #1: Camille - Complex AI-Driven Musical Performance (YouTube)