Laurence Cliffe

intermedia art, research and design

Home | Archive | Publications

mediapipe2osc

mediapipe2osc

mediapipe2osc is a Javascript solution for sending mediapipe landmark coordinates via UDP as OSC messages to any application that can receive them such as Unity, Unreal, Ableton Live, Max and more. Offers the potential to control and manipluate data, audio and visual output using body, hand or face detection and positional tracking.

This solution uses the osc.js library and builds on monlim’s MediaPipe-Hands-OSC to include face and body detection and tracking.

Current capabilities include:

  • mediapipe2osc-face-detection: sends value 1 as an OSC message when a face is detected in the camera frame and value 0 when no face is detected.
  • mediapipe2osc-face-landmarks: sends the x and y coordinates of the rightEye, leftEye, noseTip, mouthCenter, rightEar and leftEar as OSC messages.
  • mediapipe2osc-distance: sends the distance from the camera to the face in cm as an OSC message.
  • mediapipe2osc-pose: for now this sends some basic pose landmark coordinates of some main body parts via OSC
  • mediapipe2osc-hand-landmarks: this currently send the coords of the wrist and finger tips for each hand as OSC messages

You can download the repo from here.

Overall, mediapipe2osc aims to provide a quick, low-cost and accessible way to incorporate human detection, tracking and embodied interactional technology into creative projects.