MediaPipe Tutorial: Real-Time On-Device ML for Vision and Audio

Contents

MediaPipe real-time on-device machine learning vision and audio pipeline

Figure 1: Frames in, predictions out — the whole pipeline runs on this machine, not on someone's server

You need face detection, gesture recognition, or audio classification running live, right now, with no round trip to a server. MediaPipe is Google's answer for exactly that: a framework built to run vision, text, and audio ML pipelines directly on-device, in real time, across nearly every platform you'd want to ship to.

I've used it for a gesture-controlled prototype where latency actually mattered, and the built-in solutions saved me from writing a face landmark detector from scratch. Let's get you set up with it.

What MediaPipe Actually Covers

MediaPipe Solutions provides a suite of libraries and tools for quickly applying AI and ML techniques inside your applications. The part you'll use directly is called MediaPipe Tasks, which gives you the core programming interface: a set of pre-built libraries for deploying ML solutions with minimal code.

Tasks split cleanly into three domains:

Vision: face detection, hand landmark tracking, gesture recognition, object detection, image segmentation
Text: language detection, text classification, embedding
Audio: audio classification, real-time audio processing

Everything runs on-device by design. Input data (images, video, text) stays on the device and never gets sent to Google's servers. MediaPipe does send anonymized performance metrics back for improving the APIs, but your actual content stays local. That's the privacy story you'd expect, and it's worth mentioning to anyone on your team asking about data handling.

This is the same bet our Edge AI for beginners guide describes, only pushed further: instead of picking a small device that can run a model, you're picking a framework that already ships optimized pipelines for the common vision and audio jobs.

One Important Update Before You Start

If you're specifically looking for LLM inference through MediaPipe, know this upfront: the MediaPipe LLM Inference API is still available, but Google now recommends migrating to LiteRT-LM instead, and the API sits in maintenance-only mode. If your project needs conversational LLMs on-device, start with LiteRT-LM directly rather than building fresh on the MediaPipe path.

Everything else covered in this tutorial — vision and audio task pipelines — remains actively developed and is not affected by that change.

Setting Up Your Platform

MediaPipe Tasks ships platform-specific packages, so pick the one matching where you're building.

Android (Java/Kotlin)

implementation 'com.google.mediapipe:tasks-vision:latest.release'
implementation 'com.google.mediapipe:tasks-text:latest.release'
implementation 'com.google.mediapipe:tasks-audio:latest.release'

Python

pip install mediapipe

from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from mediapipe.tasks.python import text
from mediapipe.tasks.python import audio

Web (JavaScript)

<script src="https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/vision_bundle.mjs" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/tasks-audio/audio_bundle.js" crossorigin="anonymous"></script>

Or via npm, if that fits your build pipeline better:

npm install @mediapipe/tasks-vision @mediapipe/tasks-audio

Only import the module you actually need. FYI, bundling all three (vision, text, audio) when you only use one bloats your app for no reason.

Building a Vision Pipeline

Let's build a real-time gesture recognizer, since it's one of MediaPipe's flagship demos and shows the general pattern you'll reuse across every vision task.

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

base_options = python.BaseOptions(model_asset_path="gesture_recognizer.task")
options = vision.GestureRecognizerOptions(base_options=base_options)
recognizer = vision.GestureRecognizer.create_from_options(options)

image = mp.Image.create_from_file("hand_photo.jpg")
result = recognizer.recognize(image)

print(result.gestures)

Every vision task follows this same three-step shape: build options pointing at a model file, create the task object from those options, then call the task's method on your image. Swap the model and the options class, and you've got face detection, object detection, or segmentation instead.

If you want to see what a hand-tuned detector does when you push it further, our face recognition with OpenCV walkthrough covers the same problem space with a different toolchain, and object detection with YOLO shows where prebuilt solutions stop and custom training starts.

Real-Time Video vs Static Images

For live camera feeds, switch to the streaming mode most tasks support, which processes frames as they arrive rather than one static image at a time. Each task exposes running modes (image, video, or live stream) in its options, so check the specific task's documentation for the exact flag names, since they vary slightly between vision tasks.

Missing this is the single most common performance mistake: feeding a live feed through image mode means re-initializing state on every frame instead of letting the task track temporal continuity.

Testing on a desktop before you touch a phone? A Logitech C920 webcam is the boring, reliable choice — it exposes a clean 1080p MJPEG stream that won't fight your frame pipeline.

If you're deploying past the desktop, the Jetson Nano tutorial walks through running this class of model on NVIDIA's edge board, and our Edge AI hardware roundup compares the boards worth considering.

Building an Audio Pipeline

Audio classification follows the identical pattern:

from mediapipe.tasks import python
from mediapipe.tasks.python import audio
from mediapipe.tasks.python.components import containers

base_options = python.BaseOptions(model_asset_path="yamnet.task")
options = audio.AudioClassifierOptions(base_options=base_options)
classifier = audio.AudioClassifier.create_from_options(options)

audio_data = containers.AudioData.create_from_file("clip.wav")
result = classifier.classify(audio_data)

print(result[0].classifications)

Same shape: options, task creation, classify call. That consistency is genuinely one of MediaPipe's best design decisions, since learning one task teaches you most of the API surface for every other task.

Audio classification is the entry point, not the ceiling. If your actual problem is transcription rather than tagging sound events, our on-device speech recognition with Whisper.cpp guide is the better fit — it solves a different job with the same privacy properties.

Using Prebuilt Models vs Custom Models

MediaPipe ships MediaPipe Models, pre-trained and ready to run for each supported solution, so you often don't need to train anything at all. Face detection, hand tracking, and generic object detection all have solid defaults out of the box.

When the defaults don't fit your use case, MediaPipe Model Maker lets you customize models for solutions using your own data. The classic example: training a gesture recognizer on custom hand gestures you defined yourself, then deploying that customized model through the same GestureRecognizer API you'd use for the stock model. You write your training data once, and the deployment code barely changes.

Model Maker exports the same .task bundle format the prebuilt models use, which is the detail that makes the swap painless. If you end up needing a model that MediaPipe doesn't model at all, the TensorFlow Lite tutorial covers training and exporting from scratch — MediaPipe's vision tasks consume .task files built on TFLite underneath.

Web Deployment Specifics

Since browser-based real-time vision is a common use case, here's the minimal working setup for a webcam-driven task:

import { FilesetResolver, GestureRecognizer } from "@mediapipe/tasks-vision";

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm"
);
const recognizer = await GestureRecognizer.createFromOptions(vision, {
  baseOptions: { modelAssetPath: "gesture_recognizer.task" },
  runningMode: "VIDEO",
});

const results = recognizer.recognizeForVideo(videoElement, performance.now());

The FilesetResolver step loads MediaPipe's WASM runtime, which is what makes real-time inference feasible directly in a browser tab without any server involved. Ever tried explaining to a client why their browser demo needs zero backend infrastructure? This is exactly why that's possible.

Serve the .task model file alongside your bundle rather than fetching it from a third-party origin, or you'll be debugging CORS errors that look like model corruption.

MediaPipe vs Alternatives

Tool	Best For	Trade-off
MediaPipe	Pre-built vision/audio pipelines, Google ecosystem	LLM support is newer and now points to LiteRT-LM
ExecuTorch	Custom PyTorch models needing edge deployment	No pre-built task library like MediaPipe's
Core ML	iOS-only apps chasing max Apple hardware performance	Apple platforms only
ONNX Runtime	One model format across many runtimes	You assemble the pre- and post-processing yourself

MediaPipe's real strength is speed to a working prototype. If your task matches one of its pre-built solutions, you'll have something running in an afternoon instead of a week. MediaPipe excels at computer vision tasks specifically, with its vision pipeline solutions being more mature than its newer generative AI additions.

A Practical Decision Framework

Straightforward guidance for picking a path:

Does one of the stock tasks cover your problem? Face, hand, gesture, object, segmentation, text, or audio classification — if yes, start tonight with the prebuilt model and skip training entirely
Stock model close but not right? MediaPipe Model Maker on your own data, then ship the same .task file through the same API
Needs an on-device LLM? Go straight to LiteRT-LM and skip the MediaPipe LLM path
Shipping iOS-only and want maximum Apple silicon performance? Our Core ML tutorial is the better tool for that job
Model is custom and MediaPipe has no task for it? Fall back to TFLite or ONNX Runtime and write your own preprocessing

The mistake isn't choosing the wrong row — it's spending two weeks building a face landmark pipeline that already ships as a five-line task call.

Common Mistakes People Make

Bundling every task module

Recall the setup section directly — only import vision, text, or audio individually, matching what your app actually uses.

Missing running mode

Recall the real-time section directly — using image mode for live video feeds wastes performance, and you have to switch to video or live-stream mode explicitly.

Building fresh on the LLM Inference API

Recall the update section directly — start new LLM work on LiteRT-LM instead, since the MediaPipe path is maintenance-only now.

Skipping Model Maker when defaults don't fit

Recall the models section directly — don't fight a generic model's limitations for weeks when customizing it might take an afternoon.

Ignoring the anonymized metrics

Recall the privacy section directly — content stays local, but telemetry does leave the device, and someone on your team needs to know that before you sign off on a privacy review.

Recommended Books

Computer Vision: Algorithms and Applications by Richard Szeliski — the theory sitting under every prebuilt MediaPipe detector, so you understand what face landmark tracking is actually computing when you call it.
Deep Learning for Vision Systems by Mohamed Elsayed — the bridge from CNN fundamentals to the pretrained backbones these tasks wrap, written for people who want the intuition rather than the paper.
Python Computer Vision Cookbook by Massimiliano Spada — practical fallback recipes for the moment a prebuilt solution doesn't cover your case.

Want to Go Deeper?

If you want structured practice on computer vision and applied ML, Educative's computer vision courses run through hands-on labs that pair well with this kind of task-based API work. The unlimited plan is useful when you're working through several model types in one stretch.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is MediaPipe used for?

MediaPipe runs machine learning pipelines for vision, text, and audio directly on a device in real time. Its Tasks API covers face detection, hand and gesture tracking, object detection, image segmentation, language detection, text classification, embeddings, and audio classification without any server round trip.

Does MediaPipe send my images or audio to Google?

No. Input data such as images, video, and text stays on the device and is never sent to Google's servers for inference. MediaPipe does transmit anonymized performance metrics to help improve the APIs, which is worth knowing if your team is reviewing the privacy posture.

What is the difference between MediaPipe Solutions and MediaPipe Tasks?

Solutions is the higher-level umbrella covering libraries and tools for applying ML techniques inside an application. Tasks is the part you call directly: the programming interface built from prebuilt libraries that deploy a specific ML solution with minimal code.

Can MediaPipe run in a web browser?

Yes. The vision and audio JavaScript packages run WebAssembly inference inside the browser tab, so a webcam-driven demo needs no backend at all. You load the WASM runtime with FilesetResolver, point the task at a .task model file, and call the task method on each frame.

Should I use MediaPipe or LiteRT-LM for on-device LLMs?

Start new LLM work on LiteRT-LM. Google still ships the MediaPipe LLM Inference API but now recommends migrating to LiteRT-LM, and that API sits in maintenance-only mode. Vision and audio tasks in MediaPipe are unaffected and remain actively developed.

How do I use my own custom model with MediaPipe?

Train or fine-tune it with MediaPipe Model Maker on your own data, export a .task bundle, then pass that file to the same task API you would use for the stock model. The deployment code barely changes when you swap in a customized model.

Does MediaPipe work offline?

Yes. Once the package and the .task model file are on the device, inference needs no network connection at all — the browser build loads its WASM runtime from your own bundle, and the Python and Android builds are local files.

Wrapping This Up

MediaPipe hands you working vision and audio pipelines with minimal setup, all running privately on-device across Android, iOS, web, and Python. Prebuilt models get you moving fast, Model Maker covers customization, and the consistent options-then-create-then-run pattern makes learning one task transferable to the rest.

Will it replace a custom-trained model for a genuinely novel problem? No, and that's not its job. But for the vision and audio tasks it already covers — gesture recognition, face detection, audio classification — it's hard to beat the speed from zero to a working real-time demo. Grab a prebuilt model tonight and point it at your webcam; you'll have something running before your coffee gets cold :)

When you outgrow the prebuilt solutions, our best computer vision tools comparison covers what to reach for next, and the Edge AI beginners guide puts the whole on-device landscape in context.