How Does Your Phone Even Know That's a Painting? Image Tracking, Explained Simply

When you point your phone at a mural and an animation locks onto it perfectly, it can feel like sorcery. It isn't. It's a technique called Natural Feature Tracking, or NFT (not the crypto kind), and it's been quietly powering "markerless" AR for years.

Step one: find the interesting bits

Instead of scanning a QR code or a special printed marker, the software looks at the actual image — a mural, a painting, a poster — and identifies "interesting points": high-contrast corners, edges, and unique details that stand out from their surroundings. These are sometimes called image descriptors, and they're generated ahead of time from a reference photo of the artwork.

Not every image works equally well. A mural with sharp lines, varied color, and lots of texture gives the software plenty of unique points to lock onto. A flat, low-detail image (or a low-resolution photo) gives it almost nothing to grab, which is why some art tracks instantly and other art needs you to hold still for a second.

Your phone isn't recognizing a painting — it's recognizing a fingerprint made of edges.

Step two: match, then track

Once your camera feed comes in — handled by a browser API called WebRTC that requests permission to use your camera — the AR engine compares what it sees, frame by frame, against that stored fingerprint of feature points. When enough points line up, it knows not just that the artwork is there, but exactly where it is and at what angle — which lets it "glue" an animation onto the wall so it stays put even as you move your phone around.

This is genuinely different from the earlier generation of AR, which relied on printed QR-style markers placed next to the art. Image tracking skips the marker entirely — the art itself is the trigger. That's a big deal for murals specifically, since nobody wants a QR code sticker slapped next to a beautiful wall.

Why WebAR (no app download) works this way

All of this — camera access, computer vision, and rendering the animation — can now happen inside a regular mobile browser via WebXR-style APIs, meaning detection happens locally on your device without needing to send your camera feed anywhere. That's the technical reason apps like Vivia Art can skip the app-store download entirely: the browser already has everything it needs.