Watching the Watchers

A privacy analysis of automatic content recognition on U.S. smart TVs

82% of U.S. households have a smart TV that can take snapshots of whatever is on screen and report them back. We measured where that data goes, and whether turning the feature off actually stops it.

Illustration of how ACR works, not data from the study.

Automatic content recognition (ACR) is built into most smart TVs. It identifies what's playing, whether it comes from an app, an antenna, a game console or a cable box.

The TV takes short audio or video snapshots of whatever is on screen.

Those snapshots, or fingerprints made from them, are sent over the internet to the ACR provider.

The provider compares them with a library of known content to work out exactly what you're watching.

Viewing habits can be used for targeted ads and shared with or sold to other companies.

Little is publicly known about how ACR behaves in practice. Deployments are opaque, and smart TVs are hard to test automatically. We asked three questions.

We built a testbed that plays the same content on every TV without anyone at the remote, records all of its network traffic, and then works out which traffic belongs to ACR.

Run thousands of controlled viewing experiments on 12 TVs and capture every packet they send and receive.

Use the controlled content to pick out the servers that respond to what's on screen, then compare their traffic across conditions.

Twelve smart TVs from ten manufacturers cover the eight operating systems that run about 95% of U.S. smart TVs, and more than 80% of the TV brands with over 3% of U.S. households. The operating system is what we compare, so we chose TVs to cover every major OS first, adding a second TV where one OS is sold by several brands.

All traffic to and from each TV was captured with the Mon(IoT)r software and the SPHERE research infrastructure. In total we ran 5,943 controlled experiments.

A control machine sends the same content to every TV through three routes, so each viewing scenario is identical across devices.

HDMI-to-RF modulators broadcast our own custom channel to each TV's antenna input.

IP-to-HDMI converters turn video streamed from the control machine into an HDMI signal, like a cable box or console.

Google Pixel phones and a Windows laptop running Pigeoncast and Miracast cast to each TV using its own mirroring protocol.

The scenarios cover the common ways people use a smart TV. Idle, with the TV on the home screen and nobody using it, is the control for spotting baseline traffic.