Google: Measuring Heart Rate Through Smartphone Cameras
Google has published research on how technology can be used to passively measure heart rate during everyday smartphone use.
Resting heart rate (RHR) is a biomarker of cardiovascular health and long-term risk, with high RHR associated with adverse cardiovascular events and certain chronic illnesses.
With approximately five billion people around the world already owning a smartphone capable of monitoring their RHR, there is an opportunity to broaden access to health tracking.
Developing a RHR monitoring model
In 2022, Google demonstrated how smartphones can measure heart rate by placing a finger over the camera.
Its new research introduces passive heart rate monitoring (PHRM) that enables the tracking of heart rate and RHR in the background while using a smartphone.
It uses the front-facing camera to capture a video of the user’s face, using deep learning to estimate heart rate with a mean absolute percentage error of less than 10%.
This meets industry accuracy standards for people of all skin tones.
Eric S. Teasley, Product Manager and Ming-Zer Poh, Staff Research Scientist at Google Research, say: “To our knowledge, PHRM marks the first large-scale demonstration of passive HR and daily RHR monitoring during everyday smartphone use.
“As the only rPPG (remote photoplethysmography) method to meet heart rate accuracy standards for people of all skin tones – even in unpredictable real-world conditions – it sets a new standard for the field. It also represents the first use of rPPG to estimate daily RHR, achieving wearable-level accuracy across all skin tones.
“By combining an understanding of user habits with cutting-edge deep learning techniques and an inclusive design, we’ve developed a smartphone-based heart rate monitoring system that enables wearable-like heart health insights.”
Advanced heart rate technology
PHRM measures heart rate by sensing the fluctuation in how light interacts with skin each time the blood pulses through it.
It uses an on-device software pipeline that processes short clips of facial video and uses temporal shift convolutional neural networks to predict heart rate.
Previous studies have vastly underrepresented people with dark skin, as melanin makes the signals more challenging for cameras to detect.
Google’s researchers developed PHRM with more than 350,000 video clips from nearly 700 diverse participants.
It tested the participants in both laboratory and real-world settings, devoting model training to the most challenging and complex cases.
Researchers used the Monk Skin Tone Scale to ensure that participants classified with light and medium skin tones each comprised at least 25% of the datasets and participants with dark skin tones comprised at least 33%.
This makes the study the largest and most diverse rPPG study to date, allowing researchers to develop inclusive models that accurately predict RHR across all skin tones.
Testing the system
Google’s researchers trained PHRM to handle a variety of conditions in laboratory settings, recording facial video and simultaneous electrocardiogram (ECG) data from 365 study participants.
It found that PHRM significantly outperformed 15 of the leading published rPPG models on the same test.
Researchers also trained the model on real-world data, conducting a free-living study with 231 participants.
These participants installed a custom data collection app on their phones and used them as normal while wearing an ECG chest strap and a Fitbit tracker.
In this study, the app recorded an average of 231 video clips per day.
Future research could explore optimising camera exposure or increasing success when there is excessive head movement from the participants.
Google intends to catalyse further research by making its data and modelling resources available to qualified researchers.
