Do you ever glance at your trusted Fitbit, using its estimated calories burned as a deciding factor when getting that second scoop of ice cream at the end of the night? It would make sense. Say the device predicting you’ve burned 500 extra calories that day—that means you’ve earned it, right?
Not quite.
The Stanford University School of Medicine has recently released a study showing that fitness trackers, while often surprisingly good at measuring heart rate, struggle when it comes to measuring calories burned.
The research looked at seven wrist-worn fitness trackers—Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn and Samsung Gear S2—observing 60 volunteers who were walking, running and cycling while wearing the devices. When it came to heart rate, six out of seven of the devices had a margin of error less than 5 percent. The seventh, the Samsung Gear S2, had the highest margin of error, though it was still only 6.8 percent.
Unfortunately, for your chances of eating that ice cream, the devices were off by much more when it came to energy expenditure.
None of the devices had an error rate below 20 percent, and the most successful device was incorrect by an average of 27 percent. Even more shocking was the worst-performing device, the PulseOn, which was off by an average of 93 percent.
“People are basing life decisions on the data provided by these devices,” said Dr. Euan Ashley, who was the senior author of the study and is a professor of cardiovascular medicine, genetics and biomedical data science at Stanford. Because the data for energy expenditure is so far off, the study asserts that basing decisions on one of the examined wristwatches isn’t the best plan.
Because each device uses its own algorithm, and those algorithms aren’t released to the public, the authors of the study are unsure as to why the data was so off-the-mark. However, they predict that much of this is due to the algorithms making assumptions that don’t fit individuals realistically.
“All we can do is see how the devices perform against the gold-standard clinical measures,” said Anna Shcherbina, a graduate student who co-authored the study. “My take on this is that it’s very hard to train an algorithm that would be accurate across a wide variety of people because energy expenditure is variable based on someone’s fitness level, height and weight.”
The team intends to do further studies with the devices, next looking at how they perform as volunteers go about a normal day as opposed to running on treadmills in a controlled lab setting. Until then, the researchers have created a website to share the data they’ve collected and invite others doing research to submit data on device performance.
Photo: LWYang, CC-BY
Emma Korstanje is a freelance journalist based out of Athens, GA.