Artificial Eye-telligence: How Google is Using Your Eyes in Medicine

  • Post author:
  • Post published:April 3, 2018
  • Post category:New Tech

AI + EYES = ♡
Google’s New AI Can Predict Heart Disease just by Looking at You






What’s that old proverb, “eyes are the window to the soul?” Well, they aren’t; however, Google has turned them into a window to cardiac health. Scientists from Google and its health-tech subsidiary Verily have discovered a new way to assess a person’s risk of heart disease using machine learning. By analyzing scans of the back of a patient’s eye, the company’s software is able to accurately deduce data, including an individual’s age, blood pressure, and whether or not they smoke. This can then be used to predict their risk of suffering a major cardiac event—such as a heart attack—with roughly the same accuracy as a blood test.


The algorithm potentially makes it quicker and easier for doctors to analyze a patient’s cardiovascular risk, as it doesn’t require a blood test. But, the method will need to be tested more thoroughly before it can be used in a clinical setting.


So, How Does It Work?


While the idea of looking at your eyes to judge the health of your heart sounds unusual, it draws from a body of established research. The rear interior wall of the eye (the fundus) is chock-full of blood vessels that reflect the body’s overall health. By studying their appearance with camera and microscope, doctors can infer things like an individual’s blood pressure, age, and whether or not they smoke, which are all important predictors of cardiovascular health.


To train the algorithm, Google and Verily’s scientists used machine learning to analyze a medical dataset of nearly 300,000 patients. This information included eye scans as well as general medical data. As with all deep learning analysis, neural networks were then used to mine this information for patterns, learning to associate telltale signs in the eye scans with the metrics needed to predict cardiovascular risk.



Two images of the fundus, or interior rear of your eye. The one on the left is a regular image; the on the right shows how Google’s algorithm picks out blood vessels (in green) to predict blood pressure.


When presented with retinal images of two patients, one of whom suffered a cardiovascular event in the following five years, and one of whom did not, Google’s algorithm was able to tell which was which 70 percent of the time. This is only slightly worse than the commonly used SCORE method of predicting cardiovascular risk, which requires a blood test and makes correct predictions in the same test 72 percent of the time.


Still Not Convinced?



One of the exciting aspects of this study is the generation of “attention maps” to show which aspects of the retina contributed most to the algorithm, thus providing a window into the “black box” often associated with machine learning. This can give clinicians greater confidence in the algorithm, and potentially provide new insights into retinal features not previously associated with cardiovascular risk factors or future risk.


As you can see from the above image, the AI came very close in its predictions—like scary close. The researchers have demonstrated that deep learning applied to a retinal fundus image can frequently predict a number of vital information about the patient—not only smoking status and blood pressure—but, can actually predict the occurrence of a future major cardiovascular event on par with current measures.




Woman or man, adult or child, rich or poor, American or otherwise: heart disease does not discriminate. It’s the number one killer of men & women, in the United States & around the world. But rather than focusing on the endpoint of death, researchers are going back to the roots of the disease. They want to find out when and how it starts—and to make sure it doesn’t. There’s a strong body of research that helps doctors understand what puts people at risk: daily behaviors, including exercise and diet, in combination with genetic factors, age, ethnicity, and biological sex all contribute.


The prospect of machine learning applications in screening for risk or the presence of disease, as has been shown in diabetic retinopathy, suggests an exciting new frontier in healthcare. Machine learning allows researchers to read signals in data-rich environments, like images, that were previously difficult to examine, given their complexity, and enables broader, deeper scientific discovery. The opportunity to one day readily understand the health of a patient’s blood vessels with a simple retinal image could lower the barrier to engage in critical conversations on preventive measures to protect against a cardiovascular event.


At Verily, their focus has been on understanding the transition from health to disease across a range of conditions, and machine learning is one of the tools that enables this discovery. They continue to collaborate with different organizations on the early detection of risk and disease: from Google AI on this work, to the American Heart Association on the One Brave Idea Initiative (led by Dr. Calum MacRae, chief of cardiovascular medicine at Brigham and Women’s Hospital, the One Brave idea team is comprised of a team of investigators from a wide variety of specialties like engineering and data scientists and institutions including MIT and Stanford; each has experience breaking down boundaries and working outside the constraints of traditional biomedical science; their team is developing biomarkers for coronary heart disease or atherosclerosis that can define the earliest stages of the disease—such as young adulthood, or even childhood), to Duke and Stanford for the Project Baseline Study (a longitudinal study, that will collect broad phenotypic health data from approximately 10,000 participants, who will each be followed over the course of at least four years; the study is the first initiative of Project Baseline, a broader effort designed to develop a well-defined reference, or “baseline,” of health, as well as a rich data platform that may be used to better understand the transition from health to disease and identify additional risk factors for disease). Discovering strong predictive technologies will enable the preventive approach to healthcare that is core to their mission.


To read the full paper, published in Nature Biomedical Engineering, describing their work greater detail, click here.

Are you interested in learning more about biomedical technology? Let us know what you think in the form below.