With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert.
To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning?
Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims
Circulation published a nice review this week on the rationale for atrial fibrillation screening:
circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref
The main reason is that 10% of strokes are associated with undiagnosed atrial fibrillation. The patients who present to the emergency room are a pretty biased sample--for one, they're experiencing symptoms. To prevent strokes, we need to have a way to catch AF in asymptomatic people.
Part of the challenge here is that episodes of atrial fibrillation can be infrequent--in CRYSTAL-AF, for example, it took 84 days from randomization to first episode--and existing monitoring devices like Holters or Zio patches are only worn 24 hours to 2 weeks. The great thing about Apple Watch and other consumer wearables is that they're worn for months or years. That means if we can prove the algorithm is accurate, we can get higher time-coverage than a traditional medical device, catch more AF early, and prevent those 10% of strokes described in the Circulation article.
Perhaps it would be more accurate to say "we might catch more AF early, and anticoagulating those people might reduce their risk of debilitating stroke more than we increase their risk of lethal gi bleed or head bleed"?
The issue is whether the af you find through this kind of screening is associated with the same risk of stroke as conventionally diagnosed af - if not, their risk reduction from anticoagulation might not justify the bleeding risk from anticoagulating them.
The linked article suggests it might be "unethical" to even do a trial of anticoagulation vs control in af detected by such screening. That seems like a dangerous position for them to take, particularly coming from a group that is largely funded by the drug companies who sell expensive anticoagulation medication.
I would encourage readers to jump to the "sources of funding" and "disclosures" section of the linked review article to see just how many of the authors receive money from the drug companies.
None of that is a criticism of the authors of the UCSF study, this kind of technology is certainly an area worth exploring. I would just be very wary of the push by drug companies and the doctors paid by them to
1) find asymptomatic conditions
2) call it 'disease' based on older studies of patients who were much sicker
3) 'treat' them with expensive medications for the rest of their lives without actually doing research to see if it would benefit or harm the patient.
actually, the public health risk of stroke in paroxysmal AF is pretty high. Even if the expensive drugs weren't available, we'd still use coumadin which is dirt cheap. The elimination of stroke risk is worth the risk of bleeding complications.
actually, you're assuming that the kinds of patients who were diagnosed with paroxysmal AF in the studies showing increased stroke risk 10 or 20 years ago are going to be similar in risk profile to asymptomatic patients who might now be diagnosed with paroxysmal AF based on some as yet underdetermined time criteria using new technology. To quote the paper:
"Alternative arbitrary or data-derived atrial high-rate episodes burden cut points have been explored over the subsequent 10 years, ranging from 5 minutes to 24 hours.11 Uncertainty remains about the minimum burden that increases thromboembolic risk."
On the other hand, there is no uncertainty about the degree of risk from bleeding (small, but present, and occasionally lethal). So yes, patients with paroxysmal AF based on current technology and clinical standards benefit more from reduction in stroke risk than they lose in bleeding risk. That doesn't mean that all patients in the future, with any degree of paroxysmal AF diagnosed using more sensitive technologies will benefit similarly
I think what you are trying to say is that - you think there may be varying "degrees" of PAF and that certain types may be less risky? That is something that certainly can be debated, but at least most of the of thinking so far is that any presence of PAF, aside from isolated episodes after cardiothoracic surgery, carries the same, if similar risk.
I think the way my stroke colleagues look at it, is sort of the other way around - more like PAF is PAF, and as far as they can tell, based on current data, there is relatively few stratification levels that the current medical data can tell us. Meanwhile, the current assumption of risk from PAF, is so high, vs. the risk of anticoagulation (adjusted for individual patient, i.e. maybe not the patients with coagulopathies or risk of falls) that they view NOT anticoagulation as a higher risk. Because if one is wrong about it, the result is paralysis, coma or worse. If there are cheaper ways of detecting AF, all the better.
But overall, anticoagulation decisions aside, think about it - this is a potentially a cheap way to passively screen for the condition in the population. Orders of magnitude so, and possibly w/ more accuracy than a typical holter monitor or even the newer implantable devices (which in all probability use simpler rhythm analysis). Even if you are worried about the potential existence of differing levels of AF, the wealth of new data should go a long way towards further understanding AF and figuring out whether differing levels of risk exist.
This is really cool. I have an ICM (Medtronic LINQ) for something I don't believe an iWatch could detect so this isn't an option for me, but I do know they are sometimes used for AF as well, and mine cost over $60,000 to implant (billed to insurance, though my part was still much more than an iWatch costs). I have to think doing the same with an iWatch massively increases the pool of people who are capable of affording continuous monitoring.
I'm in exactly the target market here in the UK. I've had PAF in the past, I suspect it happens at night when I'm asleep, but multiple cardio traces have failed to catch it.
Short of buying my own Holter trace machine ($500, and an inconvenient mess of wires) or having an implant (possible on the NHS, and being considered, but the waiting list is long) there's no way I can be sure if PAF is enough of a problem to justify the lifestyle changes that would be caused by going on anti-coags.
Cheap continual monitoring would be a game changer.
Yeah I've used a Holter a few times they are definitely a pain to sleep in! For certain arrhythmias you need a 6-12 lead EKG though (iWatch wouldn't work in those situations as far as I understand).
Hmm, this makes me think personal insurance will at some point in the not too distant future give hefty premium hikes to those who refuse to wear a 24-7 monitoring device; as vehicle insurance in the UK is doing with monitoring devices (cameras/accelerometer-type devices).
Actually this is a huge game changer from a neurology perspective:
1) for folks who show up in the hospital, with a stroke, often the cause is not clear. So they go home on aspirin but the latest studies seem to indicate that fully 30% of these folks with "cryptogenic stroke" end up being diagnosed with PAF after 3 months on a heart monitor.
2) this heart monitor is either a cumbersome external device they have to wear for 3 months, or the Medtronic LINQ implantable loop recorder, which is nice and under the skin, but costs a lot of money for a cardiologist to implant and monitor.
3) from a public health perspective, if the Apple Watch can automatically detect paroxysmal A-fib BEFORE a stroke (i.e. permanent paralysis, inability to speak, etc)...think of the massive societal benefits this could be...
As a Paramedic here in the US, I'd be happy to see more people alerted to unusual cardiac conditions via their Apple Watch early as possible rather than finding out weeks later after damage has been done.
We don't convert afib in the field, but having an Apple Watch catch someone that's in SVT would be great. Reading down in the thread, I see that it's happened at least once.
Not the developer or founder, but a user of Apple Watch. I would like to see your questions answered as well. But as a hopeful consumer, I hope wearable devices / medtech startups direct more attention and funding to improving sensors and associated detection algorithms to such an extent that at some point in the future, it would actually be useful to real doctors and offer real medical benefits.
Wouldn't this help with pre-stroke detection? The article mentioned that 1 in 4 strokes are caused by AFib. I would imagine this would be helpful for in-patient Neuro-ICU and stroke hospitalists. On the patient side, shouldn't something like this - assuming the monitoring is in real-time - provide a preventative/detection benefits for patients with a high-risk for stroke?
Even before that - putting in on an Apple Watch potentially enables screening from a public health perspective. I'd rather prevent the stroke rather than regretfully putting the patient on coumadin after they've been paralyzed...
>With due respect to the Cardiogram developers (hi guys) as a doctor I really can't see a huge amount of value in this - my patients who present to emergency departments with paroxysmal AF are all anticoagulanted or on rate control and there has only been one instance in the last 3 years and many thousand patients when someone has presented to emergency and we have had to run through the full spectrum of echo -> anticoagulante -> cardiovert.
Isn't this a version of survivorship bias? Who about to those who aren't that lucky to manage to overcome the attack and become patients...
AF is rarely fatal, although stroke is significantly disabling. Normally (well, I don't have any figure at hand and I'm not about to go and research it at the moment, but I'm using the term in the sense of >50% but <85%) of AF instances come with some sort of symptoms... feeling faint, pounding chest, shortness of breath... good questioning can be fairly diagnostic in the doctors clinic or emergency room. But yes, as the Cardiogram guys point out in their reply to me, and in the source article, a significant number of people have strokes secondary to AF.
How are you managing regulatory issues? The Apple watch is not a legal medical device, so my understanding is that it cannot make any diagnoses recommendations to the user.
I would assume any alerts they give would be something like "we have detected an anomaly in X metric we monitor, please consult with your doctor/health professional".
Have you experimented at all with using the Apple Watch to measure blood pressure?
I have done some reading that suggests the optical sensor could measure blood pressure with some accuracy, but that Apple is hesitant to release it as a feature due to regulatory and accuracy concerns. It's my #1 wished for feature.
Measuring blood pressure (non-invasively) requires applying pressure to the artery and determining at what point that pressure obstructs blood flow. That would be a) challenging to do with a watch, and b) really annoying to the user.
Not entirely true, there is a clever method that works without (pulse transit time measurement). You measure the electrical signal of the heartbeat, and the acoustic signal, and take advantage of the fact that the acoustic signal propagates slower in blood than the electrical signal propagates through nerves. This difference is directly related to blood pressure! I don't know if you can get away with one measurement on the wrist + some clever calibration, or if you need two points (e.g. one close to the heart). If you take two measurements, you can apparently also calculate the aortal pressure, which is different from the pressure in your arm, and hard to obtain non-invasively. (I'm not an expert so I'm sorry if I got something slightly wrong :-) .)
There was a startup that worked on using this, but they failed (due to financial but also regulatory reasons), and then remotely bricked all their sold devices...
I really wish someone would develop this further. Even if it is not as accurate as a normal measurement, the implications would be huge. There are so many people running around with hypertension who have no idea. I also don't see a risk in false positives in this case, since in principle everybody is recommended to have their blood pressure checked - false positives who go to the doctor are then just like people who read an article and go to the doctor, and are weeded out there. False negatives might be a problem - but if you don't advertise it as a blood pressure measurement tool, but just implement it as an additional warning in a smart watch, you'd reduce the false sense of security people would get if it didn't work properly.
Accuracy is a useless metric for something like this. If you have binary data filled with 97% zeros (ie most of the time it is not arrhythmia) you can use the sophisticated machine learning technique of:
if(TRUE) return(0)
This will give you 97% accuracy.
EDIT:
I just read the headline earlier. Now after checking:
>"The study involved 6,158 participants recruited through the Cardiogram app on Apple Watch. Most of the participants in the UCSF Health eHeart study had normal EKG readings. However, 200 of them had been diagnosed with paroxysmal atrial fibrillation (an abnormal heartbeat). Engineers then trained a deep neural network to identify these abnormal heart rhythms from Apple Watch heart rate data."
So 1 - 200/6158 = 0.9675219. My method performs just as well as theirs if we round to the nearest percent. This is ridiculous.
Cardiogram engineer here. 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity.
These important details are often lost in the news. You can some more details on our findings in our blog post:
Just wondering, with such an unbalanced dataset (5,958 negatives, 200 positives), wouldn't have been fairer to use average precision (area under the precision-recall curve) instead of ROC-AUC?
Can you quote the part that leads you to think that? At first I was just commenting on the title, but see the edit. I would agree, my interpretation makes this the most ridiculous hype I have ever seen, so maybe I missed something.
No quote in particular, but it seems unlikely that they would miss such a blatant dataset bias and fail to instrument their ML models with metrics besides accuracy. But then again, welcome to Silicon Valley :)
1 - 200/6158 = 97% is indeed a pretty suspicious coincidence though. I would assume/hope that they've shuffled a big dataset of recorded heart events (like the image in the TC article) and that the 200 people diagnosed with paroxysmal atrial fibrillation only rarely experience AF, so the number of true positives is probably far smaller than 1% of the dataset.
We need to see the full, published study and its methods (particularly around recruitment and exclusion criteria) before we can judge it properly. Until then, the presented statistics about accuracy, sensitivity, and specificity potentially bear no relation to real world usage, if the cohort and data quality were tightly controlled, as you'd expect for an initial study involving the makers of the algorithm. A few other thoughts:
1. Even at 98% sensitivity and 90% specificity [0], which I don't think would hold up with real world usage in casual, healthy users, if AFib has a prevalence of roughly 2-3% [1] then by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib.
2. There is huge variance among people's real world use of wearable sensors, and also among the quality of the sensors. (Imagine people that wear the watch looser, sweat more, have different skin, move it around a lot, etc.) You'd likely need to do an open, third-party validation study of the accuracy of the sensors in the Apple Watch before you can expect doctors to use the data. My understanding is that the Apple Watch sensors are actually pretty good compared to other wearable sensors, but I don't know of any rigorous study of that compares them to an EKG.
3. Obviously, this is only for AFib. AFib is a sweet corner case in terms of extrapolating from heart rate to arrhythmia, because it's a rapid & irregular rhythm that probably contains some subpatterns in beats that are hard for humans to appreciate. As others—including Cardiogram themselves [2]—have pointed out previously, many serious arrhythmias are not possible to detect with only an optical heart rate sensor.
Full journal publication is coming--as you likely know, the system doesn't always move as fast as we'd like.
> quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive.
For what it's worth, about 10% of people who come in to the cardiology clinic experiencing symptoms are diagnosed with an abnormal heart rhythm. So even a 20% positive predictive value would be an improvement over the status quo.
As mentioned below, you can use other risk factors (like CHA2DS2-Vasc, or even simply age) to raise the pre-test probability, and thereby control the false positive rate.
As a meta-point, I do think we let the perfect be the enemy of the good in medicine, and that potentially scares people away who could otherwise make positive contributions. For example, many of the most common screening methods in use today are simple, linear models with c-statistics below 0.8. You can build a far-from-perfect system, and still improve dramatically over how people receive healthcare today.
My overall message to machine learning practitioners sitting on the sidelines would be: please join our field. The status quo in medicine is much more primitive than we have been led to believe, and your skills can very literally save lives.
Thanks for replying! I'll certainly be looking forward to the publication.
>about 10% of people who come in to the cardiology clinic experiencing symptoms are diagnosed with an abnormal heart rhythm
OK, but I'd be more careful about staying apples to apples in your comparisons; your app is about asymptomatic AFib. So how many of those people going to the cardiology clinic had undiagnosed AFib; for how many of those would a new diagnosis of AFib have changed the plan of care; etc. Kind of like robbiep was saying, I would be interested in actual added value from the larger perspective.
Totally appreciate your point about perfect being the enemy of the good. The danger is that these semi-medical wearables currently straddle a strange zone between medical and consumer use. The inevitable marketing strategy is to co-opt the positive reputation of medical products while acknowledging none of the pitfalls of consumer products. Most of the screening methods you bring up are used by a doctor on symptomatic patients with a suggestive history, and only as a partial component of clinical judgement. The way Cardiogram seems to make the most money, on the other hand, is to sell the product to asymptomatic, casual users. (Furthermore, CHA2DS2-Vasc costs 30 seconds of talking or reading a medical record, not $700 in Apple products.) So you're inevitably running up against some doubts among physicians [0].
And finally, I agree that more machine learning practitioners should join medical research. I hope the field works to set more reasonable expectations, however, as in: ML will solve very specific subtasks in clinical reasoning (as in the diabetic retinopathy study [1]). Instead, the headlines usually ratchet that up to "AI will replace radiology/cardiology/$specialty in X years." That tends to hurt the people currently in the trenches, since their contribution in bringing about practical, incremental change is diminished. The top answer of this Quora thread [2] has a good discussion of the many dimensions of the problem.
The diabetic retinopathy study (and the somewhat recent stanford dermatology study) were the first ML studies I had read about that blew me away in terms of their sensitivities and specificities, as compared to real doctors. Your comment on specific subtasks is perfect, and I try and use these examples when discussing ML with fellow medical students.
However, like you said, the medical field is very slow, and has quite a lot of inertia to maintain the status quo. Unless insurance companies refuse to compensate practitioners that don't use these tools, I fear that few, if any, in the healthcare field will opt to use such techniques.
And finally: How should someone with both a medical and computer science background get into ML?
I found the Statistical Learning self paced course on Stanford's site to be a great formal intro to ML algorithms implemented in R, and it is taught by the inimitable Hastie and Tibshirani: http://statlearning.class.stanford.edu
> by a quick back of the envelope calculation a positive test result is still 5× more likely to be a false positive than a true positive. With those odds, I don't think many cardiologists are going to answer the phone. You'd still need an EKG to diagnose AFib.
This is a good point, and certainly nobody should go directly to a cardiologist based on these results. It seems that this would be a good system to recommend that people get an EKG done, though.
> probably contains some subpatterns in beats that are hard for humans to appreciate
Not really, no... As you said, AFib is one of a very small number of causes of irregularly irregular heart rates (and is by far the most common). AFib is pretty easy to spot, even just by feeling someone's pulse with your fingers.
If I am reading this correctly, there were 6,158 patents with ~200 true positives, so approximately 3% of the population. 98.04% sensitivity (recall) and 90.2% specificity (%true negatives) leads to...
~4 false positives for each true positive.
That isn't bad, all things considered, but still a long way to go.
They aren't clear what they mean by "97% accuracy". Does that mean 97% of people with arrhythmia are correctly diagnosed, or 97% of people are correctly diagnosed or not-diagnosed? If it's the latter, it's not very helpful at all. The number of people in the general population with arrhythmia is significantly less than 3%, so if this Apple Watch test says you have arrhythmia it is far more likely to be a false positive than a true positive.
Here, 97% accuracy refers to a c-statistic (area under the ROC curve) of 0.9740. An example operating point would be 98% sensitivity with 90% specificity.
For reducing false positives, rather than starting with the general population, it'd be natural to start with a higher risk sub-group, e.g., people with a high CHA2DS2-Vasc score.
Earlier this week, Circulation published a review screening for atrial fibrillation:
circ.ahajournals.org/content/135/19/1851.full?ijkey=StzSPk8eljGaP2G&keytype=ref
No, accuracy doesn't tell us anything about usefulness. If the test is cheap and it brings in people for a better test and thus saves lives it is useful. The only real danger is if people decide that since the watch says there is nothing wrong so they can ignore other symptoms.
It is something like chest pain: most of the time chest pain is not a symptom of a heart attack, but it is best indicator we have so you go to the emergency room when you have chest pain. Doctors there can evaluate your situation.
On a related note, I had a very rare stroke last week, only symptom: headache for 3 days.
The doctor sent me for a CT scan without any idea of what to look for. The only clue: it was not a migraine because I missed any neurological anomaly.
On the other hand, stroke prevalence is rare, 1.3% for men, and generally don't look like mine, and my kind is 2% of all strokes, it would have been completely weird to suspect it with such a low probability and such a non-specific evidence.
Long story short, I almost got sent home with an aspirin while I had an extensive cerebral veinous thrombosis.
Having spent a couple of days in a stroke ward over Christmas, the low prevalence of strokes should never be used as a reason not to test where possible.
I suspect a lot of people don't understand how serious and crippling - mentally and physically - a bad stroke can be.
If you're lucky a bad stroke kills you. If you're not lucky you lose a good part of your brain and motor function.
In practice this means you can be left unable to move some or all of your limbs, unable to talk, unable to hear, unable to understand what's happening around you, and perhaps unable to see.
It's no exaggeration to say that it can turn life into a nightmare.
Anything that makes this less common and less likely is a good thing.
Disagree. Note that both mammogram and colonoscopy frequency recommendations have both been reduced recently, both because the cost of screening and because of the consequences of false positives resulting in unnecessary medical intervention
That is a useful point, but not reason to disagree. In this case a cheap test that indicates you need a better test is useful. The risk of false positives needs to be considered, but that is for medical professionals. Even if the ultimate answer is for most people nothing more is done it still gets people into the doctor to explain what else they need to look for before a test is done.
I've arrhythmia and I can see it on my Garmin watch by looking at the data, as well as .. my phone.
It doesn't help much though, because I don't know if its good or bad (well, actually I know but not because of the watch data).
Doctors are still needed for this, and generally that includes a bunch of controlled tests and people listening to your heart while also gathering data (similarly to the watch albeit with a more precise apparatus)
I guess it can help to tell people they might wanna see a doctor if they haven't though.
First validation study is on atrial fibrillation, although we've had users who have discovered other arrhythmias through the app. Here's on example of a person who discovered supraventricular tachycardia:
https://blog.cardiogr.am/my-apple-watch-saved-my-life-a61256...
Despite all the caveats around this work, early detection is one of the reasons HR trackers are a great investment. You can't manage what you can't monitor
To the founders: what do you see as being he end game here? Are you just looking for validation (of the concept in itself, not the app- I use it on my watch)? Is the US market so different that this is particularly useful and cost effective for detection? Do you see this as eventually displaying early warning in the instance of an early warning?
Thanks and I don't mean to denegrate your efforts, but I do see lots of Consumer Med tech as solving a problem that really isn't creating value (i.e. Proliferation of devices,wearables and algorithms that proclaim the ability to help with X but are really marginally helpful at best) and I'm wondering if I'm missing something about the actual medical benefit, or whether what I feel like is true- that they aren't after being a medical device at all but instead are chasing the consumer dollars by making medical claims