Can AI Outperform Doctors? The Reality Behind the Claims
When Microsoft claimed AI achieved 80% diagnostic accuracy compared to doctors, it stopped healthcare leaders mid-scroll.
But the real question isn't whether the technology works—it's whether it can improve patient care in the messy reality of hospital systems.
In this episode, our panel of experts—a founder, physician, educator, and CFO—examine what Microsoft's article, "The Path to Medical Superintelligence", really means for the patients they're committed to serving, the financial implications for the hospitals, and the applicability for future healthcare professionals. Ultimately, providing the clinical context needed, the real value proposition, and the commercialization aspect to separate signal from noise.
In reality, the AI landscape is a minefield of misinformation, but it doesn't have to be. We're all in the business of doing what is best for our patients, and this episode offers the grounded, peer-to-peer analysis you need to navigate the hard realities and lead with clarity.
“We want what's best for our patients. And we know that we're reaching a block. We know technology is exponential. And our laws are kind of logarithmic, and we're not there yet. And we need some change.”Harvey Castro, MD
Loading...
Highlights of the Episode
[00:20] Tension Between Hype vs. Value
[01:50] Evaluating AI as Medical Superintelligence
[05:28] First Principle of AI in Healthcare
[07:12] The Hard Truth About Implementation
[09:50] Cost Analysis of AI Implementation
[11:23] Payer-Provider Relationship
[14:21] The Future of AI in Healthcare
Resources
Microsoft AI: The Path to Medical Superintelligence
Microsoft AI Journal Article: Sequential Diagnosis with Language Models
Transcript
Junaid:
Welcome to Signal and Symptoms podcast. Today's topic is artificial intelligence in healthcare. And we're going to talk about Microsoft's recent article that actually said that we have a pathway to superintelligence.
Junaid:
Now the question is, it really true? Or is it just a hype? And again, as you know, in our conversations, we have always talked about hype versus value. I'm gonna actually let Harvey Castro, my esteemed colleague, talk about this. And then we're gonna go and do into nitty-gritty details that is it just, again, hype versus value. So Harvey, take it.
Harvey:
Well, first of all, honor to be with these guys. Really exciting to see Ed and Lise and you all together. So let's just jump right in. So fundamentally, we have an article from Microsoft that basically says, believe it or not, that the AI is around 80 % on really tough cases and us humans, the expert in the loop, doctors as myself, are around 20 % right off the cuff.
Harvey:
I know that doctors around the world are saying, you know what, is this a textbook presentation? Because if it is, of course it's going to get it right. And the other part, my data scientists around the world are looking at this and saying, well, this is kind of cheating because if the AI knows about these cases ahead of time and it's trained into the data set, then obviously it's going to perform higher than the human because it already saw the case. And so it becomes really fundamental question. Is this really hype? Or is this really something true science that we actually don't understand and that it actually can make a fundamental change in medicine? So I'm really curious what you guys think.
Junaid:
So Elise, what do you think? mean, I would let you go first. You're a combination. You're teaching to students. What do you think the future lies in terms of A, applicability, and B, these studies are really skewed to look good on media?
Elise:
Yeah, I agree with that. I think we are looking in isolation at these studies and they are not necessarily representative of real world in terms of if we're, you know, if there's a patient set or a data set, there's bias built into that when we try to duplicate that. I don't know that it can scale across the way that it's necessarily represented in the media, right? We've seen this time and again, where, you know, something is proven effective or we have a high percentage of, you know, efficacy. And then when we bring it elsewhere, there are a lot of factors that were not considered, right? Just so many variables that I think being able to really take this and fulfill that vision that I think a lot of people have when they're reading, you know, articles like this and getting excited about it is… I think we're a little far off from that.
Junaid:
Ed, what is your sentiment that when you talk to CIOs and CDOs, when these articles show up and then when you are actually talking about even founders like me, how do you approach this for both from a founder perspective and a CIO perspective?
Ed:
Yeah, I think it gets a little dangerous when we're too far out ahead of reality, because I think people can get discouraged, right? And so we've seen that with, let's pick on IBM Watson from way back when. So I was just with the Cleveland Phonics yesterday and a lot of my friends up there, physicians and researchers, and we were just kind of talking about the old days when there was all this hype that, you know, what's going to happen. And if we do too much, we get too far out ahead. I think there's a discouragement that gets set in. But that said, I think we need articles like this and research like this that shows us the way, right? But we don't want that gap to be too long. If it's 10 years, it's discouragement. And then people, when they run into a founder, they're like, yeah, everyone says that it's never going to happen. I'm not going to make the investment. I'm not going to take the time. And as a result, we don't get the digital transformation that we all see.
Ed:
So I think the article is great. I think it's good to have that future vision cast out there. And we start working towards that. just saying that we need to make sure that that time gap isn't so big that it has the opposite effect, right? It discourages founders. It discourages CXOs to make that investment, to take that chance to do that pilot. So I love the article. I love the future thinking. It's gotta be careful that we're not too far ahead of ourselves.
Junaid:
Let's go, Harvey. think we should still talk about, let's say, if we implement it today, what is the real value proposition?
Harvey:
You know, one of the core things about the study was us physicians, are we ordering too much? As an ER doctor, we'd get called into the principal's office if we're ordering too many tests that aren't unnecessary. As you as a specialist can agree that sometimes you look at the ER workup, you're like, why'd you order X, Y, and Z? You know, that wasn't necessary. You could have found that record a month ago. It was there. And so that's part of the study. The AI was being able to be more efficient with ordering the tests and that whole algorithm change.
Harvey:
One of the things that you and I have talked about in the past, and I know you're a fan of first principles, is what are the first principles in this article? Why does this article even exist? Why is it getting so much bandwidth? At the end of the day, it's because of misdiagnosis. Millions of people are being misdiagnosed. People are dying every year. And so the world, not the United States, the world is crying out for what can we do in healthcare? What is there? And here's an answer, possibly, maybe it is AI coming in.
Harvey:
Two in the morning, I'm not at my best, I'm tired. But if I have AI literally in the EMR, looking at the signs and symptoms, looking at the efficiency, looking at my ordering and saying, hey doc, did you forget X, Y, Z or why did you order Y, Z? Then I can say, you know what, probably a good point. I may not need that, here's that record. And so that's why I'm really interested in this study. I think there's such a deeper message here that we don't really realize.
Harvey:
If this is in fact reproducible, and if in fact we have access, like I'm dying to get access to this, to start playing with it and testing it out, but assuming it does, man, this opens the door to so many possibilities.
Ed:
It's great. I reviewed the paper. think it's amazing, the technology and how it's all orchestrated. That's kind of like the new buzzword in healthcare is how do we orchestrate everything? Because as we know, we're dealing with fragmentation at scale in many different areas. So what's needed is orchestration. So I love the fact that we're able to leverage AI to do this. And yeah, of course, there's going to be, that's the first thought of the CFO, right? And no matter where you are.
Ed:
You know, we're all driven to healthcare because we want to serve others. We want to see healthy communities. We want to see highest quality of care. We want to see reduction in morbidity and mortality. But at the end of the day, sadly, right, it still has to come down to dollars and cents. And so, yeah, there's going to be the eyes, you know, from a financial point of view, like how do we bill for this? How does it impact our overall revenue?
Ed:
You know, that's the other thing when we talk, you know, not just in this case, but in the future cases that we speak about is another reason why we haven't experienced the full potential of AI and other digital capabilities is the payment system is a laggard. And as long as you have that gap, that's another gap, right? As long as you have that gap that exists, it's going to retard growth for all of these amazing things that do the right thing for people. So.
Ed:
That's one thing they asked me look at. So yes, I would receive this as a member of the C-suite or as a board member of a couple of health systems. And you would say, wow, we got to do this. But again, there's going to be this review to the bottom line, right? How does this impact revenue? It's a sad thing, but a reality. And so that'll have to be carefully scrutinized. So as we look at this from a clinical point of view and a patient and clinician experience point of view, we also have to look at it from a financial lens.
Junaid:
I'm gonna go jump to Harvey for two minutes. Harvey, from an ED physician, I mean, we practice what we call defensive medicine. And we are always worried about medical legal. And some of the time, I recommend getting a CT brain on a person with known diagnosis of seizures, known non-compliance. But I said, no, we want a CT scan because we wanna do this. And that brings us, what are your thoughts in this particular? You know, if you were to implement it in 40 different EDEs or urgent care clinics, how is your perspective changes when you have defensive medicine in mind?
Harvey:
That's the sad part. Big picture, I know some physicians have picked states that have strong tort reform so that they can practice medicine and not worry about that side of the medicine. But with that said, at the end of the day, ER is really tough in that seconds count. don't have data. You may be asleep. I may not be able to call you at two in the morning and get information. So I have to do tests that I already know have been done. Unfortunately, that's just...
Harvey:
something that we have to deal with and different doctors have that bell curve. Some doctors are very conservative and they order everything under the sun and some others are like, no, I get it and they try to do their best. A tool like this, now there's two issues. One, when you look at the study, the cost is really high and I think it's just because of the token cost, the information that they're having to spend on, but.
Harvey:
That's going to change with time and I don't think that time is going to take long. So putting that aside, if that cost is low, then I see this as reducing our expenses. There's two points. Ed mentioned this. If our EMR is really well integrated to the point where I have all that information and it should be, then there's one thing I don't have to order tests. There's so many times I get patients, I'm like, hey doc, I had these labs done literally last week. Why are you doing them again?
Harvey:
Some of those labs I have to redo, but maybe 80 % I don't. And so if I have that EMR that's integrating and then have AI going into that saying, hey, yes, I see it. This lab was done. There's a low probability. Then I see me decreasing my costs and decreasing that quote unquote defensive medicine.
Junaid:
Ed, do you think that insurancers and peers, including Medicare, Medicaid, and slash or private insurance being vested in adding this technology in a way that partners with the hospitals this way they can actually decrease their costs, the overall costs, and is it something that is going be feasible?
Ed:
Yeah, I know I preface my answer to every question first from a very high level strategic point of view, and I'm going to do the same here. Of course, in the ideal world, providers and payers would work together to solve many of the things that ail us, but we don't. Right. And that's why you've seen the birth of pay biters or a Kaiser model where it's both payer and provider. but even that there's still a struggle internally with these health systems between the two parties because what typically because there's perverse incentives, right? A sentence are misaligned and what benefits one party is usually at the detriment of the other. So if there was a way, a utopian way, and I think there is where we could work together for the benefit of our population, the people that we serve, I think we could solve this. So going back to the financials now and answer your question at a more granular level, I think we could come together with payers and providers and work out a way where this becomes super feasible for the reduce in overall cost and increases quality. It hits the quadruple aim basically, and we can we can make it work at scale. But again, and I'm an optimist, but also a realist, working through the details of that in a payer provider sort of relationship is not something that many have perfected, right? I've dealt with this my entire career and I try to understand both sides. So don't pick a party who's to blame or whatnot, but I think if we could come together, leveraging an orchestration capability like this would reduce the cost and I think we would have enough revenue that everyone would come out well and ultimately the patient would benefit.
Junaid:
Harvey, thoughts.
Harvey:
Yeah, well, number one, I'm gonna put on my doctor hat. To me, all it needs is one person to save their life, to improve them, and it doesn't matter that 20 million, 100 million investment. To me, that's crap. Now I'm gonna put my CEO hat of a hospital administrator, and then I look at that and I'm like, let's do something like this because it's gonna end up saving dollars. Now I'm gonna put on my AI hat and say, dude, why don't we just convert this to a llama model and show the improvement, and if we can prove that, Man, now this is nominal. Now it literally flips this whole equation where maybe the cost is maybe a million dollars and the output that we're getting is so much better. And then I'm gonna add another layer. If we can train the doctors, train the C-suite, everyone in that healthcare experience, then I would argue that the cost will go even lower because now our output would be even better.
Harvey:
First of all, again, honor to be with this panel. Thank you all for being here and people at home watching. You know, this is an evolution of science, right? We're here and where we were last year wasn't where we are today and where we were two years is way further where we are. So let's see where we are maybe in just a month or two. With that said, at the end of the day, for me, and I can speak for the panel, we want what's best for our patients. And we know that we're reaching a block. We know technology is exponential and our laws are kind of logarithmic and we're not there yet. And we need some change. We know that if anyone on this panel comes up with an invention and there's not a CPT code or some kind of code for the government or the insurance to charge, it's not gonna happen. And we know that studies like this is saving lives, but yet we're not implementing it because of the costs. Hopefully in the near future, that disparity and gap becomes shorter and shorter to the point where we are implementing, we are changing.
Harvey:
So I'm really excited about the next time we meet. I really want to thank everyone out there. And the last thing I want to say is use AI as if it was your bike. The more you start pedaling, the more you understand, the more you start realizing, huh, this tool isn't as bad as some people make me think. And then the last thing I want to say is today you may judge AI and say it's crap, but make sure that you are using the best AI out there. Test it out so that when you judge, you're judging from the best model that's out there.
Ed:
I want to jump in and add one more thing. And this is probably going to be a common theme, I think, perhaps in many of our episodes. And what it really is going to take is courageous leadership, right? Because we've identified, even today in this episode, identified three or four major blockers that it would be easy for any one of us to say, yes, right, thing for the patients, but man, it's too hard. Forget it. It's too hard. And that's what most people do.
Ed:
I'm telling you, most people in positions of leadership, they throw their hands up in the air. They say, well, I would do this if I could. And I can't because I'm limited by all these different things that we touched on today. But it's a bunch of BS, right? We know it. I know the four of us know it. You got to go out there and make stuff happen. You got to be a courageous leader. And I think that's going to be a common theme that we're going to keep having to come back to. It's like take these ideas and make something happen. Work with founders, take risks. Don't worry about your freaking job. You do the right thing for people. And if we had more of that sort of courageous leadership and didn't lack as much, I think we'd be way ahead. So let's all go out there and just do the right thing. We got the technology, technology's not the issue. Brilliant people doing brilliant things. We just gotta make it happen from a leadership perspective.
Learn more about the work we do
© 2025 Signal & Symptoms. All rights reserved.