Good morning everyone for another episode of Signals and Symptoms. I'm glad that you are here with us and we have our expert, Harvey Castro. I want to share a study with Harvey and get his thoughts that radiologists' last exam benchmarking frontier multiple AI models is an exonomy of visual reasoning errors in radiology. And they actually basically created the toughest exam on purpose. That, hey, what are you going to do? And this is how they were actually scored.

Today's topic I wanted to talk about first before we get into this and I want you to take the sort of lead on this because I know you love this and I love too by the way, but given that I am so hooked with my keyboard, given like sort of I work now as a coder, but anyways, this is your game. Get see more without ever looking away. And I know you have your glasses and I did you see the announcement and everything. What are your thoughts on this? what do you think how is going to improve? human in that's exactly what we talk about human loop human in loop

Yeah, let's talk about it. So let me start off with my poor son. I sent him to give me a pair of glasses because I'm out of the country and he said, hey, dad, there's a long list. There's about a three to four week waiting list. So now I'm waiting and then they have to be fitted. so unfortunately, or fortunately, I'll be getting them here in about a month. But how do I see this in health care? No pun intended. Imagine these protocols. Imagine a guideline that just pushed out by the hospital. And I'm having to see my patient and I don't know my guidelines. But if I'm able to attach these to the glasses and now be able to see them on the screen, how nice would that be? One of the things I'm personally working on is trying to get the APIs for this or access to this so that I can catch emotional intelligence.

So if my patient needs X, Y, and Z, it's catching their emotional intelligence, I'm processing it here and I'm able to give better care or maybe I'm autistic, or maybe I'm having trouble with the most intelligence, having that interaction. So that's just one quick example of how this could really change the future. As an ER doctor, I know the battery life is not the best, but if I'm able to have this and in real time, I need your data, I need your labs, I need your x-rays, and I can just pull it up right there. I mean, that's life-saving. When seconds count, be able to jump in and be like, this lab just came in and I need to act.

I was literally in Denmark a couple of weeks ago, speaking at the World Federal International Pharmacy. And then one of the biggest things I've talked about is having this information in real time, connecting it to my pharmacist so that when that overdose comes in with Tylenol and I need that antidote, even real time, this information is going to the pharmacist and boom, through AI, he's already working on the antidote and sending it to me before I even ordered it.

No, mean, so we have an AI scribe, of course, in this world class, but the word that you said was two things. Number one, capture of accuracy right then and there. So if the person shows me a wound or tremors or something, I can just click a button and rather than just capturing the audio, I can just capture this video and the actual picture within the note itself. This is again, a picture is worth a thousand words. And then if the video is essentially a series of pictures. So you can say a million words right then and there. So that was something I use case that I thought, man, that would be interesting because if a Parkinson's patients comes in, it is very hard, you know, that they're going to come back in three months. I'm going to change and adjust the medication today. Is it really actually helping? And if I have video proof, I can just go and then think about it. That if I want to put a DBS, the brain stimulator, I have a whole series that this medication he tried, he failed and therefore he needs a deep brain stimulator again.

Very powerful in terms of capturing both, as you said, just pure, it's not magic anymore, like a picture and a video, but that's on your face, it's available. And as you suggested that right then and there, when I'm talking to the patient, I mean, I want to make sure that they're taking the medication, which is, don't know, Parkinson's medication is four times a day, it has to be specific timing. You don't want to take it before you sleep because you don't want, you know, just you don't tremor inside sleep. So if you're taking a nap, have to adjust your medication. And those kind of capturing is actually very difficult. It's not like regular medicine, AM and PM. And that actually also improves the capture because right then and there you have it on your face and it's recording and then going through. So long story short, that is something that is fantastic that is going to add availability. As you said, right information at the right time. That is the main job of AI, which is half the time. Like 80 % of the time we are still hunting for, you know, medications and information. So I'm going to be really impressed when these come out and hopefully we'll get a hands on it. We should reach out to Meta to give us early access to APIs so that we can actually build things on it. So let's just put that in the system and then we're going to go ahead and then go from there.

Now today's topic, the main topic I wanted to discuss was radiology. So before I go to the other topic, which we talked about agentic AI and how this is going to change the way systems are moving. And we talked about last time, essentially, let me just first start with this. This is the main topic, the algorithmic consultant, a new era of clinical AI calls for a new workforce of physician algorithmic specialists. And then the main topic is human AI and everything. And then that's why we're talking about this. But I want to, before we get into this, I want to share a study with Harvey and get his thoughts on radiologists' last exam benchmarking frontier multiple AI models and the exonomy of visual reasoning errors in radiology. And they actually basically created the toughest exam on purpose that, hey, what are you going to do? And this is how they were actually scored. So radiology's last exam, Claude Opus is actually insanely low, which is really odd. Then comes Grok for OpenAI 03, then Gemini 2.5 Pro, GPT-5 Thinking. And then this is radiology training, trainees. And then finally, we have board-certified radiologists. And this is essentially, again, by design, a hard sort of pass, but this is really where not the radiologists even did 100%.

So most radiologists got to 80 % accuracy on this test. And this is really interesting. So we have been thinking about it that, especially in radiology, AI is catching up with a very small niche and this. And then are we taking this and going into the future, which is physician algorithm specialist, let's just have a conversation on what do you think about the exam and what do you think about how things are going to move in the future?

Yeah, I think the more and more we train these models, for example, in radiology and we improve them, as you can see that that scale is not as quite as high as it should be. But if we took those radiologists and train these models, reinforce the model and created it, it's going to be scary good how important it is. And then just to tie it all together, say we did have those chips that you started the podcast with and say you put them there at the hospital system, man, in theory, it could be as powerful enough to literally sit on your phone and some. So that's kind of scary how strong it could be.

So let me break it down. First of all, is it ethical to produce a test that is designed to fail LLMs? Do you understand what I'm trying to say? I'm saying that for we have seen multiple things that for breast mass detection, that's what we're doing. X-ray, TB detection, brain bleed detection. I mean, our models are at 98 % accuracy. And I keep saying that. Everyone asks me, well, you can detect a bleed in the brain. No, I mean, I have six cold strokes going at the same time. There's a human cognitive capacity. I'm actually under the weather right now. No matter what I do, how much Tylenol I take, there's a human lapse that exists. Third, I keep saying, people say, there's human intuition. And people say that, you know, do what you can replace doctor's eyes. And I keep telling them that my eyes only see the visible spectrum.

They do not see the pre and the post, which is literally what the AI does. It does pixel by pixel identification and then brings in the picture and then goes from there. So number one, A, is it ethical to develop these kind of tests to fool people? Which is important in my opinion. Again, we need to distinguish high from this because another thing is that what the radiologist does is so underrated. So thoughts on those two comments.

Yeah, I agree with you. It's kind of, I hate when scientists can skew a study a certain way and you can manipulate the data to make it to your goal. As far as what you're saying, it'll be quite interesting. I think this, I do want to mention this real quick. There's this thing called Martex law and the skinniest this technology that we're talking about, all of it, AI is exponential in our our culture, our organizations are logarithmic. And as a result, the gap is getting bigger and bigger. What does that mean? This technology is getting way far ahead to the point where our legislators can't keep up, the FDA can't keep up, and our organizations are just kind of almost like as a straight line. And when these gaps get bigger and bigger, what's going to happen is disruption. And we're going to start hearing stories of individuals taking these LLMs to another level. no regulation and it's just going to be crazy because again, this gap and it's not because it's our society can't keep up.

So let's go back to just one more thing. And over here, I wanted to talk about this particular study in which the algorithmic consultant. And then when do you think that this is going to be a reality where a physician algorithm specialist is going to come in and how do you visualize it? I have my own things. don't want to, you know. Yes. so yeah, what would Harvey plus Harvey AI look like?

Yeah. Well, you know what's interesting to me, you and I have to sound pompous. We're at a different level right now. We understand the space. We don't need a consultant for us. We understand these models. We know which models to use and how, and we're our own interpreter in a way. And to this, when I read this, I kept thinking, ideally, we should, our goal as a physician now should be able to be able to do both, not needing someone else ideally, you know? I know when your question was, when will this happen? That's a tough one. I really think there's already certain hospitals like NYU, not NYU specifically, but NYU Tron and certain big hospital systems already have their own LLM that they're training. And to me, there's this future physicians being trained there and they already have the model-ish and they know how to use it, the good, the bad, the ugly. And they are those physicians that are one, meaning they're the interpreter, they know the algorithm. They know the medicine and they don't need that quote unquote interpreter. For the rest of the world, there's that bell curve. Junaid and I are early adopters. Like we already showed the apple and we wanna use these meta glasses and we're like pushing, pushing. But for the rest of the world is not in that bell curve. I think that this article is perfect for them.

So the way in the future I look at it is, first of all, interesting insight. And that's very important that I keep saying I learn a lot doing this and the way to learn things is doing it actively and then not be shy because because you go I can be wrong. that's why, you know, Harvey's saying, oh, you've seen him as an add to. So the way I look at it is that. Like you did it actually, you took all your books and Ted talks, etc., and put it into essentially a knowledge base, vectorized it. And then half of the time you're also asking because you're so productive. You don't even remember how many books you've written, five books or something. So the idea is that you actually have a trained Harvey AI with your bedside that actually, you know, with your talks and everything. think that that sort of vision will be that I will have a neurovascular frontier model for, let's say, my stroke work and neuro epileptologist frontier model. And then on top of that, it will have all my preferences.

It will have all of my things built into that. Like, for example, how Junaid likes it, what medication he likes, what is a group, et cetera, it is. And then while somebody calls me, calls me for a consult, my eyes going to go in, dig deep into the chart and everything, pre-plan my visit, also add some of these non-starters, for example, did you get a routine EEG? Did you get an MRI brain with and without contrast in the last three months? Because all this data is needed for me to actually come and to be a productive consultant for you. So doing all of these, know, investigations beforehand, and then I come in and then talk to the patient and spend actual time with the patient rather than reviewing the stupid chart and making summaries and dictating all that crap and then understand their all the situation, including the diagnosis, the detection, the social situation, how the medication because

I can say, you know what? You should have this medication. Great. Amazing medication, but they cannot afford it. Who cares? Right. So the way I vision this in the future is that that that Janay Harvey and Harvey and Harvey is going to come the venue consult Harvey, the AI goes, do the pre-work for you, and then you set, make sure that it is a companion to you, and then does the post work like documentation and everything. And this is gonna be interesting. I really think that is the future. I think that's where, really, what we want people to understand is that it's here to augment, not replace. It's here to actually excite.