Can the popular artificial intelligence app correctly diagnose a patient just as well as a trained medical professional? Dr. Joshua Tamayo-Sarver put ChatGPT to the test and the results might surprise you! Hear why Inflect's Vice President of Innovation — and a long-time Vituity physician partner — is excited about the use of AI in healthcare, whether there should be regulations, and why we can't think of technology in healthcare the same way as a consumer.
Make sure you like and subscribe to "For Startups, By Physicians" wherever you get your podcasts. And keep up with us on LinkedIn, Twitter, and Medium at @InflectHealth, and on the web at InflectHealth.com.
Can the popular artificial intelligence app correctly diagnose a patient just as well as a trained medical professional? Dr. Joshua Tamayo-Sarver put ChatGPT to the test and the results might surprise you! Hear why Inflect's Vice President of Innovation — and a long-time Vituity physician partner — is excited about the use of AI in healthcare, whether there should be regulations, and why we can't think of technology in healthcare the same way as a consumer.
Make sure you like and subscribe to "For Startups, By Physicians" wherever you get your podcasts. And keep up with us on LinkedIn, Twitter, and Medium at @InflectHealth, and on the web at InflectHealth.com.
[00:00:00] Lindsay Kriger: Hey everyone. This is Lindsay Kriger, director at Inflect Health, the innovation hub of Vituity, where we strive to be a catalyst for better care. I'm thrilled to be hosting For Startups, By Physicians where we share insights and guidance to healthcare startups and technologists looking to create the future of health.
[00:00:18] Lindsay Kriger: As a physician-founded firm, we have connections with clinicians and intimate knowledge of what they need and how. We will be interviewing our executives, frontline providers, and industry leaders to help your business be effective and scale. Thanks for joining and let's get going.
[00:00:41] Lindsay Kriger: Today's guest, Dr. Joshua Tamayo-Sarver, is the Vice President of Innovation at Inflect Health and has been a Vituity physician partner for over 15 years where he oversees the discovery. Development and integration of technology in the healthcare space. Josh is a Harvard trained [00:01:00] physician and a Ph.D. in epidemiology and biostatistics from Case Western University.
[00:01:06] Lindsay Kriger: Josh has the ability to translate very complex topics into conversations that resonate with the everyday person, and I look forward to our conversation around physicians ChatGPT and where AI can take. Hi, Josh. Welcome to the podcast. How are you today?
[00:01:25] Dr. Joshua Tamayo-Sarver: I am doing remarkably well.
[00:01:27] Lindsay Kriger: Let's dive right in. As a physician, do you have sentiments or reservations about this type of technology entering the healthcare space?
[00:01:38] Dr. Joshua Tamayo-Sarver: Yeah, so if we think of AI and is it cool to have AI in healthcare? Yeah, I'm really excited about having it in healthcare. If we think more specifically about ChatGPT in healthcare, I think it has some great uses. I think it has a lot of misuses, maybe is the right way to say it. And a lot of it comes down to [00:02:00] understanding what it does well and what it doesn't do well and how that aligns with what we're doing when we're practicing medicine.
[00:02:06] Lindsay Kriger: And so let's dive in a little bit. When you say, when we're practicing medicine, take us maybe through, not a whole day in the life in the emergency department, but a five-minute snapshot in the emergency department when you're entering a patient room. What you currently do today and how that could transform with some of these new technologies?
[00:02:24] Dr. Joshua Tamayo-Sarver: If I start by saying, how is healthcare different from consumer and consumer application of these technologies, I think it's easier to think of the consumer application first. So let's assume someone comes in with their teenage daughter and what they really want as a Ferrari, and you are a Ferrari dealer.
[00:02:43] Dr. Joshua Tamayo-Sarver: You probably say, yes, I want to sell you a Ferrari. In fact, I'm going to show you our bestest, most expensive, exciting Ferrari for your daughter, because that's just the right one and you're gonna sell it to them. And that's the way a consumer model works — someone comes in and what they want is we assume that is what they need [00:03:00] and that is very different than healthcare.
[00:03:02] Dr. Joshua Tamayo-Sarver: Let's assume that same family now comes into my emergency department and they wanna get a CT scan of the abdomen and pelvis for their daughter. I don't say, "that's wonderful, I'm going to give you the bestest, most expensive CT scan I can." I say, "let's talk about what symptoms are going on. What are you concerned about? What do we think the diagnosis is? Is there another way for me to make that diagnosis and address your concerns without the radiation and subsequent cancer risk of doing a CT scan on your young daughter?"
[00:03:28] Dr. Joshua Tamayo-Sarver: And so in healthcare, most of the, what we're doing is actually figuring out what does this patient — and you can think about when you've gone to the doctor or when the loved one goes to the doctor, there's blood work and there's testing, and there's an exam and there are questions and there's lots and lots of stuff. And at the end you get a pill, right? The consumer model would be you show up and say, "I want a pill," because you already know that and you get it.
[00:03:49] Dr. Joshua Tamayo-Sarver: And that's very different than what healthcare really is. So that was my long way of answering.
[00:03:56] Dr. Joshua Tamayo-Sarver: How does ChatGPT work? [00:04:00] ChatGPT is a large language model, artificial intelligence. So that means it scours the internet for what knowledge is out there. And you can think of it as an incredibly sophisticated Google search tool, right?
[00:04:11] Dr. Joshua Tamayo-Sarver: It searches Google and then it provides a response from the web to say, in natural language, this is the answer. And I'm even giving you the answer the way you want to hear it. And I know that's how you want to hear it, because that's how people write on the web when they get lots of hits. But in order to do that, it has to really understand the question, right? The question you ask, it has to be very well specified because it has no idea what it doesn't know.
[00:04:33] Lindsay Kriger: So I know you did a little experiment recently by asking ChatGPT some medical questions. Did it give you some answers that you were pleased with?
[00:04:44] Dr. Joshua Tamayo-Sarver: Yeah, so I had a string of nights, and so I took some of the HPIs in a very...
[00:04:50] Lindsay Kriger: Josh, what's an HPI?
[00:04:53] Dr. Joshua Tamayo-Sarver: The history of present illness. So my description of what the patient's coming in for and what their symptoms are and that kind of stuff.
[00:04:58] Lindsay Kriger: Your description, not the [00:05:00] patient's description.
[00:05:01] Dr. Joshua Tamayo-Sarver: Right. Someone comes in and why are you here? And half the time the answer is you tell me you're the doctor, right? So then, my job is to figure out why they're there and how I can help.
[00:05:07] Dr. Joshua Tamayo-Sarver: Okay, so I put in, what the patient was there for and if it was very clear and my description was perfect and included the entire universe of information so that there was a definitive answer, then ChatGPT did an amazing job of pulling back a definitive answer for it and gave me the exact diagnosis, and it was really good at it.
[00:05:24] Dr. Joshua Tamayo-Sarver: Most of the time those things aren't there. So for example, I had a young woman who came in with right lower quadrant pain, so pain in the right lower part of her abdomen, and I put in what I got out of the patient and ChatGPT said, "oh, this could be appendicitis, or it could be ovarian torsion, right where you're ovary twists. Or it could be a cyst on the ovary or it could be diverticulitis." All very reasonable. It did not include in there what the patient did have, which is an ectopic pregnancy because she didn't know she was pregnant, but it didn't know to ask what it didn't know.
[00:05:55] Dr. Joshua Tamayo-Sarver: Now for me, I've had that happen plenty of times. So I ordered a pregnancy test and we all [00:06:00] discovered something that day.
[00:06:01] Lindsay Kriger: So ChatGPT and Google, for that matter really, — and so many of us talk about Dr. Google and going online and making sure that we think we don't have cancer when we ask Google a question and it tells us we do, then what? But you're saying both of those models are really trying to be consumer friendly, but the issue with healthcare really is that healthcare is not really a consumer business.
[00:06:22] Dr. Joshua Tamayo-Sarver: I would probably put a distinction on there that I think healthcare is a consumer business to the extent that people choose to go where they want to go and are consuming resources.
[00:06:31] Dr. Joshua Tamayo-Sarver: When you wanna get a loan, you want to get a loan, the bank doesn't say, "you don't need a loan, you're much better off saving. You shouldn't buy that today." They say, "here's your loan." Whereas in healthcare, when someone comes in with a health concern, most of what we do as doctors and most of the art of medicine, and for that matter, even the science of medicine, is trying to figure out, what is your story? What is that medical narrative that gets all the salient features? And once that's done, the knowledge retrieval part of it, whether that's a Google [00:07:00] search or a ChatGPT large language model. Now there's a definitive thing, and I think that's actually what ChatGPT — and there's a variation of it called DocsGPT, which I've worked with as well. It does a really good job once there's a definitive history and everything else for it to get right. Once there's one answer out there, it can do it. It can't problem solve. Knowledge retrieval is amazing.
[00:07:22] Lindsay Kriger: That's all good for the present state. We at Inflect believe in technology as part of the future of medicine. Not that it's gonna replace the doctor, but that it is part of the future.
[00:07:33] Lindsay Kriger: Tell me a little bit about what you hope in 10 years this technology can do to further the place that medicine is today, both from the consumer perspective and the clinician perspective.
[00:07:45] Dr. Joshua Tamayo-Sarver: The large generative language model like ChatGPT, that's not the only way to do AI that's out there.
[00:07:51] Dr. Joshua Tamayo-Sarver: There's another more cumbersome, more challenging way, which is based on a knowledge graph. You have these knowledge concepts that are out there, and you can [00:08:00] have one in medicine. You can have a pulmonary embolism, which is a blood clot in the lungs, and then you can have another concept like pneumonia, right? And then every utterance that a patient has maps with a certain weight to those things.
[00:08:12] Dr. Joshua Tamayo-Sarver: So the patient says, "oh, I have a cough." That's more pneumonia and it gets a weight and you can put a numerical score and a little bit to pulmonary embolism, but less. And then they say, "I have a fever," and that goes more to pneumonia. And then they say, "oh, and I'm on estrogens and smoking was just got off a plane" and that perhaps more to pulmonary embolism.
[00:08:27] Dr. Joshua Tamayo-Sarver: So you can imagine in a knowledge graph setting, it knows what it doesn't know because the next step that it does is it says, what question can I ask that will do the best job of differentiating between all these knowledge concepts to get me to the diagnosis?
[00:08:39] Dr. Joshua Tamayo-Sarver: What it doesn't do well, it doesn't generate that text and that speech so nicely, like a generative model does, and it actually doesn't do as well with knowledge retrieval, oddly enough, right? Because in order to get all these concepts and this knowledge graph in there, it requires an incredible line of content that you're pushing in and that it's scouring in order to categorize it, versus a ChatGPT or a [00:09:00] language model where it's like a really good Google search.
[00:09:03] Dr. Joshua Tamayo-Sarver: So you can imagine a future where those two things are paired together, where you have this knowledge graph that helps figure out what that patient was and is or needs, and then drill that down until it's a super-specified knowledge retrieval search that it can then do with a large language model, which is a cool pairing of it when it gets to that.
[00:09:23] Lindsay Kriger: Are those two different models built because engineers and statisticians believe fundamentally in those models as superior, or have they just not converged yet in a way that's meaningful?
[00:09:37] Dr. Joshua Tamayo-Sarver: So there're two very fundamentally different ways of training the AI, and they are solving different problems.
[00:09:43] Dr. Joshua Tamayo-Sarver: One is a problem-solving AI. The knowledge graph is a problem-solving, which is mostly what we're doing as a doctor. And then the second part is knowledge retrieval, which is, now that I know exactly what this problem is, there's probably someone who has figured out when you wanna turn left, how do you turn left in a car? Someone [00:10:00] has figured that out. And ChatGPT is great. The first one would tell you what, which direction you want to turn. How do I figure out which direction I wanna turn? That's a different question.
[00:10:08] Lindsay Kriger: Do you have optimism or faith that in 10 years those models can converge to be meaningful for both the consumer and the clinician and the expert in that field?
[00:10:22] Dr. Joshua Tamayo-Sarver: Absolutely. One of our portfolio companies is a knowledge graph based. And a lot of the things that they were struggling with in order to be able to actually implement this have been largely fixed by ChatGPT. The finding was great, but I'm sure other companies are gonna see that and figure it out too.
[00:10:38] Lindsay Kriger: Yeah, that's great. The market will help solve these problems as more progress is made on an individual level to converge, so that makes sense. What do you think clinicians are most afraid of in this world today as they start to hear the buzz?
[00:10:55] Lindsay Kriger: I know you're not afraid of the buzz, but let's take someone who doesn't serve in a [00:11:00] innovation role, a frontline clinician practicing in rural America. What are they most afraid about or most concerned about for their longevity of their profession?
[00:11:10] Dr. Joshua Tamayo-Sarver: So I think on one hand there's a concern that the language models like ChatGPT, because it does such a great job of knowledge retrieval is actually a smart system. That's not, it's an amazing knowledge retrieval. But it can't problem solve.
[00:11:24] Dr. Joshua Tamayo-Sarver: Now, we just talked about that there is actually another approach to AI that works for problem-solving, and it's out there and it's real and it works. So yeah, we should be concerned that's gonna happen. But I think it's concerned with a small C. There was a really interesting thing that was done with ChatGPT, and I'm not going to get into the ethics of it because that's a whole different discussion, but there was an online mental health company called Koko, and what they did is for 4,000 of their, it's like a text-based mental health thing. And for 4,000 of their patients, ChatGPT was their therapist. And one [00:12:00] of the things they found was that the patients who had ChatGPT liked their therapist a lot more than the...
[00:12:05] Lindsay Kriger: Okay, now I'm gonna go there. You said, I'm not gonna go there about the ethics, but let's go there. What is the most concerning?
[00:12:12] Dr. Joshua Tamayo-Sarver: If you think about how ChatGPT has learned from its large language model, it did it by scouring everything on the internet so it knows exactly how you want to be talked to , right? So it talks to you much better than I ever have. We don't need you to affirm that, but really knows what someone wants to hear and it knows exactly how to say it the way they want to hear it.
[00:12:30] Dr. Joshua Tamayo-Sarver: Think there's a great potential for ChatGPT to help us communicate with patients in a way that actually resonates with patients a lot better. And then we still need to modify it and make sure it's what we want to say. Now, the part about that was interesting is once patients found out because they disclosed that they'd been talking to a computer instead of a human, they were super pissed off.
[00:12:49] Lindsay Kriger: Great.
[00:12:50] Dr. Joshua Tamayo-Sarver: Now, the ethics part was, I don't think there was any real meaningful notice that they were actually gonna be signing up for robot instead of a human. And, that's the ethics part of the way that was not really [00:13:00] like an academic IRB studying, but it's still, I think it all that aside from a, just a knowledge and learning perspective, that's pretty amazing to think.
[00:13:09] Dr. Joshua Tamayo-Sarver: So if I'm a doctor and I'm overloaded by emails, which we are, and I'm overloaded by text messages from patients, which we are, I think the ability for me to specify exactly what that patient needs. So I am still doing that problem-solving function and saying, "Tell Lindsay about the importance of follow-up for her blood pressure rechecks in two weeks."
[00:13:29] Lindsay Kriger: But in a nice way, so she doesn't think I'm a terrible doctor.
[00:13:32] Dr. Joshua Tamayo-Sarver: Yes, but in a nice empathic way. If I actually cared, I would say it and then it would do like a nice way and I could look at it and it's yeah, that's what I wanted to say, and now I hit send. So I think in terms of those things where I'm specifying that task, really finally, I think it can work super well. Discharge instructions, follow-up, reaching out, touching base, all those oddly human things.
[00:13:54] Lindsay Kriger: There is so much pressure that has led to so much burnout, the [00:14:00] expectations that are put on the very learned academic clinical people to also have the empathetic skillset. Part of the technology hope could be whether it's from the knowledge retrieval or the communication opportunity that one of these things has to give, that our clinicians cannot be expected forever and ever to uphold this level of perfection on both sides.
[00:14:25] Dr. Joshua Tamayo-Sarver: Absolutely. I agree. And you can imagine a day and 10 years, 15 years, whenever, when you actually talked to my assistant. First, who's going to actually very efficiently figure out what you have from a knowledge graph mechanism. Then connect into this knowledge retrieval system to say what should happen next and what's the ideal course of therapy. But then that human connection still needs a human, right? So you come in with your twisted ankle and you come in and it figures out really quickly that you have a sprained ankle. You still need someone to sit there and say, "Hey, you know what? This really sucks."
[00:14:58] Lindsay Kriger: Yeah.
[00:14:59] Dr. Joshua Tamayo-Sarver: And say, I've been [00:15:00] there. I sprained my ankle, let me tell you about it. And I don't think. part of our job is going to be replaced with synthetic empathy.
[00:15:07] Lindsay Kriger: Yeah, I agree. One thing that you touched on with the Koko example is this concept of not only ethics, but then how do we translate ethics into regulation? How do we make sure that the health industry is progressing? Because we know that the health industry has so many regulations that have prevented a lot of technology from disrupting it.
[00:15:29] Lindsay Kriger: So how do we find that balance between the incorporation of important new-age technology with securing the regulations that need to be in place to protect patients and providers?
[00:15:40] Dr. Joshua Tamayo-Sarver: Yeah and I happen to have a reasonably strong opinion about this because I'm really sick of clinical decision support systems.
[00:15:47] Lindsay Kriger: Okay.
[00:15:47] Dr. Joshua Tamayo-Sarver: My job is to make a decision, and as physicians, we take an oath to say that we hold your life in our hands, and we take those decisions very seriously because it, it is that responsibility.
[00:15:58] Dr. Joshua Tamayo-Sarver: And then you have some [00:16:00] technologist who doesn't really want to have any responsibility, but they're happy to get some money for it. And they make a decision support system that tries to guide me in decisions, but not in a way that there's any accountability. That's just noise. That's just burnout.
[00:16:10] Lindsay Kriger: Yeah.
[00:16:10] Dr. Joshua Tamayo-Sarver: That's just more crap for me to wade through to make some administrator somewhere feel like they purchased something to make my life better that didn't. And so that just pisses me off.
[00:16:18] Lindsay Kriger: Yep.
[00:16:18] Dr. Joshua Tamayo-Sarver: With one of our companies, they had a decision support system, but we said, Hey if I don't rely on it, I don't want it. So we took it through the FDA to get approval. And I think if you're not willing to have people's lives, hang on the decisions of your clinical decision support system and an industry that has people's lives hanging on it, then don't do it. If you can't hold it to that standard, then don't make me, as a physician, have to wade through crap.
[00:16:40] Lindsay Kriger: And I think that is a perfect segue to the question I always wanna end with, which is the advice that you would give to technologists that are building tech for health and for clinicians and for physicians because there's not that many people that have walked a day in the life of a physician who's not in that [00:17:00] specialty, but vice versa, there's not a lot of physicians that understand the nuance and the pressures of early-stage tech and development and that sort of thing.
[00:17:09] Lindsay Kriger: So how would you give your best advice from a physician standpoint to entrepreneurs and technologists who do want to make a difference in this space, but need to go about it in the right way?
[00:17:22] Dr. Joshua Tamayo-Sarver: I would say the biggest struggle I see that people have physicians or even technologists is understanding what we're actually doing and then aligning the technology to do that.
[00:17:34] Dr. Joshua Tamayo-Sarver: And once you do that, you recognize there are so many technologies that just from first principles can't work. A good example is a decision tree to figure out what the patient has that a patient's going to use. They can't go down a decision tree. When you have that life as a physician, you go, why are you here? And I'm like, oh, I have a fever and a cough, and when did the cough start? I have leg pain. That's the answer that is not going to be on the decision tree. And so we've seen a million different companies come [00:18:00] up with these decision trees to help figure out what the patient has and categorize. They never work very well because the patient doesn't already know what they need.
[00:18:08] Dr. Joshua Tamayo-Sarver: And when you realize that is actually the business of medicine is mostly figuring out what the patient needs, I think that really helps you target your technology for what we're doing.
[00:18:16] Lindsay Kriger: Yeah. I invite anyone listening that wants to walk alongside Josh in a shift to reach out to us because really and truthfully, it's one of our mission and part of what we deeply believe at Inflect, that you need to be alongside the physicians and alongside the real world in order to impact health in a meaningful way. I'll offer you up, Josh, as a mentor to any technologist that want to come see what it's like in the emergency department.
[00:18:43] Lindsay Kriger: Thank you so much for your time and your insights today. I really enjoyed having you on and look forward to more discussions as more buzzworthy technology continues.
[00:18:53] Dr. Joshua Tamayo-Sarver: Thank you. It was a pleasure.
[00:18:55] Lindsay Kriger: Thanks for joining us, and again, I'm Lindsay Kriger, director at Inflect Health. [00:19:00] Here at Inflect, the future of medical care and health delivery is not just right for disruption. It's increasingly personalized, accessible, and human.
[00:19:09] Lindsay Kriger: Make sure you like and subscribe to For Startups, By Physicians wherever you get your podcasts.
[00:19:15] Lindsay Kriger: And keep up with us on LinkedIn, Twitter, and Medium at Inflect Health, and on the web at inflecthealth.com.