QI Projects and Data Stewardship: Why understanding data is critical for designing a QI project

Name: QI Projects and Data Stewardship: Why understanding data is critical for designing a QI project
Uploaded: 2021-12-15T17:19:13.643Z
Duration: 1 h 4 min 30 s

December 15, 2021

Information

ID: 7292
To Cite: DCA Citation Guide

Download Transcript

00:00And so then, when Roberta.
00:04So I'll give Roberta,
00:05do you go ahead?
00:06She'll start recording.
00:08I'll give you a brief
00:09introduction and you get started.
00:11Let's see just
00:12so that I don't forget and then
00:13we can just edit it later.
00:26I need to will decide outside with
00:29Linda Phan where we'll post this.
00:31Most likely it's going to be someplace.
00:36Related to GME and we'll,
00:39we'll put it in our Internet,
00:42and they want anybody in our Internet.
00:44Anyone that has it?
00:46A net ID can look at.
00:53Do you have any
00:53moving parts to your presentation
00:55that you wanted to try out now?
00:57Are there any videos or animations?
01:01Well, the first thing wasn't like
01:03I have to navigate through here.
01:08Well, looks like looks good.
01:09Yeah it does. Stuff in here.
01:16You wanna put on your title.
01:17There we go, so Roberta, why don't
01:20you open it up for everybody I mute?
02:00Although I'm just going to grab a pen
02:02and realize I have nothing to write with.
03:59Recording.
04:23Although quick check when I bring up
04:25the chat, does it hide my slides?
04:32No perfect.
05:05Well hello everybody,
05:07I'm all the pressure to thank
05:09you for for joining us today.
05:11We have a great session for you.
05:15I'd like to remind you to
05:18just mute your microphones,
05:20and this session will be recorded and
05:23will be available to you later through
05:27the Department of Medicine Intranet.
05:30So today we have a.
05:33It's a great pleasure for me
05:34to introduce Dr Nita Kashyap,
05:36who's the associate chief Medical
05:38Information Officer for the health system.
05:41I've worked with Nita for many
05:43years and I'm pretty sure that.
05:45If you've had to do anything related to
05:48electronic data in the health system
05:51or with clinical decision support,
05:53you have worked with her and I'm
05:55certain you've had a great experience.
05:57She's been a great partner and what
05:59she will be doing today is discussing
06:03the importance of understanding data
06:05and where data comes from not only for
06:10quality improvement projects, but but,
06:13and this is where I come here in.
06:15In terms of my quality and safety umbrella,
06:18but this really has an impact and
06:21impact on how data are handled
06:23in the clinical realm for those
06:26involved in program development
06:28and even in clinical research.
06:30So I'm this will be a great session.
06:33I look forward to your participation.
06:35I think need two for joining us and
06:38look forward to a great discussion.
06:42Thank you, although you make it sound good,
06:45but I hope people will find it useful
06:48and after the next several minutes
06:50that we'll be spending together.
06:53So this is, uh, I guess for you,
06:55if you've ever thought about
06:57doing a process improvement or
06:59quality improvement project,
07:00and hopefully you'll get a sense of
07:02what tools you actually already have
07:05at your fingertips that you may or
07:07may not have used or take away some
07:10good information as we go along.
07:13Our goals for today or three fold up
07:18hopefully to give you an idea of some
07:20key concepts around data quality.
07:21Interpretation,
07:21sort of lose some awareness on data
07:25and reporting resources that we have
07:27and then we'll go through a very quick
07:30overview of how data and knowledge of
07:32this background information is helpful.
07:34As you design QA processes specifically
07:37around designing smart aims and
07:41success measures for your projects.
07:44Uhm, it's going to be a busy agenda.
07:46Hopefully you can all keep pace with me.
07:48And if you find I'm talking too fast,
07:50please do use the reactions in the
07:52zoom to let me know to slow down.
07:54Also,
07:55I would love to see you all and
07:58left to see reactions on zoom
08:00and chat as we go along.
08:02And I know although keep us honest and
08:05on track on time here so we'll touch on
08:09just some definitions to get us started.
08:13Talk about what data and data types we have.
08:16I'll touch on data resources and then
08:19do some fun exercise on designing goals
08:22and share with you some examples.
08:25Now starting the definitions,
08:27nothing ever starts if you don't
08:29define it well,
08:30you've all probably seen these
08:33definitions of quality improvement
08:35shared through multiple channels.
08:37Institute of Medicine defines it,
08:39and so does the Institute of Healthcare
08:42Improvement and and we've used tools
08:45and tactics from both organizations.
08:48In quality improvement processes,
08:50both relate to services and also to outcomes,
08:56so keep that in the back of your mind
08:58as you go through the rest of the
09:01information here as to what to what
09:03are you hearing about services and
09:06outcomes and where data comes in handy.
09:08When you think about data stewardship,
09:11the two concepts seem very desperate and
09:13that's why they're in two separate boxes.
09:16Quality improvement and data.
09:17I'll start by using a definition
09:20from Merriam-webster on what
09:22stewardship actually means,
09:24and it is the careful and
09:27responsible management of
09:29something interested in ones care.
09:31And this is a theme that you will
09:34see resonate through the rest of
09:37the conversation. But also U.S.
09:39National Library of Medicine at
09:41the NIH goes in a little bit deeper
09:44than what data stewardship means.
09:45I think it's a very sort of
09:48lofty definition with big words,
09:50but it doesn't give me that same ethos
09:52as the Merriam-webster definition.
09:54This, but it does go around the acquisition,
09:57storage, aggregation,
09:59Deidentification release and use of data.
10:02So sure, that's data stewardship,
10:04and hopefully by the end of
10:06this conversation we'd be
10:07able to connect these two.
10:09Concepts in a meaningful way.
10:14So let's start by understanding
10:16we're told to be data driven.
10:19What exactly does it mean? Uhm, it?
10:24Have you heard this phrase before?
10:28Any reactions, not sure?
10:30And what does it mean to you when when you
10:33hear you know based being data driven,
10:36you thinking of an image of a dashboard
10:39that you're going to or you're thinking
10:41of how you're capturing some data,
10:42or why I'm capturing this data when I'm
10:46being asked to for one more field to fill
10:49in all of those questions are valid.
10:52It starts by understanding what you
10:55want to actually do with the data.
10:59That you're being asked to talk about and.
11:02Data is only as good as the questions
11:06you ask. Data does not think it
11:08is not a thinking Organism.
11:10We are, so that's kind of the the big
11:13thing is we have to ask the question
11:15and smarter we are in asking questions.
11:17The better chance we have to
11:19glean more information from it.
11:21To me, that means being data driven
11:24is to understand that I get to ask the
11:26question and it is my responsibility to
11:29understand the circumstances around it.
11:33So if being data driven means
11:35asking the right questions,
11:37let's try to formulate the right
11:39question in the clinical context.
11:41Well,
11:41you're going to need a few basic
11:43things to formulate a question.
11:45You of course need the clinical
11:47question and the workflow that
11:49you're trying to answer some basic
11:52knowledge of Boolean operators,
11:54specifically three of them and or not,
11:57and we'll kind of run through some
11:59of those in a few minutes and.
12:01Some deep dives into what clinical
12:03data is and what types of data are
12:07important in the clinical world.
12:09You can probably think of a few
12:12different things already and I'll
12:14touch on some layers as we go along,
12:17and then other important thing is
12:19our own organizational structure,
12:21and that's helpful when you're
12:23asking the question of your data is
12:26what are you telling me and that's
12:28that should become clearer as we
12:30talk through these things.
12:32OK,
12:32starting with clinical question and
12:34I know Lydia couldn't join us today,
12:37but this is something that we worked
12:39on many years ago for Nathan Smith.
12:41Patients with HIV.
12:42So there's a need for those HIV positive
12:45patients to be seen at least twice a year,
12:49which means we could look to see who has
12:52had at least two visits in a 12 month period.
12:55But for a Qi project,
12:57if we're going to look retrospectively,
12:58we've missed the boat on seeing
13:00them twice a year, right?
13:01So what she wanted to know was?
13:04Show me all the HIV patients positive
13:07patients in this clinic who have not
13:09been seen in the past six months notice.
13:12We turned the thing a little upside down,
13:14but this way we're able to get them
13:17to be seen before that 12 month Mark.
13:20Tires so Qi projects often.
13:22We need to look at a prospective look
13:24on data and which means reformulating
13:26our clinical question so that we
13:28can have a data question that allows
13:30us to do
13:31quality improvements.
13:32Not a retrospective look. OK.
13:36Here's an example of a very simple.
13:40Definition of uncontrolled hypertension.
13:41This one comes from JNC 8 you'll
13:45see three criteria based on age
13:47and patient's condition and any
13:49one of those criteria being met
13:52defines uncontrolled hypertension.
13:53So in this case whoever is going to
13:57write the question is going to slay.
13:59I want a list of patients with
14:02uncontrolled hypertension who meet either
14:05criteria one or two or criteria 3.
14:08Keep this example in mind.
14:10We'll come back to this a
14:11couple times in the future.
14:15OK, here's one that, UM,
14:20I want this one to stay with you.
14:22Understanding your data is like
14:25understanding your children and.
14:28And why do you think I say that is
14:31because I really don't know what
14:34my children are trying to tell
14:36me unless I ask the questions and
14:38I have to be careful asking the
14:41questions because it's not always
14:43clear their reasons behind it.
14:45And then I have to keep a close
14:47eye on things because disasters
14:48can happen otherwise.
14:49So if you don't keep an eye on your data,
14:52you may find something has changed in
14:53the matrix and you didn't realize it.
14:58Now, just like our children,
15:00data is telling us a story we have
15:02to glean the information like in the
15:05image that I had on the previous slide.
15:07Something was happening in that household,
15:10but there's always a story and
15:12you're not going to get the answers
15:14unless you start asking the questions
15:17and piercing together the puzzle.
15:19So data will tell you a story,
15:22but we have to glean the information from it.
15:25And if you've heard any of these
15:27Qi lecture series or Beth Emerson
15:30makes an elegant point of it is very
15:32important to understand the workflow.
15:34Be out there,
15:35not what you think is happening,
15:37but observing it going out to the gamba.
15:39If you've heard that language
15:42in Qi lectures before,
15:44so it is really key again,
15:46even if you're trying to understand.
15:50How your data is captured,
15:51you're going to want to know
15:54that piece of information.
15:55OK, this is going to be
15:57a very meaty discussion,
15:59so I'll leave this for your perusal here.
16:02But I'll be talking a lot over
16:05these few topics so bear with me
16:08when I asked earlier clinical data.
16:12Hopefully these were some of the
16:14things that came to your mind
16:16as to what clinical data is.
16:17An all touch on five data points.
16:21There are of course several others,
16:22and observations that you
16:24normally would want.
16:25The reasons I picked these five are
16:28these are the ones we often get
16:31started with and need a lot more
16:34definitions when we have to ask
16:37for data starting with diagnosis.
16:39So everyone knows we use ICD 10.
16:42As our nomenclature for writing for ICD,
16:45I mean for diagnosis.
16:47Are you familiar with Snow Medcity?
16:52Any reactions, folks?
16:54Ah, OK, what about vysakh or?
17:00Groupers?
17:03OK, a few nods, so ICD 10 straightforward
17:10vocabulary of a collection of
17:12diagnosis arranged in some hierarchy.
17:15Snow, Med, CT, or systematic
17:18nomenclature of medicine again.
17:22Another hierarchy developed
17:24by the pathologist.
17:26It looks at disease complications,
17:30etiology, symptoms as.
17:33As starting points and then multiple of
17:38those can sort of merge into tree forms.
17:41So what it does and in our system
17:43all of those are mapped to ICD 10s.
17:45What that allows us to do is instead
17:48of looking at 40,000 ICD 10 codes,
17:50I can look at maybe 10 snow Med concepts
17:53and say these are the right ones.
17:56So for example I'm looking for
18:00hypertension rather than looking
18:02at the 110,000 ICD 10 codes.
18:05Define hypertension and its complications.
18:07I could look at the tree and say I
18:10want the situational hypertension
18:12concept taking out of my definition,
18:16but I want to keep the label
18:19hypertension concept so all of those
18:22ICD 10's related to it are captured.
18:25Uhm V SAC or values that authority center.
18:28If anyone of you are working with CMS then
18:31quality measures that are reportable.
18:33Seems requires the use of
18:37these visak groupers.
18:39For their quality measures,
18:41because they're trying to normalize
18:43data across multiple organizations,
18:45so the definitions need to be very,
18:47very specific and they want us to use
18:49codes that are in these vysakh groupers
18:52or values that authority center and
18:54groupers I guess is the general bucket
18:57of it's a box you can throw in ICD
18:5910 in there or snow Med or V SAC.
19:01So you'll have ICD 10 groupers,
19:04somet concept troopers,
19:06or VSAT groupers.
19:07How are they helpful?
19:09If I'm saying I want diabetics over
19:12the age of 65 were on metformin,
19:15I'm going to tell the analysts to say,
19:17do you have a grouper that defines diabetes
19:20and they're going to come back and say,
19:22well, we have this concept.
19:24Grouper that's has diabetes.
19:26You OK using it and your answer
19:28should be let's start there.
19:30Sure,
19:30because it gets you started off
19:33the ground much faster than trying
19:35to collate a ton of code yourself.
19:40The.
19:41Everyone with me so far,
19:43no one's fallen asleep.
19:46Next one over labs,
19:48another common thing that we look
19:51for in in data queries for clinical
19:54purposes order names fairly obvious.
19:58One thing I would suggest is if you're
20:01going to give the analyst and name
20:03or for the order you're interested
20:05in trying to go to the database,
20:07look up in Epic and look how
20:09many versions of orders we have.
20:11There are some tests that have
20:13upwards of 40 different values.
20:15Things that we can order.
20:17You may not always eat in
20:19the inpatient context,
20:20it may become obvious when you're
20:22in the outpatient context and
20:24looking in database look up results.
20:26Why do I call orders and results differently?
20:29Because some orders are ordered panels.
20:32So CBC for instance,
20:33if I'm interested in just in the hemoglobin,
20:36I don't have to pull all the CBCS.
20:37I can just ask for the result
20:39name that says hemoglobin.
20:41The easiest way to look for those
20:43is open or chart. Look at a result.
20:46See what it's called and in there
20:48somewhere in that big screen is
20:51also additional information that
20:53tells you what that means.
20:55That test result is called by the
20:58lab and also if it has a base name.
21:03So in addition to result name,
21:05you may come across two things,
21:07a common name or a base name.
21:10A common name is something that the lab
21:13might use for their internal purposes.
21:15A base name is something that
21:18someone has gone through.
21:20The effort in the database to try and
21:23bring desperate sounding things together.
21:26For example, glycated hemoglobin and
21:29hemoglobin A1C could have the same base
21:32name of A1C when you're riding your reports.
21:35If someone says I have a base name
21:38grouper for this, you're going to say,
21:39yeah, go ahead, use that.
21:41Let's start there,
21:42so please her design to collate
21:44things so you don't have to
21:45go individual cherry picking.
21:47For names anyone heard of Loinc codes?
21:50And that's an acronym.
21:54OK so link is logical observations.
22:00Something and. Names and codes.
22:06Forget what it's up anyhow, it was, uhm.
22:09I think it's identifiers.
22:11So logical. Observations,
22:12identifiers, names, and codes.
22:14Huge mouthful, but it was developed
22:16by the Regent Street Institute back
22:18in the 90s to be the competition
22:22for ICD 10 in the lab world.
22:25So if you think ICD 10s are granular,
22:28loading codes are so granular that
22:30it they go down to the level of the
22:33manufacturer and the instrument.
22:35So the advantage. There, however,
22:37is that they now become machine readable,
22:40so I don't need to necessarily
22:43say have people and eyes to say.
22:46This test is the same as this test
22:48if they're learning code matches,
22:50we can trend them on the same line.
22:53So why is that important?
22:55When you get labs from outside
22:57organizations through care everywhere,
22:58and we have the ability to map
23:02them were quality measures.
23:03If we're getting an A1C from
23:05an outside organization,
23:07we can say that this is the so the
23:09machine can say this is the same test and
23:12give credit in our quality measures cores.
23:15So something to remember as.
23:16Hi, is another things become
23:18more prominent in our state?
23:22OK, everyone with me so far.
23:26It's an OK pace ready to sing.
23:29OK, I'm moving on to procedures.
23:33Those are sort of self explanatory.
23:36Almost everyone knows to look
23:37for CPT codes and there's just
23:40no way other way around those.
23:42Medications are interesting,
23:43one.
23:43You can certainly look by medication name,
23:45but you also know that you can
23:47have generic and brand names,
23:48so you have to account for those,
23:50but there's another way.
23:52Each medication is classified under a
23:55pharmaceutical class or a therapeutic class,
23:58so I can have a sorbs.
24:00Or I can say anti hypertensives which
24:02is a bigger unit and those will pull
24:04in all of the medications of interest.
24:07If you're looking for something of that
24:09nature and just like we talk about.
24:12Loinc codes for labs.
24:14There is RX norm and codes for medications.
24:18RX norms are helpful again in the
24:21same way for interoperability
24:23and quality improvement work.
24:26The NDC's you probably are familiar
24:28with if you're giving medications or
24:30flu shots or anything in the office,
24:32you have to pick the light right NDC?
24:34Because that has the manufacturers
24:36name tide to it.
24:38When there is a recall,
24:39we need NDC information,
24:41not RX norm,
24:42which is why most recalls are
24:45handled by retail pharmacists and
24:47not clinicians who prescribe them.
24:51OK,
24:51this last one is important and
24:53I promise this will be very,
24:55very useful if you get stuck in
24:59what these things actually means.
25:01So if you're interested in certain visits,
25:04there is a distinction that is in the
25:09epic database on what is an encounter.
25:12A visit versus a visit type.
25:15What is a face to face visit?
25:17Do you want ancillary visits
25:19included in your encounters or not?
25:21Are you only interested in hospital or Ed?
25:24All of those things need
25:26to be classified because
25:28an encounter is as broad as
25:30any contact with the system.
25:33For those of you who
25:34practice in the outpatient,
25:35you do telephone encounters.
25:37You do orders only encounter any
25:40contact with the system at a
25:43patient level is an encounter.
25:45When you get to come,
25:48do you want hospital outpatient,
25:50ancillary ambulatory surgical centers?
25:52All of those things need to be
25:56defined to get to the best data
25:59source that you can get for
26:01the work that you're doing and.
26:06I'm not going to go through this table.
26:08This is for you to look at
26:10and reference well.
26:11It's a summary of what we just talked about,
26:13but also things that you must know
26:15and things that are good to know.
26:17So feel free to reference this
26:19whenever you feel the need to for
26:22a refresher and also happy to
26:26be a touchpoint anytime you have
26:29any questions on those things.
26:31This is my last thing about the data,
26:33deep dive and then we'll go on
26:35to more fun things.
26:37This blue box here is all of epic
26:40data and what the other boxes
26:42are trying to tell you is how
26:45organizationally we are structured.
26:46If you're not already clear with
26:49from left to right,
26:50we'll start with Yale Medicine,
26:52Yale Health Center.
26:53Then we have the Yale New Haven
26:55Health System,
26:56which includes any MG and the
26:59hospitals and the hospital campuses.
27:01Then we have community connect
27:03practices who are members of the
27:05medical staff who use EPIC in
27:07their offices and their data is
27:09part of the same database.
27:10We also have two FQHC's,
27:13Fairhaven and Cornell Scott,
27:14who also use Epic and their patients
27:17and data are also in the system
27:20when you're working on a project,
27:22since important for you to ask
27:25for the data where you have.
27:28I guess that you are authorized
27:31to view and also to understand
27:35your aggregate numbers will likely
27:38include this larger group.
27:41Of data set, unless you specify otherwise,
27:45the other important thing here is
27:47that epic is a patient centric record,
27:49so regardless of where the care was given,
27:53if a patient's data is recorded so
27:55it doesn't matter where the blood
27:57pressure was taken or the A1C was done,
28:00it is recorded at the patient level.
28:02The patient attribution happens
28:05based on other things and.
28:08This is probably not the right
28:10forum to go deeper than this,
28:12but just take this home that when
28:14you're trying to define which
28:16data points you're looking at,
28:17this is how the hierarchy works,
28:19and then you may find that.
28:22Not all the data you want is in epic.
28:25There may be outside sources,
28:27so if you're looking for endoscopy data,
28:29it's likely not an epic in a
28:30way that we can report on it.
28:32There may also be a need to do some
28:35manual abstraction for things that are
28:37that don't make into an electronic system,
28:40or if you're looking for
28:43other sort of data points on,
28:46say,
28:46qualitative work.
28:50OK, everyone with me so far.
28:54OK, we're going to go back to
28:56this question now that we've taken
28:57a deeper dive into the data,
28:59we're going to come back up
29:01and formulate that question.
29:02So this is how JNC is defining
29:06and controlled hypertension.
29:08Can I get a reaction from all
29:10of you if this is adequate?
29:12If this definition adequate for a data pull?
29:19Jenay.
29:29OK, I see a lot of maybes.
29:31OK, so here's here's another
29:35way to think about it. Uhm?
29:39Which blood pressure outpatient
29:42inpatient emergency room?
29:45Should I exclude certain locations
29:47where the blood pressure was captured?
29:51The way it's defined,
29:52does it mean that both systolic
29:54and diastolic have to meet the
29:56definition or it's either or
29:58could be above the threshold?
30:01What is how do I define diabetes,
30:03hypertension, secd, right?
30:04So for someone to be able to
30:06actually pull the data for you,
30:08they're going to need a little bit more
30:10granular definition in each of those.
30:12So the clinical questions need to be
30:16split into its constituent elements
30:18so that you can have a data query.
30:21So clinical question needs
30:22to be broken down further.
30:24So on the left hand side,
30:25it's that same clinical question
30:27what I've tried to do on the
30:29right hand side is a data query
30:32from that same clinical scenario.
30:34What you'll notice is that age
30:36is split outside the two blood
30:39pressure components are split,
30:40so major splitting of data elements
30:43happening and then the logic is looking
30:45a whole lot different than the one
30:47you saw before because now you have
30:49to account for all of these combinations.
30:52I mean,
30:52we didn't even go into defining
30:55diabetes because we're going to
30:57use what to define diabetes.
31:00No meds.
31:02Sure,
31:02you can use nomed by brokers of some kind.
31:08Right, so you're going to use
31:10whatever existing definitions are,
31:12because chances are they'll get you
31:13close enough, and then you can always
31:16improve upon someone else's work.
31:18So this should make life a lot easier when
31:21you're going into getting data queries,
31:24time, check for all of us.
31:26How are we doing on time, Roberta?
31:31I think we're good.
31:32It's 12:30 perfect, so we're about halfway
31:36and this is going to get into data
31:40sources and formulating data queries so.
31:44I'll keep us going. Here,
31:47so has anyone heard of Slicer Dicer before?
31:52A few nights I'm happy to hear that.
31:55And I see Richie in the audience.
31:57He's probably get got his hands
31:59on to a few of those.
32:00So if you're in the epic chart,
32:03search the magnifying glass on the top right.
32:05You type in chart search and in that
32:08chart search you type in Slicer Dicer,
32:12it gets you to this screen and what I've
32:14done here is taking that data query that
32:16we put in into this system and divided
32:20into 48 slices and this is what I got.
32:23This is all the patients in that Blue Epic.
32:26Databox who meet the criteria of
32:29uncontrolled hypertension by age group.
32:32So I have four age groups here.
32:34As you can tell,
32:35very handy tool.
32:37It takes a few seconds to compile the data.
32:39It's an aggregate level view,
32:41but it's good for hypothesis testing.
32:44And if you're wondering how to learn
32:46more about it right in that screen
32:48is a very nice tutorial that was
32:50done by one of our nurse analysts
32:52talking through this very example.
32:56OK,
32:56but if you want to look at sort of a
32:58more realistic picture of people you're
33:00familiar with patients you're familiar with,
33:02there is another tool that you can use,
33:06and that's called a reporting work bench.
33:09I will try not to go too deep in there
33:12because there is a very nice tutorial.
33:14If you have access to LMS or learning
33:16management system at the health system
33:18it's posted out there and there are
33:20some resources towards the end of the
33:22slides where you can get more information,
33:25but.
33:25Why is the reporting workbench helpful?
33:27First of all,
33:28you can get to it.
33:30From going to the epic chart or the
33:33top left epic button looking for my
33:36reports and this is just an example of
33:39patients on our opioid registry and
33:42showing some parameters on on on those.
33:46The important part of a work
33:48bench report is because I can get
33:50directly into the patients chart.
33:52From this action menu I can I can
33:55document in this patient's chart.
33:57I can place orders,
33:58send a communication to the patient,
34:00and furthermore I can do bulk
34:03actions across multiple patients.
34:04So if I selected more than one patient here,
34:06I could be placing an order
34:08on multiple patients.
34:09I could be sending in my chart
34:10communication to many, many patients.
34:12So this is a actionable patient
34:16list inside of Epic and.
34:18Again,
34:19just to give you an overview of where
34:21to start and there's other avenues
34:24if you want to take a deeper dive
34:27top left button on an epic button,
34:29look for my reports,
34:30go to library,
34:32and here's a CHEAT SHEET which you probably
34:35won't hear it elsewhere is type A1.
34:39In that search button and what it is,
34:42it's a template that one of
34:44our ambulatory analysts built.
34:45It has the usual things you would want
34:47about an ambulatory patient, like the PCP,
34:50like the last visit and things like that.
34:52And then you can.
34:55You can modify this template
34:57to suit your purpose.
34:58The other thing you can do is there
35:01are thousands of these workbench
35:02reports you can type in something,
35:04and chances are you will get a hit.
35:06So if you type HIV,
35:07there's probably an HIV report here.
35:09Look how many diabetes reports I found here.
35:12So it's easier to modify an existing
35:15one than to create a new one.
35:17So that's another Ave for looking at
35:20data that you might already be familiar
35:24with on patients that you may have
35:26already have access to their chart,
35:27and you're not violating any HIPAA
35:30here because those are somehow have
35:32a treatment relationship with you.
35:34OK, what are some of the other sources?
35:37Has anyone heard off the analytics portal?
35:43OK, so if you're on the intranet at the
35:46hospital and the address is up here,
35:48but what it is, it's collecting all of
35:50the different reports and dashboards
35:52and variety of different avenues.
35:54Some are in epic summer outside
35:57and also it tells you.
36:00Uhm, the volume of reports and analytics
36:02that we have available at our fingertips
36:05through our joint data analytics team.
36:08This gives you a 12 month rolling numbers
36:12on how many Helix requests for data we get.
36:17One thing to notice,
36:18we churn out a lot of data requests
36:21and reports and dashboards.
36:23The other thing to note is what
36:25you're looking for is probably here
36:28rather than waiting for a request.
36:30Uhm, and the other thing to note
36:32is if you're looking for aggregate
36:35data that's not very specific.
36:37Try one of those self-help tools,
36:39because chances are with these
36:41kinds of requests,
36:42they're working on something.
36:45And your request might wait a
36:47little while before they get to it.
36:50The other thing I want to touch on is an
36:53example of dashboards that are in EPIC.
36:57The way you get to them is.
36:59So the epic button look for my dashboards.
37:02UM, this is an example of a
37:05population health dashboard,
37:06and for those who are in outpatient
37:10doing quality improvement,
37:11and there are a few things I
37:13want to highlight here.
37:14One is of course this is that bar
37:17graph icon that will tell you that
37:20you're on the dashboard page in.
37:23Here is where you can search
37:25for other dashboards.
37:26So whatever your default dashboard is
37:28probably going to be the physician dashboard.
37:30Uhm,
37:30click on this little icon and see
37:33what else you can find that might
37:35be of interest to you and that we
37:36might have already have access to
37:38that you didn't know off the pop
37:40health dashboards are available
37:42for all physicians and residents.
37:44My login department and name
37:46is already pulled in here,
37:48so the data I'm showing is not mine.
37:50It's doctor Connelly who's in MG doctor
37:52and a close friend and colleague.
37:55Uh, and what it shows here.
38:00All of these widgets come from
38:02chronic disease registries that
38:03were correlated with your help,
38:05with some of the experts within the
38:08organization and they have already
38:11have standardized definitions,
38:13so they're all coming from a
38:15preventive care definition registry
38:17of chronic kidney disease, asthma,
38:19COPD, diabetes, hypertension, CDs.
38:23What am I missing?
38:24Scroll through when you get to this.
38:26Page and see what others are out there.
38:29The other thing is that each of
38:31these quality metrics are based on
38:33national quality forum definitions.
38:35Again thoroughly validated,
38:36so if you're looking for a project
38:40to work with your residents or
38:42for yourself and this you find
38:44one that's important to you.
38:46It your definition, your data,
38:49your department's data,
38:51your epic department's data are
38:53here and can be tracked overtime.
38:56Those sparklines are actually showing trends.
38:59You can look at the definition
39:01by going in this
39:03icon and remember that Workbench
39:05report that actionable list you
39:08can run one right here to find the
39:10patients who are not meeting this
39:12quality measure and maybe send all of
39:14them a message if you want it too so.
39:16Those are some of the things too.
39:19And tools that you have handy now
39:21that you could go and explore.
39:23OK, so in all of these things,
39:26if you were asked me that question and
39:28what exactly is the best data source,
39:30tell me one place this is going to
39:32be my answer, each one to their own.
39:34It's like it's like when you want to
39:36eat cake and I'm sorry I'm talking
39:38about this in the noon hour and you
39:41guys are skipping lunch because of this.
39:43But this is how it happens when
39:45we want cake in our family.
39:47One of my kids likes to bake.
39:49This is what she turns up.
39:51The other one also likes to bake but
39:53doesn't like the big from scratch.
39:55And this is me.
39:56I'm OK with the store barking,
39:58so each one to their own and you find
40:01the one that works best for you in
40:03the time that works best for you.
40:05And hopefully today you got an
40:07idea of what is out there.
40:09Yeah OK,
40:10switching gears a little bit in
40:12the last few minutes on how do
40:15you use this information.
40:16So there are three stages in a project
40:18where you might have a need for data.
40:21One is when you need some baseline
40:23information when you're planning a proposal.
40:24The second is when you actually
40:26crafting a smart aim where you need
40:28some more targeted data because
40:30you're going to put some goals
40:32out there and then the third is
40:34when you're measuring impact.
40:35So if we keep going through
40:38with that thought.
40:40Here's an example on the right
40:41hand side of a workflow.
40:43Remember, I said we're closer important.
40:45This is a workflow on data flow
40:49diagram for how we do flu data
40:52reporting for the hospitals for CMS.
40:54It's an annual exercise and we've
40:56done that for many years and it's an
40:59onerous exercise for the team that does it.
41:02So this time these last few years
41:05we've chronicled how we get this flu
41:08data and when we try to improve.
41:11The process it's important to note
41:13that there are some places that we
41:15have the flu information discretely.
41:17For everybody else,
41:19the clinician gets a survey to answer.
41:22So when we are designing a project.
41:25We could choose to say, well,
41:27I'm going to improve the active
41:30medication staff vaccination capture rate.
41:33Uh,
41:34by a certain percentage in a
41:37certain flu season.
41:38By getting more data resources right
41:42so I will supplement this AC health
41:45system by information that comes from EPIC.
41:48For instance,
41:49it needs consent and whatnot,
41:51but that was the project that we did.
41:53Or we could say one of our quality
41:55goals is that fewer people get to
41:58answer this survey because it's
41:59an additional burden for busy clinicians.
42:02So you could tackle any part of the workflow,
42:06but you're going to need the workflow
42:07to understand how to make improvements,
42:09and you may need some some data points to
42:13know what the numbers are. With me so far.
42:18OK.
42:22There we go, so measuring
42:24success is I guess the next part
42:28you all know measures come in outcome
42:31process and balancing flavors.
42:34You need inclusion exclusions and
42:36exceptions in the denominators.
42:38So far we pay a lot of attention
42:40to like who's meeting what.
42:42I want them to meet, who's not meeting
42:47a criteria or desirable action.
42:50What we forget is that the real power lies
42:54in identifying the correct denominator.
42:57So if you measure the whole of epic data,
43:00you're diluting your denominator right there,
43:02so your proportions are going
43:04to look really bad.
43:05But but this is what you're going
43:08to need is define your inclusion
43:10exclusion and exception.
43:12Exclusions are valid reasons where
43:16care doesn't need to occur there,
43:19of course removed from the denominator.
43:20Exceptions are when there's individual
43:24judgment or individual reasons.
43:27That's patient based.
43:28Keep those two in mind,
43:30but at the end of the day they're
43:32both removed from the denominator
43:34on the right side.
43:35As an example,
43:36there's a measure for depression
43:38screening that anyone having a
43:40visit during a calendar year.
43:42Uhm,
43:43should get a depression screening
43:45and most of you are probably using
43:47the PHQ 9 depression screening.
43:49So let's say we had 100 patients
43:52with a visit, five had depression,
43:5410 declined the screening.
43:56What's my denominator now?
44:02Is it 100? Is it 95?
44:07Is it 85?
44:13Sorry, I had to whip out
44:15my calculator for this one.
44:17So let's go in here it will.
44:19You will take both of these out from the UM,
44:22from the denominator,
44:24because those people one is an
44:26exclusion and the other is an exception.
44:29Uh, this is an example and
44:31I don't want to dwell on it,
44:32'cause you've probably seen it before,
44:34but it's an electronic clinical
44:37quality measure definition from
44:39centers on poor control of diabetes,
44:41but I use that as an example
44:43to say this is how they.
44:45They define how we should be
44:47pulling data for this measure.
44:50Use that as a template the next time you
44:52find yourself having to design A measure,
44:55and so we use this in here when
44:59we were doing the flu reporting.
45:01And how does that translate?
45:03So focusing on the denominator
45:06here we said OK,
45:07all active medical staff
45:09who've had the flu shot.
45:11We need to tell CMS about it.
45:13OK, well what's active Med
45:14staff is does active include,
45:16refer and follow clinicians?
45:18What about people who are on
45:20leave of absence or vacations,
45:21or sabbaticals during the flu season?
45:23What?
45:24What happens when you join the
45:25medical staff or leave the medical
45:27staff during the flu season?
45:29Are we including you in
45:30the denominator or not?
45:32So those are the kind of questions we
45:34need to tease out. Uh, in terms of.
45:37Who has had the flu shot?
45:39Well, that's easy to say flu shot,
45:43but there are people who
45:45have valid exemptions.
45:46We give medical and religious exemptions.
45:51Some people have allergies.
45:53We also talked about the
45:55self attestation survey.
45:56So are all of those going to
45:57make it to the numerator?
45:59Again,
45:59this is more to provoke the
46:01kind of thinking we all have to
46:04do to define our numerator,
46:05denominators to get to the actual measure.
46:10OK,
46:10I promise this is the one last
46:12concept I'm going to talk about,
46:14and then we'll wrap up for any questions so.
46:18Uh,
46:18this is the story about survivorship
46:21bias and you may have come across
46:24in different areas of discussion.
46:26This is a diagrammatic representation
46:28of World War Two planes and each
46:31of those tots shows where these
46:33claims got hit when they came
46:35back after their sortie.
46:36So the question for the Navy was
46:39how do we reinforce these planes
46:41so they can last us longer?
46:43Because clearly they are very effective.
46:45They can fly behind enemy lines.
46:47So what was proposed was to reinforce
46:50the areas that are shown in red dots here.
46:53But that made those planes really
46:55heavy and they couldn't possibly
46:57take off to fly and so this was
47:01the dilemma that was happening
47:03and there was one person who
47:05didn't agree with this approach.
47:07He instead proposed that we
47:09reinforce the areas that are
47:11shown by the arrows ABC and D.
47:16Anyone have a guess why and
47:18feel free to unmute yourself?
47:20It was clean, planes came back 'cause
47:23they weren't shot in those areas.
47:25So they survived. Yeah, correct
47:27exactly. So this is Abraham Wald.
47:30He was a statistician in the in the
47:32department and he this was exactly
47:34the question he says you have found
47:36the planes that came back but
47:37not the ones that didn't make it.
47:40You need to reinforce the other
47:42areas so it's always good to ask
47:44what am I not measuring is this?
47:47Is this the right sample to answer
47:49my question and fundamental things
47:51that often get forgotten when
47:53you're too close to the weeds,
47:55but just I thought it would
47:57be a helpful reminder.
47:59As promised, I was the last concept to here.
48:03I'll summarize by saying there are four
48:06important things to take away from today.
48:08One is that asking the right question is
48:11really the key in quality improvement,
48:14and you also have to turn a
48:16clinical query into a data query,
48:18which means being a big splitter
48:21in some ways.
48:22The second is you have to really
48:25know your data and keep an eye on it
48:28just like you do on your children.
48:30There are many data sources and
48:32good enough is usually enough,
48:34especially when you're starting off,
48:36so don't hesitate to take a deeper dive.
48:40And then lastly,
48:41can this data set answer my question?
48:42What's not represented in my data set?
48:47And with that I want to thank
48:48you for your attention.
48:49I know you guys have the great
48:52responsibility of improving clinical
48:53quality care and with that comes
48:55a great responsibility and that
48:57responsibility is what we were talking
48:59about called data stewardship.
49:02And I'm open to any questions,
49:05although anything on your end
49:08thank you very much.
49:09This was very informative.
49:10There are many questions in the chat.
49:12I try to piece them together and
49:16will start asking so one reminder
49:19this lecture is being recorded.
49:21It will be posted in the Department
49:23of Medicine, Intranet or you
49:24need is a meter ID to do that.
49:26You can go into the Department of
49:30Medicine webpage and you will find.
49:32A link to the Internet.
49:35Uh, so give us a few days
49:38to to get it posted.
49:40We can share the slides.
49:42I'm assuming the two would you be
49:45willing to share the slides and we will.
49:48We will need to just.
49:50Probably posted also in our intranet
49:54the so some some questions here.
49:57So the first one has to do with the
50:00dashboards and one of the and there are
50:03two questions that came related to that.
50:06One is how far are we from
50:10segmented dashboards including
50:11data on race and ethnicity?
50:16That's a big question,
50:17and I see Linda asked the question.
50:20So Linda, we have a lot of
50:22dashboards already summing up,
50:23pick some outside.
50:24It depends on the topic that you're
50:26looking at, and because race and
50:29ethnicity is captured in epic,
50:31we often can show race and ethnicity
50:34distribution for almost any data
50:36set that's coming out of epic.
50:38The one caveat there is the data
50:41is only good as as good as how
50:43it was when it was captured,
50:45so we have a lot of gaps.
50:47In our race and ethnicity data capture rate.
50:50Because.
50:52Can understand there's a lot of others,
50:54or patient refuses to answer type
50:57questions which render it a lot less useful.
51:01But because the data is captured,
51:03we can show it it's not universally
51:05displayed in every dashboard because
51:07most of the dashboards are not
51:09something the analyst designed.
51:11It's coming from a request
51:14from someone like yourself,
51:16so as the awareness for sort of
51:20diversity equity inclusion grows.
51:23It probably will become a standing
51:25part of most of our datasets.
51:29So the second one,
51:31related to two that came comes from
51:34Elaine Fajardo which has to do with the.
51:37If I understand it correctly,
51:38she wants to find out how can the
51:42architecture of a dashboard be identified.
51:46So how can one understand how
51:50the data that's in a dashboard?
51:54What the workflow of that data,
51:56the architecture?
51:57For for, for a dashboard,
51:59how can that be known by an end user?
52:03Most of the dashboards that are outside
52:06of Epic will have an about page and
52:09and which will tell you information
52:11on where this was pulling from,
52:13who requested it and when.
52:15What's the look back and standard
52:18definitions that are being used.
52:20And also I will say looking for the
52:24Helix logo on your dashboard gives
52:26you a sense that this went through
52:28that rigorous validation process
52:30because as you as you saw here,
52:34I can create a dashboard,
52:35but it's probably not as good as what
52:37our team comes with and we also have
52:39a population health team that does
52:41this kind of population data analysis.
52:43So look for that about page and
52:46then if you have any questions
52:48on sort of more detailed on.
52:51Like what definition of
52:52diabetes did you use here?
52:54That is often something deeper that
52:56may not be for a general audience,
52:58so I can help connect the dots there.
53:03Uhm? Need to a. There was a a request
53:08for share of the links, important,
53:12particularly to the analytics portal.
53:15Can you explain to people why is it that
53:18sometimes they try to log on and cannot?
53:21So can you justify what are the the
53:24the permissions that are required?
53:27Where do you have to be logging on
53:30from to get access to that please?
53:33Analytics portal is on our
53:36intranet at the hospitals.
53:37If you're on the hospital network,
53:39you should be able to get onto it.
53:41What report or dashboard you see from
53:44there depends on the level of access,
53:47so there are some executive for
53:50leadership dashboards that you won't
53:52see or may not have access to.
53:55Their might be dashboard safe for the
53:58heart and Vascular Department and so hard,
54:01and vascular leadership is seeing those,
54:03but none of us have.
54:05Access to those so it depends
54:08on your reporting access.
54:09When you get into the epic dashboards.
54:12Similar things apply.
54:14Mostly aggregate data is visible.
54:17Aggregate clinical data is
54:19visible to most clinical users.
54:22When you get to patient level information,
54:24it's it's likely limited to
54:27patients you have access to in your
54:32epic roles and responsibilities.
54:34So if you have.
54:36Uhm, if you practice, say,
54:38a digestive disease and temple,
54:40you can see patients over there,
54:41but not the Long Wharf practice or not
54:43the NMG digestive disease patients.
54:48I'll let that get to what you had in mind.
54:52Well, I think so.
54:53It was a question from there were a couple
54:56of questions from from the audience.
54:58There is a question that has to do with
55:03Workbench which is in the IT comes
55:05phrased as is the dashboard for your
55:08patients or other workbench reports.
55:11For your patients.
55:12How do you replicate the dashboard?
55:15The report for a group of providers?
55:18For example, resident or a group?
55:24And why don't you answer that?
55:26I I may be able to give some
55:28practical answers to that from
55:29an administrative perspective.
55:31Sure, I'll give the short answer
55:33then it is possible to do.
55:36If you want to get hands
55:38on to a workbench report,
55:39couldn't hurt to try,
55:41but you go into edit and then you can
55:45edit by PCP or encounter provider.
55:48And a few additional things which will
55:51take some practice to get familiar with.
55:54But technically it's possible to do.
55:56I know that some of our.
56:00Pediatric faculty do that for
56:03the residents in the clinic.
56:06So, and that question came from
56:08Christine Krueger from and and Christine.
56:10The practical answer is that yes,
56:13your any report that you build will be
56:16limited by the authority that you have
56:20to access certain a certain population.
56:23So for you it will be your patience
56:26comes under my patients. In the.
56:29If you need something that's
56:31error greater level, for example,
56:34if it's something that the hkcu
56:36would have to do to have access
56:39granted by the clinical leadership.
56:42So for example,
56:43then Tobin would be able to see
56:46the totality of the report,
56:48so and that's restricted to the epic
56:51departments that are connected to
56:53an individual section or program,
56:56so so there's a little bit of complexity,
56:59but it's doable. You just.
57:01There there needs to be added access
57:03granted to you and that's that's the
57:06the the the issue of restriction there.
57:10Sorry, my question is actually for
57:12the dashboard itself, not for.
57:14I know how to run a report
57:15and add additional people,
57:16but for the dashboard I noticed
57:17that you had some metrics on there.
57:19Can you only ever see the
57:21dashboard for yourself or is there
57:23like a department dashboard?
57:24So if you remember on the dashboard
57:26there was a left hand side and the
57:28right hand side the left hand side was
57:30the clinician, the right hand side.
57:32Was the department the login
57:34department of the Clinician? Yep.
57:37The there were some.
57:40There was a general question
57:43about from Adrian de Silva on
57:46to start a quality improvement
57:48project requiring data from EPIC.
57:51Whom should I reach out to for
57:53guidance or help to gather the data?
57:58I can take PT.
57:59Oh and need to and there's an
58:01added a second part to that
58:03question that has taboo to do
58:05with institutional approval.
58:07Whenever whether IRB
58:08approval is needed, yeah, and
58:12so the the line between quality
58:15improvement projects and research
58:17projects is fairly blurred.
58:19There is even as we speak and
58:21initiatives sort of going on with senior
58:24leadership on trying to create better.
58:27Distinction between the two
58:28and why is that helpful?
58:29Because there are different sensitivities
58:33and checks and balances needed for both,
58:35so although talked about some
58:37organizational checks and balances,
58:38so we're not violating HIPAA by
58:41looking at patient data that we
58:43don't have any right to be in,
58:45and then for research similar constraints.
58:47Does this research actually
58:49has been approved by HRPP?
58:51Does it require patient consent and so
58:54on so forth? That's all of what we.
58:57Would call data stewardship.
58:59So don't look at them as restrictions.
59:02Look at them as a responsibility to do.
59:05Yes, you can put in a request to
59:08two for data at Helix winehq.org and
59:11tell them this is for research or
59:14operations purposes, but if you want,
59:16sort of.
59:17Uhm, uh,
59:18that sort of a colleague to
59:20go through what it means.
59:22I'm happy to sort of be that
59:24consultant in the middle before
59:26you even have to put in a request.
59:28I would also encourage to say
59:30if you're planning a project and
59:32you're looking for baseline data,
59:34try one of these hands on tools,
59:38because remember,
59:39it's the same resource who will be
59:42putting up a large research data set.
59:46Who might?
59:47Actually need to pull your preliminary
59:50data for problem hypothesis testing.
59:52So for if the data that you have
59:55gives you a ballpark number and
59:58that's good enough for your proposal.
01:00:01These tools come in very handy,
01:00:03saves everyone a lot of time.
01:00:06Need to there two in the last
01:00:08two minutes that we have.
01:00:09There are two questions
01:00:11about how data are available.
01:00:13The simple one is are all medications
01:00:16in the HR available in RX norm format?
01:00:20It's a yes Margaret
01:00:22OK and the other one is how do
01:00:26we find data for orders done?
01:00:28I've heard something about logic stream.
01:00:32The question is which patients have
01:00:35mammogram ordered but not completed.
01:00:38So when we write reports on an
01:00:41so in the example of mammogram,
01:00:45whether or not this patient had a mammogram,
01:00:48the analysts will often ask you.
01:00:50Are you interested in orders
01:00:52or completion of the order?
01:00:54So any order that's completed in Epic
01:00:58will tell you whether or not that
01:01:01mammogram was actually performed.
01:01:02So there's a distinction even
01:01:04within the epic data set.
01:01:06Logic Stream is a third party
01:01:08system that we have that helps us
01:01:10do some level of slicing and dicing.
01:01:13You can certainly get access
01:01:15to it and look at it.
01:01:17Yet another system that you have to learn,
01:01:19but I think.
01:01:20For the project that you have in mind,
01:01:22Christine mammogram completion
01:01:24is a good place to start.
01:01:27Just don't forget that there are
01:01:29some mammograms that may be scanned
01:01:31into the system so other data
01:01:33sources may need to be looked at.
01:01:39So I think we covered everything
01:01:41and we have 30 seconds to go so
01:01:44need to can't thank you enough
01:01:46for this wonderful session.
01:01:48Thanks for the attendance.
01:01:49We head up to 68 people so certainly
01:01:52a lot of interest and come and
01:01:55and again I hope it was enjoyable
01:01:58to everybody as it was to me.
01:02:00Thank you once again.
01:02:01Need to and a good day to everyone.
01:02:04Thanks everyone. Thanks me too.
01:02:07I'm sure you're going to have a lot of
01:02:09people in dating you with requests now.
01:02:12Not a problem, and I'm ready
01:02:14just offered yourself up for
01:02:16every single question.
01:02:18No worries. Have a good team.
01:02:22I just a quick question
01:02:24with the IRB question,
01:02:25where does implementation science,
01:02:26'cause I can just see that that's
01:02:28another whole group of people that's
01:02:29going to use this right? Is that?
01:02:31Does that undergo IRB approval or so?
01:02:34Here's the and I'll look to.
01:02:37Although for sort of some guidance
01:02:38on how you think about it,
01:02:40if I'm ever going to write about
01:02:42it in a peer reviewed publication.
01:02:46I'm going to look for an IRB up most
01:02:49often times it's helpful to sometimes
01:02:52just reach out to the IR B for
01:02:55unclear before you put in a request,
01:02:57and also once the guidelines
01:02:59for Qi and research come out.
01:03:01They're still in development.
01:03:03We will have some very pragmatic guidance
01:03:07on which needs to go through what channels.
01:03:10Hopefully a pathway, right, right?
01:03:13I mean that that's been very
01:03:14frustrating because depend.
01:03:15It's like the.
01:03:15Flavor of the of the week.
01:03:17Depending on who is answering your request,
01:03:20some some of them will say why are
01:03:22you asking me a question 'cause
01:03:23everything that I do is quality
01:03:24improvement like this doesn't,
01:03:26and then they're like.
01:03:27Just go to this website and the
01:03:28other there's another whole group
01:03:29that sends me an official letter,
01:03:31right?
01:03:32So yeah,
01:03:34I mean they're they're in the same way,
01:03:36inundated with a lot of requests.
01:03:38I know they had to restructure
01:03:39sort of blow everything up and
01:03:41repackage it together when COVID
01:03:43requests started coming in.
01:03:44But you know, let no crisis could.
01:03:47Go to waste.
01:03:48I know there's been a lot of work
01:03:50happening on the IRB side along
01:03:52with the skins and Brian Smith
01:03:55and other leadership to sort of
01:03:58create those processes upstream,
01:04:01so so the people who want to do this
01:04:04kind of work get the guidance upfront,
01:04:07and it's not easy defining those.
01:04:11As you can imagine.
01:04:15Great guys, so thank you very much.
01:04:18Once again much more to talk in
01:04:20the future. Have a good day folks.
01:04:26Need to, I'll save the chat for you.