Qualitative research

(Julia Frost & Kristin Liabo)

Objectives

The overall objective of the qualitative research was to understand influence of modelling, including the use of machine learning techniques, in the context of the national audit, in order to support efforts to maximise the appropriate use of thrombolysis and reduce unnecessary variation.

Specifically the aims were to:

1. Explore current understanding and rationale for the use of thrombolysis for ischaemic stroke, in order to establish reasons for the variance in the use and speed of thrombolysis.

2. Understand physician perspectives on simulation and machine learning feedback, to influence how simulation can be incorporated into the Sentinel Stroke National Audi Programme (SSNAP) to have a positive impact on practice.

3. Identify potential routes for the implementation of machine learning feedback, to inform and improve future stroke management.

4. Explore how physicians interpret the potential consequences of following changes in pathway suggested by simulation.

Key findings

Qualitative research demonstrated a varying openness to machine learning and modelling techniques.

1. Broadly, those units with higher thrombolysis use engaged more positively with the research, and those with lower thrombolysis use were more cautious.

2. Clinicians from lower thrombolysing units tended to emphasise differences int their patients as the reason for lower thrombolysis. Those in mid-rate thrombolysis units tended to emphasis access to specialist resources as being key in being able to deliver thrombolysis well. Those in higher thrombolysing units tended to emphasise the work and investment that had gone in to establishing a good thrombolysis pathway.

3. Clinicians wanted to see the machine learning models expanded to predict probability of good outcome and adverse effects of thrombolysis.

4. Despite this being a small study physicians engaged with the machine learning process and outcomes, suggesting ways in which the outputs could be modified for feedback to stroke centres, and utilised to inform thrombolytic decision-making.

Data collection

Ethical approval was provided by the Health Research Authority and Health and Care Research Wales (HCRW) 19/HRA/5796.

To pilot our interview approach, we undertook a face-to-face group interview with a small group of medical registrars on a regional rotation that included stroke, in their clinical setting. Having given written consent to interview, a senior modeller (MA) provided the registrars with a demonstration of the modelling process and outcomes, prior to the qualitative researchers piloting the topic guide. Feedback from the stroke physicians suggested that this approach was appropriate and produced data that was fit for purpose. Our approach was subsequently modified for remote delivery, with all interviews conducted via MS Teams, Skype or Zoom, dependant on the media that was allowed in each NHS Trust. At the beginning of each interview participants watched a 10-minute video made by the senior modeller that contained examples of the process and outcomes of the machine modelling and pathway analysis, as a stimulus for discussion [1]. The topic guide was then used to elicit participants’ own experiences of thrombolysis and perspectives on machine learning, alongside observations of group interactions and clinical settings [2].

During the interviews, we collected data about physicians’ backgrounds, their attitudes to thrombolysis and their understanding of variance, their perspectives on machine learning, and potential loci for the implementation of machine learning feedback (within and beyond SSNAP), established the physicians’ views on possible unintended consequences, which may result from changing the acute stroke pathway, and potential means of mitigation. Our fieldnotes reflected the challenges of conducting interviews via video, with physicians often in clinical settings and sometimes wearing PPE, as well as capturing the dynamics between physicians, who were also remote working from each other [3, 4].

Towards the end of the project, and during the third lockdown for COVID-19, we undertook an online discussion of our results with a small group of physicians (N=3) who were identified via an annual meeting for trainees, organised by the British Association for Stroke Physicians (BASP). The modellers (MA, CJ) presented a further set of outputs from their analyses, and the discussion focused on how additional modelling outputs might be used to facilitate quality improvement and inform service delivery.

Data analysis

Interview data was transcribed by an independent GDPR-compliant transcriber, and fieldnotes were written up by the two researchers. All data were anonymised and managed in Nvivo for Teams [5]. Both researchers read all the transcripts to develop preliminary ideas and understanding. We developed these ideas alongside further re-reading of the transcripts, using a Framework Analysis aligned with the four broad exploratory objectives of the study, but crucially with an openness to any new insights from the physicians [6], [7]. Analytical summaries across multiple cases were created independently by both researchers, and used to explore the data. We held repeat discussions to develop the analysis, looking for negative cases, and resolving differences of opinion about interpretation [8]. In this way, we were able to examine these physicians’ accounts of their use of thrombolysis and orientation to machine learning and simulation. As our analyses developed, we also discussed our findings with members of the wider research team.

Results

Interview participants

We recruited nineteen participants, who took part in three individual and five group interviews. Eleven of the participants were consultants (specialising in stroke, neurology or elderly care) with four stroke registrars and one specialist stroke nurse. Ten participants were male and nine were female.

Thrombolysis	Interviews
Low (Site A)	Group: 2 Stoke Consultants (2 x M) + I Specialist Stroke Nurse (1 x F)
Low (Site B)	Single: Consultant Stroke and elderly medicine (M)
Low (Site C)	Group: 3 Stroke Consultants (1M/2F)
Middle (Site D/Pilot)	Group: 4 Registrars (4F) (Pilot interview)
Middle (Site E)	Single Consultant Geriatrician (M)
Middle (Site F)	Single: Stoke Consultant(M)
High (Site G)	Group: 3 Consultants: Stroke & Geriatrician (1M/2F)
High (Site H)	Group: 3 Consultants: Stroke, Neurologist, Geriatrician (2M/1F)

Current attitudes to thrombolysis use

Physicians working in hospitals with lower thrombolysis rates were more likely to suggest that a significant barrier to thrombolysis was the delayed presentation of patients, which could be magnified by suboptimal ambulance services:

Quote

“A lot of patients present outside the window of thrombolysis at the hospital.” (Site B)

“I think we rarely hit the 11% percent national numbers, probably because patients come just outside the thrombolysis window.” (Site A)

Those working in hospitals with lower thrombolysis rates were more likely to report that their patients were ‘different’ to those presenting at other centres – in terms of rurality, ethnicity, frailty, or socio-demographic factors:

Quote

“We’ve a slightly older population… we’ve slightly more bleeds than infarcts…we’re a slightly larger geographical area, so sometimes people are a bit delayed getting to hospital and we operate across two sites as well.” (Site C)

The above physician also highlighted that because of these complexities, decision making about thrombolysis was the most difficult part of their job. While population differences were also acknowledged by physicians at higher thrombolysing centres, they were more likely to articulate the centrality of patient heterogeneity in their decision making:

Quote

Consultant 1: “I wouldn’t be giving thrombolysis for various reasons… They’re often late, or got a very mild deficit, or they’ve got something that makes you feel extra wary about treating them… we’ve got a population that’s increasingly frail, they’ve got multiple comorbidities… [but] every patient is unique.”

Consultant 2: “We all have different approaches, I say to myself the first question is, if I don’t thrombolyse this patient, what is the worst neurological outcome they could have? What is the disability going to be? And then the next question is how far are we down the time pathway, what’s the risk of bleeding here? And then, what are the little things that feed into pros and cons, how does that alter the equation from a standard patient? is it that the benefits are going to outweigh the risks, how finely balanced is that decision?”

Consultant 3: “The days of people being textbook strokes are long gone… we don’t see them… we don’t have a blanket policy. We eyeball them. And if they look dodgy we park them and work out what’s going on, if they don’t look dodgy, we go straight to the scanner.”

(Site H)

Those in mid-rate thrombolysis centres suggested that some of the delays in patient presentation could be mitigated through treatment by stroke physicians rather than generalists, or by the involvement of a Specialist Stroke Nurse:

Quote

“We typically have a more deprived population, so accessing health care and time to hospital [and] our ambulance service is not as good…”, “a burden of disease due to deprivation… we do see a lot of young strokes… smoking, drinking, drug abuse… expertise is important there, so if you looked at our patients… the ones that had been thrombolysed under 30 minutes… nearly all of them had been managed by a stroke registrar or a geriatrics registrar or a geriatrics consultant.” (Site E)

“[a] stroke nurses being there increases the speed…”(Site D)

Those physicians currently working in centres with low or medium thrombolysis rates seemed more likely to emphasise the equipment that they lacked, and which they perceived would improve the accuracy and speed of their decision making.

Similarly, physicians working in hospitals with higher thrombolysis rates suggested that their higher rates were due to access to scans and other specialist facilities, as well as 24 hour stroke services:

Quote

“We’re a big teaching hospital… that’s also got a trauma centre” (Site H)

“Thrombolysis is done by registrars with consultation on the phone with some access to the imaging for the consultant… there is no dedicated stroke team at night… we have a big variation between out of hours and in hours door to needle time… it’s 38 minutes, out of hours it’s 89 minutes… don’t thrombolyse wake up stroke… MRI… perfusion scan… we don’t have the facilities.” (Site F)

“On SSNAP data, we are one of the top performing units in the country and that has happened through years of planning and hard work, where we take direct admissions, twenty-four seven, we don’t do remote assessments… it’s always face to face assessments by consultant… with a specialist nurse, to see a patient, etc. And we have access to scans directly, including vascular imaging…” (Site G)

Thus, the provision of more diagnostic tools was perceived as enabling a more nuanced approach to risk management - that went beyond tallying risk factors, and individualised patient care for more ‘marginal’ case:

Quote

“If I might manage a level of uncertainty about the onset time and some other characteristics, medications, for example, a slightly imperfect history that I have, if it’s a very severe stroke, it’s going to be a disabling stroke and I feel that the risks are outweighed by the benefits… I think that stroke severity and my perception of the ability to benefit from thrombolysis will then weigh into how much uncertainty I’m able to cope with, with the other things.” (Site G)

Although the sample interviewed was small, they were diverse in their attitudes to thrombolysis use.

Perspectives on simulation and machine learning

Physicians who identified as confident thrombolysers had an initial scepticism of both the premise and methods employed in the simulations they were shown, although this scepticism was later dispelled:

Quote

“The first thought that came to mind was [with the modelling] an innate assumption that doing more thrombolysis is a good thing… So, your machine learning may tell us how to do a lot of people who possibly don’t need it, possibly. I’m not saying that’s necessarily what you’re going to do, but it’s where it might go if we just say ‘more is better’.” (Site H)

Physicians who both worked alone and were interviewed in isolation were more anxious about how the simulation might be used to hold them to account for their decision-making and identified perceived risks.

Quote

“I’d be suspicious if such a tool was available and a patient wasn’t thrombolysed, then that might involve the lawyers and the legal teams.” (Site C)

“I think safety would be the top thing, isn’t it, it’s got to be a hundred percent safe and I think if you are close to a hundred percent safe, if you can show that, if you can show that it’s safe and it doesn’t cause any negative outcomes for patients, then -and it also enhances patient care by speeding the process up, then I think you’ve won. If there’s doubts about its safety, even if it does speed things up, people aren’t going to trust it… clinicians are always wary about litigation, as well… some of this software could be used retrospectively… it could lead to decision-making being criticised retrospectively.” (Site E)

Those who had the benefit of working in a team with both a culture of collaboration and professional challenge were more inclined to see machine learning as a resource to draw upon for their own decision-making. For example, with those in low thrombolysing centres suggesting that it might augment their decision-making, while those in higher thrombolysing centres viewing it as a positive challenge to inherent assumptions that they might have developed:

Quote

“I think it would be a help if there was a patient where, you know, maybe somebody else would have thrombolysed them and we might see something that we weren’t doing that we would then, you know, implement as an action, if there was something clearly that we could be doing, you know, that would improve the rates.” (Site B)

“if you have a computer model you might get out these things out of, well, at least you think about it if the computer says something, then you have to have a strong kind of argument to refuse, to say no. [laughs] To say, well, the computer says, well the modelling comments that they should be thrombolysed, why do you say it’s not thrombolysed, you can’t just say oh, because he’s old or whatever.” (Site F)

“it would be useful to know what would somebody else do. Now, whether that’s presented as a number or as a likelihood for thrombolysis… the hospitals…” (Site F)

In addition, those in the highest thrombolysing centre also thought that the modelling outputs could extend their quality improvement initiatives:

Quote

“It’s just a tool, isn’t it, it’s just another tool. We would never – you’d never base your decision on what the machine said! I mean, not until it’s like, you know, the Star Trek computer!… You’re generating data for improving a process and for understanding of process, so it’s very helpful for that… And it might be useful to beat the managers and say we need help with this, that and the other, but then any audit does that.” (Site H different physician)

Some participants suggested additional variables that they would like to see included in the modelling:

Quote

“There are factors there which we would use in our decision-making process which are not listed as inputs…active bleeding, head injuries, blood pressure, whether the patient assents or consents [inputs]are insufficient and superficial…” (Site G)

“The other thing that feeds in is that not everyone’s comfortable looking at CT heads and some people are waiting for that to be reported and I think that can add considerable time… especially down here, the radiology registrar is not always based in this hospital. So, they cover the whole of the [area] and they might be based in [other centre], whereas in the daytime you can just walk round the corner and speak to the radiologist reporting the scan, and say “What do you think, is it OK”, or call the consultant on call. But I think if you’re, for example, a med reg in another speciality thrombolysing at night, you wouldn’t have that confidence to say that and then having to call up a radiology reg on call in another hospital all takes time, doesn’t it… Say, at worst, half an hour.” (Site D)

Centres with middle thrombolysis rates were keen to see the outcomes of employing machine learning included in the outputs:

Quote

“How have you extrapolated your outcome data?… what I think would be useful to know is within that people of decision makers, who is making the decisions?… [I would want to see] median times with clear confidence intervals would be most useful…diagnosis at discharge comparative to decision making at the time.” (Site D)

“So I think that is, kind of, disabilities should be part of that pathway, some type of assessment for that, for instance, the things which I do is I have to try and identify a link between the disability they get and patient kind of function - that would be helpful.” (Site F)

Perspectives on simulation and machine learning varied by the size and type of unit that the physicians worked in, with some participants welcoming the addition of modelling to their decision making tool kit, with others worried about the loss of their agency.

Potential routes for the implementation of machine learning feedback

Across centres, there was an understanding that modelling had the potential to identify which changes a particular centre could invest in to improve their stroke pathway:

Quote

“Tell us we should do our scans quicker or hurry up with our CTAs”…“Placing them on the scanner table rather than wheeling them round… simple things.” (Site H)

“the SSNAP data we have is great, but it’s difficult to apply that to solutions locally. Whereas if you could apply the modelling to a local set-up and find out where the delays are consistently across a number of cases, rather than just looking at one case…if you do that across hundreds of cases in the same centre, then you find local solutions to increase speed.” (High Site H, different physician).

When asked to identify the potential routes by which machine learning might inform or improve future stroke management, physicians replied with suggestions that matched particular issues with which they were grappling.

Those in low thrombolysing centres wanted a tool that could help them to improve care with a particular patient or type of patient, via a prototypical patient:

Quote

“[I] Would value a prototype patient… where it showed you which hospital would or wouldn’t thrombolyse… I would trust the data… we could get some advice on where to improve… there might be some big gains from that, if we did it.” (Site B)

“If we had the information that over the country [about older frail people from care homes], it would probably give those hospitals that are more cautious, more confidence to give that thrombolysis”(Site D).

“People are quite afraid of the risk of bleeding and things like that. If they produce a type of individualised risk and benefit for the patient, on the information that’s provided on algorithm, that would be very helpful… and also the ability to be updated quickly, that would be very helpful, because the texts change every year and then sometimes you can’t keep up with all those protocols and pathways.” (Site H)

Physicians working in the highest thrombolysing unit, who expressed greater familiarity with SSNAP as well as other performance indicators, wanted a more sophisticated instrument that could compare treatment across consultants or centres:

Quote

Consultant 1: “internally we tend to look at consultant level data, just by looking at the thrombolysis data and picking that apart. But obviously the numbers are small, so the data can be quite varied… but I don’t mind seeing it at consultant level information as well as, then, hospital.”

Consultant 2: “You take a prototypical patient and apply them to the algorithms that you’ve constructed for our hospital and see what pops out the other end… These things rarely provide an answer; they just point you to something you can reflect on…”

(Site H)

Perspectives on the potential routes for the implementation of machine learning feedback were informed by physicians’ beliefs about their current needs, with the idea of a prototype patient proving popular. However, there was variance in beliefs about what variables should be included, and whether its objective should be direct patient care or as a quality improvement tool.

Anticipated consequences of stroke pathway feedback

Two physicians, both of whom worked on their own, in lower thrombolysing centres, were sceptical about the consequences of changes to the stroke pathway. Having identified that they found decision making about thrombolysis difficult, both then questioned the evidence base for increasing the rate of thrombolysis:

Quote

“I do think it’s about, kind of, the personality of the person deciding it, it is very subjective, is thrombolysis, I mean, I know we have all the guidelines as to who we should and shouldn’t thrombolyse, but, you know, some consultants will aggressively continue to reduce the blood pressure with as much IV medication as they can until they can thrombolyse, others will say, well, you know, a few doses and if it doesn’t come down, OK, it’s probably not meant to be, so yeah. And you know, I think I personally just sort of very much stay within the exact rules for whether you should or shouldn’t thrombolyse.” (Site B)

“more thrombolysis doesn’t mean better care… when I hear of hospitals that are thrombolysing,… twenty odd percent, I do sometimes question them. Are they really thrombolysing strokes? Is the clinical diagnosis of stroke really robust enough or are they thrombolysing mimics and then putting that into their SSNAP data anyway, just to make them look good. And then their mortality rates are lower because they’ve thrombolysed non-strokes anyway. So, some of me is – I’m a bit cynical with, of the SSNAP data sometimes, from some of the sites that appear to be doing really well.” (Site E)

In contrast, those in higher thrombolysing centres had a more balanced perspective on the perceived benefits of implementation of machine learning in the stroke pathway but identified the likely enduring challenges of thrombolysis decision making:

Quote

Consultant 1:“There’s been a gazillion studies looking at how to give TPA quicker, so the question, I think, for you guys, is what’s going to be different about this, compared to everybody else’s that tells us to get ready, get the ambulance there quicker, be more streamlined, have a checklist, der, der, der, you know, what’s going to be different?”

Consultant 2: “Outcomes data with a comparator is a disaster, what does it mean?… I think you’re going to end up with a league table, but basically we already have one with SSNAP.”

Consultant 1: “The implementation of artificial intelligence and automated reporting of scans would change the picture, would change the landscape, let’s say, of the speed of thrombolysis.”

Consultant 2: “To be fair, the only aspect of machine learning I can see in this is the thrombolysis decision making process. The rest is all straightforward factors…. The only two parts machine learning is going to help is if the machine can actually interpret the head scan for us, which is really part of the decision to treat or not treat, and that’s the only real machine learning aspect of this, the rest is not… your decision to treat or not treat… That’s the difficult part. That’s the grey area where everyone does a different thing.”

(Site H)

Discussion

We identified that physicians working in hospitals with lower thrombolysis rates identified patient factors (e.g. age and ethnicity) and patients’ time taken to travel to hospital as significant barriers to optimal thrombolysis decision making [9], [10]. These perspectives are associated with working in smaller and more rural hospitals where physicians tended to work alone, rather than in teams, and where decision making about thrombolysis is taken less frequently, potentially invoking fear of poor decision making and fear of complications [11], [12].

In contrast, physicians working in hospitals with higher thrombolysis rates identified facilitators that they employed to mitigate previously identified ‘grey areas’ of decision making (e.g. the individual-level interpretation of available evidence for the efficacy of thrombolysis) [13]. Timely access to, and adequate reporting of, computed tomography (CT scan) and access to specialists (or in some cases peers) were seen as crucial [14], [15]; as was the provision of specialist nurses who could prepare thrombolytic drugs for administration, while physicians focused on gathering diagnostic information [16]. These physicians were more likely to work in larger centres, in collaborative teams that were actively advocating for the development of stroke services [17], and be able to envisage that machine learning could be used to improve operations and logistics in their specialism [18].

A significant finding of this research is that confident thrombolysers were more likely to be attuned to the potential benefits of instigating machine learning in the acute stroke pathway, in contrast to less confident thrombolysers, and where improvement in thrombolysis use and speed is most warranted. Previous studies in applied clinical informatics have identified that health professionals have deep rooted concerns that machine learning may lead to deskilling and distortion of the physician-patient relationship [19]; in part rooted in the lack of demonstrable potential of machine learning in clinical practice, with physicians yet to see how synthetic data is actionable at the point-of-care [20]. Unfamiliarity with machine learning and lack of familiarity with how models are created is associated with lack of trust in the methodology [18]. However, previous research has identified that trust can be established through researchers and clinicians working collaboratively throughout the research process [21], [22] , and with the methods and outputs reported in a transparent way [23]. As with existing research on the implementation of guidelines in stroke care, the identification of key individuals to lead the implementation or ‘advocates on the ground’ was seen as fundamental to the adoption of machine learning for optimising thrombolytic decision making [12], [17].

Various perspectives were provided by physicians on which machine learning outputs could best optimise thrombolytic decision making, with discussions including both the form and content of any data displays, and preference for a range of presentations that could be adapted to the clinical context or physician preference. To ensure clinical face validity outputs were requested to be accessible and engaging, as well as being presented in a way that limited the cognitive burden of the physician engaging with them [20], [21], [24]. The suggestion of a prototypical patient or vignette was popular, as was suggestions that downstream patient outcomes (e.g. on discharge) be included [25]. There was recognition that currently patient outcomes are fed back at unit or centre level, and some physicians were open to receiving this feedback at an individual (or Consultant) level [12]. However, we also acknowledge that some physicians did not perceive that machine learning would diminish the uncertainty about the utilisation of thrombolysis, because of their enduring concern about the potential risks and benefits associated with the therapy [12], [15].

Strengths and limitations

A limitation of this work is the small pool of physicians that we were able to interview. Although we were able to capture a range of perspectives from differing models of service delivery and with varying rates of thrombolysis, we are hindered in the conclusions that we can draw. Prior to the COVID-19 pandemic we had difficulty gaining access to physicians at the interface of NHS Trust research and development departments and human resources departments. Despite being a low risk study, and having provided all necessary paperwork required for approval by the Health Research Authority Research Ethics Committee, we were regularly required to provide additional documents for local settings. This, and the frequent change of personnel with whom we were asked to engage, delayed the recruitment process considerably. This qualitative work was then suspended during the COVID-19 pandemic while physicians were reallocated to frontline duties. However, on resumption of this study, bureaucratic processes (e.g. temporary local NHS Trust policies) prevented us from interviewing physicians, even when they had provided us with written consent to be interviewed.

As well as benefiting from more qualitative interviews with physicians, we would have benefited from a broader model of provider engagement. Borrowing from complexity science, Braithwaite et al (2018) suggest that implementation in complex health care scenarios requires an iterative and an adaptive social science approach to identifying which levers work best in which contexts. Although we had extensive patient and public involvement throughout the design and conduct of this research, our future planned work must also include a wider group of clinical stakeholders (e.g. including physicians who work in the ED, Specialist Stroke Nurses, etc.) to ensure that we work with, rather than on, health professionals throughout the research and dissemination process, to identify the most appropriate routes for machine learning feedback to stroke services [26]. Possible models for future engagement include embedding a wider range of clinical stakeholders in all aspects of the work (e.g. by our adoption of Integrated Knowledge Translation approach) to ensure co-production of knowledge for policy and practice [27]–[29].

Conclusions

We identified a range of factors which physicians perceive of as barriers or facilitators to the use of thrombolysis for ischaemic stroke. However our key findings is that confident thrombolysers were more likely to be open to the potential benefits of instigating machine learning in the acute stroke pathway, in contrast to less confident thrombolysers - and where improvement in thrombolysis use and speed is most needed. Despite this being a small study physicians engaged with the machine learning process and outcomes, suggesting ways in which the outputs could be modified for feedback to stroke centres, and utilised to inform thrombolytic decision-making. Future research needs to engage a wider group of clinical stakeholders throughout the research process.

References

[1] N. Crilly, A. F. Blackwell, and P. J. Clarkson, “Graphic elicitation: using research diagrams as interview stimuli,” Qual. Res., vol. 6, no. 3, pp. 341–366, Aug. 2006, doi: 10.1177/1468794106065007.

[2] J. M. Eakin and B. Gladstone, “‘Value-adding’ Analysis: Doing More With Qualitative Data,” Int. J. Qual. Methods, vol. 19, p. 1609406920949333, Apr. 2020, doi: 10.1177/1609406920949333.

[3] M. M. Archibald, R. C. Ambagtsheer, M. G. Casey, and M. Lawless, “Using Zoom Videoconferencing for Qualitative Data Collection: Perceptions and Experiences of Researchers and Participants,” Int. J. Qual. Methods, vol. 18, p. 1609406919874596, Apr. 2019, doi: 10.1177/1609406919874596.

[4] J. Salmon, Qualitative Online Interviews: Strategies, Design, and Skills. Los Angeles: Sage, 2014.

[5] Q. S. R. I. P. Ltd., “NVivo qualitative data analysis software.” 2018.

[6] J. Ritchie and L. Spencer, “Qualitative data analysis for applied policy research.,” in Analyzing qualitative data (ed. {Bryman} & {Burgess}), London: Routledge, 1994.

[7] M. Miles, A. Huberman, and J. Saldana, Qualitative data analysis: A sourcebook., 3rd ed. Los Angeles: Sage, 2014.

[8] N. Mays and C. Pope, “Qualitative Research: Rigour and qualitative research,” BMJ, vol. 311, no. 6997, pp. 109–112, Apr. 1995, doi: 10.1136/bmj.311.6997.109.

[9] J. P. Sheppard et al., “The association between prehospital care and in-hospital treatment decisions in acute stroke: A cohort study,” Emerg. Med. J., vol. 32, no. 2, pp. 93–99, Feb. 2015, doi: 10.1136/emermed-2013-203026.

[10] R. G. Thomson et al., Factors that influence variation in clinical decision-making about thrombolysis in the treatment of acute ischaemic stroke: results of a discrete choice experiment. Southampton (UK): NIHR Journals Library, 2017.

[11] A. Stecksén, B. Lundman, M. Eriksson, E. L. Glader, and K. Asplund, “Implementing thrombolytic guidelines in stroke care: Perceived facilitators and barriers,” Qual. Health Res., vol. 24, no. 3, pp. 412–419, Mar. 2014, doi: 10.1177/1049732313514137.

[12] L. E. Skolarus et al., “Understanding determinants of acute stroke thrombolysis using the tailored implementation for chronic diseases framework: A qualitative study,” BMC Health Serv. Res., vol. 19, no. 1, Mar. 2019, doi: 10.1186/s12913-019-4012-6.

[13] A. De Brún et al., “Understanding clinicians’ decisions to offer intravenous thrombolytic treatment to patients with acute ischaemic stroke: A protocol for a discrete choice experiment,” BMJ Open, vol. 4, no. 7, p. e005612, Jul. 2014, doi: 10.1136/bmjopen-2014-005612.

[14] W. J. Meurer, J. J. Majersik, S. M. Frederiksen, A. M. Kade, A. M. Sandretto, and P. A. Scott, “Provider perceptions of barriers to the emergency use of tPA for Acute Ischemic Stroke: A qualitative study,” BMC Emerg. Med., vol. 11, May 2011, doi: 10.1186/1471-227X-11-5.

[15] N. Moloczij, I. Mosley, K. M. Moss, K. L. Bagot, C. F. Bladin, and D. A. Cadilhac, “Is telemedicine helping or hindering the delivery of stroke thrombolysis in rural areas? A qualitative analysis,” Intern. Med. J., vol. 45, no. 9, pp. 957–964, Sep. 2015, doi: 10.1111/imj.12793.

[16] Y. Xiao, “Artifacts and collaborative work in healthcare: Methodological, theoretical, and technological implications of the tangible,” J. Biomed. Inform., vol. 38, no. 1, pp. 26–33, Feb. 2005, doi: 10.1016/j.jbi.2004.11.004.

[17] C. Donnellan, S. Sweetman, and E. Shelley, “Implementing clinical guidelines in stroke: A qualitative study of perceived facilitators and barriers,” Health Policy (New. York)., vol. 111, no. 3, pp. 234–244, Aug. 2013, doi: 10.1016/j.healthpol.2013.04.002.

[18] S. Sandhu et al., “Integrating a machine learning system into clinical workflows: Qualitative study,” J. Med. Internet Res., vol. 22, no. 11, Nov. 2020, doi: 10.2196/22421.

[19] S. Castagno and M. Khalifa, “Perceptions of Artificial Intelligence Among Healthcare Staff: A Qualitative Survey Study,” Front. Artif. Intell., vol. 3, p. 84, Oct. 2020, doi: 10.3389/frai.2020.578983.

[20] D. Roosan et al., “Feasibility of population health analytics and data visualization for decision support in the infectious diseases domain,” Appl. Clin. Inform., 2016, doi: 10.4338/ACI-2015-12-RA-0182.

[21] J. Ross, F. Stevenson, R. Lau, and E. Murray, “Factors that influence the implementation of e-health: A systematic review of systematic reviews (an update),” Implementation Science, vol. 11, no. 1. BioMed Central Ltd., pp. 1–12, Oct. 26, 2016, doi: 10.1186/s13012-016-0510-7.

[22] S. Crowe, S. Turner, M. Utley, and N. J. Fulop, “Improving the production of applied health research findings: Insights from a qualitative study of operational research,” Implement. Sci., vol. 12, no. 1, pp. 1–12, Sep. 2017, doi: 10.1186/s13012-017-0643-3.

[23] A. De Brún et al., “Factors that influence clinicians’ decisions to offer intravenous alteplase in acute ischemic stroke patients with uncertain treatment indication: Results of a discrete choice experiment,” Int. J. Stroke, vol. 13, no. 1, pp. 74–82, Apr. 2018, doi: 10.1177/1747493017690755.

[24] S. Crowe et al., “Combining qualitative and quantitative operational research methods to inform quality improvement in pathways that span multiple settings,” BMJ Qual. Saf., vol. 26, no. 8, pp. 641–652, Aug. 2017, doi: 10.1136/bmjqs-2016-005636.

[25] H. Kareemi, C. Vaillancourt, H. Rosenberg, K. Fournier, and K. Yadav, “Machine Learning Versus Usual Care for Diagnostic and Prognostic Prediction in the Emergency Department: A Systematic Review,” Acad. Emerg. Med., vol. 28, no. 2, pp. 184–196, Feb. 2021, doi: 10.1111/acem.14190.

[26] N. J. Fulop et al., “Explaining outcomes in major system change: A qualitative study of implementing centralised acute stroke services in two large metropolitan regions in England,” Implement. Sci., vol. 11, no. 1, pp. 1–13, Jun. 2016, doi: 10.1186/s13012-016-0445-z.

[27] C. M. Johnson, T. R. Johnson, and J. Zhang, “A user-centered framework for redesigning health care interfaces,” Journal of Biomedical Informatics, vol. 38, no. 1. J Biomed Inform, pp. 75–87, Feb. 2005, doi: 10.1016/j.jbi.2004.11.005.

[28] L. Boland, A. Kothari, C. McCutcheon, and I. D. Graham, “Building an integrated knowledge translation (IKT) evidence base: Colloquium proceedings and research direction,” Health Research Policy and Systems, vol. 18, no. 1. BioMed Central Ltd., pp. 1–7, Jan. 20, 2020, doi: 10.1186/s12961-019-0521-3.

[29] T. Nguyen et al., “How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field,” Heal. Res. Policy Syst., vol. 18, no. 1, pp. 1–20, Mar. 2020, doi: 10.1186/s12961-020-0539-6.