The study aims to analyze the previous literature on government initiatives to implement health information systems (HISs).
Proknow-C (Knowledge Development Process-Constructivist) was used in the selection of the literature and in the bibliometric and systematic analysis.
The research identified a portfolio composed of 33 articles aligned with the research theme and with scientific recognition, as well as periodicals, authors, papers and keywords that stood out the most. Amongst the government initiatives in the 24 identified countries, England has been the most studied nation, and there is a certain prominence of research arising from developed countries. Electronic health records (EHRs) have been the most explored technology. Efficiency and safety of health care delivery, integration of information and among health organizations, cost reduction and economicity are the most expected benefits from government programs. The difficulties found are related to the broader context in which the system is inserted, to the management of the program, to technology itself and to individuals. The most emphasized difficulties identified in most countries were previous context marked by a lack of standardization/interoperability, acceptance of providers and users and project financing. The findings of the present article provide a theoretical framework for future studies, in addition to yielding a replicable process for future use.
This research may be considered original as it analyzes – through a constructivism-structured process (Proknow-C) – the phenomenon under investigation by gathering bibliometric and systematic review data concomitantly. The countries and technologies reported emerge from the process itself.
Keywords: · Health information system · Health information technology · Health programs · Health management · Implementation · Assessment.
The world scenario, characterized by frequent changes in the social, economic, political and technological spheres, directs organizations to constantly reviewing their roles and processes. Without being an exception, institutions related with public health end up being deeply influenced by the environment and see themselves obliged to update their management policies and practices. The perspective of such institutions, which used to be devoted to assistance and devoid of the obligation to present management evidence, changed and they are currently on the path toward an entrepreneurial orientation, guided by performance, results and transparency (Lima, Schramm, Coeli & Silva, 2009; Botega, Andrade & Guedes, 2020).
The complexity of these organizations demands the generation, processing and availability of an expressive volume of information, which is essential for their processes and make up subsidies for decision-making in public health policies (Cresswell, Bates & Sheikh, 2013). These information include those referring to patients' health records (electronic health records), medical and nursing practices, management of clinical environments, knowledge and learning management, organization of administrative practices in the environment where the health care takes place, among others (Marin, 2010). By obtaining, processing and making them available for multiple stakeholders, an effective assessment is enabled not only in terms of administrative and operational efficiency, but of the greater purpose, i.e. the well-being of society (Morrison, Robertson, Cresswell, Crowe & Sheikh, 2011).
Within such context, government initiatives to implement health information systems (HISs) through information technology (IT) have been taking place internationally in countries with different development levels in order to improve patients' health and the quality and efficiency of health care services (Ludwick & Doucette, 2009; Rozenblum et al., 2011; Sligo, Gauld, Roberts & Villa, 2017; Haried, Claybaugh & Dai, 2019). As for these countries, we mention Germany (Deutsch, Duftschmid & Dorda, 2010), the United States of America (Payne, Detmer, Wyatt & Buchan, 2010), England (Sheikh et al., 2011), Denmark (Aanestad & Jensen, 2011), Australia (Morrison et al., 2011), Greece (Katehakis, Halkiotis & Kouroubali, 2011), Canada (McGinn et al., 2012), India (Aanestad, Jolliffe, Mukherjee & Sahay, 2014), Lithuania (Vedluga & Mikulskiene, 2017), Brazil (Mussi, do Valle Pereira, de Oliveira Lacerda & dos Santos, 2018), among others.
These initiatives are commonly characterized as complex programs or megaprojects (Price, Green & Suhomlinova, 2018), as they occur on a regional or national scale (large-scale) and involve the coordination of multiple stakeholders, who usually have distinct interests and perspectives (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). Given the increasing frequency of implementation of large-scale HIS, researchers have reported the need to maximize knowledge and understanding of the perspectives utilized in such implementation (Cresswell & Sheikh, 2009; Cresswell, Williams & Sheikh, 2020b). A few scholars have been arguing (Ross, Stevenson, Lau & Murray, 2016; Sligo et al., 2017; Klecun et al., 2019) that there is a broad previous literature on punctual implementation of HIS and on local and organizational coverage of HIS. Research that addresses implementations with a broader comprehensiveness, i.e. large-scale, is still found, however in a smaller number, especially those of literature review, which is the case of the present proposal.
Therefore, this article aims to fill the literature gap identified, contributing to the selection and bibliometric and systematic analysis of a bibliographic portfolio about the phenomenon through a constructivism-structured process – Proknow-C (Knowledge Development Process-Constructivist) (Ensslin, Mussi, Chaves & Demetrio, 2015; Carvalho et al., 2020). As indicated by Carvalho et al. (2020), the selection of relevant literature through a structured process is a critical step in the development of systematic reviews.
The previous literature reviews on programs for implementation and assessment of large-scale HIS usually aim to describe the program focusing on one or more specific countries (e.g. Ludwick & Doucette, 2009; Price, Green & Suhomlinova, 2018) and on specific health information technologies. In addition, these studies are not structured around the literature selection and bibliometric and systematic analysis in an integrated way through a bibliographic portfolio.
Thus, the present research provides a broader focus and contributes to the previous literature in three different ways: (1) by integrating concomitantly bibliometric and systematic review data of published literature, (2) by analyzing international initiatives of implementation of large-scale health information technologies without predefining technologies or specific countries, which will emerge from the selection process itself and (3) by making use of a structured review process from a constructivist perspective (Proknow-C).
Therefore, we intend to answer the following research issue: What is the state of art in the qualified literature about the implementation and assessment of large-scale HISs? More specifically, we intend to (1) select a bibliographic portfolio composed of a relevant, scientific bibliography aligned with the established theme; (2) to carry out the bibliometric analysis of this portfolio, exploring variables of the selected articles and their references; and (3) to carry out the systematic analysis of the portfolio by exploring the predominant countries identified in our research, the most studied technologies, implementation approaches, expected benefits and the difficulties faced by different countries.
This study may help and direct current and future investigation processes with the same nature. The research process and the main findings may be used by different researchers aiming to develop studies on the issue and may be used as a guide for practitioners, including those involved in initiatives of implementation of large-scale HISs.
In the upcoming section, HISs and the characteristics of their large-scale implementation will be described. In section 3, we will present the methodological procedures adopted, as well as the research framework. Subsequently, the application of Proknow-C and the outcomes of such method will be presented. Then, in the last section of this article, we present the conclusions and suggestions for future research.
2. Health information systems
The purpose of HIS is to produce and organize information and knowledge generated and used in the health care area in order to support the planning, the improvement and the decision-making process of the multiple actors involved in related processes (Lippeveld, Sauerborn & Bodart, 2000). The use of HIS can improve not only the health of individuals but also the performance of health care providers, providing increased levels of quality, financial efficiency and greater participation of patients in the care of their own health (Blumenthal & Tavenner, 2010).
Several acronyms have emerged over time to designate electronic HISs, such as electronic health record (EHR), electronic medical records (EMR), health information exchange (HIE), computerized physicians order entry (CPOE), hospital information system and telemedicine/telehealth/e-health (Blumenthal, DesRoches, Donelan, Rosenbaum & Ferris, 2006; Jha, Doolan, Grandt, Scott & Bates, 2008; Rosenthal, Seeman & Gibson, 2005; Lakbala & Dindarloo, 2014; Petroudi & Giannakakis, 2011; Spil, LeRouge, Trimmer & Wiggins, 2009).
Considering the national scope of implementation of such technologies, this sort of venture comprises the government definition of policies and standards that encourage the convergence of public and private interests in the development of an effectively functional national system (Coiera, 2009). Coiera (2009) still proposes a typology for regulatory models of implementation of large-scale HIS (top-down, middle-out, bottom-up), categorizing them according to the level of influence of government authority and the level of autonomy experienced by provider institutions in the process of developing, implementing and using the systems. While the top-down approach is characterized by a centralized management accomplished by the government, in the bottom-up approach the health care institutions make their own decisions about the system to be implemented, following the minimum interoperability standards. In an intermediate way, the middle-out approach combines elements of the other two approaches (Coiera, 2009, Morrison et al., 2011).
The implementation of electronic HISs at the national level has been considered a complex process full of challenges (Greenhalgh, Morris, Wyatt, Thomas & Gunning, 2013; Rippen, Pan, Russell, Byrne & Swift, 2013; Cresswell & Sheikh, 2013, Cresswell et al., 2020a, b). Technical, human, social and organizational problems are frequently reported, which compromise the efficiency and effectiveness of these initiatives (Sheikh et al., 2011; Greenhalgh et al., 2010; Murray et al., 2011; Peute, Aarts, Bakker & Jaspers, 2010; Spetz & Keane, 2009; Warth & Dyb, 2019), which – in turn – involve several interested parties necessary for the success of the implementation.
The actors involved in large-scale implementation programs comprise the government, health organizations, health professionals, patients, IT professionals and IT companies (McGinn et al., 2012; Sheikh et al., 2011; Rozenblum et al., 2011; Haried, Claybaugh & Dai, 2019). The wide range of interrelations among these actors generates a complex network of relationships, impairing the alignment of mutual interests (Greenhalgh et al., 2013; Greenhalgh et al., 2010; Robertson et al., 2010; Klecun et al., 2019; Cresswell et al., 2020). Amongst the several difficulties found in the implementation of large-scale HISs, the following stand out: (1) slower processes with a narrower scope than planned; (2) low degree of adaptability to local demands; (3) lack of alignment with work practices and users; and (4) negative impacts on the quality and effectiveness of patient care (Sheikh et al., 2011; Murray et al., 2011; Peute et al., 2010; Spetz & Keane, 2009; Rippen et al., 2013; Robertson et al., 2010; Cresswell & Sheik, 2013).
Within this context, it is paramount that the implementations of HIS are assessed and that the relevant aspects of successful and unsuccessful initiatives are identified (Sligo et al., 2017). This article, through the methodological procedures described in the upcoming section, aims at selecting a bibliographic portfolio with scientific recognition that addresses global initiatives of implementation of large-scale HISs and at carrying out the analysis.
3. Methodological procedures
This study presents a literature review based on the constructivist perspective, utilizing a process known as Proknow-C (Ensslin et al., 2015). The first versions of Proknow-C appeared in 2007. Since then, it has been continuously improved and has aggregated features that explore the several possibilities of identification and analysis of bibliographic portfolios (Ensslin, Dezem, Dutra, Ensslin & Somensi, 2018). Proknow-C was chosen due to its constructivist orientation and its detailing and methodological rigor, in addition to its increasing applications (Ensslin et al., 2018; Carvalho et al., 2020) since it was developed. Every data collection and data analysis process of this research was guided by this instrument.
Proknow-C consists of the following macro steps: (1) selection of a portfolio of scientific articles aligned with the research topic and with scientific relevance; (2) analysis of the portfolio through bibliometric criteria; (3) systematic analysis of the content of the portfolio from specific perspectives; and (4) definition of the research issue and aim of the scientific research (Ensslin, Ensslin, Imlau & Chaves, 2014).
The knowledge construction process begins with the selection of the bibliographic portfolio. This stage is intended to produce a research framework related to the phenomenon under investigation. The outcome of this stage is the set of articles (bibliographic portfolio) considered relevant by researchers and aligned with the research theme (Ensslin, Mussi, Chaves & Demetrio, 2015).
For the definition of the bibliographic portfolio, the starting point is the construction of a gross basis of articles. Initially, the research axes are defined, which represent the subthemes of the research. Through these axes, the parameters to be followed during the process of structuring the theme are defined, represented by a set of keywords. Keywords are arranged through a Boolean expression, according to the research axes to which they are related.
Subsequently, the databases where the search for scientific articles will be carried out are identified. The chosen databases must be aligned with the research theme, considering their relevance in the explored field of knowledge. Such representativeness can be evaluated through the number of articles resulting from the search through the defined Boolean expression. Additionally, filters are applied to the search to limit, for instance, a certain period and the types of document to be identified. The initial selection of articles must be validated by comparing the keywords used for the search with those presented in the articles found (adhesion test). If necessary, the search may be repeated including new research terms, which must be related to the research axes.
Then, the gross basis of articles goes through a refinement process with the application of some sequential filters: (1) elimination of redundant articles; (2) alignment of the title with the research theme; (3) scientific relevance, which corresponds to the number of citations of the article; (4) alignment of the abstract with the research theme; and (5) alignment of the full text with the research theme.
In the present research, in order to analyze the alignment of the articles with the research topic, the following inclusion criterion was defined: empirical or theoretical articles (i.e. that explore cases of the empirical literature) that portray, analyze or discuss programs of implementation and/or assessment of large-scale health information technologies carried out or supported by government initiatives. For these articles, the following exclusion criteria were defined: (1) studies focused on a single aspect of a HIS, such as usability; (2) planning, pre-implementation, or initial phase processes for the implementation of HISs; (3) assessment of the level of adoption of HIS in a given region and/or country; (4) eminently technical descriptions of HIS with little or no approach to implementation and/or assessment; and (5) studies focused on the description of approaches and/or methodological paradigms to investigate the implementation and/or assessment of HIS.
After establishing the bibliographic portfolio, the bibliometric analysis process begins (Van Raan, 2003), which consists in the survey and quantitative evidence of key information in the set of articles that make up the portfolio. In this stage, we analyze the distinctive variables of the selected articles as well as their references; more specifically, the scientific relevance of the periodicals, the relevance of the articles, the relevance of the authors and the most used keywords (Lacerda, Ensslin & Ensslin, 2014, Ensslin et al., 2015).
The next step, known as systematic analysis, consists in analyzing the content of the portfolio based on the variables of interest to the researcher (Ensslin et al., 2018). The framework used in this research (Section 3.2) describes the key variables of the bibliometric and systematic analyses.
The described steps were carried out with the support of two different applications: EndNote X7.1 (Thomson Reuters, 2013) and Zotero 4.0 (Zotero, 2019). Endnote (Thomson Reuters, 2013) is a reference management software package; through its use, the identification of repeated articles in the portfolio was carried out, in addition to listing authors, utilized keywords and periodicals. This software was also used to make abstracts and full texts available for reading. Zotero 4.0 is a reference management software program to manage bibliographic data (Zotero, 2019); through its use, it was possible to identify the number of citations registered in Google Scholar not only for each article of the portfolio, but also for their references.
3.2 Research framework
The analysis variables are graphically represented in Figure 1, which illustrates the framework that guided the data analysis of this research. Based on the selection of a bibliographic portfolio representative of the research theme, the respective analyses were accomplished.
The bibliometric analysis involved: (1) the analysis of the relevance of the periodicals where the articles of the bibliometric portfolio were published, as well as the bibliographic references in each of the articles of this portfolio to identify and characterize those with greater relevance to the research theme; (2) analysis of the scientific recognition of the bibliographic portfolio articles and their references, considering their number of citations; (3) identification of authors and their relevance to the research topic of the portfolio and to the references; and (4) identification of the most frequent keywords.
The systematic analysis was carried out based on the analysis of the articles of the bibliographic portfolio considering the following variables: (1) countries where the initiatives for the implementation of the prevailing HIS took place; (2) the most studied health IT; (3) implementation approach adopted (top-down, bottom-up, middle-out); (4) expected benefits from national programs; and (5) main difficulties/barriers faced by the countries.
4. Application of Proknow-C and results
The steps of the Proknow-C application process referring to the selection of the bibliographic portfolio are detailed in Section 4.1 and presented in Figures 2 and 3. The bibliometric and systematic analysis are described in Section 4.2 and 4.3, respectively.
4.1 Selection of the bibliographic portfolio
4.1.1 Definition of research axes, keywords and Boolean equation
Three research axes representative of the theme analyzed were defined (Figure 2). The first axis represents the analysis process, i.e. implementation and assessment. The second axis elucidates the object to be implemented or evaluated, i.e. HISs. The third and last axis indicates the scope of the process, i.e. large-scale.
For each established research axis, guiding keywords were defined (Figure 2) based on the adhesion tests required by the method, gradually making them more aligned with the research theme. The combination of the three sets of keywords, representative of each of the three research axes, resulted in 144 combinations.
The Boolean equation that represents such combinations was applied to Scopus, IsiKnowledge, Ebsco, PubMed and ProQuest databases, limited according to the following criteria: (1) time horizon: publications from 2008 to 2020; (2) search fields: titles, abstracts, keywords; (3) type of publication: scientific articles; (4) source of publication: scientific periodicals; and (5) publication language: English. The search activities were carried out in November 2020 and yielded a total of 7,265 articles, which correspond to the gross basis of articles.
4.1.2 Filtering the gross basis of articles
The first action related to filtering the gross basis of articles was the identification and exclusion of 1,567 duplicated publications, which yielded a total of 5,698 articles to be analyzed according to the alignment of their titles to the research theme. In this step, the titles of the 5,698 articles were evaluated and, according to the inclusion/exclusion criteria (Section 3), 5,447 articles had to be eliminated. Thus, the basis of scientific articles, not repeated and aligned with the research theme through the titles, was composed of 251 articles. The procedure described so far is summarized in Figure 2.
In the next step, where the scientific relevance of the articles is evaluated, we verified the number of citations presented by each of the 251 articles according to Google Scholar. We determined a representativeness of 90% of the citations of these articles, which means that we only selected articles that had at least 40 citations. This set was composed of 57 articles and called Repository K (with title alignment and verified scientific recognition).
The other 194 articles of the portfolios with title alignment and not selected in this step compose the so-called Repository P (i.e. with title alignment but without verified scientific recognition). These articles were reassessed in a subsequent stage.
In the next stage, the 57 articles with scientific recognition and aligned titles were evaluated in terms of alignment of their abstracts with the research theme. Again, the analysis was guided by inclusion and exclusion criteria (Section 3). Reading the abstracts of the 57 articles (Repository K) led to the exclusion of 33 articles. Thus, the databank of non-repeated articles with title and abstract aligned with the research theme and with verified scientific relevance – called Repository A – comprised 24 articles. Additionally, we identified a set of 68 authors in the 24 selected articles, making up the author databank of Repository A.
At this point, the process retrieved the 194 articles from Repository P (with title alignment and without verified scientific recognition), which were submitted to a reassessment process. The aim of such reassessment was to identify eventual articles that could be incorporated into the bibliographic portfolio. The 194 articles were segregated according to the following filters: (1) articles published between 2019 and 2020; thus, for being recent, they still lack scientific recognition; and (2) articles published before 2016 by authors pertaining to the author databank of Repository A, i.e. authors who had already published papers with verified scientific recognition.
Hence, 71 articles were identified according to filter (1) and other 16 articles consistent with filter (2). The resulting 87 articles were submitted to the alignment criteria according to their abstracts; 14 aligned articles were identified, forming Repository B (non-repeated articles with aligned abstracts and potential scientific recognition). Then, they were incorporated into the articles of Repository A, constituting the so-called Repository C, with 38 non-repeated articles aligned with the research theme considering their titles and abstracts.
Subsequently, the 38 articles were fully read and five had to be eliminated for being misaligned according to the inclusion and exclusion criteria (Section 3). Thus, the primary portfolio was created, which comprises 33 non-repeated articles aligned with the research theme according to their titles, abstracts and full texts. Figure 3 illustrates the described procedures for the composition of the primary bibliographic portfolio.
The next step consisted of testing the representativeness of the primary bibliographic portfolio. The purpose is to identify if, within the references of the articles from this portfolio, there are other relevant articles with scientific recognition that should be incorporated into the portfolio itself.
4.1.3 Representativeness test of the primary portfolio
The representativeness test consists in listing all references from the primary portfolio's articles applying the same filters used to define the initial portfolio. The analysis of the 33 articles pertaining to such portfolio identified 2,020 references. Thus, 1,534 references had to be eliminated either because they were duplicated, because they did not correspond to the articles, or because they were published outside the time frame (2008 to 2020), which resulted in 486 articles. The analysis of the title excluded another 341 articles, while another 17 – despite being aligned – were already part of the primary portfolio, being also eliminated from this analysis. Therefore, the analysis yielded a total of 128 articles stemming from the references of the primary portfolio's articles, which were not duplicated, not part of the primary portfolio and aligned with the research theme according to their titles.
The 128 articles were then classified by the total number of citations on Google Scholar. We established a cut-off line representing the articles responsible for 80% of the citations. Then, the criteria of inclusion and exclusion described in Section 3.1 were applied to the abstract of the 29 resulting articles. In this assessment, however, we identified no article that could be incorporated into the primary bibliographic portfolio. The main exclusion factor was the fact that most of these articles were classified as theoretical and not directly related to empirical research. Thus, during this latest review, no article was added to the final bibliographic portfolio.
Hence, the selection of articles for the bibliographic portfolio identified 33 studies aligned with the research theme according to their titles, abstracts and full texts and with scientific recognition. These articles are listed in Table 1.
Afterward, we carried out the bibliometric analysis of the bibliographic portfolio and the articles in their references.
4.2 Bibliometric analysis
4.2.1 Relevance of the periodicals
The analysis of the periodicals where the articles and references pertaining to the bibliometric portfolio were published identified those with greater relevance to the research topic. The most representative periodical was the International Journal of Medical Informatics (four articles in the portfolio), followed by BMJ– British Medical Journal and JAMIA–Journal of the American Medical Association (with three articles each), Journal of Health care Engineering, BMC–Medical Informatics and Decision-Making, Health Policy and Journal of Information Technology (with two articles each), as presented in Figure 4a.
Similarly, we analyzed the periodicals listed in the references of the portfolio's articles. For such, all references identified as scientific articles published between 2008 and 2020 and aligned with the research theme were taken into consideration.
Like the assessment of the composition of the periodical databank of the bibliographic portfolio, the analysis of the periodicals where the articles of the references were published enabled us to emphasize the following: JAMIA–Journal of the American Medical Informatics Association (12 articles), International Journal of Medical Informatics (nine articles), Health Affairs (eight articles), Journal of the Association of Information Systems (seven articles), BMJ–British Medical Journal (6 articles) and New England Journal of Medicine (five articles). Subsequently, other two periodicals stood out, namely BMC–Medical Informatics and Decision-Making and Journal of Innovation in Health Informatics (with four articles each). The results are shown in Figure 4b.
Four periodicals stood out (both in the articles and in references): International Journal of Medical Informatics, Journal of the American Medical Informatics Association (JAMIA), Medical Informatics and Decision-Making (BMC) and British Medical Journal (BMJ), emphasizing the relevant connection of these periodicals with the research topic.
Another relevant issue to be mentioned herein about the periodicals where the articles were published is the impact factor. In this research, we analyzed JCR – Journal Citation Reports (JCR) and Scientific Journal Rankings (SJR), verified through the databases Web of Science (JCR) and Scopus (SJR). Figure 5a presents the number of articles of each periodical of the portfolio, as well as its corresponding JCR impact factor. It is noticed that the periodical with the highest JCR impact factor is the British Medical Journal (BMJ) (JCR = 30.31), which is the second journal with the highest number of articles of the portfolio (3 articles).
The periodicals with the 5 largest JCR impact factors are British Medical Journal (BMJ) (30.31), Canadian Medical Association Journal (CMAJ) (7.74), Journal of Strategic Information Systems (5.23), Journal of Medical Internet Research (5.03) and Journal of General Internal Medicine (4.59).
Figure 5b presents the number of articles of each periodical of the portfolio, as well as their corresponding SJR impact factor. The periodicals with the five highest SJR impact factors are Information Systems Research (3.24), Journal of Strategic Information Systems (2.70), Journal of the Association of Information Systems (2.46), British Medical Journal (BMJ) (2.05) and Social Science and Medicine (1.94). The analysis of the graphs in Figure 5 does not indicate a direct relationship between the number of articles of the bibliographic portfolio and the impact factor of the periodicals.
4.2.2 Scientific recognition of the articles and their references
To demonstrate the scientific recognition of the articles of the bibliographic portfolio and their references, we analyzed the number of citations related to each of them. For such, the software Zotero was used, which identifies the number of citations of each article registered in Google Scholar. The number of citations attributed to the articles of the bibliographic portfolio is registered in Figure 6a.
The outcome of the relevance analysis of the 33 articles highlighted some studies, namely the ones developed by Ludwick & Doucette (2009) (770 citations), Sheikh et al. (2011) (317 citations) and Greenhalgh et al. (2008) (307 citations). The first article relates to a systematic literature review focused on the analysis of empirical cases of implementation of HISs in several countries. The second is a qualitative longitudinal research based on multiple case studies and presents perceptions on the first processes of the implementation of HISs linked to the National Health Service (NHS) by the English government. The third evaluates through multiple case studies the perceived impacts on institutions that are pioneers in patients' EHRs in England.
Regarding the articles mentioned in the references of the bibliographic portfolio, 128 articles with title alignment were analyzed. From those, we highlight Damschroder et al. (2009) (5,681 citations), Blumenthal and Tavenner (2010) (2,231 citations) and Petter, DeLone and McLean (2008) (5,681 citations). The study by Damschroder et al. (2009) provides a framework to guide research on implementation of HISs. Blumenthal and Tavenner (2010) present proposals for the application of HISs encourage by the HITECH initiative proposed by the American government. Petter, DeLone and McLean (2008) present the results of a literature review about successful information systems, consolidating the utilized measures to assess success and analyzing their relationships in the individual and organizational context.
The analysis of the prominent articles of the portfolio and references indicates an alignment with three different research axes: assessment and implementation; computerized health systems and large-scale processes. Figure 6b identifies the 20 most cited articles among the references of the bibliographic portfolio.
4.2.3 Identification of authors and their relevance to the research theme
In order to identify the most emphasized authors within the portfolio and considering the set of references, we verified the number of articles of each author in each of these two sets. The analysis of the authors responsible for the 33 articles of the portfolio, which is composed of 117 researchers, emphasizes A. Sheikh (participating in eight articles), K. Cresswell (responsible for seven articles) and A. Robertson, W. L. Currie and Z. Morrison (who participated in three articles each), conforming to Figure 7a.
We also identified the authors of the 128 articles pertaining to the references, which were aligned with the research topic. This databank of authors, composed of 346 researchers, highlights the participation of A. Sheikh in 12 articles, K. Cresswell in nine and T. Greenhalgh in six, which is illustrated in Figure 7b.
Figure 7c allows the observation of the emphasized participation of some authors in the articles of the portfolio, as well as in the references of these articles. The upper quadrant on the right allows the identification of authors highlighted by the volume of articles not only in the portfolio, but also in the references.
Regarding the authors stressed in the bibliometric analysis, we highlight that Aziz Sheikh, who is responsible for eight articles in the portfolio and other 12 in the references, is professor and director of the Usher Institute, linked to the University of Edinburgh. His research interests comprise studies of asthma, health inequalities and use of health IT. Kathrin Cresswell, with seven papers in the portfolio and other nine articles in the references, is a psychologist and social scientist, who is also the director of innovation at the Usher Institute, linked with the University of Edinburgh. Operating in the field of information technologies for clinical studies, her main research interests are the use of health IT, sociotechnical analysis, qualitative methods, health innovation and information systems. Trisha Greenhalgh, highlighted for having two articles in the portfolio and six in the references, is a physician and professor of primary care health sciences at the University of Oxford. She dedicates to research related to the integration between social sciences and medicine, emphasizing health care organizations. Ashish K. Jha stood out for the citations of his work in the references of the bibliographic portfolio (five articles). He is professor of global health and director of Harvard Global Health Institute, linked with the Harvard School of Public Health, being active in health policies.
4.2.4 Keywords used in the bibliographic portfolio
The use of the EndNote X7.1 software (Thomson Reuters, 2013) allows the identification of 100 keywords utilized to characterize the articles of the bibliographic portfolio; the most frequent ones are presented in Figure 8.
Amongst the thirteen keywords defined based on the axes of the research, seven are to be found in this list. Such fact corroborates the alignment between keywords and the research topic, which validates its use in the search process. The most repeated keywords are implementation and EHR.
4.3 Systematic analysis
The aim of the systematic analysis was to highlight in the 33 articles of the portfolio the countries and technologies studied, the implementation approaches adopted, the expected benefits and the difficulties faced. We identified a set of 24 countries that were the focus of articles (Table 2) with a predominance of research in developed (18) in relation to developing countries (6).
The most widely studied national programs for the implementation of health IT are those in the United Kingdom, as they represent 22 articles of the portfolio. In this context, the British initiative to implement a national system of EHRs linked to the national IT program (National Program for Information Technology – NpfIT) is the most approached research issue. Considered one of the most significant initiatives ever documented, it involved a total of 168 hospitals and 73 health care environments, corresponding an investment of £ 12.7 billion (Robertson et al., 2010). Such initiative, however, did not achieve the expected success, being discontinued in 2011 (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). Apart from the countries that make up the United Kingdom, the other most studied countries are the United States (eight articles) and Canada (seven articles).
EHRs are the most studied technologies, which indicate the countries' search for integration and improvement of patient care. EHRs are defined as longitudinal medical records of patient health information, utilized as a basis for the integration of health care (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019). These records follow pre-established patterns that take into consideration the possibility to share the patient history with other health organizations geographically dispersed, providing an integrated view of the patient's health (Klecun, Zhou, Kankanhalli, Wee & Hibberd, 2019; Ludwick & Doucette, 2009).
Improvements in the quality of health services are among the benefits expected by the national programs of the 24 countries identified in this research. Through the implementation of HISs, the countries expect improvements in the efficiency/agility of patient care, safety of health care delivery and patient's safety, privacy and security of health information, increased patients'degree of choice over their health care, improvements in the access of citizens to health services, integration and sharing of information between different health organizations; advances in the definition of health indicators and public policies, cost reduction, more consistent and efficient use of health resources and reduction of the gap between supply and demand of health care.
Regarding the adopted approaches to implement those systems, we identified the three approaches proposed by Coeira (2009) in the set of countries analyzed in this study. Coeira (2009) points out that the top-down approach used by several national initiatives to implement HISs may be the main reason of their frequent failures. Among the problems identified, one highlights the fact that a unique, standardized and imposed (by the government) solution does not apply to the variety of local problems and asymmetries. Thus, the author defends the use of the middle-out approach as the most appropriate way for governments to achieve goals related to national policies, preserving the interests of involved institutions and providing solutions to local asymmetries.
In line with such perspective, the outcomes of this study show that the adopted implementation approach is among the difficulties identified in the different countries. In the case of England, for instance, previous research (Robertson et al., 2010, Morrison et al., 2011) indicated that one of the causes of failure of the national IT program (NPfIT) lies in the top-down approach, which generated delays in the process and user frustrations. In the case of the United States, the bottom-up approach was unable to stimulate the adoption of technology by large providers (Wilson & Khansa, 2018). Failures resulting from the approaches utilized seem to have affected the future directions of some countries. England and the United States, for example, have been directed more efforts toward the implementation of a middle-out approach (Fragidis & Chatzoglou, 2018, Price, Green & Suhomlinova, 2018).
In addition to the implementation approach adopted, it was evident that all countries – regardless of their level of development – faced a set of difficulties or barriers when implementing health information technologies nationally, which strengths the dimension and complexity of this sort of program. Of the articles analyzed, we identified reports of 34 main difficulties (Table 3).
When assessing the set of difficulties faced by the countries, the ones that stood out in more than half of this set are the previous context of lack of standardization and interoperability and the consequent difficulty in integrating systems, regions and health suppliers; resistance, skepticism, acceptance of health care providers and users; and project financing and budget constraints. It is important to highlight herein that most part of those difficulties are related to the management capacity of the programs – whether at the government or at the organizational level – and to the conciliation of the different needs of the several stakeholders involved.
In general, the difficulties identified are related to a wider context of system implementation (e.g. instabilities in the country's political and economic contexts, changes in market structures, suppliers and technologies), to the way the program is managed and conducted (e.g. project management, change management, implementation approach adopted), to technology itself (e.g. usability, performance, alignment with work practices) and to individuals (e.g. resistance, skepticism, acceptance of the system, previous experience with computers).
These findings corroborate other studies (Cresswell, Williams & Sheikh, 2020a, Greenhalgh et al., 2010) that emphasize that this sort of initiative involves a set of social, organizational and human transformations that must be considered to ensure that the benefits expected by the different stakeholders are, in fact, achieved. As pointed out by Price, Green & Suhomlinova (2018), national, effective IT systems require the ability to understand and manage complex and dynamic relationships of the several organizations involved.
5. Final considerations
The purpose of this article was to carry out the selection, bibliometric and systematic analysis of a bibliographic portfolio about the topic “implementation and assessment of large-scale health information systems” through a structured process (Proknow-C). The selection stage – based on a set of established criteria – identified a databank of 33 articles aligned with such research topic.
The bibliometric analysis allowed the identification of the periodicals International Journal of Medical Informatics, Journal of the American Medical Informatics Association (JAMIA), Medical Informatics and Decision-Making (BMC) and British Medical Journal (BMJ) as those with more publications about the research theme. These journals stood out for the expressive volume of articles in the bibliographic portfolio and in the references.
In the author databank identified in the portfolio and in the references, two authors stood out: Aziz Sheik, PhD (eight articles in the portfolio and 12 articles in the references) and Kathrin Cresswell, PhD (seven articles in the portfolio and nine in the references). Both are researchers at the Usher Institute, linked to the Edinburgh University.
In the selected portfolio, the ranking of academic relevance (carried out based on the number of citations) highlighted the studies accomplished by Ludwick & Doucette (2009) – with 770 citations on the Google Scholar platform –, Sheikh et al. (2011) with 317 citations and Greenhalgh et al. (2008) with 307.
The bibliometric analysis allowed for the identification of the most frequently used keywords in the selected articles, namely: Electronic Health Records and Implementation. In addition, the set of keywords identified in the articles corroborates the adhesion of the keywords initially used in the search process.
The systematic analysis indicated a set of 24 countries that were the focus of national initiatives for the implementation of health IT – it is observed, however, a prominence of studies carried out in developed countries. England is the mostly studied country. The most discussed technology identified in the articles is the EHR, which indicates that countries are aiming to provide patients with an integrated health care system. We observed a shift in the orientation of some countries, which initially used to adopt top-down and bottom-up implementation strategies, toward the middle-out approach.
Through the implementation programs of HISs, the different countries intend to improve the efficiency and safety of health care delivery, to integrate information and health organizations, to reduce costs and to provide resource savings. Improvements in service quality to citizens are expected by all countries.
In the pursuit of these benefits, national governments have been facing several difficulties. From the analyzed articles, 34 main difficulties emerged, which are related to a broader context of system implementation, to the management of the program, to individuals and technology itself. The difficulties related to resistance and acceptance of health organization and users; to the previous context of lack of standardization and interoperability; and to project financing and budget constraints were the most frequent ones identified in most part of the countries.
In general, the difficulties emphasize the complexity of implementing national systems of health IT; in some cases, they are underestimated by countries and end up compromising the success of these initiatives and the pursuit of expected benefits. An effective management of the program focused on integration and on a shared vision between the diversity and heterogeneity of those involved stands out as a basic requirement.
Among the theoretical implications, the results presented herein contribute to the creation of an updated and systematically constructed theoretical framework on the research topic. The findings add knowledge to literature on implementation and assessment of large-scale health IT, as well as on management of complex programs and projects. Additionally, the study also provides a replicable process, which can be used in its own update.
In portraying government initiatives from different countries, this research adds to managerial implications for government agencies and health institutions. The benefits expected from these initiatives are achieved through the effective management of the difficulties identified in the reported experiences. These difficulties may be translated into lessons to be learned for those involved in this sort of government program. Thus, the effective management of these programs has a direct impact on the quality of health care service and individuals' well-being.
Ultimately, this research has implications for public policies, indicating the need for the development of health national policies and e-health, which must support and be aligned with the implementation of large-scale health IT in order to enhance the expected benefits and minimize the barriers inherent in this kind of initiative. These benefits make a difference in the quality of the health care service provided to citizens and can contribute to atypical contexts, such as what is being experienced with the pandemic caused by COVID-19. These contexts require efficiency and safety of health care delivery, reliable and fast information and the integration of systems and among health institutions.
We reaffirm that the aim of this study is not to create a theoretical framework per se, but to contribute to the creation of knowledge through the provision of a theoretical basis about the research topic, which was elaborated through a rigorous, structured and systematic process. Despite following a structured process, the selection of articles may also be influenced by a few subjective factors inherent in researchers, which already indicates one of the limitations of this study. Another limiting factor is that the main difficulties faced by the countries when implementing their programs refer to the selected bibliographic portfolio. Thus, the fact that other difficulties have not been identified in this set of countries does not mean that they were not faced by them.
Based on the findings of this study, we identified a few suggestions for future research, namely: (1) the study of initiatives to implement HISs in developing countries in order to expand knowledge about these countries; (2) comparative studies between developing and developed countries; (3) analysis of strategies or actions adopted by countries to minimize the difficulties identified in this study; and (4) update or periodic replication of the process carried out by us to maintain a timely analysis on the subject.
|1||D. A. Ludwick, J. Doucette||Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries||2009|
|2||T. Greenhalgh et al.||Introduction of shared electronic records: Multi-site case study using diffusion of innovation theory||2008|
|3||A. Sheikh et al||Implementation and adoption of nationwide electronic health records in secondary care in England: Final qualitative results from prospective national assessment in “early adopter” hospitals||2011|
|4||A. Robertson et al.||Implementation and adoption of nationwide electronic health records in secondary care in England: Qualitative analysis of interim results from a prospective national assessment||2010|
|5||M. Aanestad, T. B. Jensen||Building nationwide information infrastructures in health care through modular implementation strategies||2011|
|6||K. M. Cresswell, D. W. Bates, A. Sheikh||Ten key considerations for the successful implementation and adoption of large-scale health information technology||2013|
|7||M. Tsiknakis, A. Kouroubali||Organizational factors affecting successful adoption of innovative eHealth services: A case study employing the FITT framework||2009|
|8||R. Rozenblum et al.||A qualitative study of Canada's experience with the implementation of electronic health information technology||2011|
|9||E. Deutsch, G. Duftschmid, W. Dorda||Critical areas of national electronic health record programs-Is our focus correct?||2010|
|10||C. Sicotte, G. Paré||Success in health information exchange projects: Solving the implementation puzzle||2010|
|11||W. L. Currie||Institutional isomorphism and change: The national programme for IT-10 years on||2012|
|12||T. L. Box et al.||Strategies from a nationwide health information technology implementation: The VA CART STORY||2010|
|13||T. Greenhalgh et al||Introducing a nationally shared electronic patient record: Case study comparison of Scotland, England, Wales and Northern Ireland||2013|
|14||K. M. Cresswell, A. Worth, A. Sheikh||Integration of a nationally procured electronic health record system into user work practices||2012a|
|15||C. A. McGinn et al.||Users perspectives of key factors to implementing electronic health records in Canada: A Delphi study||2012|
|16||Z. Morrison et al.||Understanding contrasting approaches to nationwide implementations of electronic health record systems: England, the USA and Australia||2011|
|17||K. Cresswell et al.||Anything but engaged: User involvement in the context of a national electronic health record implementation||2011|
|18||L. L. Fragidis, P. D. Chatzoglou||Implementation of a nationwide electronic health record (EHR): The international experience in 13 countries||2018|
|19||E. Deriel et al.||Success factors for implementing and sustaining a mature electronic medical record in a low-resource setting: A case study of iSanté in Haiti||2018|
|20||T. Vedluga and B. Mikulskiene||Stakeholder driven indicators for eHealth performance management||2017|
|21||N. Pouloudi, W. Currie, E. A. Whitley||Entangled stakeholder roles and perceptions in health information systems: A longitudinal study of the U.K. NHS N3 network||2016|
|22||M. Aanestad et al.||Infrastructuring work: Building a state-wide hospital information infrastructure in India||2014|
|23||W. L. Currie||Translating health IT policy into practice in the UK national health service||2014|
|24||K. M. Cresswell, A. Worth, A. Sheikh||Comparative case study investigating sociotechnical processes of change in the context of a national electronic health record implementation||2012b|
|25||D. G. Katehakis, S. Halkiotis, A. Kouroubali||Materialization of regional health information networks in Greece: Electronic health record barriers & enablers||2011|
|26||W. L. Currie, D. J. Finnegan||The policy-practice nexus of electronic health records adoption in the UK NHS: An institutional analysis||2011|
|27||T. H. Payne et al||National-scale clinical information exchange in the United Kingdom: Lessons for the United States||2011|
|28||C. Price, W. Green, O. Suhomlinova||Twenty-five years of national health IT: Exploring strategy, structure and systems in the English NHS||2019|
|29||E. Klecun et al.||The dynamics of institutional pressures and stakeholder behavior in national electronic health record implementations: A tale of two countries||2019|
|30||K. Cresswell, R. Williams, A. Sheikh||Developing and applying a formative assessment framework for health information technology implementations: Qualitative investigation||2020|
|31||T. H. Payne et al.||Status of health information exchange: a Comparison of six countries||2019|
|32||K. Wilson, L. Khansa||Migrating to electronic health record systems: A comparative study between the United States and the United Kingdom||2018|
|33||P. Burne||The introduction of electronic medical records in France: More progress during the second attempt||2018|
|United Kingdom (UK) (England, Scotland, Ireland and Wales)||1, 2, 3, 4, 6, 9, 11, 13, 14, 16, 17, 18, 21, 23, 24, 26, 27, 28, 29, 30, 31, 32||EHR, EMR, EPS, CDS, CPOE, HIE||Top-down3||√ sharing patient data to support the health care √ efficiency and safety of health care delivery √ quality of the health service √ cost reduction √ increased patients'degree of choice over their health care √ updating the national health service √ privacy and security of health information √ alignment of health care practices|
|Canada (CA)||1, 8, 9, 10, 15, 18||EHR, EMR, HIE||Top-down||√ efficiency of care √ patient safety √ quality of the health service √ improvements to the access of citizens to health services √ sharing patient data to support the health care √ more consistent and efficient use of health resources √ reduction of the gap between supply and demand of health care|
|United States (US)||1, 6, 12, 16, 18, 27, 31, 32||EHR, EMR, EPS, CART||Bottom-up3||√ sharing patient data to support the health care √ cost reduction √ quality of the health service √ efficiency of care √ safety of health care delivery √ privacy and security of health information √ use of data to define health indicators and public policies|
|Denmark (DK)||1,5,9,18||EHR, EMR, EPR||Middle-out||√ efficiency and agility of health care √ sharing patient data to support the health care √ increased patients'degree of choice over their health care √ integrated care|
|Australia (AU)||1, 9, 16||EHR, EMR||Middle-out||√ quality of the health service √ efficiency of care √ increased patients'degree of choice over their health care √ sharing patient data to support the health care √ cost reduction √ safety of health care delivery √ Continuity of care|
|Germany (DE)||9, 18||EHR||Bottom-up||√ efficiency of care √ quality of the health service √ increased patients'degree of choice over their health care|
|Sweden (SE)||1, 18||EHR, EMR||Bottom-up||√ safety and effectiveness of health care delivery|
|New Zealand (NZ)||1, 18||EHR, EMR||Middle-out||N/I4|
|Greece (GR)||7, 25||HIS, EHR||Top-down||√ quality of the health service √ more consistent and efficient use of health resources √ cost reduction √ sharing patient data to support the health care √ efficiency and agility of health care √ safety and effectiveness of health care delivery|
|India (IN)||22, 31||HIS, HIE||Bottom-up||√ sharing patient data to support the health care √ more consistent and efficient use of health resources √ efficiency of care √ use of data to define health indicators and public policies √ quality of the health service|
|Switzerland (CH)||18, 31||HER||Bottom-up||√ quality of the health service √ increased patients'degree of choice over their health care √ patient satisfaction|
|Haiti (HT)||19||EMR||N/I||√ to improve the access to information √ quality of the health service √ support the national HIV treatment program|
|South Korea (KR)||18||EHR||Middle-out||√ IT structure to integrate systems and the sharing of patient data|
|Lithuania (LT)||20||e-health||N/I||√ quality of the health service √ reduction of the administrative burden of the national health system by integrating the different types of e-health used in the country|
|Singapore (SG)||29||EHR||Top-down||√ integration of data and institutions √ quality of the health service|
|China (CN)||31||HIE||Top-down||√ quality of the health service|
|France (FR)||33||EMR||N/I||√ coordination and quality of the health √ cost reduction √ modernization of the health system|
O presente trabalho foi realizado com apoio do Conselho Nacional de Desenvolvimento Científico e Tecnológico – Brasil (CNPq) – Número do Processo 429497/2016-4.
Aanestad, M., & Jensen, T.B. (2011). Building nation-wide information infrastructures in healthcare through modular implementation strategies. The Journal of Strategic Information Systems, 20(2), 161-176.
Aanestad, M., Jolliffe, B., Mukherjee, A., & Sahay, S. (2014). Infrastructuring work: building a state-wide hospital information infrastructure in India. Information Systems Research, 25(4), 834-845.
Blumenthal, D., & Tavenner, M. (2010). The ‘meaningful use' regulation for electronic health records. New England Journal of Medicine, 363(6), 501-504.
Blumenthal, D., DesRoches, C.M., Donelan, K., Rosenbaum, S.J., & Ferris, T. (2006). Health information technology in the United States: The information base for progress. Princeton, NJ: Robert Wood Johnson Foundation.
Botega, L.A., Andrade, M.V., & Guedes, G.R. (2020). Perfil dos hospitais gerais do Sistema Único de Saúde. Revista de Saúde Pública, 54(81), 1-10.
Carvalho, G.D.G., Sokulski, C.C., Silva, W.V., Carvalho, H., G., Moura, R.V., … , Francisco, A.C., & Veiga, C.P. (2020). Bibliometrics and systematic reviews: A comparison between the Proknow-C and the methodi Ordinatio. Journal of Informetrics, 14(3), 101043.
Coiera, E. (2009). Building a national health IT system from the middle out. Journal of the American Medical Informatics Association, 16(3), 271-273.
Cresswell, K., & Sheikh, A. (2009). The NHS Care Record Service (NHS CRS): recommendations from the literature on successful implementation and adoption. Journal of Innovation in Health Informatics, 17(3), 153-160.
Cresswell, K., & Sheikh, A. (2013). Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. International Journal of Medical Informatics, 82(5), e73-e86.
Cresswell, K., Sheikh, A., Franklin, A.D., Krasuska, M., Nguyen, H.T., Hinder, S., … & Williams, R. (2020a). Theoretical and methodological considerations in evaluating large-scale health information technology change programmes. BMC Health Services Research, 20(477), 1-6.
Cresswell, K., Williams, R., & Sheikh, A. (2020b). Developing and applying a formative evaluation framework for health information technology implementations: qualitative investigation. Journal of Medical Internet Research, 22(6), e15068.
Cresswell, K.M., Bates, D.W., & Sheikh, A. (2013), “Ten key considerations for the successful implementation and adoption of large-scale health information technology”, Journal of the American Medical Informatics Association, 20(e1), e9-e13.
Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A., & Lowery, J.C. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science, 4(1), 50.
Deutsch, E., Duftschmid, G., & Dorda, W. (2010). Critical areas of national electronic health record programs—Is our focus correct? International Journal of Medical Informatics, 79(3), 211-222.
Ensslin, S., Ensslin, L., Imlau, J., & Chaves, L. (2014). Processo de mapeamento das publicações científicas de um tema: Portfólio bibliográfico e análise bibliométrica sobre avaliação de desempenho de cooperativas de produção agropecuária. Rev. Econ. Sociol. Rural [online], 52(3), 587-608.
Ensslin, L., Mussi, C.C., Chaves, L.C., & Demetrio, S.N. (2015). IT outsourcing management: The state of art recognition by a constructivist research process and bibliometrics. JISTEM-Journal of Information Systems and Technology Management, 12(2), 371-392.
Ensslin, L., Dezem, V., Dutra, A., Ensslin, S.R., & Somensi, K. (2018). Seaport-performance tools: an analysis of the international literature. Maritime Economics & Logistics, 20(4), 587-602.
Fragidis, L.L., & Chatzoglou, P.D. (2018). Implementation of a nationwide electronic health record (EHR) the international experience in 13 countries. International Journal of Health Care Quality Assurance, 31(2), 116-130.
Greenhalgh, T., Stramer, K., Bratan, T., Byrne, E., Mohammad, Y., & Russell, J. (2008). Introduction of shared electronic records: Multi-site case study using diffusion of innovation theory. Bmj, 337, a1786.
Greenhalgh, T., Stramer, K., Bratan, T., Byrne, E., Russell, J., & Potts, H.W. (2010). Adoption and non-adoption of a shared electronic summary record in England: a mixed-method case study. Bmj, 340, c3111.
Greenhalgh, T., Morris, L., Wyatt, J.C., Thomas, G., & Gunning, K. (2013). Introducing a nationally shared electronic patient record: case study comparison of Scotland, England, Wales and Northern Ireland. International Journal of Medical Informatics, 82(5), e125-e138.
Haried, P., Claybaugh, C., & Dai, H. (2019). Evaluation of health information systems research in information systems research: A meta-analysis. Health Informatics Journal, 25(1), 186-202.
Jha, A.K., Doolan, D., Grandt, D., Scott, T., & Bates, D.W. (2008). The use of health information technology in seven nations. International Journal of Medical Informatics, 77(12), 848-854.
Katehakis, D.G., Halkiotis, S., & Kouroubali, A. (2011). Materialization of regional health information networks in Greece: Electronic health record barriers & enablers. Journal of Healthcare Engineering, 2(3), 389-403.
Klecun, E., Zhou, Y., Kankanhalli, A., Wee, Y.H., & Hibberd, R. (2019). The dynamics of institutional pressures and stakeholder behavior in national electronic health record implementations: A tale of two countries. Journal of Information Technology, 34(4), 292-332.
Lacerda, R.T.D.O., Ensslin, L., & Ensslin, S.R. (2014). Research opportunities in strategic management field: a performance measurement approach. International Journal of Business Performance Management, 15(2), 158–174.
Lakbala, P., & Dindarloo, K. (2014). Physicians' perception and attitude toward electronic medical record. Springerplus, 3(1), 63.
Lima, C.R.D.A., Schramm, J.M.D.A., Coeli, C.M., & Silva, M.E.M.D. (2009) Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cadernos de saúde pública, 25, 2095-2109.
Lippeveld, T., Sauerborn, R., & Bodart, C. (2000). Design and implementation of health information systems. Geneva: World Health Organization.
Ludwick, D.A., & Doucette, J. (2009). Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics, 78(1), 22-31.
Marin, H.F. (2010). Sistemas de informação em saúde: considerações gerais. Journal of Health Informatics, 2(1), 20-24.
McGinn, C.A., Gagnon, M.P., Shaw, N., Sicotte, C., Mathieu, L., Leduc, Y., … & Légaré, F. (2012). Users' perspectives of key factors to implementing electronic health records in Canada: a Delphi study. BMC Medical Informatics and Decision Making, 12(1), 105.
Morrison, Z., Robertson, A., Cresswell, K., Crowe, S., & Sheikh, A. (2011). Understanding contrasting approaches to nationwide implementations of electronic health record systems: England, the USA and Australia. Journal of Healthcare Engineering, 2(1), 25-41.
Murray, E., Burns, J., May, C., Finch, T., O'Donnell, C., Wallace, P., & Mair, F. (2011). Why is it difficult to implement e-health initiatives? A qualitative study. Implementation Science, 6(1), 6.
Mussi, C.C., do Valle Pereira, C.D., de Oliveira Lacerda, R.T., & dos Santos, E.M. (2018). Pre-implementation evaluation of a nationwide information system for university hospitals: lessons learned from a study in Brazil. Behaviour and Information Technology, 37(3), 217-231.
Payne, T.H., Detmer, D.E., Wyatt, J.C., & Buchan, I.E. (2010). National-scale clinical information exchange in the United Kingdom: lessons for the United States. Journal of the American Medical Informatics Association, 18(1), 91-98.
Petroudi, D., & Giannakakis, N. (2011). New technologies in hospital information systems. Clinical Technologies: Concepts, Methodologies, Tools and Applications, 2029-2034. IGI Global.
Petter, S., DeLone, W., & McLean, E. (2008). Measuring information systems success: Models, dimensions, measures, and interrelationships. European Journal of Information Systems, 17, 236–263.
Peute, L.W., Aarts, J., Bakker, P.J., & Jaspers, M.W. (2010). Anatomy of a failure: a sociotechnical evaluation of a laboratory physician order entry system implementation. International Journal of Medical Informatics, 79(4), e58-e70.
Price, C., Green, W., & Suhomlinova, O. (2018). Twenty-five years of national health IT: exploring strategy, structure, and systems in the English NHS. Journal of the American Medical Informatics Association, 26(3), 188-197.
Rippen, H.E., Pan, E.C., Russell, C., Byrne, C.M., & Swift, E.K. (2013). Organizational framework for health information technology. International Journal of Medical Informatics, 82(4), e1-e13.
Robertson, A., Cresswell, K., Takian, A., Petrakaki, D., Crowe, S., Cornford, T., … & Prescott, R. (2010). Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. Bmj, 341, c4564.
Rosenthal, D., Seeman, E., & Gibson, S. (2005). Applying theoretical frameworks to explicate physician acceptance of computerized physician order entry (CPOE). Journal of International Technology and Information Management, 14(4), 5.
Ross, J., Stevenson, F., Lau, R., & Murray, E. (2016). Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implementation Science, 11(1), 146.
Rozenblum, R., Jang, Y., Zimlichman, E., Salzberg, C., Tamblyn, M., Buckeridge, D., … & Tamblyn, R. (2011). A qualitative study of Canada's experience with the implementation of electronic health information technology. Cmaj, 183(5), E281-E288.
Sheikh, A., Cornford, T., Barber, N., Avery, A., Takian, A., Lichtner, V., … & Morrison, Z. (2011). Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in ‘early adopter’ hospitals. Bmj, 343, d6054.
Sligo, J., Gauld, R., Roberts, V., & Villa, L. (2017). A literature review for large-scale health information system project planning, implementation and evaluation. International Journal of Medical Informatics, 97, 86-97.
Spetz, J., & Keane, D. (2009). Information technology implementation in a rural hospital: a cautionary tale. Journal of Healthcare Management, 54(5), 337-347.
Spil, T.A., LeRouge, C., Trimmer, K., & Wiggins, C. (2009). IT adoption and evaluation in healthcare: Evolutions and insights in theory, methodology, and practice. International Journal of Healthcare Information Systems and Informatics (IJHISI), 4(3), 69-96.
Thomson Reuters (2013). EndNote X7.1 help user guide. Retrieved from: http://clarivate.libguides.com/ld.php?content_id=33227814.
Van Raan, A.F. (2003). The use of bibliometric analysis in research performance assessment and monitoring of interdisciplinary scientific developments. Technology Assessment-Theory and Practice, 1(12), 20-29.
Vedluga, T., & Mikulskiene, B. (2017). Stakeholder driven indicators for eHealth performance management. Evaluation and Program Planning, 63, 82-92.
Warth, L., & Dyb, K. (2019). eHealth initiatives; the relationship between project work and institutional practice. BMC Health Services Research, 19(520). doi: 10.1186/s12913-019-4346-0.
Wilson, K., & Khansa, L. (2018). Migrating to electronic health record systems: A comparative study between the United States and the United Kingdom. Health Policy, 122(11), 1232-1239.
Zotero (2019). Software documentation. Retrieved form: https://www.zotero.org/support/.