Abstract
To assess the incidence of outpatient examinations delivered through a web portal in the Latium Region in 2 years and compare socio-demographic characteristics of these users compared to the total of examinations performed. All radiological exams (including MRI, X-ray and CT) performed from March 2017 to February 2019 were retrospectively analysed. For each exam, anonymized data of users who attended the exam were extracted and their characteristics were compared according to digital access to the reports. Overall, 9068 exams were performed in 6720 patients (55.8% males, median age 58 years, interquartile range (IQR) 46–70) of which 90.2% residents in Rome province, mainly attending a single radiological examination (77.3%). Among all exams, 446 (4.9%) were accessed, of which 190 (4.4%) in the first and 5.4% in the second year (p < 0.041). MRI was the type of exams mostly accessed (175, 7.0%). Being resident in the provinces of the Latium Region other than Rome was associated with a higher access rate (OR = 1.84, p = 0.001). Considering the overall costs sustained to implement a web portal which allows users a personal access to their own reports, if all users would have accessed/downloaded their exams, an overall users’ and hospital savings up to €255,808.28 could have been determined. The use of a web portal could represent a consistent economical advantage for the user, the hospital and the environment. Even if increasing over time, the use of web portal is still limited and strategies to increase the use of such systems should be implemented.
Keywords: Cost savings, eHealth, Electronic medical record (EMR), Health level 7 (HL7), Hospital records
Introduction
In recent years, development in informatics have allowed the distribution of laboratory, pathological anatomy and consultation reports with innovative methods through dedicated web portals, e-mails, social networks or messaging, both directly to the patients and/or their doctors [1–4].
Several studies have assessed the impact of these technologies in the medical field, especially in the clinical laboratory, on the appreciation of doctors and patients, on the reasons for the disparity of their use by age, wealth and race [5, 6].
In the radiological field, the risk of misinterpretation of the report, because of the loss of direct communication between doctors and patients, due to the online distribution of reports and images and the actual usefulness of using this model of transmission has been assessed; the appreciation of patients who download the radiological reports online has also been compared with those of the clinical pathology laboratory [7–10]. In a very recent guideline, the World Health Organization recommended the use of digital modality to improve people’s health [11].
In Italy, a strategy to promote digital growth is being developed, and recently, the Italian government defined a strategic and economic policy document for a 3-year plan (2019–2021) for Information Technology in Public Administration [12], which also defined the actions of intervention dedicated to digital health and the main solutions for the evolution of the public administration information system, aimed at improving health services, limiting waste and inefficiencies, improving the cost-quality ratio of health services and reducing differences among territories.
This plan includes the dematerialization of diagnostic imaging reports, with the aim of creating digital documents with full legal value, which will lead to the replacement of all paper equivalents, in order to reduce costs and improve the quality of service to citizens.
An electronic document managing system is based on a digital signature—involving the use of smart cards—which allows the transmission through the network of a coded document, its retrieval and/or its digitalization on optical supports, in compliance with the laws and regulations.
In Italy, this type of procedure has already been successfully implemented and nowadays available in some regions such as in the North-eastern region of Veneto through the Veneto ESCAPE (Electronic Signature in Care Activities for Paper Elimination) project [13] financed by the Agenzia per l’Italia Digitale - Agency for Digital Italy (AgID), the technical agency of the Presidency of the Council of Ministers also devoted to the eHealth Innovation. Other regions such as Latium (central Italy) have started the same process a few years ago with the Lazio ESCAPE project [14].
The Lazio ESCAPE project foresees that the reports and images are made available to the user in digital format for a period of 45 days, allowing the patient to view the report in PDF format (Portable Document Format) and the images in JEPG/DICOM format. The patient can also download the radiological examination report and images (see Fig. 1 for a schematic diagram of the system implemented).
Through the Lazio ESCAPE portal, in the page were images of his/her exam can be viewed and/or download, the user has the possibility to download the DICOM viewer, free of charge with clear instructions on its use.
After 45 days, the link is deactivated irreversibly, and the user cannot request a new access. However, the patient can still pick up the CD at the hospital free of charge.
The objective of our study was to assess the incidence, on the total of examinations performed in a 2-year period (2017–2019), of those delivered through the online portal of the Latium Region, comparing socio-demographic characteristics of accessing and not accessing users. Moreover, based on the first 2 years in which this IT mode was available in our Imaging Diagnostics department, a raw economic estimation of potential savings reachable by a universal access scenario was carried out.
Material and Methods
From March 2017 to February 2019, all consecutive diagnostic imaging exams performed by the Imaging Diagnostics Department of our Institute, including X-ray (XR), computed tomography (CT) and magnetic resonance imaging (MRI), were retrospectively analysed. Ultrasound and bone mineral density study exams were excluded because reports and images are delivered to the user/patient immediately after the exam. Exams performed on patients aged less than 18 years were excluded.
Anonymized exams data of users who attended the exam and those who accessed (view and/or download) the reports and images are extracted from RIS/PACS workstation (Agfa). For each exam, the following variables were included: age, gender, user’s residence (as indicated from Local Health Unit of residence), type of exam and recorded access to the portal. Anonymization did not allow any analysis of computer literacy of wealth of patients.
The CDA2 (Clinical Document Architecture) and HL7 (Health Level Seven) standards have been used to structure a document created to facilitate the exchange of clinical information between the various actors involved in the provision of health services [15]. A CDA document can contain texts, images, sounds and other multimedia content and its structure is formally derived from the Reference Information Model (RIM) of HL7 v3 which ensures the necessary flexibility also in relation to future developments of the standard; the images are encrypted with digital signature in PKCS#7 and XML-CDA2 format [16, 17]. This standard has been already implemented and updated in our Institute in the last 10 years, as well as digitally signing of all reports/images being already compliant with the ESCAPE standards before its implementation.
A raw economic estimation of the costs derived by a standard delivery of reports or by the use of such portal was performed, projecting in our context all the direct costs of the staff preparing and delivering the reports, of the consumables (toner burners, paper, envelopes, CDs, electricity) and of the structures (necessary spaces in square metres), based on what was estimated by the Veneto ESCAPE project [18]; in detail, the average cost for delivering the diagnostic imaging exams was estimated at 1.65 euros per exam. For the users, two types of costs were considered: those incurred by transportation to collect the report quantified as 3.52 euros and time lost (average 2 h) quantified as 11.51 euros hourly wage for a total amount of 26.56 euros.
No other additional costs were considered since no supplementary storage for reports and media was required. After 45 days, all data are cancelled from the shared storage, but still available on the single Institution. Since also the access to the portal is permanently cancelled, not providing the possibility to restore the access did not required any further staff cost to reassign access.
The study was approved by the Institute’s ethical committee (n°31/2018).
Result
Sociodemographic Analysis
In the 2 years of observation, a total of 9068 examinations were performed in 6720 patients and included in the analysis, almost equal distributed among X-rays (3325, 36.7%), computed tomography (CT; 3234, 35.7%) and magnetic resonance imaging exams (MRI; 2509, 27.7%).
Out of 9068 exams carried out, 446 (4.9%) were accessed at least one time (see Table 1), of which 190 (4.4%) in the first year, with an increased proportion in the second year of activity (256, 5.4%—p < 0.041).
Table 1.
Total | Exam result accessed N (%) |
p | |
---|---|---|---|
All exams | 9068 | 446 (4.9) | |
Type of exam | |||
Magnetic resonance imaging, MRI | 2509 | 175 (7.0) | |
X-ray, XR | 3325 | 124 (3.7) | < 0.001 |
Computed tomography, CT | 3234 | 147 (4.5) | |
Year of activitya | |||
First | 4290 | 190 (4.4) | 0.041 |
Second | 4778 | 256 (5.4) | |
Users’ residence | |||
Rome province | 8199 | 380 (4.6) | < 0.001 |
Other Latium provinces | 543 | 45 (8.3) | |
Other Regions | 326 | 21 (6.4) |
aYears of activity: first year, March 2017 to February 2018; second year, March 2018 to February 2019
The type of exam most accessed was MRI (175, 7.0%) followed by CT (147, 4.5%) and X-rays (3.7%) (p < 0.001). When considering differences between the 2 years of activity, the significant increase in exams access is essentially due to an increase in CTs access (5.2% vs 3.8%, p = 0.052) and secondly of MRI (7.6 vs. 6.3, p = 0.208), while X-rays exams were constantly accessed over this period (3.7%) (Fig. 2).
We recorded a single access (by the patients, the attending physician, or the specialist) in 155 exams (34.8%) and 5 or more accesses in 106 exams (23.8%) (Table 2). The average number of access to the single exam was 3.58 access/exam (3.48 in the first year and 3.66 in the second, data not shown); the distribution of number of accesses according to type of exam was significantly different and higher for MRI where the proportion of exams with 5 or more accesses was 30.9% compared to 21.1% for CT and almost half for XR (16.9%) (p = 0.030) (Table 2).
Table 2.
Type of exam | Total | N (%) | Number of web accesses to the exam | ||||
---|---|---|---|---|---|---|---|
1 | 2–4 | 5+ | Median (IQR) | Average | |||
MRI | 2509 | 175 (7.0) | 61 (34.9) | 60 (34.3) | 54 (30.9) | 2 (1–6) | 4.04 |
XR | 3325 | 124 (3.7) | 42 (33.9) | 61 (49.2) | 21 (16.9) | 2 (1–3) | 3.13 |
CT | 3234 | 147 (4.5) | 52 (35.4) | 64 (43.5) | 31 (21.1) | 2 (1–4) | 3.44 |
Total | 9068 | 446 (4.9) | 155 (34.8) | 185 (41.5) | 106 (23.8) | 2 (1–4) | 3.59 |
MRI magnetic resonance imaging, XR X-ray, CT computed tomography, IQR interquartile range
Among the 6720 patients considered in the 2 years of observation, male gender was predominant (55.8%) and the overall median age was 58 years (interquartile range (IQR) 46–70). They were mostly resident in the municipality of Rome (90.2%) and to a lesser extent in other provinces of Latium Region (6.0%). The vast majority of patients performed a single radiological examination in the period of interest (77.3%).
Table 3 shows subjects’ characteristics according to the access to the portal, examining separately those who withdrew the exam personally by hand, and of those who accessed/downloaded them from the web-portal of the Latium region. Globally, 424 (6.3%) performed at least one access. Analysing the characteristics of the patients who consulted the portal, there are no substantial gender and age differences between the two groups of patients, as well no difference in portal access between the first and second year of activity. A statistically significant proportion of exams where accessed through the portal from patients living in provinces of the Latium Region other than Rome (OR = 1.83, p < 0.001) (Table 3); this difference was evident overall and when compared among MRIs and CTs exams only (data not shown),
Table 3.
At least one consultation | Total | OR (95% CI) | p | aOR (95% CI) | p | |
---|---|---|---|---|---|---|
N (%) | ||||||
All subjects | 424 (6.3) | 6720 | ||||
Gender | ||||||
M | 235 (6.3) | 3750 | 1 | 1 | ||
F | 189 (6.4) | 2970 | 1.02 (0.83–1.24) | 0.871 | 1.02 (0.84–1.25) | 0.813 |
Age | ||||||
< 45 | 94 (5.9) | 1589 | 1 | 1 | ||
46–65 | 185 (6.6) | 2900 | 1.08 (0.84–1.40) | 0.540 | 1.07 (0.83–1.39) | 0.587 |
66+ | 145 (6.5) | 2231 | 1.11 (0.85–1.45) | 0.461 | 1.11 (0.85–1.46) | 0.448 |
Users’ residence | ||||||
Rome province | 363 (6.0) | 6064 | 1 | 1 | ||
Other Latium provinces | 42 (10.5) | 401 | 1.84 (1.31–2.57) | 0.001 | 1.83 (1.31–2.57) | < 0.001 |
Other Regions | 19 (7.5) | 255 | 1.26 (0.78–2.04) | 0.337 | 1.28 (0.79–2.07) | 0.318 |
Year of activitya | ||||||
First | 216 (6.0) | 3577 | 1 | 1 | ||
Second | 208 (6.6) | 3143 | 1.10 (0.91–1.34) | 0.330 | 1.11 (0.91–1.35) | 0.322 |
OR univariate odds ratio, aOR multivariable adjusted odds ratio using all shown variables in the final model, CI confidence intervals
aYears of activity: first year, March 2017 to February 2018; second year, March 2018 to February 2019
Considering the number of access according to the users’ characteristics, users aged less than 60 years show a high rate of single access to the portal, while subjects aged 60 or above have a higher proportion of 5 or more accesses (chi-square for trend, p = 0.0011) (Table 4). In the second year of activity, a trend was observed for patients performing 5 or more accesses to their exams (25.8% vs 21.1%).
Table 4.
At least one consultation N (%) |
Number of web accesses to the exam | |||
---|---|---|---|---|
1 | 2–4 | 5+ | ||
All exams | 446 | 155 (34.8) | 185 (41.5) | 106 (23.8) |
Gender | ||||
M | 249 | 89 (35.7) | 106 (42.6) | 54 (21.7) |
F | 197 | 66 (33.5) | 79 (40.1) | 52 (26.4) |
Age | ||||
< 60 | 241 | 96 (39.8) | 102 (42.3) | 43 (17.8) |
60+ | 205 | 59 (28.8) | 83 (40.5) | 63 (30.7) |
Users’ residence | ||||
Rome province | 380 | 126 (33.2) | 161 (42.4) | 93 (24.5) |
Other Latium provinces | 45 | 21 (46.7) | 15 (33.3) | 9 (20.0) |
Other Regions | 21 | 8 (38.1) | 9 (42.9) | 4 (19.0) |
Year of activitya | ||||
First | 190 | 69 (36.3) | 81 (42.6) | 40 (21.1) |
Second | 256 | 86 (33.6) | 104 (40.6) | 66 (25.8) |
Type of exam | ||||
MRI | 175 | 61 (34.9) | 60 (34.3) | 54 (30.9) |
XR | 124 | 42 (33.9) | 61 (49.2) | 21 (16.9) |
CT | 147 | 52 (35.4) | 64 (43.5) | 31 (21.1) |
MRI magnetic resonance imaging, XR X-ray, CT computed tomography
aYears of activity: first year, March 2017 to February 2018; second year, March 2018 to February 2019
Economic Analysis
In our experience with this system of downloaded online, we estimate that the Institute has saved €735.9 in 2 years; if all 9068 exams had been downloaded online, the estimated savings would have been € 14,962.20.
For each patient who accessed the portal, the personal saving can be estimated on average €26.56 for a total of €11,845.76; if all the 9068 exams had downloaded online, the savings would have been € 240,846.08.
The overall savings for the users and for the Institute eventually obtained, if all users had downloaded the exams online, would have been € 255,808.28 in 2 years.
Discussion
Development in informatics have allowed the distribution and consultations of reports with innovative methods through dedicated web portals, e-mails, social networks or messaging, both directly to the patients and/or their doctors [1–4]. This possibility addresses also a recent guideline of the World Health Organization which recommends the use of digital modality to improve people’s health [11], promoted by the Italian government which also defined those actions dedicated to digital health, aimed at improving health services, including the dematerialization of diagnostic imaging reports [12].
Beside the appreciation of patients and professionals, the real impact of these systems of reports consultation is still debated.
Some studies point to users’ fear of an inadequate level of confidentiality of personal information and health data, lack of awareness of access to a portal and negative experience on first access to the portal, for example from a lack of ease use [4]. Socioeconomic factors also could influence the use of the portal, such as the low level of education, limited access to the Internet and belonging to racial or ethnic minorities [5, 6].
However, once the initial problems are resolved, a recent paper demonstrated that after a 1 year of use, 99% of the subjects wanted to keep the users active as they find important benefits from their use, such as a better understanding of their planned treatments [19].
In the management of chronic disease, access to web-portal improved adherence to drugs in patients with diabetes or patients with HIV receiving antiretroviral therapy [20].
In this paper, we present our experience on web-portal implemented within the Lazio ESCAPE project for the digitalization and web access of radiological reports. We focused the analysis on the first 2 years (2017–2019) of the services.
Our data indicate that the percentage of exams that were actually accessed through the web-portal was 4.9% overall, with a significant increase in the second year of study reaching 5.4% (vs. 4.5% in the first year, p = 0.041). Even if the majority of users attending the Radiology Department were obviously from the municipality of Rome and its province, exams accessed were more likely from users living in other provinces of the Region or in other Regions (p < 0.001). MRIs were the exams mostly accessed (7.0% of total) and only secondly CT exams (4.5%).
There are some entirely European and especially Italian peculiarities that explain the low confidence with the digital services: in Europe, currently, only 56% of citizens aged between 16 and 74 have basic computer skills, and overall Italians are ranked 24th out of 28 for computer literacy rate and digital skills [21, 22].
A study by the European community performed in 2013 on deployment of eHealth among general practitioners recently updated showed the extreme variability of services, dissemination and information systems in healthcare with a low overall use of portals: only 1% of users use it for all reports, 4% only for some and the remaining 95% do not use it; however, the study underlines the poor use in radiology and the availability of online images [23].
According to age, users older than 45 years of age are those more likely to access the service, in accordance with what was found in the Veneto ESCAPE project [24].
In diagnostic imaging, Miles et al. in 2016 showed that 51% of patients accessed their radiological reports when they were available [25]. Furthermore, in their paper, they found that the use of the portal varied substantially according to the socio-demographic characteristics: the typical user is a native English speaker, a woman, with an age of 25–39 years. Compared to a previous study performed in 2006 [26], Miles and colleagues’ study showed an increase in overall access to radiological reports from 35% of the previous experience.
The most significant experience on this field is the one implemented in a collaboration between the National Institutes of Health (NIH) and the Radiological Society of North America (RSNA) through the RSNA Image Share Network (ISN) project, in order to make images and reports available in the USA.
The project not only allows patients to access the images but also to make them available to clinicians and attending doctors also for a “second opinion” allowing the access to the single exams up to 30 days. Over a period of 4 years (2012–2015), more than 40,000 radiological examinations performed on roughly 9000 patients were uploaded. Among these exams, roughly 12,000 (30%) were firstly accessed/downloaded by patients, other medical or research centres. Through a simple e-mail sent by the server as “alert” to the user for not accessing promptly, the exams sent after the end of the 30-day period of access, the number of reports access raised to 30,000 (70%) [27].
Familiarity with the use of a computer is certainly one of the determining factors in the choice of how to obtain images, and in fact elderly patients, who are the population with the greatest use of health care, often perceive or experience significant barriers in their use of information technology. But it is also interesting to note that when a help desk activity was activated to help patients using ISN in 3 years, only 1128 problems were detected on 13,458 patients, which suggests that patients were largely able to access images [28].
Considering the increased proportion of users accessing the portal in the second year, as well as an increased quote of multiple accesses (see Table 4).
It can be speculated that once a greater dissemination of information and awareness of users of this new method is achieved over time, augmented confidence of users and professionals in this system has determined this increase. The possibility that the users shared his/her credentials with the attending physician/specialist would have allowed a further wider dissemination and use of the service.
These data are substantially in agreement with the literature and with results presented in the European Commission reports [23], but lower to previously cited US data [28], even if in their report authors referred to a system that allowed the access to the exams not only to the single user (as in our system) but also to the prescribing clinician and other physicians.
A very recent study performed in US collecting data from 80 hospitals, however, showed that the patient’s direct online access to their radiological imaging results was even lower than in our experience (4%) while the remaining patients still preferred the standard retrieval of their exams through a burned CD/DVD [29].
The possibility of downloading radiological reports online also opens the debate on the opportunity to differentiate the timing of the disclosure of sensitive health information, such as those contained in the radiological report, for example by giving priority to the attending physician on the patient to allow the preparation of the necessary therapeutic treatments and avoid unnecessary periods of anxiety and research and frantic internet searches [30–34].
The digitization of reports entails the saving of consumables. In addition, the possibility of downloading the report online would determine a reduction in travel costs of users towards, in our situation, the hospital for withdraw their reports, thus causing a lower pollution production.
As reported in the reports from the Veneto ESCAPE project, it can be speculated that if the totality of radiological reports delivered from the Latium Region would be accessed online, our environment would be spared of a consistent amount of greenhouse gases produced not only for users’ travel but also for the production and delivery of paper reports and digital supports for the images [24].
This study presents some limitations firstly due to lower numbers resulting from a recently started project if compared to other National and International experiences, but our results are consistent with other bigger experiences. Even if limited to 2 years of experience, an overall increase in access to the system was observed overall. Our experience wants to represent an evaluation of the initial impact in our context of a new methodology in order to benchmark its evolution over time, correlating it with future technological, cultural and financial progress.
The availability of the online report can be considered a concrete contribution to the improvement of the quality of life, especially for those who are older or non-autonomous, since it makes it possible to simplify some practices and reduce the movement of patients and carers [24].
Digitization with online access to their own reports is an undoubted advantage for users, especially if they are not autonomous, sparing time lost for travel and costs deriving. It would determine economic savings for the Institutions in terms of consumables and personnel costs, allowing to release consistent resources that can contribute to the sustainability of the health system. Digitalization of health records would determine finally in positive environment effects in terms of contribution to the reduction of atmospheric pollution.
In our opinion, a strategy to increase the spread of electronic distribution of reports and images could be to incentivize their use, also economically: both with ad hoc funds for public health companies only (divided between Information and Communications Technology and personal investments) and by reducing the cost of the service for users who use the portal.
Abbreviations
- CDA2
Clinical Document Architecture
- DICOM
Digital Imaging and Communications in Medicine
- ESCAPE
Electronic Signature in Care Activities for Paper Elimination
- HL7
Health Level Seven
- ICT
Information and Communications Technology
- ISN
Image Share Network
- JPEG
Joint Photographic Experts Group
- NIH
National Institutes of Health
Portable Document Format
- RIM
Reference Information Model
- RSNA
Radiological Society of North America
Compliance with Ethical Standards
Conflict of Interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Footnotes
Key Points
• The growth of radiology will result in an increased demand for image and report sharing.
• Frequency of online access to medical reports and radiological is still low but increasing over time.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Goldzweig CL, Orshansky G, Paige NM, et al. Electronic patient portals: evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med. 2013;159(10):677–687. doi: 10.7326/0003-4819-159-10-201311190-00006. [DOI] [PubMed] [Google Scholar]
- 2.Shaw RJ, Ferranti J. Patient-provider internet portals--patient outcomes and use. Comput Inform Nurs. 2011;12:714–718. doi: 10.1097/NCN.0b013e318224b597. [DOI] [PubMed] [Google Scholar]
- 3.Wen KY, Kreps G, Zhu F, et al. Consumers’ perceptions about and use of the internet for personal health records and health information exchange: analysis of the 2007 Health Information National Trends Survey. J Med Internet Res. 2010;12:e73. doi: 10.2196/jmir.1668. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res. 2015;17(6):e148. doi: 10.2196/jmir.4255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Perzynski AT, Roach MJ, Shick S, et al. Patient portals and broadband internet inequality. J Am Med Inform Assoc. 2017;24(5):927–932. doi: 10.1093/jamia/ocx020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gordon NP, Hornbrook MC. Differences in Access to and Preferences for Using Patient Portals and Other eHealth Technologies Based on Race, Ethnicity, and Age: A Database and Survey Study of Seniors in a Large Health Plan. J Med Internet Res. 2016;18(3):e50. doi: 10.2196/jmir.5105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Arenson RL. Building a Better Online Portal for Patients to Obtain Radiologic Results. Radiology. 2019;290(1):144–145. doi: 10.1148/radiol.2018182218. [DOI] [PubMed] [Google Scholar]
- 8.Li KC, Marcovici P, Phelps A, et al. Digitization of medicine: how radiology can take advantage of the digital revolution. Acad Radiol. 2013;20(12):1479–1494. doi: 10.1016/j.acra.2013.09.008. [DOI] [PubMed] [Google Scholar]
- 9.Johnson AJ, Frankel RM, Williams LS, et al. Patient access to radiology reports: what do physicians think? J Am Coll Radiol. 2010;7:281–289. doi: 10.1016/j.jacr.2009.10.011. [DOI] [PubMed] [Google Scholar]
- 10.Henshaw D, Okaw G, et al. Access to Radiology Reports via an Online Patient Portal: Experiences of Referring Physicians and Patients. J Am Coll Radiol. 2015;12:582–586. doi: 10.1016/j.jacr.2015.01.015. [DOI] [PubMed] [Google Scholar]
- 11.World Health Organization. Recommendations on digital interventions for health system strengthening https://www.who.int/reproductivehealth/publications/digital-interventions-health-system-strengthening/en. Accessed 2 Feb 2020 [PubMed]
- 12.Agenzia per l’Italia Digitale (AGID). Piano Triennale Sanità digitale: https://www.agid.gov.it/it/piattaforme/sanita-digitale. Accessed 2 Feb 2020.
- 13.Veneto's Research Centre for eHealth Innovation. Veneto ESCAPE project. http://www.consorzioarsenal.it/en/web/guest/progetti/veneto-escape/il-progetto. Accessed 2 Feb 2020
- 14.Regione Lazio. Agenda Digitale della Regione Lazio. Linee guida per lo svilupp del Lazio Digitale. Progetto Lazio ESCAPE. https://gse.it/normativa_site/GSE%20Documenti%20normativa/LAZIO_DGR_n284__16_06_2015.pdf. Accessed 2 Feb 2020.
- 15.Health Level Seven Italia. Implementation Guide Clinical Document Architecture (CDA) Rel. 2 http://www.hl7italia.it/hl7italia_D7/. Accessed 2 Feb 2020
- 16.Cram D, Roth CJ, Towbin AJ. Orders- Versus Encounters-Based Image Capture: Implications Pre- and Post-Procedure Workflow, Technical and Build Capabilities, Resulting, Analytics and Revenue Capture: HIMSS-SIIM Collaborative White Paper. J Digit Imaging. 2016;29(5):559–566. doi: 10.1007/s10278-016-9888-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Mendelson DS, Bak PR, Menschik E, Siegel E. Informatics in radiology: image exchange: IHE and the evolution of image sharing. Radiographics. 2008;28(7):1817–1833. doi: 10.1148/rg.287085174. [DOI] [PubMed] [Google Scholar]
- 18.Buccoliero L, Calciolari S, Marsilio M. A methodological and operative framework for the evaluation of an e-health project. Int J Health Plann Manage. 2008;23(1):3–20. doi: 10.1002/hpm.881. [DOI] [PubMed] [Google Scholar]
- 19.Mishra VK, Hoyt RE, Wolver SE, Yoshihashi A, Banas C. Qualitative and Quantitative Analysis of Patients' Perceptions of the Patient Portal Experience with OpenNotes. Appl Clin Inform. 2019;10(01):010–018. doi: 10.1055/s-0038-1676588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Kahn JS, Hilton JF, Van Nunnery T, et al. Personal health records in a public hospital: experience at the HIV/AIDS clinic at San Francisco General Hospital. J Am Med Inform Assoc. 2010;17(2):224–228. doi: 10.1136/jamia.2009.000315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.European Commission (2019) Human Capital and Digital Skills https://ec.europa.eu/digital-single-market/en/human-capital Accessed 2 Feb 2020.
- 22.European Commission (2017) Europe's Digital Progress Report 2017. https://ec.europa.eu/digital-single-market/en/european-digital-progress-report. Accessed 2 Feb 2020
- 23.European Commission (2013) Benchmarking Deployment of eHealth among General Practitioners. http://ec.europa.eu/information_society/newsroom/cf/dae/document.cfm?doc_id=4897. Accessed 2 Feb 2020
- 24.Centro Veneto Ricerca e Innovazione per la Sanità Digitale (Veneto's Research Centre for eHealth Innovation) (2013). Progetto Veneto ESCAPE. http://www.consorzioarsenal.it/documents/10157/332925/Quaderno+2-2013+Veneto+ESCAPE. Accessed 2 Feb 2020
- 25.Miles RC, Hippe DS, Elmore JG, Wang CL, Payne TH, Lee CI. Patient Access to Online Radiology Reports: Frequency and Sociodemographic Characteristics Associated with Use. Acad Radiol. 2016;23(9):1162–1169. doi: 10.1016/j.acra.2016.05.005. [DOI] [PubMed] [Google Scholar]
- 26.Weingart SN, Rind D, Tofias Z, Daniel Z. Who Uses the Patient Internet Portal? The PatientSite Experience. J Am Med Inform Assoc. 2006;13(1):91–95. doi: 10.1197/jamia.M1833. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Langer SG, Tellis W, Carr C, et al. The RSNA Image Sharing Network. Digit Imaging. 2015;28(1):53–61. doi: 10.1007/s10278-014-9714-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Awan OA, Awan YA, Fossett J, Fossett R, Mendelson D, Siegel EL. Patient Engagement: The Experience of the RSNA Image Share Patient Help Desk. J Am Coll Radiol. 2015;12(12 Pt A):1289–1292. doi: 10.1016/j.jacr.2015.05.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Lye CT, Krumholz HM, Eckroate JE, et al. Evaluation of the Patient Request Process for Radiology Imaging in U.S. Hospitals. Radiology. 2019;292(2):409–413. doi: 10.1148/radiol.2019190473. [DOI] [PubMed] [Google Scholar]
- 30.Woolen SA, Kazerooni EA, Steenburg SD, Nan B, Ma T, Wall A. Optimizing Electronic Release of Imaging Results through an Online Patient Portal. Radiology. 2019;290(1):136–143. doi: 10.1148/radiol.2018180883. [DOI] [PubMed] [Google Scholar]
- 31.Reicher JJ, Reicher MA. Implementation of Certified EHR, Patient Portal, and "Direct" Messaging Technology in a Radiology Environment Enhances Communication of Radiology Results to Both Referring Physicians and Patients. J Digit Imaging. 2016;29(3):337–340. doi: 10.1007/s10278-015-9845-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Lee CI, Langlotz CP, Elmore JG. Implications of Direct Patient Online Access to Radiology Reports Through Patient Web Portals. J Am Coll Radiol. 2016;13(12 Pt B):1608–1614. doi: 10.1016/j.jacr.2016.09.007. [DOI] [PubMed] [Google Scholar]
- 33.Gefen R, Bruno MA, Abujudeh HH. Online Portals: Gateway to Patient-Centered Radiology. AJR Am J Roentgenol. 2017;209(5):987–991. doi: 10.2214/AJR.17.18291. [DOI] [PubMed] [Google Scholar]
- 34.Greco G, Patel AS, Lewis SC, et al. Patient-directed Internet-based medical image exchange: experience from an initial multicenter implementation. Acad Radiol. 2016;23:237–244. doi: 10.1016/j.acra.2015.10.012. [DOI] [PMC free article] [PubMed] [Google Scholar]