138
GRAJEK ET AL., Evaluating IAIMS at Yale
JAMIA
Original Investigations
Evaluating IAIMS at Yale:
Information Access
SUSAN E. GRAJEK, PHD, PASCAL CALARCO, MLIS, SANDRA J. FRAWLEY, PHD,
JAMES MCKAY, PERRY L. MILLER, MD, PHD, JOHN A. PATON, PHD,
NANCY K. RODERER, MLS, JOSEPH E. SULLIVAN
A b s t r a c t Objective: To evaluate use of information resources during the first year of
IAIMS implementation at the Yale – New Haven Medical Center. The evaluation asked: (1) Which
information resources are being used? (2) Who uses information resources? (3) Where are
information resources used? (4) Are multiple sources of information being integrated?
Design: Measures included monthly usage data for resources delivered network-wide, in the
Medical Library, and in the Hospital; online surveys of library workstation users; an annual
survey of a random, stratified sample of Medical Center faculty, postdoctoral trainees, students,
nurses, residents, and managerial and professional staff; and user comments.
Results: Eighty-three percent of the Medical Center community use networked information
resources, and use of resources is increasing. Both status (faculty, student, nurse, etc.) and
mission (teaching, research, patient care) affect use of individual resources. Eighty-eight percent
of people use computers in more than one location, and increases in usage of traditional library
resources such as MEDLINE are due to increased access from outside the Library. Both survey
and usage data suggest that people are using multiple resources during the same information
seeking session.
Conclusions: Almost all of the Medical Center community is using networked information
resources in more settings. It is necessary to support increased demand for information access
from remote locations and to specific populations, such as nurses. People are integrating
information from multiple sources, but true integration within information systems is just
beginning. Other institutions are advised to incorporate pragmatic evaluation into their IAIMS
activities and to share evaluation results with decision-makers.
n J Am Med Inform Assoc. 1997;4:138 – 149.
Affiliations of the authors: Yale University School of Medicine,
New Haven, CT (SEG, PC, SJF, JM, PLM, JAP, NKR); Yale –
New Haven Hospital, New Haven, CT (JES).
Supported in part by NIH grant G08 LM05583 from the
National Library of Medicine.
Presented in part at the 19th Annual Symposium on
Computer Applications in Medical Care, New Orleans,
Louisiana, October 31, 1995.
Correspondence and reprint requests to: Susan Grajek, PhD,
Medical Information Technology Services, 333 Cedar Street,
P.O. Box 208000, Yale University School of Medicine, New
Haven, CT 06520-8000. e-mail: susan.grajek@yale.edu
Received for publication: 6/26/96; accepted for publication:
11/6/96.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
Research Paper n
Journal of the American Medical Informatics Association
Volume 4
Background
Information Access at Yale
Located in New Haven, Connecticut, the Yale – New
Haven Medical Center includes the Yale University
Schools of Medicine and Nursing and Yale – New Haven Hospital. The Medical Center’s primary missions
of patient care, education, and research are pursued
by a community of approximately 1,300 full-time faculty, 1,105 students, 900 post-doctoral trainees, 400
residents, 1,250 nurses, 3,350 full-time staff at the Hospital and 2,400 full-time staff at the schools (including
1,000 full-time managerial and professional staff *).
The Medical Center has a campus-wide, high speed
data network based on Ethernet and token ring segments. This network is connected to the Internet via
the University network and thus provides access to
both local and remote information resources.
Public Workstations
Public workstations are available for use by people as
they move about the Medical Center. The user interface must have a similar framework in different locations so that retraining is unnecessary, and identification must be required for access to restricted
information. At Yale, we developed menu software,
NetMenu,2 to provide this functionality. It logs user
activity, it has an online survey feature, it supports
scripting of connecting and disconnecting to online
resources, it launches applications such as computerassisted instruction (CAI), and it can be configured
to take over the screen and hide the underlying operating system from the user while protecting the
*The 900 post-doctoral trainees at the School of Medicine do not
include the 400 Hospital residents. The 1,105 students consist
of 486 medical students, 414 graduate students, 141 nursing students, and 64 physician assistant students. The 3,350 full-time
Hospital staff do not include the 1,250 nurses.
Mar / Apr 1997
139
workstation from user changes. The same menu software is used on public library workstations (23 Macintosh and Windows-based devices) and on public
clinical workstations in the Hospital (71 Windowsbased devices). These two sets of public workstations
share a common set of resources, such as MEDLINE,
electronic mail, and the World Wide Web (WWW). We
chose to tailor the content and structure of the menus
to meet the different information needs, security requirements, and license restrictions in the two environments. The public library workstations include an
array of bibliographic, full text, and Macintosh-based
medical education resources, while the public clinical
workstations provide access to hospital information
systems for order entry and results reporting and a
variety of online medical textbooks and guides, and
procedure recorders. (In addition, there are 1,300 dedicated hospital workstations in high traffic nursing
stations to enable staff to quickly query its patient care
and clinical laboratory systems and MicroMedex.)
The Windows-based public library workstations also
contain InfoFinder,3 an integrated information resource search tool created at the Yale Center for Medical Informatics and based on the National Library of
Medicine’s Information Sources Map (ISM), a component of the Unified Medical Language System
(UMLS).4 InfoFinder enables users to search for and
select appropriate online resources to query.
Private Workstations
All personal computers connected to the University
network are configured with icons to access Ovid5 (the
software delivering MEDLINE, CINAHL, 15 full-text
journals, and two other bibliographic databases), Orbis (a NOTIS-based system containing the Yale Library catalog and three additional bibliographic databases), the World Wide Web via Netscape, electronic
mail, the biomedical mainframe computer (to run molecular biology programs, statistical software, and
other applications), the University personnel directory, the Nexis/Lexis databases, and FTP software.
These frequently used resources are also accessed
from MedMenu, a Web-based menu developed by
the Medical Library to mirror the menu on the public
library workstations (with the exception of medical
education software). This Web-based menu (http:
//www.med.yale.edu/medmenu) has many of the
advantages of our original NetMenu. It runs on both
Macintosh and Windows-based devices, it provides a
common core menu of resources, and the core menu
can be maintained centrally. In addition, the Web
menu can be coordinated with the users’ own customized menus of resources, and the Web software is in
the public domain, whereas the NetMenu software requires commercial software components.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
The Yale – New Haven Medical Center is in the process of developing a comprehensive information environment based on the National Library of Medicine’s vision of an Integrated Advanced Information
Management System (IAIMS).1 A fundamental goal of
the IAIMS vision and of our efforts is to provide relevant information when and where it is needed in
support of our missions of education, biomedical research, and patient care. In this paper we present usage data by type of information resource, type of user,
nature of task, and location. We discuss the use of our
data to guide investment decisions and to determine
factors that affect utilization.
Number 2
140
Evaluating Information Use and Its Impact
Within an IAIMS Environment
To evaluate progress toward our IAIMS goal of providing information users need in the course of their
work, we asked four basic questions:
1. Which information resources are being used?
2. Who uses information resources?
3. Where are information resources used?
4. Are multiple sources of information being integrated?
The questions build on one another and parallel
stages in the development of an IAIMS environment.
At the first stage, simple usage is monitored to determine how heavily information resources are being
used. At a second stage we ask how different types of
users (e.g., students or teachers) use these information
resources. At a third stage we ask whether we are
‘‘providing information when and where it is
needed.’’ Finally, we ask whether people are integrating information from multiple sources for a single
task such as diagnosing a patient. Integration marks
the most ambitious of IAIMS goals, and this report
focuses on the simplest form of such integration —
that which the users perform themselves.
Our evaluation strategy has two underlying principles:
1. Collect information necessary to act. Our evaluation
efforts need to be focused and pragmatic; we need to
have a concrete use for any data we collect. Our resources for evaluation are limited, so we need to be
very careful to ask only those questions whose answers will influence later resource allocation. Furthermore, we do not want to burden members of the
community or service providers with requests to
participate in evaluation unless there is a clear benefit
to them. We issue a monthly report on usage and
trends (http://www.med.yale.edu/computing/acadcomp/iaims/reports.html). The report is distributed
to senior Medical Center administrators, library and
computing professionals, and information providers.
We have used the data to help us make decisions
about the number and location of public workstations;
about adding, deleting or modifying delivery of specific information resources; and about increasing offerings in categories such as full-text databases and
medical education software.
2. Use multiple methods and take multiple measurements.
By comparing results from a variety of evaluation
methods, we assess the reliability of our data. We also
conduct our evaluation in several phases to recognize
trends.
This evaluation extends previous work at IAIMS sites
in three ways. First, we collect and analyze usage in
greater detail, examining patterns and trends in different environments and among different groups of
users. Second, evaluation at Yale is an ongoing activity that is integrated into the management of information resources. Finally, we include users and nonusers of online information systems in our evaluation
activities to measure the impact of networked information upon the entire Medical Center community
and to identify any barriers to our goal of benefits for
all.
Methods
Measures
We collect evaluation information from usage logs,
online surveys of users of public library workstations,
a Medical Center-wide survey, and user comments.
Usage Data
Each month we collect usage data for major resources
(e.g., MEDLINE, electronic mail) and for all information services available on the public library and clinical workstations (see Table 1 for a summary).
The most complete usage information is available for
the Ovid databases (MEDLINE, CINAHL, Health
Planning and Administration, and 15 full-text journals), Current Contents, the WWW server, and the
Yale biomedical gopher. Those applications have internal logs and may be tracked network-wide. For
each of the Ovid databases, we collect data identifying
the users, their departments, the number of searches
they conducted, the number of times they accessed
Ovid, and whether they accessed Ovid from the Library or from some other location.
The Current Contents server logs the number of sessions, the user, the number of sessions that were conducted from within libraries, and the number of automatic searches that were run. (Current Contents
enables users to specify that a specific search be run
automatically each week when the database is up-
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
Previous reports from other IAIMS sites6,7 have emphasized a variety of institution-specific goals and related strategies for evaluation including cataloguing
accomplishments8,9 and lessons learned,10 analyzing
costs and benefits,8,9 assessing costs versus utilization,11 reporting usage,9 and recording oral history.12
In the present report we emphasize a detailed analysis
of information use across classes of users, means of
access, and purpose (e.g., clinical care).
GRAJEK ET AL., Evaluating IAIMS at Yale
Journal of the American Medical Informatics Association
Volume 4
Number 2
Mar / Apr 1997
141
Table 1 n
Usage Data Sources
Major Resources
Available on Clinical
Workstations
Available on Library
Workstations
Bibliographic databases and
search tools: 16 applications (1
Windows only) in Library, 4 in
Hospital
Clinical assistance (3 applications
in Library, 1 in Hospital)
E-mail/news/networks (7 applications in Library, 3 in Hospital)
Full-text and factual databases: 30
applications (1 Windows only)
in Library, 9 in Hospital
Word processing and utilities (8
applications in Library, 4 in
Hospital)
Hospital information systems and recording
tools (9 applications)
Grant and research information (4 applications)
Medical Education (49
Macintosh applications)
dated. Users receive citations resulting from the automatic searches by e-mail.)
Usage data for the Medical Center World Wide
Web server are generated by MUSAGE (http:
//www.blpes.lse.ac.uk/misc/musage.htm), a perl
script created at the London School of Economics and
Political Science. MUSAGE logs the total number of
pages accessed, the number of times each page is accessed, and the IP address accessing each page (enabling us to track usage from Yale, other educational
institutions, major network providers, U.S. commercial sites, U.S. non-profit sites, U.S. government sites,
and foreign countries).
Usage logs for the Yale biomedical gopher record the
number of times Yale users and the number of times
external users access each gopher document.
Although there are several desktop- and mainframebased electronic mail systems in use at the Medical
Center, most e-mail traffic is processed through the
School of Medicine’s VAX 7610 minicomputer by
PMDF-MTA13, a mail transfer agent from Innosoft International. PMDF logs the number of messages sent
to and from the various email systems. Mail sent to
the Hospital and some internal desktop electronic
mail messages are not processed (or logged) by
PMDF.
Usage data for the information resources accessed
through the public library and clinical workstations
are generated by the NetMenu software. Each time a
user clicks on a ‘‘connect’’ button to access an information resource, the menu software logs the date,
time, machine, and resource name.
As objective as they might seem, usage data are subject to limitations and measurement error. Our electronic mail data are almost but not quite complete.
The Web data include accesses from people developing and testing the pages and so at times usage for
individual pages or directories is artificially inflated.
For some resources, we can only track usage from
public workstations because no usage data are collected at the resource level. Our means of logging application usage from public workstations is dependent
on counting the number of clicks on the ‘connect’ button. Examining the Ovid usage data has taught us that
this does not always mean a user actually uses the
application. Finally, on rare occasions, logging is disrupted and usage goes uncounted. With these limitations in mind, we interpret most usage data in relative rather than absolute terms, examining trends
over time, comparing usage levels by different users
of the same application, or comparing usage levels of
different applications collected by the same mechanism (e.g., the public menu logs).
Online Surveys
We wrote a questionnaire in Visual Basic to survey
users of public library workstations. When a survey
is active (typically for short periods of time), the
NetMenu software displays the survey every 20th
time someone tries to connect to a resource. The user
must complete the survey before being connected to
the resource. To lessen the interruption of work, the
survey occupies a single screen, and consists of only
four or five questions. Questions to provide demographic information are repeated during each survey
period. Topical survey questions vary from one sur-
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
Current Contents
Electronic mail
Ovid databases: CINAHL, Health
Planning and Administration,
MEDLINE, and 15 full-text biomedical journals (e.g., JAMA,
NEJM, Science)
WWW server
Yale biomedical gopher
Available on Public Library and
Clinical Workstations
142
vey to another. Using this online survey we collected
over 900 responses during three 2-week surveys between December 1994 and May 1995, and 321 responses over a 5-week period in the spring of 1996.
Medical Center-wide Survey
The survey consisted of questions about library use
(whether respondents used library resources, libraries used, frequency of use, purposes for using) and
computer use (whether respondents used computers,
locations where they were used, frequency of use,
purposes for using, specific applications used, and
computer ownership).
Medical Center (including 5 nurses), 10 were out of
town for the summer, 9 had unlisted or disconnected
telephone numbers, and 21 (including 6 students and
5 residents) did not return the phone calls. A preliminary analysis to determine whether people who returned their surveys earlier had different patterns of
library or computer use than later respondents revealed no significant differences, suggesting that nonrespondents neglected to participate for reasons unrelated to the survey’s content.
User Comments
We value written and oral feedback from our users as
a means of validating our quantitative results, suggesting issues to explore in the future, and, most important, confirming benefits implied by usage data.
We collect users’ comments (1) in the surveys mentioned above, (2) with an electronic suggestion box on
the public library workstations, and (3) from reference
librarians and computing user support professionals.
Validity
The names of faculty, postdoctoral trainees, and M&P
staff were drawn from the University’s Human Resources database. Students’ names were drawn from
medical, nursing, physician assistant, and graduate
student enrollment lists supplied by their registrars.
Residents’ names were supplied by the Hospital’s
Medical Staff Office. The Hospital drew a sample of
nurses registered in their patient care system database. Samples were drawn by selecting every xth person from the target population, where x = (population
size)/(desired sample size). Every original list or database was sorted alphabetically by last name except
the populations in the Human Resources database,
which were sorted by university ID number (social
security number except for foreign nationals). We
mailed people up to three copies of the survey, over
a period of 6 weeks, until they returned a complete
survey. People who did not respond to any of the
three mailings were telephoned as often as three times
and asked to either return the survey or complete it
over the phone.
We found a high degree of validity in our data, based
on comparing current results with similar questions
from different evaluation sources. We compared eight
measurements from the present surveys and usage
data with results from surveys and inventories that
we had conducted in the past 5 years. We chose measurements that were unlikely to have changed over
the time intervals of the comparisons to avoid confounding validity with trends. Comparisons included
data on computer usage and ownership, word processor and medical education software usage, interest
in electronic journals, and points of access needed.
None of the comparisons was significantly different
using binomial tests; the median difference was 3.5
percentage points, ranging from a minimum difference of 1 percentage point (for computer ownership)
to a maximum of 11 (for desire for off-site access to
network resources).
With the possible exceptions of medical students and
nurses, the people who responded seemed to be representative of the Medical Center. The overall response rate was 80%. Only 55% of medical students
and 35% of nurses responded. (Our sample of nurses
was poorly drawn, as five of the nurses had left the
Hospital, and another four could not be located for
the telephone calling. We later learned that the database from which they were drawn included ex-employees for record-keeping purposes.) Of the 49 people who did not complete the survey, 9 had left the
Which Information Resources are Being Used?
Results
By all measures, Medical Center information resources are being used widely and with increasing frequency. For most resources, usage follows a yearly cycle and increases from year to year. Figure 1 displays
usage per month of the public library workstations
over the past 2 years. The fluctuations in usage from
month to month follow a predictable pattern for an
academic institution; however usage over the past
year has increased an average of 14% from the previous year.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
We administer an annual questionnaire to a random,
stratified sample of 240 members of the Medical Center faculty, postdoctoral trainees, students, nurses, residents, and managerial and professional (M&P) staff.
The questionnaire enables us to learn what proportion
of the Medical Center community are users, to learn
how various subpopulations use IAIMS resources and
to identify barriers to usage.
GRAJEK ET AL., Evaluating IAIMS at Yale
Journal of the American Medical Informatics Association
Volume 4
Use of some of the major resources accessible over the
entire Medical Center network is also growing (Fig.
3). MEDLINE use is increasing by more than 30% per
year, from 8,536 sessions in May 1995 to 11,259 sessions in May 1996. Although we have only recently
begun tracking use of the Medical Center’s World
Wide Web server and electronic mail, those services
are also expanding. Use of the Yale biomedical gopher
has begun to level off as the Web grows in popularity
among users and information providers. About 15%
of usage of the Web server is from within Yale; the
remaining 85% of usage is external. MedMenu is one
of the sites most heavily used by both Yale (300 – 500
times a month) and external users (700 – 1,000 times a
month). Other frequently used sites include those of
the Medical Library and academic departments for
Yale users, and the Medical Center and School of
Medicine main pages, the Center for Advanced Instructional Media’s Web Style Manual, information
about applying to the School of Medicine, academic
departments, and the Library for external users.
Mar / Apr 1997
143
F i g u r e 1 Use of Library NetMenu workstations: total
connections per month. (Source: usage data from library’s 21 workstations).
worked applications as MEDLINE, the University library catalog, Internet access tools (e.g., Netscape),
molecular biology computing tools, and the Business
Management System, a networked collection of administrative databases. Use of networked resources
varied among subpopulations from only 57% of
nurses to 100% of students, residents, and research
faculty (x2 = 31.98, p > .001) (Table 2).
Special resources for specific populations. Next we
examined usage of resources whose primary purpose
is to support either research, patient care, or education
to measure the impact of IAIMS on the primary missions of the Medical Center. For example, how many
physicians are using clinical information systems? As
use of the relevant class of resources by researchers,
clinicians, or teachers and students becomes widespread, IAIMS becomes integral to achieving the Medical Center’s primary missions. Low use of relevant
resources becomes a warning signal that we may need
Who Uses Information Resources?
Use of information resources is widespread and high,
according to our 1995 survey. Most of the Medical
Center community is using library resources (82% of
all faculty, students, nurses, residents, and managerial
& professional staff), computers (96%), and networked information resources (83%).
Electronic information and tools are as varied as the
people who use them. Resources range from tools and
applications that are potentially useful to almost all
of the Medical Center community, such as electronic
mail, to specialty-specific applications such as molecular biology software. In our analysis we focused on
network based information resources which are more
indicative of an IAIMS environment.
Who uses global applications and tools? We asked
annual survey recipients whether they used such net-
F i g u r e 2 Use of Hospital NetMenu workstations: connections per workstation per month. An asterisk indicates an estimated figure due to interrupted logging.
(Source: usage data from clinical workstations).
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
Public workstation usage in newly established sites is
increasing more dramatically. The School of Nursing’s
NetMenu workstation was used 85% more in 1996
than in 1995. The Hospital has undertaken a major
expansion of its public clinical workstations, adding
more workstations (from 16 to 71 with at least 30 more
planned) and applications (from 12 to 30 so far) and
training users. With so many additional workstations,
an overall increase in usage is to be expected. What
is noteworthy is the increase in public clinical workstation monthly usage per workstation (Fig. 2). Individual public clinical workstations were used an average
of 75% more in the summer and fall of 1995 compared
with 1994.
Number 2
144
GRAJEK ET AL., Evaluating IAIMS at Yale
to look for possible barriers to information delivery
of those resources to the people who need them. To
explore this issue, we first looked at mission-related
uses of the public library workstations. Then we
moved beyond users to the Medical Center community by examining questionnaire results.
Based on our May 1995 online survey of public library
workstations, most usage is clearly related to support
Table 2 n
Use of Networked Applications
Status
% Using
N
Research faculty
Students
Residents
Teaching faculty
Post-doctoral trainees
Clinical faculty
Research M&P staff
Administrative M&P staff
Nurses
100
100
100
94
88
87
80
61
57
19
165
13
16
16
23
20
19
7
of research, clinical care or education, with research
being the most common goal. Almost half (46%) of all
321 respondents reported using the workstations to
find information for a research project. One fifth (20%)
of respondents were looking for course-related information, and 16% were looking for information to treat
a patient. Analyzing the data separately for faculty,
postdoctoral fellows and residents, and students, we
learned that although research was still the primary
reason for using the public library workstations, the
three groups had quite different information needs
(Table 3).
Faculty were the most likely to be using library workstations for patient care and students the least likely
(x2 = 15.9, p < .001). Significantly more postdoctoral
fellows and residents, and fewer students, used library workstations for research (x2 = 14.5, p < .001).
There were no significant differences in the groups’
use of library workstations for education. Students
had a fourth major reason for using the workstations:
electronic mail. The library is many students’ primary
point of access for electronic mail. Forty-three percent
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
F i g u r e 3 Use of selected resources.
Journal of the American Medical Informatics Association
Volume 4
of medical students use email solely from the library,
and only one-third of medical students connect to
their email accounts via modem from home.14
For people conducting research (research faculty, postdoctoral trainees, research M&P staff, and graduate
students), we selected five resources: using the library
to find research information, using computers to find
research information, analyzing research data, using
computers to prepare grant proposals, and molecular
biology computing. Overall, significantly more researchers than non-researchers used four of the five
research resources (x2 ranged from 9.476– 17.388, p <
.01), but we found no differences for use of computers
to prepare grant proposals (x2 = 1.71, p = .191). More
graduate students (50%) reported using molecular biology computing than other respondents, followed by
research faculty and research M&P staff (;27%).
Fewer than 10% of post-doctoral trainees used molecular biology computing.
For patient care, we compared residents, clinical faculty, and nurses with other respondents on their use
of libraries to find clinical information, of computers
to find clinical information, and of the Hospital’s
CCSS patient care system. In all cases, significantly
more patient care providers used those resources than
non-clinicians (x2 ranged from 14.07– 52.57, p < .001).
However, nurses were far less likely than clinical faculty or residents to use libraries or computers to find
clinical information. Residents were heavy users; each
resource was used by over 80% of residents.
We compared teaching faculty and students with
other respondents in their use of three educational resources: computers to prepare presentations, libraries
to prepare for courses, and medical education software. As a whole, teachers and students were significantly more likely to use educational resources than
all other respondents (x2 ranged from 4.02– 18.82, p <
.05). Physician assistant (PA) students were not significantly different from non-educational respondents
in their use of computers to prepare presentations.
The primary users of educational software were PA
students and medical students; fewer than 20% of all
other respondents reported using it. Over 70% of
medical students used each of the three resources.
We drew three conclusions from our analyses of why
resources are used:
1. IAIMS resources are being used to support research, clinical care, and education.
145
Mar / Apr 1997
Table 3 n
Reason for Using Public Library Workstations
N
Research
Education
Patient care
Electronic mail
Total
Faculty
Postdoc/resident
Student
66
49%
23%
32%
0%
;100%*
44
66%
16%
25%
5%
;100%*
165
35%
19%
11%
34%
;100%*
*Totals slightly exceed 100% because 8% of respondents reported using the workstation for more than one reason.
2. There is room for growth. For example, only about
two-thirds of research faculty are using computers
to prepare grant proposals, only 71% of nurses are
using the Hospital’s online patient care system,
and, only 12% of nursing students are using medical education software. We need to determine
whether we can improve tools, training, or points
of access to facilitate these groups’ use of these resources.
3. Most resources have secondary, as well as primary
uses and users. For example, although clinicians
are the primary users of clinical systems, the same
information is also used by students as part of their
training, and by faculty for clinical research.
Where are Information Resources Used?
Where are people using computers? From the 1995
annual survey, the most common location was University offices, where 75% of non-students reported
using computers. More than half of respondents said
they used computers on the Hospital floors (clinicians
only), at home or in the library (Fig. 4). Eighty-eight
percent of respondents use computers in more than
one location, with some using them in as many as
seven.
Use of the MEDLINE database dramatically illustrates
what people do when barriers to information access
are removed. Used by an estimated 72% of the Medical Center community, MEDLINE is the Medical Center’s most popular database. Several recent efforts
have made MEDLINE more accessible: (1) The introduction of OVID MEDLINE in April 1993 was the first
time that a full MEDLINE file was provided on a networked system without direct charges for Medical
Center users. (2) The Hospital, which added MEDLINE to its public clinical workstations in 1994, is in
the process of increasing its public clinical workstations from 6 in June 1993 to a present total of 71. (3)
The School of Medicine created a Desktop Computer
Support Unit to provide and support standardized
tools for connecting to networked resources, including
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
We analyzed data from the annual Medical Center
survey to learn the extent to which researchers, teachers and students, and clinicians were using information resources relevant to their domain.
Number 2
146
Are Multiple Sources of Information
Being Integrated?
We have three indications that people are integrating
information from multiple sources. First, in our December 1994 online survey of users of the public library workstations, over half of the 325 respondents
reported using more than one application during their
session (Fig. 6). We are reasonably certain that these
people are using multiple applications to meet a single
information need, because in our May 95 online survey, 92% of respondents were using the workstation
for a single purpose.
Second, most respondents (79%) in the 1995 annual
survey reported using both libraries and computers
when looking for either clinical or research information. A small percentage (17%) use only libraries and
only 4% use only computers to find such information.
Third, 79% of the respondents to the online survey on
InfoFinder considered InfoFinder to be a useful tool.
People most commonly use InfoFinder to supplement
MEDLINE and online catalog searches for research information (x2 = 13.3, p < .001). People using InfoFinder
for this reason rated it as more useful than people
using it for other reasons (x2 = 6.02, p < .05), such as
searching for patient care information or needing to
find information but not knowing exactly where to
look.
Discussion and Conclusions
Factors Affecting Usage
Many factors help determine whether an information
resource is used. Our experience and the present evaluation lead us to believe that cost, location of workstations, use of menus and search tools, and developments in information technology all affect usage.
Providing information when and where it is needed
apparently led to increased use, as indicated by the
dramatic increase in use of information resources
when public clinical workstations were introduced in
the Hospital. We hypothesize that what has changed
is not the number of times people would like to consult MEDLINE, but instead the number of situations
in which they are able to do so easily.
Improved access also affects how work is done. For
example, a professor of cell biology has told librarians
that he no longer uses a reprint file now that he can
conduct MEDLINE searches whenever and wherever
he needs to. He has also begun advising his graduate
students to use MEDLINE searching as a replacement
F i g u r e 4 Where people use
computers.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
MEDLINE. (4) The University and the Hospital are
improving tools for connecting from off-site locations.
While Figure 3 demonstrates that overall MEDLINE
usage has been increasing, the trend lines in Figure 5
reveal the growth is due to the fact that people are
increasingly accessing MEDLINE from Hospital, office, laboratory, and home computers (R2 = .76). We
estimate that 8% of non-library use is from public clinical workstations, and 92% is from office, laboratory,
and home computers.
GRAJEK ET AL., Evaluating IAIMS at Yale
Journal of the American Medical Informatics Association
Volume 4
Number 2
Mar / Apr 1997
147
for their own reprint files. An emeritus professor of
genetics conducts a regular monthly search of MEDLINE to update a comprehensive bibliographic database. He performs the search at the most convenient
location, which might be his office, his home in New
Haven, or his home in Florida.
Menus help users select and access frequently used
resources. In addition menus highlight important online information resources and thus encourage their
use. In one experiment we moved the InfoFinder to
the top of the menu and it moved from the eighteenth-most-used resource to the eighth-most-used resource. This suggests that there may be some merit in
customizing menus for different environments, as we
have done for public library and clinical workstations,
and in defining an explicit goal or strategy for organizing the content of menus.
Although people are using computers in a variety of
locations, there is clearly room for improvement in
outreach, especially in providing access to information resources in outpatient care settings and Hospital
offices, where fewer than half of clinicians currently
use computers. Anecdotal data suggest that locating
electronic information resources in the midst of a patient care setting can provide indirect as well as direct
benefits. For example, one Hospital resident reports
he can now make productive use of slow moments on
the floors by using nearby public clinical workstations
to conduct MEDLINE searches and access Internet resources.
Integration Issues
Although most integration currently occurs in the
minds of people, integration among information sys-
Search tools such as InfoFinder and various Web
search engines are of great help to the users we surveyed in finding the ever-increasing number of online
resources beyond MEDLINE.
Areas for Improvement in Information Delivery
Our evaluation has identified several areas for improvement in access to online information. We need
to improve the delivery of specific resources to particular subpopulations. For example, we need to learn
whether problems with access, training, and/or resources themselves are limiting nurses’ use of networked information resources. Addressing that issue,
Yale – New Haven Hospital is adding electronic mail
access to its public clinical workstations and is providing nurses and residents with e-mail accounts.
F i g u r e 6 Number of different applications used in a
single session.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
F i g u r e 5 MEDLINE usage.
Source: MEDLINE usage
data.
148
Integration also occurs within clinical systems. Most
departmental clinical systems can now be accessed
through a common order entry and results reporting
system for inpatient care. In addition data repositories
are being developed to consolidate clinical data
within the Hospital and to consolidate financial and
clinical data within the Medical School.
Plans for Future Evaluation Activities
In our second year of IAIMS implementation, we are
continuing our basic evaluation strategies of collecting
evaluation data based on its capacity to inform the
decision-making process and of using multiple measures at multiple times to improve reliability and assess trends. However, as our IAIMS environment matures and as the nature of information technology and
applications evolves, those strategies are beginning to
lead us in new directions. We are changing the format
of our IAIMS monthly report to focus more on overall
use (network-wide and remote) and less on use at any
one location such as the library. Upcoming evaluation
projects will explore the effectiveness of the World
Wide Web and the impact of medical logic modules
on clinical care.
We believe we have begun to answer the questions:
which resources are being used, who uses resources,
and when and where are resources being used? We
are less satisfied with our knowledge of integration
and its impact, primarily because we are still in the
process of developing integrated applications. Thus
evaluation of the impact of systems integration will
be a goal of the next stage of our evaluations.
We are also designing an evaluation of the impact of
information delivery on research, patient care, and education. We will begin by assessing users’ perceptions
of the impact of information resources on their work,
and then we will try to relate specific outcomes measures to specific changes in the information environment.
Concluding Remarks
We believe our evaluation is not simply documenting
the impact of IAIMS upon the Yale Medical Center
community’s access to information; it is also shaping
that impact. We ensure that key decision-makers receive and discuss evaluation information by regularly
putting it on meeting agendas. Evaluation data have
helped justify initiating projects (providing connectivity to the medical students’ dormitory), expanding
projects (the Hospital’s deployment of public clinical
workstations), and refining projects (a series of computer-based medical education quizzes).
The benefits we have derived from our evaluation
lead us to encourage other IAIMS sites to incorporate
evaluation into their own activities. Specifics will differ, but we believe the key components to a successful
evaluation are collecting data that will be used to
make decisions, sharing and discussing evaluation
data with administrators and other decision-makers,
instituting processes for the ongoing compilation of
information about usage and users, mounting formative evaluation studies on an as-needed basis, and
periodically reviewing the evaluation process itself.
Any IAIMS effort is a major undertaking, pulling together the
efforts of many individuals throughout the institution. The authors thank all those who contributed to the work reported in
this paper; in particular, Lisa Miller of Academic Computing
and Janet Miller of the Medical Library.
References n
1. Matheson NW, Cooper JA. Academic information in the academic health sciences center: roles for the library in information management. J Med Educ. 1982;57:1 – 93.
2. Shifman, MA, Clyman JI, Paton JA et al. NetMenu: experience in the implementation and evaluation of an institutional menu of information sources. In: Proceedings of the
Seventeenth Annual Symposium on Computer Applications
in Medical Care. Washington DC. November 1993, 554 – 8.
3. Miller PA, Frawley SJ, Wright L, Roderer NK, Powsner SM.
Lessons learned from a pilot implementation of the UMLS
information sources map. J Am Med Inform Assoc. 1995;2:
102 – 15.
4. Rodgers RPC. Automated retrieval from multiple disparate
information sources: the World Wide Web and the NLM’s
Sourcerer project. JASIS. 1995;46:755 – 64.
5. Jacso P. OVID/MEDLINE: a gourmet choice. Database.
1994;17:31 – 8.
6. Lindberg DAB, West RT, Corn M. IAIMS: an overview from
the National Library of Medicine. Bull Med Libr Assoc.
1992;80:244 – 6.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
tems is beginning to occur in some situations. The library has an online database system from OVID that
provides access to MEDLINE, CINAHL and a core
collection of 15 biomedical journals (text plus graphics). People can use the OVID interface to run the
same search strategies on several databases and, with
a keystroke, move from a MEDLINE citation to the
full text of the article if it is one of the 15 core journals.
Even when MEDLINE sources are not among the online journals, users can eliminate a visit to the library
by requesting a document via a Web form. We may
be able to achieve another advance in integration with
the release of Web front-ends to all our bibliographic
and many full-text databases, including MEDLINE
and Yale’s online library catalog. We hope to experiment with adding links to InfoFinder on search result
pages to suggest additional information sources to
users if they need them.
GRAJEK ET AL., Evaluating IAIMS at Yale
Journal of the American Medical Informatics Association
Volume 4
Mar / Apr 1997
149
11. Clayton PD, Anderson RK, Hill C, McCormack M. An initial
assessment of the cost and utilization of the Integrated Academic Information System (IAIMS) at Columbia – Presbyterian Medical Center. In: Proceedings of the Fifteenth Annual Symposium on Computer Applications in Medical
Care. Washington DC. November 1991, 109 – 113.
12. Gorry GA. IAIMS development at Baylor College of Medicine. Bull Med Libr Assoc. 1992;80(3):247 – 52.
13. Snyder J, Trumbo J. A backbone to the world. Info World.
1996;18:68 – 77.
14. Grajek S. Results of an online survey of Biomed VAX use
by Medical Library users. Internal report, March 1996.
Downloaded from https://academic.oup.com/jamia/article-abstract/4/2/138/731189 by guest on 03 June 2020
7. Clayton PD. Integrated Advanced Medical Information Systems (IAIMS): payoffs and problems. Meth Inform Med.
1994;33:351 – 7.
8. Roderer NK, Clayton PD. IAIMS at Columbia – Presbyterian
Medical Center: accomplishments and challenges. Bull Med
Libr Assoc. 1992;80(3):253 – 62.
9. Broering NC, Bagdoyan HE. The impact of IAIMS at
Georgetown: strategies and outcomes. Bull Med Libr Assoc.
1992;80:263 – 75.
10. Stead WW, Bird WP, Califf RM, Elchlepp JG, Hammond
WE, Kinney TR. The IAIMS at Duke University Medical
Center: transition from model testing to implementation.
MD Computing. 1993;10:225 – 30.
Number 2