Verification Validation BME DesignII
Verification Validation BME DesignII
Verification Validation BME DesignII
LECTURE SIX
Dr. Daniel Akwei Addo
In the next two sections of this lecture you will learn how to plan and conduct tests appropriate for your medical
device and analyze test results. Although you may have engaged in feasibility tests earlier in the design
process, testing takes center stage during verification (are technical specifications met?) and validation
(are customer/user needs met?). These tests aim to demonstrate that a new product is both safe (not
harmful) and efficacious (performs as required). Some of the most impactful academic design projects
are simple; still, the design and execution of the tests and the analysis of test results present significant
technical challenges.
First section of this lecture is focused on benchtop and laboratory testing in nonliving systems. You will learn how
testing is critical in verifying specifications, validating your design with customers, determining the
probabilities and severity of known failure modes, performing a risk analysis, and discovering new
failure modes. Guidelines are provided on electrical tests, static and dynamic mechanical tests, testing
to failure, and using testing standards. In addition, you will learn how to simulate biological fluids, tissues,
and organs in nonliving systems, as well as the role of aging and shelf life studies.
The next section builds upon the guidance for testing in nonliving systems but focuses on the unique
challenges in conducting tests in vitro (cell and tissue) and in vivo (biocompatibility, animal, and human
clinical trials). Testing in living systems is often required for verification and validation of certain
specifications and user needs, respectively. Furthermore, data from living systems is often necessary in
obtaining regulatory clearance, convincing health insurance companies to reimburse for use of the
device, and creating promotional materials. Working with living systems brings with it ethical and legal
protections for test subjects. You will therefore learn how you can obtain the proper approval and safety
training to perform tests in living systems.
Page 1 of 19
By the end of Both sections you will be able to:
• Design and conduct the appropriate tests in nonliving and living systems to verify each of your
specifications.
• Design tests that validate that your solution will meet the needs of users.
• Select and execute appropriate data analysis methods for design verification and validation.
• Present test results in a way that will convey the efficacy and safety of your design.
• Become qualified in engaging in living system testing and in drafting proposals to university committees for
in vitro testing and in vivo testing.
• Complete a risk analysis of your design solution and create mitigation plans for reducing risk.
Medical device design aims to create a safe (not harmful) and efficacious (does what is claimed)
solution. Following a robust design process, complying with design controls and regulatory requirements,
and testing the design solution will ensure these goals are met. There are several reasons why
testing is an integral part of the design process. It is an important requirement of design controls,
provides support for regulatory and reimbursement submissions and sales of a new product, helps
improve the product and prevent failures and recalls, and can save time and money when used to
detect design defects before they reach the customer. These reasons are presented in greater detail in
this section.
Testing is an essential component of design controls. It is an iterative process that often guides the design process.
Test results determine how close a design solution is to meeting all technical and nontechnical requirements and
helps identify any changes needed for the next design iteration. It often indicates a need to change the design to
improve performance, increase safety, and reduce residual risk. Once design changes have been implemented,
testing is repeated to confirm that the revised design performs as expected and does not introduce new problems.
Test results often determine if design iterations and additional testing are needed and should be added to the project
schedule.
Testing allows engineers to discover potential design defects in a new product before it reaches the
customer. Simple design changes may unknowingly introduce a new variable that results in decreased
performance or safety. Failure to repeat testing of the new design iteration can lead to product recalls
caused by design changes. Testing can help identify hidden problems, improve safety, reduce risk, and
reduce the potential for product recalls. Generally, the extent of testing before market introduction will
increase with higher potential risk.
Most new medical devices will be tested to determine how they could fail. In the previous lectures, Failure
Modes and Effects Analysis (FMEA) is introduced to identify potential failure modes and their effects
(impact). To perform a risk analysis of a design (without actual test results), the probability of occurrence
for each failure mode and the severity of the failure’s impact are estimated, and an estimated risk
value is calculated. Testing on final (or nearly final) devices can determine actual probabilities, allowing
for a more accurate risk analysis to be conducted in which actual probabilities are used instead of
estimates. Methods for determining more accurate probabilities are presented later in this lecture.
Page 2 of 19
§ Legal and Ethical Reasons for Testing
There are ethical and legal considerations associated with the testing of a medical device. If a device
failure causes harm to a user, is the designer legally or ethically responsible? Proper documentation of rigorous
failure testing not only protects end users from harm but may also provide designers and companies with some level
of legal protection while reducing their liability.
There are several ways in which testing and ethics intersect. First, in the context of a medical device,
using a market introduction of a commercialized medical device is not an acceptable alternative to
premarket testing as a way to find defects in medical devices. Some popular innovation methods advocate
introducing a Minimally Viable Product (MVP) into the market as soon as possible with the intent
of quickly identifying failure modes reported by their customers.
Improvements are then implemented, and field testing is repeated. First, while this method may be
acceptable in some industries, it is ethically and legally unacceptable for medical devices. Second, simple
design changes are often assumed to have no impact on product function or safety, and additional testing
is not conducted. These changes, however, may unknowingly introduce a new variable that results in
decreased performance or safety. Failure to repeat testing of the resulting design iteration has often
resulted in product recalls caused by design changes. Third, the engineering profession, through various
engineering societies (e.g., National Society of Professional Engineers, Biomedical Engineering Society,
Institute of Electrical and Electronic Engineers), and most companies have strict ethical codes that include testing
as a pathway to protect the public from harm.
Regular and iterative testing can help reduce project risks by identifying defects early in the design
process. The 1-10-100 rule shown in Figure above illustrates the benefits of detecting and correcting
defects as early in the design process as possible. If detected before a design is finalized and transferred
to production, assume that the relative cost of detection and correction would be $1. If detected after
the design is finalized and production has begun, the relative cost would be $10. This increase often
includes the cost of revised or new tooling needed to correct the design defect. If detected after market
Page 3 of 19
release, the relative cost could climb to $100 depending on how many units have been shipped to customers. This
increase is due to the costs associated with retrieving product from the field (e.g., shipping
product back to the manufacturer, replacing returned product) and any legal actions that could
result. From a financial perspective, this illustrates the importance of detecting a design defect as early
as possible in the design process. Early detection not only protects the end user of the device (and complies
with a company’s ethical responsibility to produce a safe and efficacious product) but also makes
good business sense.
The adage “time is money” applies to the design process. As you navigate your design process,
perhaps your most precious resource is time. In this regard, the detection of a design flaw or defect can
seem like a setback. However, by detecting the problems and analyzing risks early in the design process
through testing, you will avoid wasting time and money on concepts that will not meet customer needs
or solve the customer’s problem.
Another reason for testing is to support sales. Test results regarding device performance and capabilities
are often cited in marketing and sale literature and publications in medical journals. If results are favorable, sales
personnel may use them to persuade customers to purchase the new device. If testing includes evaluation of
competitive devices, results are often used to compare the new device’s performance with that of the competitive
devices (competitive benchmarking).
Two significant barriers to the commercialization of a new medical device are obtaining regulatory
approval and convincing health insurance providers to reimburse for use of the new product. Without
reimbursement from health insurance providers, the patient would pay the often-high price of a medical device,
making it very costly and thus significantly limiting the product’s potential market. A significant portion of most
medical device regulatory submissions will include test results to demonstrate the safety and efficacy of the device,
as well as evidence that design controls have been followed and risk analyses have been conducted. Reimbursement
decisions are based on evidence that the new medical device is reasonable and necessary.
Testing appears in many forms throughout the design process. It determines how close a design
solution is to meeting all performance and other requirements and helps identify what changes
may be needed for the next design iteration. In this respect, the purpose of the testing presented in
this section is to measure some aspects of performance. Various forms of performance testing are
conducted at different stages in the design process, starting with feasibility tests involving rough
prototypes or analytical models, moving on to tests of more detailed and final prototypes, and ending
with tests of the final product. When testing indicates the need to revise the design, some of
these tests will need to be repeated. This section focuses on the forms of performance testing and
when they are typically conducted, to help you consider the most appropriate tests for your design
solution.
Feasibility Testing
It is important to identify concepts that are not technically feasible as early as possible in the design
process to avoid wasting time and money on those that will not meet (or have the potential to meet)
customer needs or solve the original problem. Feasibility testing, often called proof-of-concept
testing, occurs early in a design process to determine if a design concept has the potential to meet performance
requirements and technical specifications. It helps you answer the question, “could our design concept
possibly work?” These tests often involve rough prototypes or analytical models.
Page 4 of 19
Feasibility tests of physical prototypes are often conducted to determine if one critical
specification needed to meet performance requirements can be (or has the potential to be) met or if a
particular function or subsystem of the design has the potential to function as required.
Analytical models can be used to determine the technical feasibility of a design concept. The use of
basic mathematical models such as free body diagrams, finite element, circuit, and stress analyses, as
well as computer simulations, are helpful tools for identifying feasible concepts worth further development
and exploration. For example, to determine the feasibility of a new syringe design, a manual
calculation or computer simulation can be used to calculate the minimum pressure needed to produce
the required flow rate of liquid out of the attached needle and into the patient’s body. This will depend
on several factors such as the needle and barrel diameters, lengths of the barrel, syringe tip, and needle,
viscosity of the liquid, and resistance of the body tissue. Once this minimum pressure has been calculated,
the diameter of the plunger can be used to calculate the minimum plunger force required to produce
the desired flow rate. This can then be compared to anthropometric and biomechanical data of the
typical forces that physicians, nurses, and other healthcare providers can apply to a syringe plunger to
determine if the new design could function as required.
Prototype Testing
Prototype testing is typically conducted on the first fully functional prototype in a laboratory setting to
determine if a design solution meets specifications and performance requirements. At this stage of a
project, it is often found that only some requirements have been met. Performance testing of these
prototypes helps identify what, if any, changes are needed to improve the design. The detailed design
approaches presented in the previous lecture are then applied, a new prototype is constructed, and testing of the
improved prototype is repeated. Some animal testing might begin, as presented in the subsequent lecture, on more
advanced prototypes.
Product Testing
When testing advanced prototypes confirms that the latest design iteration is acceptable, the design is
temporarily frozen. Depending on the device and its clinical application, it may be transferred to production
to begin scaling up for manufacturing. Additional prototypes may be built to support animal or
clinical studies prior to manufacturing. Product testing is conducted to determine if the product, manufactured
using actual production processes and final materials, meets performance requirements and is
safe and efficacious. These tests may be performed at different stages during the manufacturing scale-up
process and are used to determine the potential effects of any changes introduced by the manufacturing process on
the performance or safety of the new device. These effects may be caused by new variables that were not present in
the previously tested prototypes (and thus were not tested) such as mold release agents, lubricants, new production
tooling, new assembly procedures, sterilization processes, or packaging. Ideally, testing of final production units
(manufactured per the final design and materials) using the final production, sterilization, and packaging processes
would be included in the testing plan for a new medical device.
The decision as to when to test a product may be affected by financial considerations. For example,
there would be a high financial risk to investing in expensive multi-cavity injection molds before receiving
acceptable results from a clinical study. If the study indicated that design changes were needed,
depending on the complexity of the changes, the tooling might need to be scrapped or revised at considerable
cost. To reduce financial risk, many companies will produce devices for clinical studies using
less expensive temporary tooling. They will wait to receive positive clinical study results before freezing
the design and committing to building expensive production tooling, after which product testing of
final production units would be conducted. However, other companies might be willing to take this risk
if they feel that it is outweighed by the potential benefits of saving time and allowing for a quicker
product introduction.
Page 5 of 19
Design Verification
Design verification answers the question, “did we make the product right?” It is typically accomplished
by testing prototypes to experimentally confirm that the design meets the most recent version of the
specifications. In the language of design controls, this means that design input = design output, and design
specifications are met. Verification testing also helps you identify specifications that still need to be met and
provides a level of assurance that the product will function as required.
Typically, a test is designed and conducted, and results are compared to ideal and marginal values
for each specification. For example, consider a specification for a handheld device that states that it
must not weigh more than 5 lb (2.27 kg) and ideally would weigh 3 lb (1.36 kg) or less. A test to weigh
the device would be included and conducted as part of verification testing. If the device weighed 5 lb
or less, it would pass this requirement, and the specification would be met. If the device fails this test,
you would then determine how to reduce the weight or decide if a slightly higher weight would be
acceptable. Prototypes are refined until all specifications are met through the iterative design process.
In this section, guidance is provided on designing and conducting verification tests applicable to
your device. Included are the use of testing standards, bench testing in laboratory settings, and packaging,
distribution, and shelf-life testing. The test protocol for your device should include only those tests needed
to verify that each specification has been met. For example, tests to determine if a product is waterproof
are not necessary if being waterproof is not a requirement included in the list of specifications.
Standards play an important role in design verification. They include test procedures that have been created,
revised, validated, and approved by experts and practitioners. They provide you with industry-accepted
and endorsed test procedures in one place. Standards can prevent your team from “reinventing the wheel” and
creating custom test procedures that may exist, saving time and effort and ensuring that the appropriate test
methods will be used. Some standards include performance requirements (minimally acceptable values) that you can
use to determine if your design performs acceptably. Ideally, these requirements have been incorporated into the
list of target specifications established.
Testing per standards allows medical device companies to advertise that their products comply with
a particular standard (or standards). This is often a customer requirement and is also important to regulatory
and reimbursement agencies. The use of standards benefits customers. For example, when considering which brand
of ureteral stent to purchase, a urologist may compare the loads required to straighten the proximal curls (located
in the kidney) between stents from different manufacturers. This performance characteristic is associated
with resistance to migration of the stent. If all companies test per their own standard, then direct
“apples to apples” comparisons of their products would not be possible. Using the same standard test
procedures avoids confusion and allows for direct comparison between stents from different manufacturers,
thereby allowing the urologist to make a better, more informed purchasing decision.
§ Bench Testing
The term “bench testing” refers to tests done on a laboratory bench or in a laboratory setting outside of
living systems. Bench tests may be conducted during any phase of the design process. They can be used
to evaluate one particular performance specification early in the design process to test a new feature or
functionality or verify the entire detailed design solution prior to moving on to animal and human clinical
studies. In this section, we discuss some common forms of bench testing.
The goal of bench testing is to simulate, in a laboratory setting, the actual use, service environment,
Page 6 of 19
and function of the device. Bench tests should attempt to simulate the physiological service environment, including
mechanical and electrical loading, lubrication and wear conditions, fluid composition, concentration, temperature,
and pH, magnetic fields, and other energy sources (radio frequency, gamma radiation), as appropriate. Prior to
testing, test samples and test equipment should be inspected and calibrated, respectively. During testing, it is helpful
to monitor changes in performance characteristics over time such as flow rates (tubular devices where fluid is
transported, such as catheters and stents), volume of wear debris generated (in designs where parts contact each
other, such as joint replacements), or voltage output and leakage current of electronic devices (such as
ECG monitors). Upon completion of all testing, test devices should be inspected for signs of degradation
and changes in physical properties over time. This would include electronic devices with decreased
sensitivity and accuracy and values that are out of calibration, implantable devices showing signs of
corrosion and wear, product failures (product no longer functions as intended), loss of physical properties
such as strength and flexibility of polymeric devices, discoloration of materials, obstruction and
kinking of tubular devices, and fracture of devices.
Single-use, disposable medical devices, such as ECG electrodes, are designed to be used once and then
discarded. Other devices such as ECG monitors and some surgical instruments are designed for multiple
uses. It is important to confirm that all devices perform as required for the expected service life of the
device (single or multiple uses). For multiple-use devices, it is critical to know if and how their performance
changes over time. These devices could be tested for a minimum number of individual uses or
hours of operation, based on the expected service life of the device, or they could be tested until failure.
Nondestructive tests leave the device intact, functional, and typically undamaged after testing.
These tests are often conducted when there is a limited number of test samples needed for other purposes
such as demonstrations to customers, sponsors, and clients. Data from these nondestructive tests
can confirm that the product will meet performance requirements but will not yield information regarding
how long the device can function until failure.
Destructive testing results in damage to the device. It is conducted to confirm device performance
and determine how and when a device will fail. Results of destructive testing can be used to determine
more accurate values for probabilities of occurrence used in the design FMEA and risk analysis process, and the
need for product warnings.
Comparisons of results from nondestructive and destructive tests are necessary to confirm factors of safety. For
example, assume that a device will be subjected to a maximum compressive load of 45.4 kg (100 lb) during normal
use, and a test demonstrates that the device can withstand this load with no damage. To determine the factor of
safety, the device must be tested to failure. If the load at failure is 90.8 kg (200 lb), then it can be concluded
that a factor of safety = 2 exists for the maximum compressive load.
In an academic design project, you may be required to deliver a functional prototype to a sponsor or
client. If you have enough funds to create only one prototype, then destructive testing would not be
appropriate because it would destroy your only prototype. For this reason, you may only want to perform
tests that do not damage, weaken, or decrease the device’s service life. In many situations, destructive
testing is not necessary. You can demonstrate that your device meets specifications (design verification)
through nondestructive testing. However, you should be aware that rigorous testing (both destructive and
nondestructive) is often required before a device is allowed to be used in a healthcare setting.
Many medical devices will be subjected to various forms of mechanical stress. This includes devices
that do not have a primary mechanical function. For example, a pacemaker’s electronics, wires, and
sensors will experience mechanical loading while in the body. Most mechanical tests can be classified
Page 7 of 19
as either static or dynamic.
Static testing, in which a mechanical component or device is loaded with a single static mechanical
load (sometimes until failure) is used to determine mechanical properties and the maximum load to
failure. This type of testing is most appropriate for single-use medical devices that will be loaded once
and then discarded. Knowing the maximum load to failure and the expected load during use of the
device allows design engineers to calculate a factor of safety for a single-use device.
For load-bearing, implantable devices such as joint replacements, static testing may have less
value. These devices will be repeatedly loaded and unloaded (cyclical loading) during each gait
cycle when in the body. For this reason, cyclical loading better simulates the service environment
and actual loading conditions of these implants. It typically involves mechanical loads that produce
fluctuating tensile and compressive loads and can lead to fatigue failure due to the formation and
propagation of surface cracks. This type of testing using dynamic loads is often referred to as
fatigue testing. When designing a dynamic fatigue test, load magnitude, load frequency, number of
cycles, range of motion, and environmental variables such as temperature, pH, and composition of
test fluids are chosen to simulate the expected service environment most accurately.
To determine the time or number of cycles to failure, testing must continue until the product fails
(no longer functions as required). This form of destructive testing, if conducted with an appropriate
sample size, can yield a distribution of time-to-failure for the design. It allows design engineers to predict
the percentage of devices that will exceed the required service life and determine the typical failure
modes. This can be useful information when exploring ways to extend the device’s longevity and
reduce the failure rate.
Testing until failure of devices at different mechanical loads allows construction of an S-N
(stress vs. number of cycles) curve that can be used to understand the relationship between the number of loading
cycles until failure and the stress that is applied to a component, device, or material sample during
mechanical loading. Knowledge of this behavior allows the determination of the fatigue limit of a
device or material, which indicates the stress below which it will not fail due to mechanical loading and
resulting fatigue stress. It also allows determination of fatigue life, which is the maximum number of
loading cycles until a fatigue failure occurs. Fatigue strength (or endurance limit) is the stress that will
produce a fracture after the fatigue life is reached. Materials such as some steels and titanium alloys
will not fracture if stresses are below the fatigue limit.
Most materials do not exhibit a distinct fatigue limit. Decreasing stress in these materials increases
fatigue life, and increasing stress decreases fatigue life. In other words, materials can withstand a much
higher single, static load than they can withstand a fluctuating load applied through many cycles. This
relationship implies that fatigue strength (involving many loading cycles) will be much less than a
material’s ultimate tensile strength (involving one loading cycle). For example, the ultimate tensile
strength of Ti6Al4V alloy is 930 MPa (135 kpsi), but its fatigue strength is only 420 MPa (61 kpsi) at
108 cycles, which is 45% of the ultimate tensile strength. Many implantable materials have a fatigue
strength between 30% and 50% of their ultimate tensile strength.
• Electrical Testing
There are a variety of tests required for electronic medical devices that evaluate device performance and
safety for the patient or user. Safety concerns include electrical shock, burns, and electrocution hazards.
For example, some electrical components can produce significant amounts of heat, which can present
burn hazards to the patient or create a fire hazard in a clinical setting. Tests for these and other potential
hazards may include thermal and leakage current measurements and determining the potential for electrical
shorts if a wire fails or becomes exposed. Standards for testing electronic medical devices provide
approved and recommended test procedures and define the maximum safe levels of leakage current and
other related electrical hazards. IEC 60601-1 Medical electrical equipment – Part 1: General requirements
for basic safety and essential performance is an international standard that covers electrical
Page 8 of 19
equipment used in medical practice, emphasizing basic safety and performance requirements.
Electrical testing is also conducted to evaluate device performance in the expected service environment.
For example, there are tests to determine the immunity of an electronic device to electromagnetic
interference (EMI) that can occur when electromagnetic fields produced by one electronic device interfere
with the normal function of a nearby electronic device. ISO 14117: 2019 Active implantable medical
devices—Electromagnetic compatibility—EMC test protocols for implantable cardiac pacemakers,
implantable cardioverter defibrillators and cardiac resynchronization devices is an example of an
international standard that includes tests for electromagnetic compatibility of cardiovascular devices.
It is often possible to test medical devices in a biological service environment without testing in living
systems. It is important to know which aspects of the biological environment are critical to simulate in
these situations. This section discusses two common methods for simulating this environment, including
use of biological fluids and duplication of human anatomy.
Some medical devices will be in contact with biological fluids during use. To simulate the service
environment, devices should be exposed to these fluids to determine their potential effects on the
device. However, it can be difficult (and potentially unsafe) to use actual body fluids during testing. In
many situations, it is possible to test the device in a simulated fluid that exhibits similar physical and
chemical properties to the actual fluid. For example, it is possible to simulate the non-Newtonian
rheological properties of blood. Companies such as Nasco Life/Form sell blood-like substances and animal
blood (www.innov-research.com). For many applications, it is possible to simulate blood using materials
that can be easily found in a grocery store. For example, glycerin is colorless and nontoxic and used
in many medical and cosmetic applications. Xanthan gum is a powder polysaccharide food additive
used in cooking as a thickening and binding agent. Two simple blood-like fluids can be made with
mixtures of ∼75/25% (water/xanthan gum by weight) or ∼60/40% (water/glycerin by weight). More
sophisticated formulations use a combination of glycerin and xanthan gum. The percentages can be
changed to roughly estimate other biological fluids (e.g., saliva, lymph, semen). Standard test solutions
for simulated testing in urine and other bodily fluids also exist.
Some device testing requires access to a tissue model that simulates some aspect of human anatomy
and physiology. The model used will depend upon the test required. For example, it may be helpful to
have an accurate anatomical model of a particular organ or organ system. Anatomic models can also be
purchased.
Other device testing may require a physical model that mimics a critical property of tissue. For
example, in situations that involve ultrasound, it is important to simulate the appropriate acoustic properties
of the bulk tissue. Phantoms with acoustic properties that mimic different tissue or bone might
also be needed. Similarly, the mechanical homogeneity and consistency of tissue are often required; in
this case, BFX ballistics gelatin can be used with a plaster mold (first sprayed with cooking spray) to
replicate parts of the body. The mixtures can be altered to mimic various densities. Thin-walled tubes
within the gel can be used to mimic veins and arteries.
At times the simplest way to test a device is to use real tissue. The next section discusses testing in living systems,
but several models exist that can very closely mimic human tissue and do not require approvals to use.
For example, you might try chicken skin and muscle (readily available at a grocery store or butcher shop) as
a surrogate for living biological tissue. You might also check a local butcher shop for animal organs. If there
are ongoing animal tests at your institution, the researcher conducting those tests might also be a source for
tissue and organs. Approval is not required to perform experiments on animals that are no longer alive.
Page 9 of 19
§ Cadaver Testing
In some situations, you may want to test the usability of your device and determine how well it will
interface with the human body. In these situations, it can often be helpful to conduct tests in a human
cadaver. For example, a team wanted to test the usability and accuracy of their handheld patellar tendon
graft knife. Bench testing on a laboratory knee model with simulated patellar tendon material (to emulate the
physical properties of tendon tissue) yielded some useful information; however, they
wanted to determine accuracy and ease of use in an actual human knee. Testing in a live human model
would require a clinical study, but cadavers are not subject to the same regulatory restrictions. For this reason, the
team decided to test their device in a cadaveric knee which allowed them to use the device and gain firsthand
knowledge of the difficulties experienced by orthopedic surgeons as they attempt to resect the middle third of the
patellar tendon for use as a graft to replace the anterior cruciate ligament. As a result, they were able to generate
new ideas for improving the accuracy and usability of the graft knife.
Cadaver testing may not be appropriate for all types of devices. For example, nonliving tissue will
degrade over time, resulting in changes in physical properties such as stiffness, strength, electrical
conductivity, and thermal properties. Thus, cadaver testing does not simulate all physical properties of
live human tissue and may not be a good option for quantitative testing. For example, the graft resection
procedure described above required minutes to complete, and the stiffness of the patellar tendon tissue
was adequate to test the knife properly. However, due to biochemical and physical changes to cadaver
tissue over time, cadaver testing would not have helped determine the long-term stability, function, and
biological performance of an implanted orthopedic device.
If cadaver testing would be helpful in testing your device, you should ask a mentor or advisor about
the possibility of conducting these types of tests. Cadaver laboratories are often found in academic
institutions associated with medical schools, nursing schools, and biomedical science programs that
offer gross anatomy courses.
The previous lecture discusses packaging design as a critical consideration in designing a medical product. As a
reminder, medical packaging is just as much a part of the product as the device itself and packaging design
considers sterilization, product protection, efficient storage, distribution stresses, and usability. It is unlikely you
will have the time or resources to conduct package and distribution tests as part of an academic design project.
However, considering the package testing requirements for your device will make you aware of this common form of
testing conducted by medical device companies.
Page 10 of 19
As presented in the previous lecture, medical device packaging needs to protect the product during shipping,
on the shelf in a warehouse, and in a hospital storeroom. As shown in the figure above, the product will be
exposed to various distribution stresses associated with shipping and storage, from the time it is manufactured
until it is opened and used. Package testing aims to ensure that the device will meet performance requirements when
the customer uses it. This requires testing of the device and its packaging, distribution simulation, and aging studies.
Package testing is required by regulatory bodies; documentation of these tests are included in the Design History
File (DHF). After the product is manufactured, it is often shipped to an off-site sterilizer. If shipped using ground
transportation, the product (including its packaging) will be subject to various distribution stresses,
including mechanical stresses such as vibrational and handling stresses (packages dropped, tossed onto
surfaces, and potentially crushed by boxes stored on top of them), and environmental stresses such as
heat, cold, or humidity. The packaged devices are then shipped back to the manufacturer’s (or a distributor’s)
warehouse and subject to additional distribution stresses. When an order is placed, the product
is shipped to the customer (subjecting it to additional distribution stresses) and then stored in a
hospital storeroom until it is ready for use.
Product stored in a manufacturer’s warehouse or hospital storeroom will often be stored in a temperature
and humidity-controlled environment. However, storage conditions can vary during shipping.
Product shipped via ground transportation across a long distance is often stored in a truck or temporary
warehouse along the way. Depending on the time of year and location, these warehouse temperatures
can climb to as high as 140°F (60°C) or drop below freezing. For devices using temperature-sensitive
polymeric materials, these conditions could soften the materials (high temperatures) or cause the
material to crack (low temperatures), affecting product performance. Humidity-sensitive
materials (such as nylon) can absorb water in the air and cause them to swell, thus resulting in dimensional
changes to the product stored in high humidity environments. Shipping products via air transportation
can result in similar effects. Not all freight cabins are pressurized, nor are they temperature-or
humidity-controlled.
§ Distribution Simulation
Packaging tests should be conducted on the final, sterilized product, in the actual packaging intended
to be used with the product. Environmental variables such as temperature, humidity, and pressure
(when appropriate) are simulated to represent the potential extremes that the product will be exposed
to. Through testing, the distribution history is also simulated to represent the mechanical and handling
stresses that the product will be subjected to during shipping and storage. These include manual handling,
stacking, and vehicle vibration tests per ASTM D999-08 (2015) Standard Test Methods for
Vibration Testing of Shipping Containers. Each test involves testing an entire shipping container that
contains many individually packaged devices.
The term shelf life refers to the length of time that a product can sit on a shelf with no significant
changes in physical properties or performance. This period typically begins after sterilization and
applies to the device and its sterile packaging. Some polymeric materials can degrade (especially if
sterilized with radiation) while in storage, and sterile packaging will not maintain sterility indefinitely.
Changes in physical properties are often referred to as aging.
There are two approaches to shelf-life testing: real-time and accelerated aging studies, both of
which are discussed below.
Real-time testing involves first conducting a distribution simulation. Instead of inspecting the product
immediately after packaging tests, the packaged product is stored in the expected storage conditions
(simulated storage environment). The product may be stored on the shelf for several years, with samples
Page 11 of 19
being tested periodically to evaluate the product and packaging for changes in properties and function.
If a company wants to claim a shelf life of 2 years, then real-time testing must be performed for at least 2 years.
Accelerated aging studies are designed to accelerate the aging process. They are typically conducted
similar to real-time studies but at higher temperatures which can accelerate the aging process. By testing
at a higher temperature, changes in material properties occur sooner. By correlating these changes
with real-time studies, the shelf life can be extrapolated from accelerated aging data. For example, suppose
a significant change in material properties (i.e., a 20% reduction in tensile strength of a polymeric
catheter material) occurs after 2 years of real-time testing at ambient temperatures and a similar reduction
occurs after 6 months of accelerated testing at a higher temperature. In that case, it may be possible
to conclude that 6 months of accelerated testing at higher temperatures would be equivalent to 2 years
of real-time testing at ambient temperatures. This correlation could be used to determine the shelf life
of similar products tested using accelerated aging studies. The advantage of this approach is that it takes
much less time and would not delay a product introduction while real-time shelf-life studies are being
conducted.
Design Validation
Design validation involves determining if a new design meets customer needs and answers the question:
“did we make the right product?” It focuses on the customer and relies on feedback representing
the user’s perspective to determine if customer needs have been met. In industry, it is common for engineers to be
involved in design validation which ideally would involve direct contact with intended users. In an academic design
project, such direct contact may or may not be possible. You may need to accept feedback from a client, sponsor, or
mentor as a substitute in this situation.
Previous lectures introduces you to interviews, surveys, focus groups, and other social science tools to
understand the problem from the perspective of the customers, clients, caregivers, and other users. You
can use these same tools to obtain validation feedback from these same stakeholders as well as project
sponsors, faculty advisors, and technical and clinical mentors. Design validation can be as simple as
giving the final prototype to stakeholders and asking if the design meets their needs. If the answer is
positive, then the design has been validated by those asked. If not, then you will have information to
guide you in improving the design. Once improvements have been made, the validation process is
repeated until stakeholders are satisfied with the final design. Remember that stakeholders may not be
aware of the technical constraints of your project and may request design changes that are not possible.
The goals of testing discussed earlier in this chapter are rarely achieved with raw data. For this reason,
the methods you use to analyze and interpret test results are just as important as the design and execution
of the test itself. In this section, we provide guidance on the analysis and interpretation of test data.
A rigorous plan and well-executed set of test procedures must be coupled with some form of data
analysis. The nature of this analysis will depend upon the tests, and there are some essential aspects
of testing and analysis that you will want to consider. First, the test protocol and analysis methods
should work together; therefore, it is good practice to determine both simultaneously. A common
mistake made by design teams is to generate data and then realize—during data analysis, when it is
too late—that the incorrect tests were run. This mistake can be costly, require additional testing, and
Page 12 of 19
illustrates the importance of identifying and conducting the correct tests. Second, when designing a
test, the number of trials must be determined. In some cases, the answer may simply be one trial (e.g.,
weighing the handheld device described earlier in this chapter). In other cases, a more elaborate
power analysis should be conducted to determine the smallest sample size needed to achieve a given
statistical significance. Third, the nature of your analysis will vary depending upon your tests and the
data generated. You will need to choose the appropriate statistical methods, recognizing the strengths
and limitations of each. Fourth, some analysis methods process raw data to extract a parameter of
interest. Matlab, MiniTab, Python, R, and other languages have many extensions or toolboxes
designed to perform statistical analyses of the data collected. Some tests will require image capture
and analysis. For example, a tool such as imagej.net may help you determine the size and motion of
bubbles in a fluid captured by a video recording. In other situations, your data may be used as the
input to a model or calculation, with the intent of indirectly determining values for a variable that is
difficult to measure directly (e.g., an internal force, flow, temperature, or concentration). These
should be planned before testing begins.
Engineers are comfortable reporting results based on numerical, graphical, and tabulated data. Design
decisions are sometimes based on more nuanced and qualitative interpretations of engineering results.
For example, assume a mechanical part of your device includes a metal cantilever that you analyzed
and found the maximum stress to be below the metal’s yield strength. That result is straightforward to
report. However, if you are asked if the device is mechanically safe for all loading conditions at a design
review, that is a more nuanced question and will be difficult to answer. Learn to express your professional
opinion in a way that conveys your level of confidence in that opinion.
As the interpretation of results is built more on a combination of context and intuition, there are a few
important rules to remember. We offer two tried and true methods to help you improve your intuition
skills. First, a good mentor or advisor will help you interpret your data and explain the rationale behind
their interpretation. Listen carefully as you will gain insights to help you build your own intuition for
future designs. Second, all results can be viewed through two different lenses: (1) a generous lens whereby
the results suggest that you are on the right track and you can advance in the design process, and (2) a critical
lens whereby the results are not entirely trusted, making further testing necessary. Many problems
with navigating the design process during testing stem from not applying one of these lenses. Being too
generous can lead one to speed through testing and find a critical flaw when it is too late. Being too critical
can cause the design process to stall. It is essential to try to balance the views from these two lenses.
Understanding the interactions between medical devices and living systems is critical in determining
safety and efficacy, a step that is necessary before regulatory bodies can clear a medical device. As
biological systems respond in complex ways and often adapt over time, tests in nonliving systems cannot
capture the variability or responsiveness of a physiological environment. Some adverse effects on
a patient, and failure modes of a device, only arise after an extended period and thus, can only be
revealed with testing in living systems.
Page 13 of 19
Tests in living systems are conducted in an analogous way to bench testing. As in bench tests, you should have a
clear purpose, a detailed protocol, the proper approvals, and appropriate analysis and documentation methods
before conducting tests. However, the significant difference is that you are testing in living systems and have a duty
to protect the safety and well-being of your test subject(s), whether they are cells, animals, or humans. Working with
living systems can expose you and others to risks, such as physical harm. For these reasons and others, you
must have a sponsor (e.g., faculty member, physician) obtain approval of a study proposal by an
institutional committee. If you are participating in studies, you will need to get the proper training
and certification. Another difference is that, unlike bench tests that typically assess the impact of one
variable at a time, testing in living systems allows you to determine the device’s effect on the whole
living system.
This section of the lecture introduces the types of biological and clinical tests conducted on living systems,
including biocompatibility and biodurability studies, animal studies, validation studies, and clinical
device testing and clinical trials. The chapter also details the process of obtaining approval to perform
tests in living systems from the appropriate university committees. Familiarity with these tests and
applicable terminology, procedures, and considerations will help you in your current project and as you
advance in your career.
To commercialize most medical devices, it is required by law to demonstrate safety and efficacy, which
generally involves testing in living systems. In the context of a design project, some specifications can
only be tested in a living system. For example, animal testing will likely be needed if a specification
dictates the time to perform a surgical procedure. Some specifications target a clinical outcome (e.g.,
reducing the incidence of bedsores), requiring human testing to determine if the specification is met.
Some devices do not interact directly with patients (e.g., medical simulators, test equipment,
service robots), and they do not need regulatory clearance. However, they do require validation testing with target
users. These validation tests are considered a form of human subject research. For these
reasons, nearly every medical device will undergo testing in living systems before commercialization.
In this section, we discuss in detail the types and purposes of testing in living systems.
There are three types of tests in living systems that are most relevant to medical device testing. First,
tests in cells and tissues—the most basic living structures—are known as in vitro tests, from the Greek
“in glass.” They are similar to bench tests in that they are performed in a laboratory. Second, tests in
whole multicellular organisms (e.g., zebrafish, rats) are known as in vivo (“within the living”) tests. A
best practice in animal testing is to begin testing with the lowest phylogenetic form (e.g., using a mouse
instead of a pig) if such testing will provide you with the desired information. Third, in medical device
design, human subjects are often used for human factors testing (e.g., validation
of device use) and performance testing (e.g., verification of a specification timing how long it
takes to use a device). As indicated earlier, tests on cadavers are not living system tests. They only need the same
level of preparation as a bench test, with accommodation for handling biohazardous waste.
Testing in living systems can help guide your design decisions as you navigate the design process.
For example, the same specification might be tested in increasingly rigorous ways as your device is
refined. An early benchtop performance test may determine if a silicone part meets a particular mechanical
specification. However, a biocompatibility test or a later biodurability test in an animal model would verify that the
physiological environment (e.g., body pH, temperature, biochemistry) does not degrade the mechanical performance
Page 14 of 19
of a device over a specified period. More rigorous and long-term testing will likely require a clinical study in
humans. On the other hand, some testing in living systems is prohibited in humans; examples include tests that
involve deliberately causing harm (e.g., nerve injury, fracture), stopping treatment of a life-threatening
event (e.g., hemorrhage), or intentionally not killing cancer cell. The only way to test these interventions
may be in an animal model.
Consider using a new grade of silicone in a tool that will be used in open surgery. Although this
material will only temporarily contact parts of the anatomy, it is essential to know how this new material
will respond mechanically and chemically to changes in the physical environment (e.g., body pH,
blood). You might initially address this question in a bench test with an environment that mimics the
service environment. Later verification tests might use an appropriate animal model. More rigorous and
long-term testing may require a clinical study. An equally important question would be, “how will the
contact with the silicone part affect the patient long-term (e.g., causing adhesions that take days to form
and longer to reach final status)?” The results of these successive tests may lead to rethinking design
decisions about material selection and perhaps lead to a material selection change.
Testing in living systems can serve dual purposes in validation testing. Assume you plan to make a
series of refinements in your surgical tool design and print a few 3D prototypes, each one successively
closer to the final design. After each prototype, you seek feedback from a user, or a group of them, using the tool
during a surgical procedure and intend to use that feedback to make each subsequent
refinement.
As introduced in the previous lecture on design risk management, a Failure Mode and Effects Analysis (FMEA)
begins by listing failure modes and then estimating the probability of failures and the severity of the
effects. Testing in living systems can allow more accurate quantification of the probable rate of occurrence of
known failure modes and, because living systems are responsive, discover previously unknown failure modes.
When a device is first tested in its actual biological service environment, the range of possible failure
modes expands. The reason for this is a device that performs well in a controlled setting may fail
when exposed to the unique service environments found in biological organisms. Some failure modes
can only be detected in a living system. For example, cytotoxicity tests require living cells. Devices
may also lose performance when subjected to biological environments. For example, if corrosion due
to low pH were tested in a controlled laboratory experiment, the compounding effects of chloride ions,
proteins, and other organic molecules would still be unknown.
Living systems adapt over time, revealing new failure modes that do not present in short-term studies.
Long-term tests of living systems aim to determine the types of adaptive responses that may be
unexpected as a device is used over long periods. Often long-term tests are conducted after a product
has been commercialized and is already being used in patients.
Testing in living systems requires careful consideration of subjects and controls to detect new failure
modes and estimate failure probabilities. Disease can be studied using an animal with a gene
knocked out. For example, if the insulin gene is knocked out in a mouse, murine offspring will all suffer
from diabetes. A diabetic population of mice can be used to conduct experiments on various treatments
for diabetes. If using a homogeneous population (e.g., diabetic mice), realize that you may inadvertently
miss essential failure modes that would only appear in a more heterogeneous population. In a
design project, the analog would be testing your technology with several healthcare providers (or even
across several healthcare systems) instead of only one. Before commercializing a device, tests should
be completed on a heterogeneous population to reduce the probability of unanticipated failure modes.
Page 15 of 19
Testing for Regulatory Approval and Reimbursement
To commercialize a novel technology, two types of organizations need to be convinced of safety and efficacy.
First, the regulatory agencies (e.g., Food and Drug Administration [FDA], EMA, and PDMA) must clear or
approve a medical device before being sold. Regulatory agencies often require clinical trials (investigational
studies) to demonstrate safety and efficacy. Second, health insurance providers will determine if a newly
commercialized device warrants reimbursement, a significant factor in whether or not healthcare providers will
purchase and use a new device.
With successive testing in living systems, the cost, time, effort, and rigor increase. For this reason, the
results of initial tests are often used to determine if a device will pass from one stage-gate
phase to the next. Companies carefully consider whether or not a test is justified. Consider that patient testing is
costly for many reasons, including:
• The equipment, space, and training involved often are beyond those required for nonbiological tests.
• Clinical staff is needed for patient recruitment, randomization, data collection, and analysis.
• Companies may be held liable for adverse events during a clinical study of a new medical device.
• In industry, for device commercialization, 150 to 300 test subjects are required to achieve results with
sufficient statistical significance for regulatory agencies and ultimately receive regulatory approval.
There are, however, many tangible benefits for patient testing. It can generate trust in the product
among customers and users. Statistics generated from clinical test data are often used in promotional
materials. For example, a company would likely publicize that clinical testing of their implantable
device demonstrated a 23% reduction in the development of an adverse biofilm. Carefully designed and
conducted living systems testing can accelerate the regulatory approval process and increase acceptance
by healthcare providers and hospital purchasing agents and from health insurance organizations,
who determine purchasing and reimbursement for the device.
In Vitro Testing
In vitro tests involve live cells or organs and are conducted in a laboratory setting. Cell tests may be
undertaken in preparations involving isolated cells or intact tissues and range from “blank” cells (e.g.,
oocytes) to more specialized cells (e.g., rat liver cells in culture). Tests in intact tissue from an organism
(e.g., dissected muscle from a rabbit) or even artificially created tissue (e.g., tissue-engineered
Cardiac muscle) are also performed in a laboratory setting.
For medical devices, the most common reason for in vitro testing is to determine the biological
performance of a new material to be used in a device. biological performance includes biocompatibility and bio-
durability. This section provides guidance on testing these host and material responses by testing material
interactions with living cells and tissues.
§ Biocompatibility Testing
A device is biocompatible if it evokes the intended host response without creating a toxic or harmful
reaction. Biocompatibility in practice refers to the short-and long-impact(s) of a medical device in patients. You
may have selected a material that is considered to be biocompatible because it is already in use in another medical
device, thinking that no biocompatibility testing will be required. However, that is only true if the same material
formulation, including chemical composition, additives, colorants, release agents, sterilization technique, and
Page 16 of 19
radio-pacifiers, has been tested previously. There have been metals, polymers, glass, and ceramics that have been
extensively tested and were determined to be biocompatible for a specific medical device application. However,
slight changes in the chemical composition, manufacturing processes, nature of patient contact, or
sterilization methods would require confirmation by additional testing for biocompatibility. For this
reason, virtually all device materials must undergo biocompatibility testing.
ISO 10993-1 outlines the types of biocompatibility tests that may apply to your device. Some tests
involve cell cultures, while others involve animals. Some tests require a sample of material, while others
require an extract of the material. Biocompatibility tests never involve humans and generally do not
require an entire intact device. A brief overview of the 13 types of tests is provided below.
1. Cytotoxicity. Evaluate the material’s effect on cell growth, cell depth, and other effects. All 72
combinations must be tested for cytotoxicity.
2. Sensitization. Determines the potential for contact sensitization as small amounts of material
can produce allergic or sensitization reactions when in contact with skin and other tissues. All
permutations must be tested for sensitization.
3. Irritation. Determines the potential for irritation in a suitable model using sites such as skin, eye,
and mucous membranes. Except for long-term
indirect ECDs, all other permutations must be
tested for irritation.
4. Intracutaneous reactivity. Assesses the localized reaction of tissue to a material. Applicable to
devices that can breach the skin and contact circulating blood and other tissues.
5. Acute systemic toxicity. In an animal model, determines the potentially harmful effects of single
or multiple exposures of less than 24 hours each.
6. Subacute and sub-chronic toxicity. Determines the effects of exposure for a period not less than
24 hours and not greater than 10% of the animal’s life span used for testing.
7. Genotoxicity. Uses cell cultures or other techniques to determine whether a material induces
gene mutations, changes in chromosome structure or number, or other DNA or gene toxicities
resulting from exposure to the test material. One genotoxic material is chromium, as it can interact
with DNA and can be carcinogenetic.
8. Pyrogenicity. Assesses a material’s potential to induce fever-producing reactions in test animals.
Fever-inducing is limited to devices causing biological effects.
9. Implantation. Evaluates local pathological effects on tissue at both gross and microscopic levels
of a material specimen (not a medical device itself) that is surgically implanted in the tissue of
an appropriate test animal.
10. Hemocompatibility. Evaluates the effects of blood contact with the test material. Tests assess the
potential for hemolysis (lysing of red blood cells) and thrombogenicity (the ability of a material
to form blood clots).
11. Chronic toxicity. Determines the effects of single or multiple exposures to the material during at
least 10 % of the life span of the test animal.
12. Carcinogenicity. Assesses the tumorigenic potential during the significant portion of a test animal’s
life span. Carcinogenicity tests should be performed only if other sources suggest that the
test material may induce malignancies.
13. Reproductive toxicity. Assesses genotoxicity gene mutations, chromosomal abnormalities, DNA
effects. Also assesses endocrine toxicity, which in turn can impact reproductive functions such as
ovulation and spermatogenesis.
Biodurability is concerned with the effect of the host (living tissue) on the material and is defined as the
ability of materials to resist alteration in vivo. While device durability can be evaluated and verified
through bench tests (in nonliving systems), biodurability requires testing in living systems to determine
Page 17 of 19
if specifications and performance requirements can be met in the context of a living system. As with
many of the tests in this lecture, biodurability tests may start with in vitro and then progress to more
rigorous animal and human tests.
Consider that medical grade silicone is biocompatible and biodurable, and thus has been widely used in
extracorporeal applications due to its material properties. Specific properties include high molecular weight
and surface tension, hydrophobic nature and hemocompatibility. It is considered a biodurable material as
well because of its chemical stability and thermal stability and virtually no thermal degradation below
150 °C. There are many medical-grade types of silicone and selecting the optimal type is one of many design
decisions; it is usually a balance between optimal mechanical properties and potential degradation when in
use. As noted previously, use of a material in one situation does not guarantee that it will be biodurable and
biocompatible in another. Bench and in vitro testing may be required for verification and validation.
Indwelling devices, such as ureteral stents, Foley catheters, left ventricular assist devices, and some
orthopedic screws and plates, are not intended to remain in a patient permanently. A first step in assessing
the biodurability of these devices may be short-term implants in animal models. The devices can
then be retrieved for inspection and an evaluation of biodurability. Evaluation tests include those for
wear (as in the hip implant), oxidation (e.g., of some polymeric insulation wires), corrosion (e.g., of bare metal
stents), and fiber and particle release, (e.g., OR towels used in surgery). Fiber and particles resist biochemical and
physiological clearance mechanisms in the body. They also cause scarring adhesions, which may inhibit normal
function and may cause chronic pain.
Implantable devices such as heart valves and joint replacements are intended to remain in a patient’s
body for decades, sometimes permanently. Assessing the biodurability of these implantable devices
likely requires retrieval, inspection, and testing of the device after a period of time. For ethical and
practical reasons, premature retrieval of these devices is not possible in a human study due to the
trauma and risk associated with the early removal of an implanted device. In general, implant retrievals
are only possible in long-term animal studies to resist unwanted changes such as material degradation
and loss of physical properties. Some devices, such as resorbable stents or absorbable sutures, are
intended to degrade within the body. In these situations, degradation is the desired change.
Animal Testing
Animal models have been used for nearly all of recorded history to gain new knowledge about anatomy and
physiology and test healthcare interventions. In the context of medical device design, testing in animals generally
occurs after in vitro testing but before testing in humans. In some cases, animal tests are used to detect
potential negative outcomes, either to a patient or to the device in a physiological environment. In other cases,
a test can only be conducted in an animal model. In the earlier section, for example, biocompatibility tests 5,
6, 9, 11, and 12 can only be performed using live animals instead of cells or tissues. Test #13 requires the use
of pregnant animals. Likewise, some bio-durability tests require a device to be implanted in an animal for
some period of time, subsequently explanted, and then subjected to additional bench verification testing.
Human Testing
The goal of all medical device design projects is to introduce a safe and effective new device to the
healthcare ecosystem. Benchtop and animal testing can verify many of your specifications, however,
some specifications can only be verified through testing in humans. In addition, formal validation of
your device can only be gained through surveys, focus groups, and ethnographic observations with the
intended users. Both clinical studies and validation studies are a form of human subject research. Even
if you do not plan to engage in human studies as a part of your current design project, it is important to
understand the factors that must be considered in conducting tests with human subjects as it is almost
always an important part of a medical device design process. One challenge with performing tests on
patients is that you are not allowed to communicate with them without specific permission to do so.
Page 18 of 19
The justification and oversight of a proposed study are required to reduce the risk of an adverse
event occurring with a test subject. The effort involved in a human subject study is significant. Beyond
the testing of your technology, you need to be concerned about recruiting patients, ensuring patient privacy, and
ensuring safety. One possible exception is if you have an industry or clinical mentor who
has an existing approved clinical study. For example, if you are developing a new walker for disabled
children, your physician advisor may already have an ongoing clinical study for the same patient population.
Page 19 of 19