Avionics Fligth Test

CHAPTER
What is Avionics Flight Test

and Why Do We Need It 1
Chapter Outline
1.0 Avionics Flight Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Historical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Flight Test Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Flight Test Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 The Need for Flight Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 Classifying the Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Contractual Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.7 Test Team Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8 Team Member Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.9 Understanding Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.10 Contractor and Subtier Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.11 Customer Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.12 Formulating the Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.14 Selected Questions for Chapter 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.0 AVIONICS FLIGHT TEST

There is little written on the specific subject of avionics and weapons systems testing,
but there are articles and tutorials available that address the general subject of testing
and experiments. One source on testing is The Logic of Warfighting Experiments, by
Richard A. Kass (Washington, DC: CCRP Press, 2006), which is available online at the
time of this writing at http://www.dodccrp.org/files/Kass_Logic.pdf. The U.S. Air
Force, and in particular the 46th Test Wing at Eglin Air Force Base, Florida, has been
promoting the use of ‘‘design of experiments’’ (DOE) as the preferred method of con-
structing and analyzing test programs. As a reference, the reader is encouraged to read
any of the numerous papers on this subject by Gregory T. Hutto and James M. Higdon.
The most recently published is American Institute of Aeronautics and Astronautics
(AIAA) paper 2009-1706, ‘‘Survey of Design of Experiments (DOE) Projects in
Development Test CY07-08,’’ which is also available online at http://pdf.aiaa.org/
preview/CDReadyMTE09_2104/PV2009_1706.pdf.
1
2 CHAPTER 1 What is Avionics Flight Test and Why Do We Need It
1.1 HISTORICAL BACKGROUND

The flight test community as a whole has been driven to do more with less in recent years.
It has been forced to become more efficient with the available dollars for testing and
evaluation. In the past, flight testing was divided among three separate groups of testers:
● Contractor flight testing
● Development test and evaluation (DT&E)
● Operational test and evaluation (OT&E)
Each of these groups was assigned a charter and was responsible for the testing and
reporting of their respective tasks. There was a bit of overlap of tasks and, as a result,
some of the testing was repetitive; different results from similar tests sometimes led to
heated confrontations. In all fairness, the test communities from each of these groups
were seeking to evaluate similar tasks from different perspectives.
Contractor flight testing of avionics systems is first and foremost developmental in
nature. Perhaps as much as 90% of the test program is dedicated to development,
whereas the final 10% is the ultimate demonstration of the finished product. Even this
demonstration is a ‘‘test to spec program,’’ where the contractor is on the hook to show
specification compliance. Completion of the program, and therefore payment, is con-
tingent upon this successful demonstration. It should be obvious that the contractor
will be reluctant to perform any testing that is above and beyond specification com-
pliance (performance testing, for example), since this will reduce the time available for
specification compliance testing. The contractor is also not being paid to demonstrate
performance.
Development test and evaluation (DT&E) is chartered with ensuring that the system
under test is indeed ready for operational test and evaluation (OT&E). Since OT&E is
interested in the true performance and functionality of the system under real-world
scenarios, the DT&E community is saddled with a rather large burden. DT&E has to
validate the specification compliance testing previously done by the contractor and then
accomplish sufficient performance testing to ensure that OT&E will not dismiss the
system outright. Specification compliance verification is used as a baseline; additional
performance testing would be moot if the system cannot perform to specifications.
Operational test and evaluation (OT&E) has no interest in specification compliance
verification and is only interested in the system’s capability to perform the desired
mission. For example, OT&E was interested in the ability of the F-14D to perform fleet
defense, not in its ability to detect a nonmaneuvering target at high altitude at some
specified range. The latter is a specification compliance issue and not a performance or
operational utility issue. In many cases, OT&E would determine that the specifications
were so poorly worded that the system, although it met the specifications, could not
perform the mission. Of course, the opposite is also true; the system may fail to meet
specification compliance but is found to be acceptable in meeting mission requirements.
The existence of these three independent organizations necessitated three periods
within the master schedule for flight testing. In fact, most test programs included the
contractor test period interspersed with DT and OT periods. These DT and OT periods
were scheduled at major milestones within the schedule coinciding with incremental
capabilities of the system. These periods lasted for weeks, and sometimes months,
and severely impacted contractor development testing, because during these periods
1.1 Historical Background 3
software was frozen and further development work was halted. The rationale here was to
not allow the contractor to fritter away all of the test and evaluation (T&E) monies
without periodic progress checks. This rationale worked fine as long as the T&E sche-
dule was not shortened and development proceeded as expected. In reality, the test
schedule was always shortened and development was always lagging. These real-world
developments caused considerable consternation among the three groups, since none of
the test organizations had any inclination to giving up scheduled test time.
In the mid-1980s, an idea to combine some of the testing among the three groups
was floated to the test community. The idea was to incorporate some or all of the DT&E
testing with the contractor testing. This way the separate DT&E test periods could be
absorbed by the contractor test periods and thereby shorten the total T&E effort. The
contractor would simply have to include the DT&E test objectives in the contractor test
plan. Since the objectives would be combined, the next logical step would be to combine
the test teams. It was from this initial proposal that the Combined Test Force (CTF),
Joint Test Team (JTT), or any of many other names this union was called was born. Note
that these were not integrated product teams (IPTs); they will be addressed later.
The initial CTFs had some growing pains. The first attempts at participatory flight
testing simply deleted the DT&E test periods and shortened the T&E effort accordingly.
Unfortunately, the contractor test periods were not expanded, even though it was
expected that they would wholly incorporate the DT&E effort. In the ideal world, many
of the DT&E test objectives and test points would match one for one with the contractor
test plan (specification compliance). In the real world, the DT&E community structured
its testing to try and accomplish both specification compliance and performance with the
same test objective. This created quite a dilemma for the contractor test organization,
especially since, during this period of cooperation, no one changed the contracts. Con-
tractors were still bound to satisfy a contract that was rooted in specification com-
pliance; there were no stipulations for additional testing. Contractor test teams were
routinely caught in the middle of the ‘‘customer satisfaction’’ and program direction
battle. On the one hand, they tried to incorporate the customer’s testing; on the other,
they were accountable to program management to test only what was in the contract.
There were other ancillary problems which became evident during these phases of
joint testing. Since testing would be combined, the teams would be combined. This idea
would have been a good one if it had not been tinkered with. the premise of this idea was
that consolidation would save money. The most obvious savings were reaped by elim-
inating the DT&E periods. A second method of savings, courtesy of the accountants,
was the elimination of manpower due to consolidation. Contractor test teams were
reduced by the number of DT&E personnel which were assigned to the JTT and vice
versa. Once again, the charters of each of these organizations were never adjusted, so the
workload remained the same, but the manpower was reduced. Each organization was
still responsible for such things as data reduction, analysis, and generation of test
reports. The contractor organization was still responsible for the instrumentation of the
aircraft, formulation of the test plan, telemetry station setup, and engineering unit (EU)
conversion of data. Additional tasks of training new DT&E members and converting
data into a format usable by the DT group was added.
As time went on, it was determined that even more time and money could be saved
if the OT&E community was also brought into the fold and a true CTF could be formed.
Problems similar to those previously presented rose anew. One unique problem with this
setup involved the specific charter of the OT&E community. Any member of the OT&E
community will tell you that they are chartered by the U.S. Congress to ensure that the
taxpayers are getting what they pay for and the users are getting a system that is capable
of performing the designated mission. This is an integral part of the checks and balances
of the acquisition system, but it leads to serious problems when trying to consolidate the
test community. How can the OT&E testers be melded with the CTF and still retain their
autonomy? In reality, operational testers have everything to gain and nothing to lose in
this situation. They will not give up their dedicated periods of testing and are able to
interject special testing into the combined test plan. This special testing is over and
above the development/specification compliance test plan.
The greatest problem with consolidation is the failure of all team members to
understand that avionics testing programs, by nature, are developmental. That is, soft-
ware-dependent systems are incremental. There is no such thing as 100% software (full
system capability), especially with the initial deliveries. If this fact is understood, spe-
cification compliance and validation testing will never be performed early in a program.
In fact, this testing should occur at the end of the program. The differences between
development, demonstration, qualification, and performance are covered later in this
chapter. OT&E organization and testing is covered in chapter 14.
1.2 FLIGHT TEST ORGANIZATION

Ideally flight test should be an independent organization (i.e., not reporting to the
Engineering Department) tasked with impartially evaluating and reporting on a vehicle
or system. the discussion here will address avionics and weapons systems. The per-
sonnel within this organization should be knowledgeable of the system under test. They
should understand the basic operation of the system, the interfaces with other avionics
systems aboard the aircraft, and where and how the system will be employed. In addition
to basic knowledge of the system, the tester must also be knowledgeable of applicable
system specifications. Specifications may be defined as the minimum standard to which
the system must perform. Many of the MIL-STD specifications we have been familiar
with in the past have given way to minimum operating capabilities, desired objectives,
or essential elements of capability (EEC). It should be noted that the specifications for a
program were probably formulated many years earlier by individuals who may no longer
be involved with the program. The specifications may be poorly written or, worse yet, be
seemingly unattainable. These problems lead to a more difficult task of test plan pre-
paration for the flight test community. An example is the inputs to the cockpit design by
the Pilot’s Working Group. In many test programs, a Pilot’s Working Group is estab-
lished during the design phase to solicit input from operational users for the design of
controls, displays, and the human machine interface (HMI). These inputs are used by
design engineers and incorporated into the final product. Five years later, when the
pilots have all moved to different assignments and the system enters flight testing, no
one can fathom the reason for the display presentations.
It is with these specifications that we must formulate the foundation of our test plan.
Above all, this organization of testers must have the ability to ‘‘call somebody’s baby
ugly.’’ This is a whimsical way of stating that the organization has the ability to remain
independent of Development Engineering and maintain objectiveness and impartiality.
Some flight test organizations within corporations and defense agencies are organiza-
tionally within the Engineering Department. This is unfortunate because the Engineering
1.3 Flight Test Objectives 5
Department more often than not is responsible for designing and implementing the
system. It is pretty hard to call somebody’s baby ugly when you and the baby have the
same father.
In the ideal world once again, the flight test organization will be a composite test
team. This team will put aside all parochial interests and focus on the common goal.
Members of the team would include flight test engineers, aircrew, and analysts from the
contractor and customer development and operational test organizations. These mem-
bers will provide the core of the test structure. In addition to flight test engineers, the
team may include the design engineers, who act as consultants and provide the technical
expertise. There may be some vendor or subcontractor engineers for those unique sys-
tems not designed by the primary contractor—an inertial navigation system (INS) or
radar, for example. Instrumentation personnel also provide a key role on the team,
ensuring that data are available in a format that is usable. Site support personnel such as
maintenance, integrated logistics support (ILS), and range support provide an orderly
flow to test operations.
1.3 FLIGHT TEST OBJECTIVES

Perhaps the most important goal of the flight test team is to validate that the system
being tested meets the customer’s needs and requirements. We are not validating the fact
that the customer knows what they want, but rather that our system will fulfill their
requirements. This is accomplished by ensuring the top-level specifications are satisfied
and by performing some measure of system performance testing. In an avionics pro-
gram, the first step is to identify system problems and anomalies. Some readers may
know this portion of the program as ‘‘fly-fix-fly,’’ which indeed it is. Unfortunately,
once again, there are some acquisition experts who believe that fly-fix-fly is a product of
an age long since passed. With all of our investment in state-of-the-art simulators, some
believe that all problems can be worked out in the laboratory prior to the flight test,
hence there is no need for a fly-fix-fly or development program. This tenet is, of course,
ridiculous to those who have been involved in the testing and evaluation of avionics
systems.
Along with the identification of problems and anomalies comes providing trouble-
shooting assistance. The most helpful thing a flight test engineer can do in the area of
troubleshooting is to provide data. It cannot be emphasized enough that the identifica-
tion of problems is only half of the job. If a problem is presented to an engineer without
proof (a picture of the baby), it has a very small chance of being corrected in a timely
manner. Flight test engineers should live and die by data; it is the fruit of their labor. In
addition to providing data, the test engineer should be ready to assist with the problem
resolution. The test engineer should be the closest individual to the problem. It was the
engineer who should have designed the test, then collected, reduced, and analyzed the
data. Yet they are quite happy to report that a problem exists in a system without
assisting in finding a solution to the problem. As mentioned previously, engineering
judgment is issued to you at graduation. It is OK to act like an engineer and provide an
educated guess as to where the problem lies and how it can be fixed.
In many cases, problems can be as simple as a busy bit being set continuously. An
obvious solution might be to reset the bit. The flight test engineer should know this
by looking at the data. The design engineer, upon hearing of the problem, but without
supporting data and the flight test engineer’s insight, may spend as much time
researching the problem as the test engineer in order to come to the same conclusion.
This may seem a trivial matter, however, if multiplied by a large number of anomalies,
one can see how the system can easily be bogged down. This logjam can be broken up
with good flight test engineering.
The final goal of the test team should be fair and accurate reporting of all tests
conducted and the results obtained. These reports should document the system perfor-
mance versus specified requirements and must be supported by the data. Once again, the
independence of the organization is critical in providing an impartial report.
1.4 THE NEED FOR FLIGHT TEST

The previous section addressed the foolishness of eliminating the fly-fix-fly concept
from flight test planning. Some quarters feel that there is no need for flight testing at all,
and here is a good place to put that issue to rest. Simulation and laboratory development
are excellent tools in placing a system into service. In fact, experience has shown that
approximately two-thirds of all problems discovered in an avionics program are found in
development labs and simulators. This is an excellent method for reducing flight time,
and with the advent of more aircraft-representative development labs, the percentages
should increase, but to what degree? Avionics systems are becoming increasingly
complex, with hard-to-simulate dependencies on other avionics systems and environ-
mental conditions.
There are two major drawbacks to labs and simulators. The first hurdle is the lab’s
approximation of the aircraft; space constraints, lack of available flight-qualified hard-
ware and software, and line runs/losses between line replaceable units (LRUs) are a few
examples of laboratory limitations. Real-world variables such as acceleration, vibration,
acoustics, and loads cannot be adequately modeled without collecting flight data first.
The second hurdle is the simulation itself (i.e., simulation is not real; it is a simulation).
Simulations cannot account for all real-world dynamics. It would be impossible to
evaluate the human factor implications of a system’s operation in moderate turbulence
in a simulator; these are things normally discovered during actual operation in a flight
test environment. Attempts are made to make the simulations as real as possible, how-
ever, the number of variables can be astronomical. These considerations are responsible
for the number of ‘‘lab anomalies’’ that are seen in all flight test programs. Even more
interesting is the fact that some functions fail in lab testing and pass in flight testing, and
some which are successful in the lab may fail on the aircraft.
Simulations can be compared to software development. As a rule, most simulations
are not developed by the user, and the programmers attempt to keep as many variables as
possible constant in the algorithms. Man-in-the-loop simulators can reduce the problems
caused by nonusers developing the simulation. The second problem is a little trickier to
reduce. The number of variables that can affect an algorithm can be large. A simulation
attempting to replicate the real world would entail developing a program that would be
quite complex. The simple bombing problem which follows demonstrates how the
variables can multiply very rapidly if we attempt to replicate real-world situations.
If we attempt to drop a bomb from an aircraft with the intention of hitting an
object on the ground, the pilot must consider a number of variables which directly
affect the impact point. The first of these variables involves the ballistics of the bomb.
1.4 The Need for Flight Test 7
These ballistics determine the flight characteristics of the bomb. By knowing how the
bomb will fly, downrange can be computed by knowing where in space the bomb was
released (altitude, airspeed, g, and attitude). If the pilot deviates from any of these
desired conditions, the impact point of the bomb will change. Lead computing sights and
target tracking boxes on today’s fighter aircraft compute a constantly changing impact
point based on changes in the input variables.
If a programmer wants to simulate these conditions, he has to take into account the
four variables mentioned. But are there only four variables? Altitude can be above
ground level (AGL), mean sea level (MSL), or height above the ellipsoid (utilizing any
one of a number of earth models). Airspeed is measured in three axes, normally North,
East, and Down, or East, North, and Up (depending upon the aircraft convention).
Affecting velocity is wind and wind gusts, also measured in three axes. Acceleration (g)
is also measured in three axes. Attitude refers to the pitch, roll, and yaw of the platform;
the true attitude of the platform may differ from the reference the platform is using for
the computations (INS, global positioning system [GPS], attitude heading reference
system, etc.). By taking all of theses components into account, the number of variables
grows to 12 or more. If assumptions are made—the pilot will always be at the correct
altitude, 1g level flight—the computations for the simulation become much simpler.
Although the preceding example may be an oversimplification of how simulations
are developed, it does explain the principle. By eliminating variables, the simulation
becomes easier and the coding more manageable. On the flip side, the simulation becomes
further removed from the real-world aircraft. The infrared (IR) signature of an F-5 aircraft
at a given aspect in clear air on a standard day is probably well known, but that same F-5
flying in desert haze over mountainous terrain in August is probably not known. It is
probably not known because all of the variables have changed and cannot be accounted
for in a simulation. The only way a database can be built for a simulation would be to
collect a massive amount of data for varying terrain, haze, and temperature conditions.
This is performance data that can only be obtained in a flight test program.
One last example: radar detection and classification of rain. If the capability of a
radar to detect rain is a requirement, it may be advantageous to test this capability within
the confines of a lab. After all, the size of raindrops and reflectivity based on frequency
are known quantities. The problem with this task is that no attention has been given to
the terrain that the rain is falling over. The performance of the radar will change if the
terrain is a runway, forested, a sand dune, mountainous, or the ocean. A massive data-
base would have to be obtained for varying rain rates over various terrains in order to
quantify true radar performance or specification compliance. Once again, a flight test
program would probably be more efficient than creating an additional test program for
the purpose of collecting rain data.
In today’s test environment, labs, simulators, and flight tests coexist and work
together in order to meet program goals. Simulations and mathematical models
attempt to predict system performance to the extent of their own capabilities, and are
excellent tools for discovering gross anomalies in the system. Flight tests provide data
on functions that cannot be evaluated in the lab, and the data that are collected can help
to validate or improve the models used in the lab. A constant supply of real flight data
allows the models and simulations to mature and become better predictors of system
performance. It can be reasonably assumed that simulation will never supplant flight
testing. It is also readily apparent that flight testing should not be undertaken without
first undertaking extensive lab evaluation.
1.5 CLASSIFYING THE PROGRAM

Avionics test programs are of two types: subsystem/system development and system
integration. Subsystem/system development encompasses the development and laboratory
testing of a new or improved avionics system. An example of this classification would be
the development of a new airborne radar or the development of an upgraded radio system.
System integration is the type of program which most flight test engineers are familiar
with. These programs entail the installation and integration of a proven system into an
existing platform. A prime example would be the installation of GPS into an AV-8B. GPS
is a proven system, more than likely provided as government-furnished equipment (GFE).
A full-scale development (FSD) program may consolidate these two programs under one
umbrella where there is some new development as well as system integration. FSD pro-
grams are becoming increasingly rare, as military services and corporations are electing to
direct monies to upgrade programs rather than the development of new airframes.
Examples of recent FSD programs are the F-35, A-400 and the F-22.
Unlike vehicle programs (when the term ‘‘vehicle programs’’ is used it refers to
performance and flying qualities programs and analog-type systems such as hydraulics,
fuel, pneumatics, etc.), avionics programs deal extensively with software. This implies
that the program will be incremental in nature; software is never delivered to test with
full system capabilities. In fact, there may be many deliveries, or builds, sent to the test
organization. If we take the case of radar software in a subsystem/system development
program, the deliveries may be scheduled as
● Build A: raw sensor development

● Build B: system integration
● Build C: weapon support
● Build D: special modes
● Build E: reliability/maintainability/qualification/sell-off
In this scenario, the first delivered build to the test community (Build A) will con-
tain only basic capabilities for the radar. The system will not play with the other aircraft
systems and may be good only for a ‘‘smoke check’’ (turning the system on does not
cause a fire) in the aircraft. Build B allows for system integration, which may provide
the first opportunity for viewing radar information on the displays, employing basic
radar modes, and possibly establishing track files. The system will not support weapons
nor will it feature special modes such as frequency agility. Build C is the first oppor-
tunity for the radar to support weapons delivery and a missile in flight. Build D provides
all radar functions with the exception of final sell-off and qualification. Build E is the
sold-off version of the radar software and will have been subjected to all environmental,
mean time between failure (MTBF), and false alarm tests.
The schedule of these deliveries is normally determined at the start of the program
and will drive the conduct of the flight test program. In the previous example, the
deliveries may be scheduled at 6 month intervals for a 3 year test program (6 months of
tests per delivery). Figure 1.1 shows the relationship of radar development to the overall
schedule.
1.6 Contractual Requirements 9
Vendor Development ∆ ∆ ∆ ∆ ∆ FIGURE 1.1 ¢

(Builds A Through E) Software Delivery as
Contractor Integration ∆ ∆ it Affects Flight
Testing
Contractor Software ∆ ∆ ∆ ∆ ∆
(Builds 1 Through 5)
Development Flight Test ∆ ∆
Demonstration Flight Test ∆ ∆
DT/OT ∆∆
The figure also shows the interdependence of the flight test organization on the
contractor and vendor software deliveries. The integration contractor must await a new
radar software delivery prior to proceeding with its own integration work. This inte-
gration work will entail melding the radar software with the aircraft software. A soft-
ware release for testing by the contractor will include the last version of radar software
received. Should any of these milestones (D) slip, one of two things must happen:
1) Each successive integration or test period must move to the right; or 2) Each suc-
cessive integration period or test period must be shortened by the length of the slip. For
those individuals working integration, the news is not bad; the integration period merely
slips to the right. For those individuals performing flight testing, the news may not be so
good; the flight test period may be shortened by the length of the slip.
The rationale is not that hard to comprehend. The milestones after the flight test
period (in our case, DT/OT) are the final milestones on the master schedule. These
milestones indicate a few things. The first is that this is the end of the program and this is
where the customer will make a final decision on the product. The second implication is
that this is where the money typically runs out. Both of these scenarios are true and they
dictate that the final milestones after flight testing are generally prohibited from moving
to the right. It is OK to move to the left, and this is seen quite often when budgets seem
to be overrunning and costs need to be trimmed. Therefore the first rule of avionics
flight testing is that, for testers, the final milestone must never move to the right. As
flight testing is compressed to the right, certain testing may be deleted and some func-
tions may not be evaluated at all. This is a problem for flight testers if they do not
address the problem with program management. Program management must be advised
that with a reduction in testing comes a reduction in confidence level. Confidence level
is your belief in the accuracy of the data you provide in the final report. Most flight test
programs are estimated based on an 85% to 90% confidence level. A reduction in the
amount of data collected will cause a reduction in confidence, and this fact needs to be
included in the final report.
1.6 CONTRACTUAL REQUIREMENTS

Whether you are a civilian contractor or a Department of Defense (DOD) tester, each
flight test program has contractual obligations that must be met. A contractual item that
your organization may be responsible for may be an interim or final report. In the
civilian world, responsibilities of the test organization may be found in the contract data
requirement list (CDRL) section of the contract. Each CDRL item has an associated
data item description (DID) which explains in detail what the contractor is required to
submit, in what format, to whom, when, and how many copies. DT&E and OT&E
organizations are typically required to submit interim and final reports on the system
being tested. Format and submittal of these CDRLs is governed by regulations or pro-
cedures. As each test program becomes more and more consolidated, the responsibility
for meeting the CDRL submittal will fall to the CTF. Some programs have moved away
from the formal specification compliance submittals common in past programs. These
programs have attempted to define system performance in terms such as a statement of
objectives or essential employment capability (EEC). Although the names and termi-
nology have changed, the responsibility for reporting progress and performance to the
customer is still a requirement.
Depending on the program, the first report that may be delivered to the customer is
an instrumentation list. As mentioned previously, compliance and performance can only
be proven when substantiated with data. The instrumentation list provides a list of all
parameters that will be collected, reduced, and analyzed during the program in order to
meet this goal. In addition to data parameters taken directly from the aircraft or system
under test, the list must also include data parameters that will be used for ‘‘truth.’’ This
list includes parameters that are obtained from pulse code modulation (PCM) encoding
systems, data busses, and other data collection systems, along with their respective data
rates or sample sizes, accuracy, and scaling factors.
In addition to identifying the parameters that will be used for compliance or per-
formance assessment, the customer is also interested in how you intend to manipulate
the data. You must provide signal conditioning information, data reduction routines,
filters, and processing algorithms in order to show that the data will not be over-
massaged. If data are to be obtained in real time, the customer will want to know merge
routines, sample rates, and if there is any data latency or senescence in the processing
system. The time alignment system you have chosen, system time (GPS, satellite,
manual), and clock drift should also be provided.
A newer report that has routinely been requested in recent programs is the weapons
system accuracy report (WSAR). This report is delivered in two sections, one at the
beginning of the program and the other at the completion of the program. The first
section deals with the accuracy of the time space position information (TSPI), or truth
data. As systems become more and more accurate, it is becoming increasingly difficult
to provide a truth system which is at least four times as accurate as the system under test.
Yes, four times as accurate (which is the second rule of avionics testing). In many cases
we are forced into using some form of sensor fusion in order to obtain the accuracy we
require. The customer may request that sensor fusion routines, accuracies, filters,
weather corrections, and earth models used for truth be provided in this report. The
second phase of this report is issued after the test program has been completed, and
addresses the overall system accuracy based on the results of testing. For example,
although the accuracy of a GPS is 15 m while on the bench, after being integrated into
our platform the accuracy may now be 17 m. The 2 m degradation can be a result of
many things: timing, latency, senescence, update rates, etc. Regardless, the total system
performance is 17 m, and that is what will be reported.
The submittals that most test engineers are already familiar with are flight test plans
and reports. Flight test plans are submitted prior to the start of the actual test program
1.7 Test Team Composition 11
and are normally subject to the approval of the customer. Depending on the organization
or affiliation, the formats of these reports will vary. Historically, contractors have been
able to submit their test plans in ‘‘contractor format.’’ As CTFs become more the norm,
test plans must fit the format required by the customer. The U.S. Air Force (USAF), as
well as the U.S. Navy (USN), require that the flight test plan conform to a format
specified by regulation. The USN, for example, uses Naval Air Instruction (NAVAIR-
INST) 3960.4B, ‘‘Project Test Plan Policy and Guide for Testing Air Vehicles, Air
Vehicle Weapons, and Air Vehicle Installed Systems.’’
Reports are divided into three categories: pilot reports, flight reports, and final
reports. Pilot reports are normally a requirement for all avionics programs and are
submitted on a per flight basis; these reports are sometimes called dailies in some test
organizations. Depending upon the program, flight reports may or may not be required.
These reports may be required only for the initial flights of the program, all flights of the
program, or may be in the form of status reports which are required after each block of
testing. Final reports are always required, and as with the test plan, may be in either a
‘‘contractor format’’ or customer format. The test community is usually responsible for
the latter two reports, while the test aircrew normally submits the pilot reports. The DID
addresses these requirements, as well as the currency and time period allotted for
submittal.
1.7 TEST TEAM COMPOSITION

Prior to the start of the test program, or even the test planning phase, a coherent test team
must be put into place. Although this sounds like common sense, in many programs it is
often overlooked. As mentioned in section 1.5, flight testing is the last operation in the
program master schedule. As such, funding for flight testing does not ramp up until the
flight test period is imminent. Prior to this funding, a skeleton crew is normally
authorized to perform planning, instrumentation development, data reduction routine
preparation, and scheduling, which is no small task. It is the responsibility of the flight
test manager to present the case to program management in order to obtain the required
funding and manpower early in the program. Failure to do so may cause the flight test
community to be behind schedule, which may eventually lead to failure.
Task teaming needs to be implemented as early as possible in the flight test cycle. It
has been proven on numerous occasions that early teaming is essential to the success of
the program. The first requirement of teaming is to identify all groups that are required
for success. Today’s jargon states that all members who either own or contribute to the
‘‘process’’ need to be included. Once individuals or groups are identified, the team must
identify the goals of the group and team member responsibilities, and assign account-
ability. Most importantly, the group must be colocated. Excessive travel and distance is
detrimental to a group’s progress, and in many cases, actually prevents a team from
carrying out its charter successfully. This situation is best demonstrated if we compare
test site colocation to the children’s game of telephone. In the game, a group of children
is placed in a line and a little ditty is told to the first child, who then whispers it to the
next child. When the last child receives the message, we ask that child to tell us what he
heard. Everyone gets a big chuckle as the child tells us something that does not sound
anything like what we told the first child. This is true when the test team is not colo-
cated; communication gets garbled in transit. This is the third rule of avionics testing.
Whether test facilities are located 2 miles apart or 20 miles apart, the results are always
the same. Communications become garbled when the test group is not colocated.
The task team should outline a new organizational structure at the outset. A team
leader should be appointed and given the responsibility and the authority to carry out the
goals of the team. All too often team leaders are given the responsibility for the team’s
actions, but they are not given the authority required to lead effectively. In this scenario,
we have a built-in scapegoat for anything that goes wrong in the program. This scenario
also weakens a leader’s ability to assign accountability to individuals within the group.
Within CTFs or JTTs, some unique problems may occur with the team leadership. A
CTF or JTT may combine contractor, DT&E, and OT&E testers under one umbrella. In
some cases, an individual from each of the groups may be assigned as a joint team
leader. As the charters of these groups are different, there will be some jockeying for
control of the direction of the team. In the end, it may be the strongest personality that
wins out.
The individuals and groups assigned to the flight test task team are those individuals
previously mentioned in section 1.2: development and software engineering, flight test
engineering, subcontractors and vendors, and DT&E and OT&E customer representa-
tives. Each of these groups have specific responsibilities to the task team and are con-
tributors to the overall goals of the team.
The goals are set by first establishing the priorities of the program. The team must
determine if the program is one of specification compliance, development, or system
performance evaluation, or some combination of the three. Once the priorities are set,
schedules can be developed in order to meet these goals. In formulating these schedules,
it is important for the team to exercise some judgment in realism. A careful review of
similar previous programs should give an indication of how long and how difficult the
program will be. The bottom line is to make schedules realistic rather than optimistic.
The foregoing description may be recognizable as an IPT, which is very much in
vogue today. The premise is the same as task teaming, and in fact, IPTs are an outgrowth
of the task team. The greatest difference is that IPTs are formed for all ‘‘processes,’’ and
have been used in many organizations to improve flow efficiency and, therefore, cut
costs. These teams have been highly effective in areas such as assembly lines and cus-
tomer assistance, but their effectiveness as a tool for the flight test effort is in doubt.
Under an IPT-based flight test program, changes must be coordinated through all
affected IPTs. If, for example, an anomaly is detected during test and a possible solution
is determined by data analysis, the fix and retest may take several months to accomplish.
The problem and fix must first be brought before the avionics IPT, who determine if the
problem and the associated correction are acceptable. Assuming they are acceptable, the
engineering change proposal (ECP) IPT is next to be briefed. This IPT will determine if
the fix is required (i.e., within scope), how much it will cost, the timeline involved, and
if it will fit in the available schedule. If approved, the fix is presented to the software
IPT, which determines if the fix can be done, how much it will cost, the timeline
involved, and if it will fit in the software build schedule. Should the fix make it past this
hurdle, it is then presented to the flight test IPT, who determines if the procedure
requires modification to the existing test program, if additional safety concerns materi-
alize, if additional testing is required, and whether or not the test can be accomplished
within the existing testing budget for that particular system. If it is agreed that the
test can be accomplished, it is forwarded to the execution IPT. This IPT determines
if the test can be accomplished with available resources and test assets and in the
1.8 Team Member Responsibilities 13
available schedule. If approved here, the final stop is the integrated test team IPT, who
determines if the test is in the best interest of the contractor, DT&E, and OT&E. If it is,
then the test is scheduled. Each of these processes is on the order of weeks if the IPTs
meet once a week; the entire iteration is a long and tedious one. When the schedule of
software builds is taken into account, the delivery of the fix may be several months away
if a build was missed while the fix was in the IPT loop. Thus, by establishing an IPT for
each process, we have impeded the progress of flight testing by an order of magnitude.
There are other problems associated with IPT-based flight test programs. Because
there are so many identifiable processes within the flight test community, there tend to
be a like number of IPTs. This phenomenon leads to two distinct problems. The first is
the sheer number of personnel required to staff these teams. The personnel problem can
be handled in one of two ways: either hire more staff or allow the same staff to become
team members of multiple IPTs. Most flight test organizations utilize a combination of
these solutions, and the results are equally disappointing. More personnel are hired to
create a process that is supposed to save money, or personnel are so busy attending to
IPT business that there is no time to perform the actual job of flight testing.
Because of the number of IPTs created, there exists an overlap of responsibilities.
This particular problem is amplified as individuals serve on many different teams. If
there is no one responsible person who can resolve problems between teams, warfare
between the fiefdoms will ensue. Unlike the autocratic flight test of a few years back,
where the program manager controlled the purse strings and had absolute authority,
today’s IPTs are generally independent, with their own budgets. As a result, IPTs tend to
become self-serving and an impediment to the process they were designed to assist.
1.8 TEAM MEMBER RESPONSIBILITIES

Each member of the task team is assigned specific responsibilities that must be
accomplished in order to successfully complete the flight test program. Some liberty has
been taken here to identify each member’s (group) responsibilities to the team. Some
organizations may place these responsibilities in different organizations; nevertheless,
all of these tasks must be completed by someone.
Development and software engineering have the primary responsibility for provid-
ing the task team with all applicable documentation. They are tasked with preparing and
maintaining the design, performance, and integration specifications as well as mea-
surement descriptor blocks (MDBs). An MDB is the description of a parameter on the
data bus in relation to where the parameter comes from and where it is going to. It
identifies the 1553 (MIL-STD-1553 military data bus, covered extensively in chapter 3
of this text) message number, word, start bit, and stop bit for every parameter on the bus.
This information is the life blood of the test team, because this is how data are extracted
from the system. For example, if we are testing the radar altimeter, we would have to
know where the parameter ‘‘radar altitude’’ is on the data bus. The MDB provides us
with this information. MDB information is contained in the integration design specifi-
cation (IDS). This specification must be updated with each release, as the locations of
the parameters on the data bus are likely to change with each new release (parameters
are added, deleted, etc.).
The program performance specification (PPS) identifies the total system operation
and is similar to the system operations section of military technical orders (Dash 1 for
the USAF, NATOPS for the USN). When testing a navigation system, it is imperative
that we understand the hierarchy of the systems and what can be expected as navigation
systems gracefully degrade. This information is found in the PPS.
The functional requirements document (FRD) describes the operations that take
place within the system line replaceable unit. This information is important to the flight
tester because it determines what functions will need to be tested. In testing system
compliance, we must know what is happening to the data within the system in order to
determine its performance. An FRD is written for each avionics system in the aircraft
which also describes the hardware interfaces and data flow between modules.
It should be evident that this documentation is mandatory information for the test
team. Instrumentation tasking depends on up-to-date parameter information. In most
programs, the first iteration of these documents will be the last time that the information
is current. Unfortunately, documentation is probably the last priority for development
and software engineering. It is for this reason that a close liaison must be developed
between the flight test team lead engineer and the cognizant engineer. This liaison will
ensure that changes in parameter location and identification are transmitted to the test
team in a timely manner.
In addition to supplying the applicable documentation to the test team, the engi-
neering community is also responsible for a number of other tasks. They are the
responsible agency for accepting and testing government furnished equipment/contractor
furnished equipment (GFE/CFE). All upfront integration testing is conducted by the
engineering group. This task includes the integration of software and hardware on all
GFE/CFE systems, as well as vendor unique systems. They prepare a lab test plan and
conduct the actual lab tests. These tests are normally conducted in an avionics integration
facility (AIF); also known as an avionics integration lab (AIL), software integration lab
(SIL), avionics development lab (ADL), and software support and integration (SSI).
Results of lab testing are furnished in final and interim reports. Successful integration and
lab testing will result in a software release to test. This release will contain the func-
tionality of the software, affected systems, and limitations to test. A final task of the
engineering community is to provide a point of contact (POC) for each system under test.
The flight test engineer’s primary task is to ensure compliance with the top-level
specifications, statement of objectives, or EECs. In order to accomplish this task, they
must formulate a test plan, coordinate the plan through the task team, prepare schedules
to meet the plan, write test cards, brief, conduct and debrief the flight. The flight test
organization is responsible for collecting flight data, reducing it to a usable format, and
analyzing the data with respect to the task team goals. Data analysis provides the basis
for meeting the primary goal. An accurate representation of the results of the test are
provided in interim and final reports. Ancillary tasks are the documentation of anomalies
discovered during the testing phase and assisting in the troubleshooting of these
anomalies with the engineering group. Troubleshooting involves supplying data to the
cognizant engineer as well as attempting to pinpoint the cause of the anomaly. Planning
and retesting after problem resolution as part of the fly-fix-fly evolution, as well as the
formulation of a regression test plan, complete the flight test tasks.
The subcontractor and vendor test representatives provide onsite technical expertise
for their respective systems. These engineers are critical to the test effort, in that most of
the problems experienced during the integration of these vendor systems should have
been seen before in previous vendor applications. They have corporate knowledge for
their systems. Another important task of these representatives is the interpretation of
1.9 Understanding Terms 15
data collected by vendor-unique data collection systems. Many systems that are devel-
oped by subcontractors contain data collection routines, outputs, and recording instru-
ments that can only be interpreted by vendor-unique data reduction systems. These
systems are highly specialized, and purchasing one may be cost prohibitive. In these
situations, the vendor may also be contracted to reduce and analyze data from this
unique system. This flies in the face of the need for the flight test community to be an
independent organization. In this case, we have the contractor who developed the system
now providing analysis as to whether the system is performing to specifications.
Although this is similar to the fox guarding the chicken coop, safeguards can be
employed in order to ensure the integrity of the flight test process. The simplest way of
avoiding conflict is to assign an individual from the team as a data analyst to assist in the
vendor’s analysis. Personalities play a key role here, as the vendor is naturally going to
be somewhat protective of his system. This is human nature, as DT&E and OT&E
personnel assigned to a contractor test team can attest.
The customer representatives (DT&E and OT&E) should be fully integrated with the
task team. The ideal team will be comprised of analysts from all agencies, sharing the
workload of collection, reduction, and analysis in support of the common goal. In an FSD
program, where multiple systems are being tested, it is not uncommon to have a con-
tractor engineer leading the radar team, a DT&E engineer leading the navigation team, an
OT&E engineer leading the electronic warfare team, a vendor engineer leading the
missile integration team, and so on. In addition to performing analyst functions, customer
representatives normally provide operational aircrew to support the tests and may have
the authority to recommend approval for specification compliance demonstrations.
1.9 UNDERSTANDING TERMS

There is great confusion in the flight test community regarding the true meaning of
terms in relation to a flight test program. One of the first tasks of the task team is to
identify realistic goals. This can be a difficult endeavor if the type of program is not well
defined. Aside from system development or subsystem integration, a tester needs to
know if the program is either development or demonstration (sometimes identified as
‘‘qualification’’). There are major differences between the two, and a failure to realize
the correct type may cause an underbidding of the program by as much as an order of
magnitude. A development program is the type previously described as fly-fix-fly.
A demonstration program entails a proof of specification compliance, with no devel-
opment work involved. It is a straightforward test program whose satisfactory comple-
tion normally is approval for production.
A development program is heavily dependent on the software delivery schedule and
the maturity of each of the releases (that is not to say that hardware deliveries may also
impact program schedules). In this type of program, the primary task of the test team is
to identify problems in hardware and software as early as possible, isolate the cause, and
retest as necessary. Figure 1.2 shows the fly-fix-fly process and its relationship to the
test community. In order to successfully complete any avionics flight test program on
time and within budget, the cycle time of this process must be made as short as possible.
It is readily apparent from the figure that the tester could get into an infinite loop if a
discovered anomaly never gets resolved to 100%. This fact highlights the fourth and
fifth rules of avionics flight testing: 4) the duration of the flight test program is directly
FIGURE 1.2 ¢ Tape Release Flight Development Flight Demonstration

The Fly-Fix-Fly
Functional Does Yes Demo Flight
Process Evaluation It Work? Test
No
Development Issue Problem Analyze

Lab Duplicate Report Analyze Data
Data Submit
Report
Development Problem No Yes

Lab Fix Definition
Does It Work?
Lab Fix Evaluate Fix

Validation In Aircraft
proportional to the duration of the fly-fix-fly process, and 5) no software release is ever
100% mature. In other words, testers must do everything in their power to accelerate the
fly-fix-fly cycle, as well as know when to stop testing a particular function. The decision
to stop the evaluation process on a particular function is one of the most difficult tasks a
tester is presented with. More often than not, liaison with the customer will alleviate
some of these difficulties. The customer (end user) will know better than anyone if the
fleet/squadron will be able to perform its desired mission with less than optimum
performance of the particular function. The tester may also face pressure from the
engineering/software development community, who will be adamant that the function is
fixed in the next build.
I digress a moment here in order to explain what is meant by a software build. The
software build is called many things, and we frequently interchange these expressions in
flight test discussions. This text may also inadvertently use different terminology to
explain this concept. The software build may also be called a tactical release, operational
flight program (OFP), build release, interim operational flight program (IOFP), or
software release. Aside from the OFP, all of these terms relate to an incremental release
of software, with each subsequent release containing more mature functionality. The
OFP is normally released to the customer for operational use (i.e., the test program is
complete), and therefore this term will not be used in our discussions.
The term software build conjures up visions of a floppy disk or nine-track or some
other sort of electronic media used to program the aircraft mission computer. This is true
to the extent that the build normally arrives at the aircraft in some sort of media format,
but it contains much more than mission computer coding. In fact, a software build will
contain code for all processors aboard the aircraft. Figure 1.3 shows a typical example of
a hypothetical software build delivered to an aircraft for test. Avionics build 1.0 contains
all of the software programs for each processor aboard the aircraft. If any of these sub-
builds within Avionics build 1.0 are changed, then the overall build number must also be
changed.
In the case cited in Figure 1.3, the software in Display Processor 1 was changed,
perhaps to correct an anomaly found in testing, which necessitated a change to the
overall software build. Upon closer reflection, it should become painfully obvious why
AVIONICS BUILD 1.0 AVIONICS BUILD 1.1 FIGURE 1.3 ¢

Mission Computer 1 MC1.0 Mission Computer 1 MC1.0 Software Build
Mission Computer 2 MC2.1 Mission Computer 2 MC2.1 Contents
Display Processor 1 DP1.0 Display Processor 1 DP1.1
Display Processor 2 DP2.0 Display Processor 2 DP2.0
Radar Signal Processor R1.2 Radar Signal Processor R1.2
Radar Display Processor R D2.0 ANY CHANGE Radar Display Processor RD2.0
EW Processor E3.5 EW Processor E3.5
Stores Management Processor S1.0 Stores Management Processor S1.0
Navigation Processor N2.0 Navigation Processor N2.0
Command Launch Computer C5.1 Command Launch Computer C5.1
the fly-fix-fly cycle is normally a slow, tedious process. If any small change in the
subsystem software requires a major build release, it is more efficient for the software
community to collect a group of changes and incorporate all of these changes at once. In
this way, only one major release is required instead of the many it would take by
incorporating changes as they come in. Although good for the engineering software
community, this logic can be a killer for the test community. Testing often comes to a
grinding halt while testers await new software.
How then can we accelerate the development cycle in order to complete the test
program on time and within budget? One way is to allow the testing to continue with
software patches. A patch allows a specific portion of the software to be modified
without impacting the entire build. This method allows testing to continue with fixes that
ordinarily would not be seen until a new software release. A problem with this philo-
sophy is that eventually all of these patches must be incorporated into a major release.
This process is known as a software recompile, and it can create serious problems for the
software tester. After a recompile, serious regression testing must be accomplished on
the new build to ensure two things: 1) that all of the patches were incorporated correctly
in the new release, and 2) and most difficult, to determine if any other portion of the
code was affected by the incorporation of these patches. Remember that patches do not
affect the entire build because they are only written for a specific portion of the soft-
ware; they can be thought of as an addendum to the code. The recompile, however,
replaces code. Regression testing in the lab will determine the extent, if any, of the
impact of patch incorporation. Experience has shown that software recompiles that
incorporate large numbers of patches are functionally inferior to the software they
replace, and two or three iterations may be required in order to regain that functionality.
Historically the U.S. Navy has allowed patches on avionics integration programs,
whereas the U.S. Air Force is currently not allowing patches.
It should also be remembered from the previous discussions that data acquisition
software is also dependent on software releases. Each software release will entail
changes to the software used to collect data. The data acquisition routines that are used
for the current software release may only be good for that release. Subsequent releases
may require updates to these routines, and a new version number must be assigned with
each new release. The data acquisition routines are married with the software releases
and are cataloged for future reference. This is configuration management, which will be
addressed in a later chapter.
Now that software releases are understood, we can continue our discussion on the
differences between development and demonstration programs. In addition to the fly-
fix-fly concept of the development program, there are other objectives that the test
community must address. The flight test team will validate engineering test results. That
is to say, the team will determine if the systems in the aircraft work the same as in the
lab. Section 1.4 touched on this issue briefly while addressing the need for flight testing.
The development test program validates the lab simulations and mathematical models as
well as provides flight data to enhance these simulations. Since there will be many
conditions and functions that are not testable in the lab, the development test community
incorporates these functions in the flight test plan. The development team also has the
responsibility of uncovering any design discrepancies not found in the lab, providing
data, and supporting the troubleshooting of anomalies. The final task is to incorporate
the refly effort into the flight test plan and account for regression testing.
Diametrically opposed to the development test program is the demonstration, or
qualification, flight test program. In this type of program, the tester’s responsibility is to
ensure that the system is compliant with the applicable specifications. It is often
assumed, albeit wrongly, that there is no development work required for this type of
program. A program is usually deemed as demonstration when proven hardware/soft-
ware is to be installed in a proven aircraft or if a decision is made that all development
work will be accomplished in the lab. There are fallacies in each of these assumptions;
the latter was debunked in section 1.4. Although proven hardware/software is being
installed in the proven test bed, there is still the small matter of integration which must
be addressed. An installation of a MIL-STD-1553 compatible tactical air navigation
(TACAN) system into an aircraft today is much more involved than previous tasks of
ripping out an old system and widening the hole for a new system. Avionics systems of
today are not stand-alone. These systems must communicate on the bus with more than
one system and are required to interface with the host aircraft’s software design proto-
cols. If this task were as simple as it sounds, then GPS integration into any aircraft would
not take upwards of a year to complete. If a mistake is made, and a development pro-
gram is classified as demonstration only, it is likely the budget will run out a quarter of
the way through the program or some serious compromises will have to be made.
Historically the ratio of development to demonstration is 4:1, and this is the sixth basic
rule of avionics flight testing.
Even though there is no such thing as a demonstration-only avionics flight test
program, the tasks associated with a demonstration program must still be satisfied.
Normally the program will be set up for development and demonstration, with the
demonstration tasks being held back until development is complete. Demonstration is
extremely important, because 99% of the time, final payment is based on successful
completion of the qualification phase (the contractual obligations have been met). In this
phase, the test community is responsible for ensuring that the system can satisfy the top-
level specifications. These top-level specifications identify the minimum level of
acceptable performance. In the past, top-level specifications were the degree to which
the system had to be demonstrated and were used as the basis for formulating the flight
test plan. In today’s environment, many of the top-level specifications have been
replaced with design objectives or EECs. Although the names have changed, the
essential information and requirements remain the same for the test community. It is
imperative that testers understand the demonstration requirements, for without them the
program can not be intelligently planned.
Normally, top-level specifications are of the MIL-STD variety and are very specific
as to what the avionics systems are and how they should perform. An example of a top-
level specification is MIL-D-8708B(AS); the AS stands for ‘‘as supplemented.’’ This is
the specification for fighter aircraft and identifies all of the possible systems that may be
incorporated into an aircraft. MIL-D-8708 has many addenda, and each of the addendum
represents a particular aircraft. Addendum 100, for example, refers to the F-14D. The
addenda instruct the reader as to which of the paragraphs in MIL-D-8708 are applicable
to a particular type of aircraft. The addenda also refer the reader to a subsystem docu-
ment if there are further specifications to be met. For example, under Radar in Adden-
dum 100 to MIL-D-8708B(AS), the reader is referred to a radar specification, SD-4690.
SD-4690 contains the specifications to be met by the aircraft-specific radar; in this case,
the APG-66. SD-4690, in turn, refers the reader to another specification that contains
additional specifications for the APG-66 operating in special modes. This document is
classified SECRET and therefore cannot be included in SD-4690. This third document is
SD-4619. This flow of documentation is typical of specifications for military and federal
agency avionics systems. It is important for the tester to understand this flow and not be
deceived into thinking that specification compliance is meeting the objectives of
the basic specification (in this case MIL-D-8708B(AS)). The following is an example of
the level of detail a tester might expect to see in these specifications:
● MIL-D-8708B(AS), Addendum 100 might state:

‘‘the onboard radar shall be able to detect and track targets in the pulse Doppler
mode.’’
● Referring to the same paragraph in MIL-D-8708, SD-4690 may state:

‘‘the APG-XX radar shall detect and track a 2 square meter target at 35,000 ft
with a total closure of 1200 knots under maneuvers to 5g at a range of 95 nautical
miles in the pulse Doppler mode.’’
● SD-4690 may in turn refer the reader to SD-4619, which may state:
‘‘the APG-XX radar shall detect and track a 2 square meter target at 35,000 ft
with a total closure of 1200 knots exhibiting angle deception noise swept
amplitude modulation (NSAM) at a range of not less than 50 nautical miles with
elevation and angle accuracy of not less than 1.5! in the pulse Doppler mode.’’
Each of the cited paragraphs above are bona fide requirements that must be
demonstrated by the test team. If the tester elects to only demonstrate the basic speci-
fication (MIL-D-8708), the test plan is going to be very simple and very incomplete. The
plan would be so simple in fact that the total test program would not be much longer
than one flight, and that flight would basically be a ‘‘smoke’’ check of the radar. As long
as the radar did not burn up after it was turned on, it would probably meet the specifi-
cations. The second paragraph is more representative of what the test plan should be
built on, as it calls for some performance numbers that must be verified. The third
paragraph is even more specific and requires more test planning and effort in order to
meet the desired specifications. Each of these paragraphs is a mandatory criteria that
must be demonstrated. Satisfaction of the basic specification does not meet specification
compliance for the system.
The foregoing discussion should trigger some thinking; prior to initiating any test
planning activity, an in-depth review of the specifications must be accomplished. If this
review is not accomplished, and all of the test requirements are not known, serious
underestimation and underbidding of the program will occur. On a related note, consider
the effect of a poorly written specification on the quality of the product. If the basic
specification is the only specification issued, would we expect the same product as if we
had issued all three specifications? In some quarters, the answer is yes. Some agencies
honestly believe that if contractors are given only the minimum guidance, they will
produce the best possible product to meet the customer’s needs. It is believed that there
is no so such thing as a ‘‘lying, cheating contractor,’’ and given half a chance, the
contractor will work miracles. The thought process is to release the contractor of the
burden of specifications and let them build the best product possible in the shortest
possible time. Obviously it is the contractor who knows best how to build the best
system. The problem, of course, is what happens at the end of the flight test program
when the money and assets are gone and the system is not exactly what the customer
needs? All the contractor needs to do is point to the contract, and the lack of detail, and
say the contract has been satisfied. The customer really has no options for recourse
except to deny future contracts. This points out the fact that the customer ultimately
‘‘owns’’ the risk in a development program. The customer must ensure that the
requirements are accurate, complete, verifiable, and fully understood by the contractor.
On the other hand, requirements equate to cost, so the customer should not have more
requirements than is absolutely necessary.
This is not to say that this scenario will happen; however, the option is there and
customers may wind up paying twice for a system they desire. Contractors, as a rule,
will not offer this concept to their subcontractors, and this leads us into the next dis-
cussion on contractor and subcontractor specifications.
1.10 CONTRACTOR AND SUBTIER

SPECIFICATIONS
In addition to validation of the top-level specifications previously discussed, the test
community is also tasked with the responsibility of validating contractor and sub-
contractor specifications. These are sometimes called subtier or C-level specifications
and are used to guide the design of avionics systems and their interfaces with other
avionics systems aboard the aircraft. For the sake of completeness, A-, B-, and C-level
specifications can be defined as
● A-level: high-level customer requirements.
● B-level: requirements levied on the prime contractor (perhaps withholding some
margin, or removing some requirements the customer might choose to delay until
future variants).
● C-level: the specification documents the prime contractor levies on subcontractors.
The IDS, PPS, and FRD were addressed in section 1.8. The prime contractor is
responsible for these specifications. Should the prime contractor subcontract work to
other vendors, an interface specification must be written for each of the systems that
have been subcontracted. These documents describe the interfaces required between
subsystems and prime systems to allow functionality. A simpler way of describing these
documents would be to say that the interface specifications are memorandums of
agreement between the two companies.
1.11 Customer Expectations 21
Since one of the primary jobs of the avionics tester is to validate the travel of data from
the sensor through each of the related LRUs to the final output to the aircrew, they must in
fact validate the interface specifications. Testing may uncover discrepancies or incompat-
ibilities between the IDS and the subcontractor interface specifications. Any incompat-
ibilities will likely cause a failure of the system being tested. At first glance it may appear
as though the system under test is at fault; however, under closer scrutiny we may find that
the system is performing as designed. An example may clarify this point: In one particular
program, the entire avionics system was to perform to class A power specifications. An
electronic system can be designed to operate under three classes of power (A, B, or C).
These classes may be thought of as operation under differences in power quality, transients,
and recovery time. An LRU specified to operate to class A may be required to accept a
larger transient of power for a longer period of time without adverse operation than a
system designed to meet class B power. In our particular example, the specification given
to the subcontractor stated class A power requirements; however, the chart included with
the text was one for class B. The subcontractor designed the LRU to the weaker require-
ment (class B). Of course, the first time the aircraft transferred from external to internal
power, the system dropped offline. Who is at fault? Is the system unacceptable?
To the user, the system is definitely unsatisfactory, yet to the subcontractor, the
system was built as designed and performs to specifications. The prime contractor in this
case is stuck in the middle with ‘‘egg on their face’’ and is forced to fund an engineering
redesign (this particular venture cost the contractor well over a million dollars and a year
of lost time). An avionics flight tester identified the problem and cause, and pinpointed
the documentation error.
1.11 CUSTOMER EXPECTATIONS

The test community’s customer may be the final customer (the buyer) or it may be local
program management. Each of these customers has different expectations of the avio-
nics test program. The final customer will expect, at a minimum, that the top-level
specifications have been satisfied and that all CDRLs have been fulfilled. Since the test
team is more than likely a task team that includes DT&E and OT&E members who were
involved in the test plan preparation, the final customer will also expect some measure
of performance data.
Local program management is a completely different customer than the buyer. Local
program management is tasked with maintaining fiscal accountability and adherence to
the program master plan. Fiscal accountability is easy to understand; there is only x
amount of money allotted to the program and there must be somewhat of a profit margin.
Normally program management maintains a contract-oriented regimen that only permits
testing strictly called out in the contract. Program management is also concerned with
maintaining progress in the test program. Progress is measured against milestones, and
the final milestone is looming ever larger on the horizon. The test community is technical
by design, and communicating with an organization (i.e., program management) that is
not technical in nature can be a frustrating experience.
In order to maintain program discipline and fiscal accountability (and mental
sanity), a constant liaison between the program office and the technical test team
must be maintained. The first consideration in this liaison is maintaining a balance
between the program milestones and technical merit. This can best be illustrated by the
‘‘flight around the flagpole’’ (a flight accomplished for the sole purpose of demonstrating
to the world that your program is alive and well), of which all test programs have been
guilty. Unfortunately one of the methods used to gauge progress in a flight test program is
flight rate; the more you fly, the more progress you are making, and vice versa. One of
the worst things that can happen in a flight test program is for the test vehicle to be on the
ground, either intended or unintended. The unintended portions include weather, main-
tenance, or instrumentation problems, and are nonproductive periods of time that cannot
be avoided. But what happens if the test team suggests that flying be halted for a period
of time so data can be analyzed, or worse, the team needs new software and is devoid of
constructive tests. More often than not, this halt in flying indicates a halt in progress
which can be easily alleviated by a continuance of flying. Therefore the test team is
directed to resume flying (progress). This example may seem silly to some, but rest
assured, it does occur in today’s environment. It is the test engineer’s responsibility to
plead their case to management and present the rationale for standing down. The test
engineer must be prepared to justify every test. Conversely, the test engineer must also be
prepared to explain why a test is a waste of time/money, even while program manage-
ment is demanding that testing start/resume in order to ‘‘show progress.’’
It must also be remembered that flight testing is dependent on the configuration of
the hardware as well as software in the test bed. Asset requirements, hardware as well as
software capability, must be relayed to program management in a timely manner in order
to preclude flying with the wrong configurations. If the test plan calls for testing data link
at a certain period of time and the hardware is not available, the applicable milestones
showing these tests must be moved to the right on the program schedule. Failure to do so
sets up a conflict between the program and the test team when the tests are not complete.
Similarly, when the software does not support the test milestones, a similar scenario will
occur. Above all, it must be remembered that scheduling, briefing, and conducting a
nonproductive mission takes as much time, effort, and manpower as a productive flight.
1.12 FORMULATING THE PROGRAM

Understanding customer requirements, test team goals and responsibilities, and the nature
of the test program allows us to formulate, or estimate, the program. It is important to
realize that undertaking an estimate without first determining these aspects is a recipe for
failure. Each estimate should follow a tried and true methodology: determine the type of
program and the required outcome, determine the assets required to perform the task in
the time allotted, and estimate the task. We have discussed the factors that affect the type
of program and will now address the assets required for the time allowed.
By knowing the nature of the development, demonstration, or performance
requirements of a system, we can set out to identify each of the following:
● The number and type of test vehicles required
● Instrumentation
● Support facilities
● Size and composition of the test team
● Special hardware or software requirements
● Schedule of software releases
1.12 Formulating the Program 23
Normally the number and type of test vehicles are predetermined by program
management, the customer, fiscal constraints, or some combination of these factors.
In estimating a program where the final determination of assets has not been ascertained,
the tester may be asked to propose a number which he feels is necessary in order to
complete the program in the time allowed. The key factor here is time.
It is an excellent idea to first research past programs with similar requirements and
examine the total sortie (flight) count and total time for program completion. This
information is archived in many locations within test organizations and is just begging to
be read prior to a new test program. These past histories probably reside next to the
‘‘lessons learned’’ notes from the same program. Once again, it is unfortunate that many
testers would rather reinvent the wheel than learn from others’ experiences. Past history
and lessons learned from similar previous programs provide invaluable insight into what
can be expected in the program you are estimating if the lessons are applied appro-
priately to the test activity at hand. There are a series of reports available for reference
regarding the aforementioned discussion. The Rand Corporation has been involved with
Project AIR FORCE and has issued a number of reports documenting trends and lessons
learned on T&E projects. The first one that I would like to reference is Test and Eva-
luation Trends and Costs for Aircraft and Guided Weapons which was published in 2004
and covers many of the topics previously addressed. The second Project AIR FORCE
report is entitled Lessons Learned from the F/A-22 and F/A-18E/F Development
Programs, Rand, 2005. Quite an interesting read.
Consider the case where a previous radar program spanned 3 years and 300 sorties
and the data time per flight would have been 1 hr. On closer examination, we see that
only one test bed was used for the program, and that the initial schedule called for the
test bed to fly 10 sorties per month. The actual flight rate was something less, about
8.33 sorties per month. This information tells us a great deal about a radar program. It
not only tells us the total number of sorties required, but it also tells us that there should
be some contingency factor of about 20% to account for loss of scheduled sorties from
the plan. This loss could be due to maintenance (scheduled or unscheduled), weather,
software, instrumentation, or personnel problems (sick, holidays, vacations, etc.). If we
used an historical program that was accomplished at Edwards Air Force Base, where
there are on average 330 days of flyable weather, and compare it to a current program
being conducted at Patuxent River, MD, where weather does pose a problem, some
adjustments would have to be made. A contingency of 20% is realistic and can be used
in estimating any avionics or weapons system test program. The numbers also show that
if flight time can be increased or if assets can be added, the total program time will
decrease. If our historical program used an F/A-18C as a test bed and our test bed is an
F-15E, the flight time will indeed be doubled and the program should be halved. Care
must be taken in the amount of time compression we anticipate in a test program. As we
compress time, we place a larger burden on the flight test analysts to turn data around
more rapidly and on software engineering to compress the fly-fix-fly cycle. Either of
these burdens may be eased by increasing manpower, but the incurred cost may out-
weigh the benefit of compressing the test program.
In order to show compliance, troubleshoot anomalies, and report performance,
we must provide data. Data can only be provided if the test bed is instrumented.
A second phase of formulating the program is determining what instrumentation is
required, in what format data should be presented, and whether these data need to be
transmitted to the ground in real time or be postflight processed. In addition, the type of
recording—video and data—needs to be determined, as well as time synchronization

and whether the development lab needs to be instrumented. The PCM format and
whether edited data bus or full data bus will be recorded also needs to be determined.
Each of the items in the instrumentation package directly impacts the length and
cost of the avionics program. Real-time data will shorten the program considerably;
however, the cost may be prohibitive. Actual labor to instrument the aircraft may take
longer than the program allows or equipment may not be available. Special instru-
mentation such as onboard simulators and stimulators have very long lead items and
need to be accounted for in the estimate if they are mandatory for success. As is the case
with test vehicles, the test engineer must be aware of the trade-offs between time,
money, and manpower affected by the type of instrumentation package chosen.
Often overlooked when trying to formulate a test program are the seemingly benign
support facilities. Test range support needs to be addressed, and the level of support that
is necessary for successful program completion needs to be accurately conveyed to the
range directorate. This is addressed in some detail in section 2.16 of this text.
In addition to consolidation of manpower within the test teams, the current policy in
the United States is to consolidate test sites as well. Flight testing of the future may be at
either an East Coast or West Coast range facility. Smaller government and contractor
facilities will be forced to unify under one of these two facilities. Work in this area is
already well under way, and it is yet to be seen who the big winners are.
Nothing is more distressing for a tester than to arrive at a test facility and find out
that there is no place to sit, or there’s no phone or computer access, or that the test article
has to brave the elements because there is no hangar space. All of these scenarios are
distinct possibilities if there was no upfront planning and coordination on the part of
the test team. Do not be lulled into a false sense of security and into believing that since
the test program has been directed to use a designated facility that all is in readiness. In
fact, there are additional costs that need to be accounted for. Other than workspace areas
and telephone and computer access, there are other areas that must be considered and
which are often overlooked.
Clearances and certifications may not be transferable from site to site; unless pre-
coordinated, program clearance at one site does not automatically grant access to
another. The ability to transmit radio frequencies requires a clearance and a deconflic-
tion. In the United States, this is obtained through the regional frequency council; radio
frequencies also cover telemetry frequencies. A clearance to transmit data will also be
required, and in many cases can only be done if the data is encrypted, which can lead to
a cascading list of additional requirements. Encryption requires additional hardware and
software, and in most cases raises the security classification. With enhanced security
comes enhanced security measures in terms of access, storage, and handling. Data must
be logged, controlled, and stored in special facilities and access to the data must be
controlled. Crypto keys are controlled, change from day to day, and require specialized
personnel to load the keys into aircraft and telemetry processors. All of these additional
requirements will have an impact on the cost of the program.
Certification in a particular field, such as hydrazine handling, may not be valid at
the test site, especially if you have a contractor certificate going to a government test
site. Military personnel are not exempt either, because in many cases the Navy does not
recognize Air Force certification, and vice versa.
Data reduction and processing facilities at the test location may not be able to
handle the installed instrumentation system, or the data rate, or support rapid turnaround
1.13 Summary 25
of data. In the worst-case scenario, a test team estimates the length of a program based
on a 24 hr turnaround of data, only to find out later that data will not be available for
analysis for 72 hr. In a similar vein, if data must shipped to the engineering group for
confirmation of results or for troubleshooting purposes and they are located many miles
away, the program will be delayed as well. This can be disastrous to the efficient flow of
the program. There are two options: maintain the schedule even though the data will be
delayed, or slip the schedule to accommodate the delay of data. From our previous
discussions, you know that you are not going to slip the program, because that costs
money. On the other hand, if you maintain the schedule, you will probably be flying
without the benefit of knowing the last flight’s performance. The end result of such a
miscalculation is wasted or lost flights and a program coming to an end without
obtaining all the necessary data. Knowing all of the limitations of a test program, not just
system limitations, is imperative if you intend to be successful.
The requirements for special hardware and software drive the program schedule and
cost. Special instrumentation, such as data bus editors or word grabbers, may cost some
additional funds, but more importantly, they require a long lead time. Times of one year
or more to design, fabricate, install, and test a special piece of instrumentation are not
uncommon. You cannot hinge the success of your program on the acquisition of key
pieces of data if you do not know how you are to capture that data. Never assume that all
data are always available with off-the-shelf, time-tested instrumentation. This is another
reason why instrumentation personnel need to be brought into the program early.
As previously mentioned, knowledge of the capabilities and limitations of the
installed software is imperative to efficient testing. Similarly, you must know the
schedule of builds in order to put together a rational schedule. Remember, incremental
builds are the norm in avionics testing, and the length of the test program is directly
proportional to the functional capabilities available in each build. The schedule will also
stretch with the amount of time it takes to fix a problem in the software. How does
today’s tester understand the capabilities and limitations of software? They need to be in
the lab during validation and verification testing long before the software gets to the test
bed. This is the only way a tester can understand the maturity of a software system.
Armed with this knowledge, a realistic schedule can be formulated.
1.13 SUMMARY
Before putting pen to paper or scheduling your first test, it is important to realize that
there are many variables to be considered in the world of avionics systems flight testing.
The rules you may have learned in vehicle testing may not apply. The single largest
difference is that you are dealing with a nasty little item called software. Do not ever
believe the adage that software changes are ‘‘transparent to the user,’’ or that ‘‘we didn’t
change anything in that module.’’ Software has extremely long tentacles, and changes in
one module may affect other seemingly unrelated modules. Scheduling test programs
can be an interesting effort, as there are so many items that can affect them: software
maturity, fix cycle, delivery schedule of builds, data turnaround time, etc. It takes a
tester adequately versed in these parameters to make an avionics test program work
properly. The tester that recognizes these variables, reads the ‘‘lessons learned’’ file, and
applies them appropriately to the test activity under consideration is the one who
will succeed.
Remember the six basic rules when performing avionics and weapons systems
testing:
1. The final milestone (D) never moves to the right.

2. Truth data (TSPI) needs to be four times as accurate as the system under test.
3. Whether test facilities are located 2 miles apart or 20 miles apart, the results are
always the same: communications are garbled when the test group is not colocated.
4. The duration of the flight test program is directly proportional to the duration of the
fly-fix-fly process.
5. No software release is ever 100% mature.
6. Historically the ratio of development work to qualification testing is 4:1.
1.14 SELECTED QUESTIONS FOR CHAPTER 1
1. What are the major differences between vehicle testing and systems testing?
2. Name three major objectives of flight testing.
3. What is a measurement descriptor block?
4. What is the bulk of testing in today’s environment?
5. What is an avionics build (software release)?
6. How come it takes so long to receive new/updated software?
7. Software is incremental. How does this fact affect systems testing?
8. What is a CDRL?
9. What is a DID?
10. What is the difference, if any, between DT&E development, demonstration, and
performance testing?
11. Why are data important when identifying problems?
12. Who has the responsibility of ensuring that the system meets the specifications?
13. What is meant by the term ‘‘composite test team’’?
14. What are ‘‘lessons learned’’ and how do they apply to flight testing?
15. What is fly-fix-fly?
16. What factors affect the length of a system’s flight test program?
17. How would you describe avionics systems flight testing?
18. Name the advantages of using real-time data.
19. What is an IPT? What is its purpose?
20. Who are the key individuals that should be a part of the flight test team?
21. What is OT&E responsible for?
1.14 Selected Questions for Chapter 1 27
22. Why can simulation not be used in lieu of flight testing?

23. Give three examples of objectives that cannot be accomplished in the simulator.
24. Is combining DT&E and OT&E a good idea? Why or why not?
25. What is a top-level specification?
26. What is the program manager’s purpose?
27. Will there ever be a difference of opinion between flight test and program
management?

Avionics Fligth Test

Uploaded by

Document Informationclick to expand document information

Copyright:

Available Formats

Avionics Fligth Test

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Avionics Fligth Test

Uploaded by

Copyright:

Available Formats

CHAPTER

What is Avionics Flight Test

1.0 AVIONICS FLIGHT TEST

1.1 HISTORICAL BACKGROUND

1.2 FLIGHT TEST ORGANIZATION

1.3 FLIGHT TEST OBJECTIVES

1.4 THE NEED FOR FLIGHT TEST

1.5 CLASSIFYING THE PROGRAM

● Build A: raw sensor development

Vendor Development ∆ ∆ ∆ ∆ ∆ FIGURE 1.1 ¢

Demonstration Flight Test ∆ ∆

1.6 CONTRACTUAL REQUIREMENTS

1.7 TEST TEAM COMPOSITION

1.8 TEAM MEMBER RESPONSIBILITIES

1.9 UNDERSTANDING TERMS

FIGURE 1.2 ¢ Tape Release Flight Development Flight Demonstration

Development Issue Problem Analyze

Development Problem No Yes

Lab Fix Evaluate Fix

AVIONICS BUILD 1.0 AVIONICS BUILD 1.1 FIGURE 1.3 ¢

● MIL-D-8708B(AS), Addendum 100 might state:

● Referring to the same paragraph in MIL-D-8708, SD-4690 may state:

1.10 CONTRACTOR AND SUBTIER

1.11 CUSTOMER EXPECTATIONS

1.12 FORMULATING THE PROGRAM

recording—video and data—needs to be determined, as well as time synchronization

1. The final milestone (D) never moves to the right.

1.14 SELECTED QUESTIONS FOR CHAPTER 1

22. Why can simulation not be used in lieu of flight testing?

You might also like