W E I B AY E S T E S T I N G : W H AT I S T H E I M PA C T I F A S S U M E D B E TA I S I N CO R R E C T ?

1
David Nicholls, RIAC (Quanterion Solutions Incorporated) Paul Lein, RIAC (Quanterion Solutions Incorporated) Weibayes is a one-parameter Weibull analysis technique developed by Dr. Robert Abernethy and other engineers at Pratt & Whitney Aircraft in the 1970s to solve problems when traditional Weibull analysis has large uncertainties or cannot be used because there are no failures.[1] A basic assumption governing the accuracy of Weibayes analysis and its associated test regimens, as stated by Abernethy, is that the value of the Weibull shape parameter, b, is known or can be reasonably estimated. Knowledge of b can be derived from historical failure data, prior experience, or from engineering knowledge of the physics of the failure. Engineering knowledge of failure physics and consistent use of accurate, representative b values is best supported by an historical library of Weibull beta plots based on actual corporate and product experience. where, R(td) = Reliability at the design life, td td = Design life to be demonstrated h = Characteristic life at CDF = 63.2% b = Weibull shape parameter For Weibayes analysis, the characteristic life is expressed by Abernethy [2] as: 1

N T = i i=1 r

(2)

where, Ti = Test time for each sample r = Number of failures N = Sample size h, b As defined above In order to determine an appropriate test time per sample, Equations (1) and (2) can be combined and rearranged to give:
t d Ti = N * ln( R( t d )) 1

Defining the Problem

In the absence of good engineering knowledge of b, inaccuracies in the assumption of its value can have a significant impact on (1) the Weibayes test regimen used to define a reliability requirement, and (2) the interpretation of Weibayes test results used to demonstrate whether that reliability requirement has been met. Marketplace pressure towards decreased test times, fewer test samples, and zero- or sudden death-based test regimens are intuitively beneficial because they lower the total test cost and allow decisions to be made earlier. As one hypothetical example, assume that we are given a stated reliability requirement of 0.90 at 1000 hours (i.e., the B10 life). A Weibayes zero failure test plan is formulated using a sample size of 5 and an assumed value of 1.5 for b. The test time required to demonstrate that this requirement has been met is calculated to be 1533 hours per test sample. What are the corresponding implications and risk in the interpretation of the test results if the true value of b is something different? The purpose of this paper is to examine and quantify the risk of assuming an incorrect value of b that is higher than the true value when performing Weibayes zero-failure or sudden death testing, and the subsequent impact on the analysis and interpretation of the results. Reliability at a specified design life can be calculated based on Equations (1) and (2). From the standard Weibull equation:

(3)

Substituting the values from the Introduction yields a 1533-hour test requirement per sample for our Weibayes zero-failure test example to demonstrate that the design life of 1000 hours at 0.90 reliability has been achieved. There are other test options available to demonstrate this requirement, however. Sample sizes can be changed. As an alternative, since the true value of b is unknown, a different assumed value of b can be used. The question then becomes, what is the best test plan to use, given that the 1000-hour design life at 0.90 reliability is a firm requirement and there are cost, resource and schedule constraints to be considered. Using Equation (3), a range of possible test scenarios can be generated to provide visibility into options for sample sizes that include potentially better assumptions for b. Table 1 illustrates one example of potential zero-failure test scenarios, based on assumed values of 1.5 and 3.0 for the Weibull shape parameter.

R( t d ) = e

t d

(1)

1 . This article is adapted, with permission, from the 2009 Proceedings of the Annual Reliability and Maintainability Symposium. 2009 IEEE.

THE JOURNAL OF THE RELIABILITY INFORMATION ANALYSIS CENTER

FIRST QUARTER - 2009

Design Life (td) = 1000 Hours Maximum Likelihood Estimate (MLE) Sample Size (N)
1 2 3 4 5 10 15

Per-Item Test Time (Ti) to Demonstrate B10 Life (R = 0.90) b = 1.5

4483 2824 2155 1779 1533 966 737

b = 3.0
2117 1680 1468 1334 1238 983 859

for the smaller b. For sample sizes of N equal to 10 and 15, however, this relationship is reversed. What is the reason for this? Simply put, the intersection of the assumed and true beta plots represents a pivot point that can influence decisions about Weibayes test plans that may subsequently result in bad decisions and unacceptable, but unidentified, risk based on conclusions about the demonstrated reliability. The two parameter Weibull distribution [3] defines the Cumulative Density Function (CDF) as:

F ( t ) = 1 e
where,

(4)

Table 1: Weibayes Zero-Failure Test Scenarios Example For this hypothetical example, lets say that organizational constraints dictate that we cannot afford to use more than 5 test samples, and there are significant schedule pressures to get this product to market. From the table, under the b = 1.5 column, using fewer test samples requires additional test time per sample (unacceptable within our schedule constraints). Using more samples reduces test time per item, but this has already been excluded from consideration. One remaining option, since the true value of b is unknown anyway, is to assume a different value for b. This allows us to use the same number of samples, yet we gain some relief in the test time per sample (approximately 300 hours if we test concurrently, or 1500 hours if we test serially). In the competitive marketplace, every hour counts. How much impact can using this different value of b really have, anyway? The remainder of this paper will provide some insight into the answer.

t = Time (in hours) h = Characteristic life at CDF = 63.2% b = Weibull shape parameter Mathematical manipulation of Equation (4) leads to Equation (5), which represents the straight-line solution to be plotted on Weibull graph paper [4].

1 lnln = ln( t ) ln() 1 F ( t )

(5)

1 lnln 1 F ( t ) ln( t ) = + ln()

The location of the intersection point of two Weibull solution lines with different betas (based on the same number of hours, t) requires that ln(t1) be set equal to ln(t2) and:

The Relationship Between Test Sample Size and the Beta Intersection Point
If you review Table 1 for a given sample size and compare the required test times at b = 1.5 and b = 3.0, you will observe that, for sample sizes less than or equal to 5, the per-sample test times for the larger value of b are noticeably shorter than those

1 1 lnln = lnln 1 F ( t1 ) 1 F ( t 2 )
Substituting

(6)

1 y = lnln 1 F ( t )

and equating the two beta

equations (derived from Equation 5) yields: continued on next page http://theRIAC.org 3

WEIBAYES TESTING: WHAT IS THE IMPACT IF ASSUMED BETA IS INCORRECT? continued from page 3

y=

1 2 (ln(2 ) ln(1)) 2 1

(7)

Table 2 provides a reference for the beta intersection point for various test sample sizes.

For Weibayes zero-failure testing with equivalent test times, the number of failures, r, from Equation (2) is equal to 1 and the summation term becomes the sample size, N. Substituting the expression for h from Equation (2) into Equation (7) yields the two-beta equation:

y=

1 1 1 2 ln( NT 2 ) 2 ln( NT 1 ) 1 2 1

(8)

The intersection of the two beta lines can then be algebraically shown to occur at the By life percent value, where:

By life % = e y *100% =

1 *100% N

(9)

Figure 2: Beta Intersection Point for Sample Size = 10 Beta Intersection CDF (By Life)
50.00 33.33 25.00 20.00 10.00 4.00 2.00 1.00 0.10

This result is shown graphically in Figures 1 and 2 for a sample size of 5 and 10, respectively.

Sample Size (N)

2 3 4 5 10 25 50 100 1000

Table 2: Beta Intersection Point as a Function of Sample Size

Figure 1: Beta Intersection Point for Sample Size = 5 Each Weibull graph includes a line plotted at b = 1.5 and b = 3.0. For N = 5, the beta intersection point occurs at approximately CDF = 20% (or the B20 life), disregarding any inherent small sample bias that is associated with the Weibull Maximum Likelihood Estimate (MLE). For N = 10, the beta intersection point occurs at CDF = 10% (or the B10 life). Note that the beta intersection point is independent of both the design life, td and the test time, Ti (assuming that the test time on each of the N samples is equal). In other words, different values for the design life will shift the beta plot pair left or right along the x-axis, but the beta intersection point will not deviate from the By life value.

The Relationship Between the Confidence Level and the Beta Intersection Point
What happens to the beta intersection points when lower-sided confidence bounds are introduced into the analysis? Simply stated, with increasing confidence level the beta intersection point will shift vertically at design life, td. While it will be the focus of a future paper to determine the mathematical relationship that governs this shift, Figures 3 and 4, respectively, illustrate the concept. In this example, the design life is set at 1000 hours. The resulting CDF is 37% at the 80% lower confidence bound (LCB) and 49% at the 90% LCB (compared to the CDF of approximately 20% at the MLE based on the sample size of 5).

THE JOURNAL OF THE RELIABILITY INFORMATION ANALYSIS CENTER

FIRST QUARTER - 2009

 In

considering the use of confidence bounds, as the desired lower confidence bound increases, the CDF value of the beta intersection point also increases.

Based upon these general relationships, we can now make some specific observations regarding the presence and quantification of risk associated with Weibayes testing if there is a difference between the assumed and true values of beta that govern the test length and the subsequent interpretation of the test results.

Figure 3: Beta Intersection Point at 80% Lower CB

Risk as it Relates to the Difference Between the Assumed and True Values of Beta
As alluded to in the introduction and problem definition sections of this paper, there is risk associated with using an assumed value of b in establishing Weibayes test plans and interpreting Weibayes plots when the true value of b is something different. The risk discussed in this paper is limited to underestimating the required Weibayes test time to demonstrate a specified design life by assuming a value of b that is higher than the true value. First, a general statement:

The higher the CDF level of the beta intersection point,

the greater the risk of reaching an overly optimistic conclusion for a fixed By life below that intersection point. For the purposes of quantifying the risk, we will work with Figures 5 and 6. This hypothetical example is based on a B10 life requirement of 1000 hours. Due to organizational budget and schedule constraints, it was decided to proceed with a Weibayes zero-failure test based on a sample size of three. The organization does not have a Weibull library of beta values for its products, or adequate knowledge of the predominant physical failure modes of the design, so it needs to rely on engineering judgment to establish an appropriate value of beta for the test (the true value of beta is unknown). Best engineering judgment concluded that, at the 1000-hour B10 design life, the product would exhibit signs of wearout, so it was decided that b = 3.0 would be appropriate (especially since it resulted in a shorter test time that supported management budget, resource and schedule constraints). Based on these criteria, the three samples were tested for 1468 hours each (based on Table 1 for b = 3.0).

Figure 4: Beta Intersection Point at 90% Lower CB

General Relationships of the Beta Intersection Point

Upon examination, the following general comments can be made for a given design life:

There is an intersection point between Weibayes plots of

two values of b, below which the required test time to demonstrate that the design life has been met is shorter for the higher value of beta than the lower value (see Table 1 for n < 5). The opposite is true above the beta intersection point (see Table 1 for n = 10 and 15). The smaller the Weibayes test sample size, the higher the CDF value at which the beta intersection point occurs (see Figures 1 and 2 and Table 2).

continued on next page http://theRIAC.org 5

WEIBAYES TESTING: WHAT IS THE IMPACT IF ASSUMED BETA IS INCORRECT? continued from page 5

failures for this example is to simply read the results directly from Figure 6. At T = 1000 hours, the assumed b = 3.0 line indicates a CDF of 10%, indicating that 10% of the population are expected to have failed by that time. The true b = 1.5 line at T = 1000 hours indicates that approximately 17% of the population will have failed.

Figure 5: Quantifying Risk When Assumed Beta is Higher Than True Beta The test was run and no failures were experienced, so the organization was confident that the design life requirements had been met. Unfortunately, returns from the field during warranty did not support this conclusion. What might have gone wrong? If the organization had been in a position to apply the resources needed to collect and analyze their data and characterize the products dominant physical failure modes, they might have discovered that the true beta for their product was actually 1.5, meaning that the product wasnt wearing out as fast as engineering judgment had assumed. Superimposing a Weibayes plot with a beta of 1.5 on Figure 5 provides some clarification of the impact of this assumption. From Table 1, the organization tested their product for the required time based on a sample of three and an assumed beta value of 3.0. If they had known that the true beta was actually 1.5, then they would (or should) have tested each of the three samples for 2155 hours (with zero failures) to support a conclusion that the design life requirement had been met. As a consequence, each sample was undertested by 687 hours (2155 hours 1468 hours). The impact of insufficient testing on the conclusions drawn from the analysis can be read directly from the Weibayes plot in Figure 5. Instead of demonstrating a B10 life of 1000 hours, the testing actually demonstrated a B10 life of only 685 hours, which is 68.5% of the requirement. The analysis can easily be extended to determine the increase in the expected number of failures over any time period and population size of interest. Additionally, the increase in associated repair and support costs resulting from the lack of knowledge of the true beta value can also be determined. One approach to estimating the increase in the expected number of

Figure 6: Increase in Expected Number of Failures Mathematically, the exact values can be found from Equations (1) and (2). Using the results from the analysis above and substituting them into Equation (2) yields:

3 1468(1.5) 1.5 = = 3054 hours 1 i=1

where, 3 = Sample size, N 1468 = Test time per sample, Ti 1.5 = True Weibull shape parameter, b Substituting the value of h into Equation (1) results in:

(10)

R(1000) = e

1000 1.5 3054

= 0.829

CDF = 1 R(1000) = 0.171 or 17.1%

where, 1000 = Design life, td 3054 = Characteristic life, h 1.5 = True Weibull shape parameter, b 0.829 = Reliability at the design life, td 0.171 = CDF at the design life, td = (1-R(td))

(11)

Suppose that there are 10,000 products (defined as P) in the field and it costs $5,000 to repair each returned item. The

THE JOURNAL OF THE RELIABILITY INFORMATION ANALYSIS CENTER

FIRST QUARTER - 2009

increase in the expected number of returns at 1000 hours, based on a true beta of 1.5, will be: ([CDF(btrue)*P] [CDF(bassumed)*P]) = (1710) (1000) = 710 additional returns (12)

the location of the design life CDF requirement in relation to the beta intersection point, and (3) the relative difference between the values of the assumed and true beta. This paper addresses only the risk associated with using an assumed beta value that is higher than the true beta value, as this represents the most damaging technical and financial risk scenario for the organization. As such, the statement can be made that the greater the difference between the values of the assumed and true values of beta (with the assumed beta being higher), the greater the organizations risk through undertesting a product to demonstrate that a required design life has been met, and being overly optimistic in the interpretation of the demonstrated reliability. General recommendations that can be made through this investigation are that:

At $5,000 per repair, these additional returns will result in approximately $3.55 million in unanticipated cost to the organization.

MS Excel Spreadsheet
Quanterion Solutions Incorporated (QSI), in its role in the operation of the Reliability Information Analysis Center (RIAC), has developed a MS Excel spreadsheet based on the concepts presented in this paper. The spreadsheet supports:

Calculation of a Weibayes zero or sudden death failure test

plan as a function of (1) sample size, (2) required design life, (3) the value of assumed beta, and (4) the potential value of true beta. The output is the required test time to demonstrate that a design life requirement has been met under the stated conditions. Calculation of the error in the estimated design life based on the value of an assumed beta and the potential true beta. This error is translated into the actual design life based on the true beta in comparison to the expected design life based on the assumed beta. Graphical representation of the error and resulting design life for a range of true beta values in comparison to the single assumed value of beta Calculation of the increase (or decrease) in the expected number of failures over a user-defined time period based on the difference between the assumed and true value of beta. Simple calculation of the cost impact associated with the increase (or decrease) in the expected number of failures over the user-defined time period

Each

organization should strive to thoroughly understand and differentiate the physical modes and mechanisms associated with the root failure causes of its products in order to identify their true beta values Each organization should establish its own library of Weibull plots and beta values that are representative of its specific products and applications, and are based on the physical modes and mechanisms associated with the root failure causes of its products Where cost and schedule permit, an organization should use the exact number of test samples that correspond with the By design life that is to be demonstrated, as that value determines the intersection point between the assumed and true value of beta. For example, to demonstrate that a B10 life requirement is met, use a sample size of 10, as the intersection of the assumed and true beta plots will be at the required design life (i.e., no risk if the assumed and true values of beta are different).

The spreadsheet is available for download from http://www. theriac.org/informationresources/demosanddownloads.swn

References
1. 2. 3. Abernethy, R.B., The New Weibull Handbook Fifth Edition, Dr. Robert B. Abernethy, June 2007, pg. 6-1 Abernethy, R.B., The New Weibull Handbook Fifth Edition, Dr. Robert B. Abernethy, June 2007, pg. 6-2 Weibull, W., A statistical distribution function of wide applicability, J. Appl. Mech.-Trans. ASME, September 1951, 18(3), pg. 293-297 Nicholls, D. (Editor/Co-Author), System Reliability Toolkit, Reliability Information Analysis Center/Data and Analysis Center for Software, December 2005, pg. 526

Conclusions and Recommendations

We have shown that there is a mathematically supported relationship between the intersection point for an assumed and true value of beta that can be used to assess and quantify risk when performing Weibayes zero failure or sudden death testing and analysis. The level of risk is a direct function of (1) the vertical location of the beta intersection point on the Weibayes plot (mathematically proven to be a function of the number of samples tested), (2)

4.

http://theRIAC.org