This paper reviews the process by which the Population Estimates Branch of the U.S. Bureau of the Census creates population estimates for multiple layers of geography, and introduces a method being developed for evaluative purposes known as a "loss function".

The Population Estimates Branch (PEB) of the U.S. Census Bureau is responsible for producing state, county and subcounty population estimates for the entire United States. A combination of population estimate techniques is used, primarily because available input data and different techniques have historically been shown to create the most accurate estimates at different geographic levels. In developing and improving population estimate techniques, one of the most difficult elements is not the creation of estimates via different methods, but, rather, a sound evaluation program. In many small area estimates programs, hundreds or even thousands of estimates may be generated for geographic entities. Customarily, statistical measures such as Mean Average Percent Error (MAPE), Mean Algebraic Percent Error (MALPE), and visual evaluation have been used to review estimates and compare them to previous versions or to other sets of estimates. It has been argued, ho! ! ! wever, that measures such as MAPE are invalid for evaluating error in that they may be strongly affected by skewed distributions of error (Swanson et al. 1999) and that they have a difficult time evaluating data series with wide ranges of population values. Other evaluative measures such as visual examination and error ranking are plagued by inefficiencies and the inability to detect the extent and magnitude of error.

These shortcomings suggest that these evaluation tools have been pushed to the limit of their usefulness and that new tools should be examined. This paper provides a preliminary examination of such a tool, the loss function, which is under development by the PEB for use in its estimation evaluation program. Before describing this tool, however, the paper sets the context by first providing a brief review the population estimate system used by PEB.

An effective system for making national population estimates must have the support of the national government and be based in law. The legal requirement for population estimates in the United States is given in Title 13 of the U.S. Code, which states, "...for the purpose of administering any law of the United States in which population or other population characteristics are used to determine the amount of benefit received by State, county, or local units of general purpose government, the Secretary shall transmit to the President for use by the appropriate departments and agencies of the executive branch the data most recently produced and published under this title."

The Census Bureau generates subnational population estimates for general purpose functioning governmental units: these have elected officials who can provide services and raise revenue. The estimates are vitally important, and have a wide variety of uses. These uses include Federal and state funds allocation (Martin & Serow 1979), denominators for vital rates and per capita time series, survey controls, administrative planning, marketing guidance, and descriptive and analytical studies. (Long 1993)

By any measure, developing an estimation system to meet these requirements is a formidable challenge. The United States is covered by a wide range of nonuniform geopolitical units, many with unstable boundaries. It is important to recognize that the geography used to create population estimates is unique, and does not conform exactly to the decennial census geography. The PEB develops population estimates for counties, which are the primary legal divisions of most states; Minor Civil Divisions (MCDs) legally defined county subdivisions; and Incorporated Places, which include cities, boroughs, towns and villages (with exceptions). In summing estimates for these levels and their pieces, there are over 80,000 pieces of geography for which the PEB makes estimates.

The difficulty of making accurate estimates for multiple, and oftentimes overlapping, levels of geography is compounded by the use of different methods and data sources. The results of these different methods and sources often must be reconciled in order to compile final estimates. These estimates are generated and reviewed concurrently, and are not necessarily produced in descending geographic order. A brief review of the estimation techniques is as follows:

• National estimates are generated using the cohort-component method, taking the 1990 enumeration of resident population, adding births, net international migration and net movement of U.S. Armed Forces and civilian citizens to the United States, and subtracting deaths.
• County estimates are created using a component change procedure called the tax return method (formerly called the administrative records method), which combines estimates of the population under 65 and 65 and over, group quarters, and household population. Birth and death statistics are combined with a migration rate, calculated from IRS personal income tax return data. The resulting county estimates are then controlled to the new U.S. population total.
• States are treated as tabulation geography rather than estimates geography, in that their estimates are created by summing controlled county estimates.
• Sub-county population estimates are made using a variation of the housing unit method known as the distributive method, which uses tabulations of housing stock as derived from administrative records, and calculates person-per-household (PPH) and occupancy rates to estimate household population, to which group quarters population is added. Household population totals are then controlled to county household populations as calculated by the tax return method.

A population estimation system involves the collection of necessary data, a sound statistical procedure to create the estimates and an evaluative system to ensure the estimates are reasonable. The evolution of the U.S. population estimation system spans nearly 100 years. Over this time, potential data sources and competing techniques have frequently been tested and compared to ensure the highest accuracy within the constraints of timeliness and resources. Most frequently, alternate data sources and estimate techniques have been tested against the decennial census. While the variables of time and cost are relatively easy to quantify, measures of accuracy are often muddled - especially for large series of estimates. Estimates may be considered accurate if they are "close to" the parameters they are estimating. Unfortunately, these are often unknown. The Panel on Small-Area Estimates of Population and Income (1980:10) advises that a variety of measures of c! ! ! loseness or accuracy can be defined. Ideally, an estimation system should produce: 1) low average error, 2) low average relative error, 3) few extreme relative errors, 4) absence of bias for subgroups.

The first criterion "low average error" is perhaps the easiest to measure, as it refers to the arithmetic mean of the percent difference (MALPE), regardless of the sign. The fourth measure, bias, is measured as the direction and quantity of error in subgroups of estimates relative to each other. In numerous rounds of U.S. county and subcounty population estimates, tests have well documented that significant biases exist (Panel on Small-Area Estimates of Population and Income 1980:Chapter 1, Davis 1994).

The second criterion, "low average relative error" and the third criterion, "few extreme relative errors" are significantly more difficult to measure. The definition of "relative" is subjective and can be measured by any number of scales and by countless parameters. Traditional methods of measuring relative error group estimates by population size. However, this does not provide the analyst an opportunity to evaluate one entire series of estimates against another. For example, if the estimated populations in a series range from 10 to 1,000,000, then it is anticipated that large percent errors would exist for small areas, and small percent errors would exist for large areas. By grouping the estimates into population ranges, the relative and extreme errors within ranges may be evaluated, but evaluation of the estimate series as a whole is still problematic. The resulting average errors of a whole series are not only biased by the distribution of the population! ! ! values in a given set of estimates, but also lack consideration for the magnitude of the absolute errors existing in places with large populations. As with any evaluative system focusing on relative errors, the effect of outliers must be considered. An outlier can be defined as "an observation which deviates so much from other observations as to arouse suspicions that it was generated by a different mechanism." (Hawkins 1980) Barnett and Lewis (1994:1) state that there are essentially three ways to contend with outliers: reject them (with the loss of genuine information); accept them (with the risk of contamination); or utilize robust methods of inference which employ all data but minimize the influence of outliers.

In developing an evaluation of the Census Bureau estimate procedures, it is assumed that each system is reliable and valid in its supporting data and results. In addition to the conventional measures of evaluating percent and numeric differences between estimates, a robust and versatile method known as the loss function has been implemented by the Census Bureau for evaluating large series of estimates.

The use of loss functions and optimization criteria is a time-efficient approach to quantifying the low average relative error and few extreme relative error measures in evaluating estimates. A total Loss Function (1) creates a value representing a combination of weighted absolute and weighted percent difference between known and ex-post facto estimate values.

The total loss function may be used to evaluate different series of ex-post facto estimates, {e_i}, against known population figures, {k_i}, where i indexes the n areas. An important feature underlying loss functions is that relative comparisons of differences can be made across all sizes of data values. The arguments of may be given by the Loss Function (2):

where L(e_i,k_i) is the value of the loss function area i. Abs(e_i,k_i) is the absolute difference between e_i and k_i and a is its weight. P(e_i,k_i) is the percent difference between e_i and k_i and b is its weight. When this function is applied, symptomatic values of relative difference are created. In addressing criterion two, "low average relative error", the average values of L for a series may derived to create. In addressing criterion three, "few extreme relative errors", individual values of L may be evaluated within series to detect outliers.

In establishing relative weights for the absolute and percent values, it is intuitive that the greater the range of values in a known data series, the greater the relative impact of absolute differences and the smaller the relative impact of percent differences when compared to a test estimate. Smaller ranges of data are naturally influenced more by percent differences than by absolute differences. The influence of the percent or absolute difference on the resulting loss function value is determined by the weights that are applied to each. Greater weight results in a greater influence on the result. The robust nature of the loss function formula allows the analyst to develop "optimization criteria" to define the relative impact of either the absolute or percent error on L. If estimates are to be used for distribution of funds, and the minimization of misallocation is sought, a greater weight may be applied to the percent error. If estimates are to be ! ! ! used by planners or decision makers for the evaluation of infrastructure needs, then a greater weight may be applied to the absolute error.

While weights may be subjectively applied in this manner, it is recommended that a preliminary index be developed, whereby weights are determined by the range of the values in the data set, with any increase in weight of the absolute differences between the {e_i} and {k_i} be offset with a corresponding decrease in weight of the percent differences. An example follows. Extensive testing of preliminary estimates data in the PEB evaluation program has shown the weights as generated by Functions (3a) and (3b). Function (3a) shows a fraction a between 0 and 1, which is used to weight the absolute difference. Function (3b) shows b=1-a, the corresponding weight of the percent change. These are used to create L values, which are useful for detection of outliers and to generate for evaluation of multiple estimate series.

Using the loss function formulas, multiple sets of estimates, {e_i¹}_,...,{e_i^m} may be compared against the known populations, {k_i}. In Table 1, two alternate sets of potential estimates are compared. The range of the population values is very similar (1,539,050 for e_i¹ and 1,528,900 for e_i²) the results of Functions (3a) and (3b) are identical to the third digit. The derived weight for the absolute difference of both are .57, with the resulting weight for the percent difference being .43.

Table 1. Example of Loss Function as Evaluative Measure for Different Estimates

It can be seen in Table 1 that there are important similarities and differences between the two sets of estimates. While both create net overestimates, their sums are nearly identical. In viewing the sums of the differences, it would appear that with a total of 3,250 persons closer to k_i, and a smaller MAPE, e_i¹ is a better set of estimates. While both the absolute differences and the MAPE at this point are useful information for making a determination of which series is a superior estimate, further "relative" information from the application of the loss function may be used to satisfy criteria two and three.

While having the second to smallest percent error at 2.7%, the relative impact of a 40,000 person difference clearly is the most substantial relative error in e_i¹. The percent absolute difference between e₃¹ and e₃² for Area 3 is negligible at .7%, but compounded by the absolute difference of 10,000 persons creates a substantially higher loss function value for e₃¹. The resulting loss function value of 88 contributes significantly to the average loss function value of 31. Thus valuable information on average relative error and extreme relative errors is gained, which may give us further insight to which estimate system is superior.

When compared to alternate measures of relative error, such as weighted averages, the loss function provides two distinct advantages. First, the loss function transformation combines the magnitude of the absolute change with a weighted percent change to guide the result upwards or downwards. Variations of weighting schemes, such as weighted MAPE, universally lower average estimates of error (Galdi 1985:6,10), making comparisons relative to size very difficult. Second, loss functions directly consider the magnitude of the change in relation to the individual estimates, not the sum of a series of estimates. Weighted averages and other evaluative measures often become less meaningful when there are numerous population estimate values which contribute to the total population. In cases where thousands of estimates are combined for a population total, the weight of large contributing values diminishes relative to small contributing values.

The loss function has some drawbacks. It is not well known nor has it been widely used outside of statistics and operational research, and is not an easy and intuitive way for novices to judge and understand the extent of error. It lacks explanatory power, in that the individual results of a loss function as well as statistics calculated upon them are essentially meaningless except in the context of the evaluation being performed. Insofar as it is a "relative measure" it does not give a good indication of a set of estimates relative to series for other time periods or pieces of geography. However, it is emphasized that the loss function be used as a contributive measure in an overall process of data evaluation. The loss function is designed as a simple method of contributing to a better understanding of two of four defined measures of accuracy.

Future research on the loss function should focus on two areas. First, empirical rules of weighting need to be developed and classified based on the optimization criteria being defined. Second, comparisons with the results of MAPE variations, such as MAPE-T and MAPE-R (Swanson, 1999); and other similar, but more complex measures of error such as M-Estimators (Hoaglin et al. 1983) may yield further insight to the reliability of the evaluative qualities of the loss function.

The creation of population estimates for the entire United States is a difficult task. By establishing the optimization criterion of the loss function, evaluative measures can be developed to select the best series of estimates and detect potential outliers within estimate series. It is emphasized that optimization criteria are meaningful insofar as they represent the desires of the producers of the estimates for different kinds of accuracy. Explicit formulations of optimization criteria are useful for representing preferences for trade-offs in accuracy. (Panel on Small-Area Estimates 1980:89) The loss function presented is thus a useful evaluative supplement to conventional statistical measures.

Barnett, Vic. And Toby Lewis. 1994. Outliers in Statistical Data. Wiley and Sons Ltd. West Sussex, England.

Davis, Sam T. III. 1994. "Evaluation of Postcensal County Estimates in the 1980's". Population Division Working Paper No. 5. Washington DC: U.S. Bureau of the Census.

Galdi, David. 1985. "Evaluation of 1980 Subcounty Population Estimates". Current Population Reports, Series P-25, No. 963. U.S. Bureau of the Census.

Hoaglin, David C., Frederick Mosteller and John Tukey. 1983. Understanding Robust and Exploratory Data Analysis. New York, NY: John Wiley and Sons.

Hawkins, D.M. 1980. Identification of Outliers, 1. Chapman and Hall. New York, NY.

Long, John. 1993. "Postcensal Population Estimates: States, Counties and Places". Technical Working Paper No. 3. Washington DC: U.S. Bureau of the Census.

Martin, Julia H. and William Serow. 1979. "Conflict and Cooperation in Producing Population Estimates." State Government Volume 52 Number 4 Autumn.

Panel on Small-Area Estimates of Population and Income. 1980. Estimating Population and Income of Small Areas. National Academy Press. Washington, D.C.

Swanson, David, Jeff Tayman and Charles Barr. 1999. "On the Measurement of Accuracy for Subnational Demographic Estimates" Paper presented at the U.S. Bureau of the Census Population Estimates Conference June 8, 1999.