Statistical Policy Working Paper 3 - An Error Profile: Employment As Measured By The Current Population Survey

Federal Committee on Statistical Methodology
Office of Management and Budget

Statistical Policy Working Paper 3 - An Error Profile: Employment As Measured By The Current Population Survey

Statistical Policy Working Papers are a series of technical documents prepared under the auspices of the Office of Federal Statistical Policy and Standards. These documents are the product of working groups or task forces, as noted in the Preface to each report.

These Statistical Policy Working Papers are published for the purpose of encouraging further discussion of the technical issues and to stimulate' policy actions which flow from the technical findings. Readers of Statistical Policy Working Papers are encouraged to communicate directly with the Office of Federal Statistical Policy and Standards with additional views, suggestions, or technical concerns.

Office of Joseph W. Duncan Federal Statistical Director Policy and Standards Statistical, Policy Working Paper 3 An Error Profile: Employment As Measured By The Current Population Survey Prepared by Camilla A. Brooks and Barbara A. Bailar, U.S. Bureau of the Census Subcommittee on Nonsampling Errors Federal Committee on Statistical Methodology U.S. DEPARTMENT OF COMMERCE Juanita M. Kreps, Secretary Courtenay M. Slater, Chief Economist Office of Federal Statistical Policy and Standards Joseph W. Duncan, Director Issued: September 1978 Office of Federal Statistical Policy and Standards Joseph W. Duncan, Director George E. Hall, Deputy Director, Social Statistics Gaylord E. Worden, Deputy Director, Economic Statistics Maria E. Gonzalez, Chairperson, Federal Committee on Statistical Methodology Preface This working paper was prepared by Camilla A. Brooks and Barbara A. Bailar, U.S. Bureau of the Census, members of the Subcommittee on Nonsampling Errors, Federal Committee on Statistical Methodology. The Subcommittee was, chaired by Monroe G. Sirken, National Center for Health Statistics, Department of Health, Education and Welfare. Members of the Subcommittee are listed on the following page. The Subcommittee considered various measures of the errors in survey results. One of the alternatives is the preparation of error profiles, that is, a systematic and comprehensive account of survey - operations that yield survey results. The errors in those results are discussed in the error profile. To illustrate the error profile approach, the Subcommittee decided to prepare such a profile for employment statistics based on the Bureau of the Census' Current Population Survey. We expect that the error profile approach will prove useful to both users and producers of statistics. Thus, it should help to enhance the users' appreciation of the limitations of the statistics. In addition, an error profile may guide the producers in their, efforts to identify those survey Operations which need to be re- designed and/or controlled better in order to improve the quality of the survey results. We hope that in the future error profiles will be prepared for other important Federal statistical series. iii Subcommittee on Nonsampling Error Monroe Sirken (Chair) National Center for Health Statistics Barbara Bailar Bureau of the Census Camilla Brooks Bureau of the Census John Cremeans Bureau of Economic Analysis Tore Dalenius (Consultant) Brown University and Stockholm University Richard Deighton U.S. Postal Service James Duffet U.S. Postal Service Maria Gonzalez (ex officio) Office of Federal Statistical Policy and Standards Tom Herzog Social Security Administration Alexander Korns Bureau of Economic Analysis Lillian Madow Bureau of Labor Statistics D. H. McElhone Civil Service Commission Ron Poland U.S. Postal Service Morton Raff Bureau of Labor Statistics Jack Scharff Health Care Finance Administrator Frederick Scheuren Social Security Administration Otto Schwartz Internal Revenue Service Gary Shapiro Bureau of the Census Joel Stubbs Internal Revenue Service Robert Tortora, U.S. Department of Agriculture Rolf Wulfsberg National Center for Education Statistics Acknowledgements This paper was written as a result of the work of the Subcommittee on Nonsampling Errors of the Federal Committee on Statistical Methodology. The authors wish to thank both the members of the Subcommittee and the full Committee for their suggestions, comments and support in writing this paper. However, it represents the compilation of a great deal of work by Bureau of the Census staff members. Many of these persons are listed in the references and are referenced in the text. The authors acknowledge their vast contributions to this paper. Though many persons who helped make this paper possible are listed in the references, not a persons who work with the Current Population Survey (CPS) author papers or memoranda. We have, listed below persons who are involved in the various survey operations and we express our appreciation to them. Several,persons listed also gave the authors helpful comments and suggestions, made available requested information, and suggested sources of additional information. Thomas Plewes of the Bureau of Labor Statistics and Estela Dagum of Statistics Canada were the major authors of Section V.B. on "Seasonal Adjustment". Specification of Definitions Earle Gerson, Stanley Greene, Gregory Russell, and Marvin Thompson of the Demographic Surveys Division; Arthur Young and Aneda France of Housing Division; Meyer Zitter, Murray Weitzman, Charles Johnson, Roger Herriot, and Arno Winard iv of Population Division; Charles Jones, Gary Shapiro, Maurice Kniceley, and Irene Montie of Statistical Methods Division. Sampling Design Gary Shapiro, Maurice Kniceley, Margaret Schooley, Christina Gibson, Wayne St. John, Charles Jones, David Diskin, Leonard Baer, Christine Jorgenson, Jerome Roth, and Peter Bounpane of Statistical Methods Division. Data Collection Procedure Gregory Russell, Ronald Dopkowski, George Gray, and Marvin Thompson of the Demographic Surveys Division; Marvin Postma, Richard Bitzer, Jon Spendlove, William Clark, and Lincoln Steigerwalt of Field Division. Questionnaire Design Gregory Russell, Ronald Dopkowski, Paul Michalak, and Ronald Taylor of Demographic Surveys Division; Erne Wilkins of Engineering Division; and Irene Montie, John Paletta, and Tacho Aguilar of Statistical Methods Division. Data Collection Staff-Regional Offices Joseph R. Norwood, Charlotte, North Carolina; Robert G. McWilliam, Detroit, Michigan; Rex L. Pullin, Kansas City, Missouri; John E. Tharaldson, Seattle, Washington; Richard C. Burt, Denver, Colorado; C. Michael Long, Los Angeles, California; Porter S. Rickley, Philadelphia, Pennsylvania; Percy R. Millard, Dallas, Texas; Stanley Moore, Chicago, Illinois; John C. Cullinane, New York, New York; Arthur G. Dukakis, Boston, Massachusetts; Thomas W. McWhirter, Atlanta, Georgia; and the 1300 interviewers and 125 Supervisory Field Representatives throughout the country. Interviewer Training Paul Michalak, Ronald Taylor, Kenneth Riccini, Kathryn Creighton, and Gregory Weyland of Demographic Surveys Division; Lincoln Steigerwalt, Lynn Minneman, Sally Spurlock, and John Mahan of Field Division; Irene Montie, John Paletta, and Tacho Aguilar of Statistical Methods Division. Quality Control of Field Work Leo Schilling, Lincoln Steigerwalt, Marvin Postma, William Clark, Richard Bitzer, and Jon Spendlove of Field Division; Irene Montie, John Paletta, Tacho Aguilar, John Linebarger, Irwin Schreiner, Douglas Moye, and Dorcas Graham of Statistical Methods Division. Data Input Operations Eleanor Cooper, Ralph Hughes, Harley Bean, and Lowell Wrucke of the Demographic Surveys Division; Erne Wilkins of Engineering Division; and Kathern Clay, Patricia Clark, and Harold Garwood of the Data Preparation Division in Jeffersonville, Indiana. Cleaning, Editing, Imputation Gregory Russell, Lowell Wrucke, Carl Jablin, Maxine Perry, and Philip Hampton of the Demographic Surveys Division; Gary Shapiro, Maurice Kniceley, Margaret Schooley, and Christina Gibson of Statistical Methods of Division. Weighting Procedures Gregory Russell and Lowell Wrucke of Demographic Surveys Division; Gary Shapiro, Maurice Kniceley, Margaret Schooley, Christina Gibson, Leonard Baer, Carrie Pasewark, and Pearl Deimel of Statistical Methods Division. Estimation Procedure Charles Jones, Gary Shapiro, Maurice Kniceley, Robert Jewett, George Train, and Larry Cahoon of Statistical Methods Division; Thomas Plewes of the Office of Current Employment Analysis of the Bureau of Labor Statistics; and Estela Dagum of Statistics Canada. Analysis and Publication Diana Harley of Statistical Methods Division; John Bregger of the Division of Employment and Unemployment Analysis at the Bureau of Labor Statistics and members of his staff. The authors also thank other persons not directly involved with the CPS, but who contributed to their, understanding of various aspects of the CPS and gave useful comments on the paper. These persons are Nash Monsour of Business Division; Carol v Corby and Leroy Bailey of the Research Center for Measurement Methods; Dennis Schwanz, David Bateman, William MacKenzie, Mason Malmuth, James Dinwiddie, and Charles Edwards of Statistical Methods Division; Donny Rothwell, Anitra Rustemeyer of the Statistical Research Division and Harold Nisselson, Associate Director for Statistical Standards and Methodology. In addition, the authors would like to express their gratitude to Christine Walker, Myrtle Rice, and Karen Scott of the Research Center for Measurement Methods (RCMM) for their help in the preparation of tables for this report; and to Patricia Petrick and Mary Jane St. John, also of RCMM, for the typing, reproduction, and distribution of this paper throughout the many phases of its development. The authors extend special thanks to Robert Hanson, formerly of the Statistical Research Division, who was kind enough to allow them the use of a draft copy of the revision of Technical Paper No. 7 which is now published as Technical Paper No. 40 under the title The Current Population Survey: Design and Methodology. A considerable part of the survey description was taken from this work, and it served as a guide to the CPS survey operations throughout the compilation of this paper. Indeed, without it this paper would have taken a great deal more time and would probably have suffered in both accuracy and comprehension. vi Authors' Preface The Subcommittee on Nonsampling Error of the Federal Committee on Statistical Methodology decided to illustrate the ways in which nonsampling error could affect survey statistics by constructing "error profiles". An error profile has as its objective the listing of the survey operations with the investigation of the potential sources of error for each operation. Ideally, the impact of the errors on the survey statistics would be presented. The ideal is rarely possible because the measurement of nonsampling errors is rarely undertaken. This error profile describes the potential sources of error in the Current Population Survey (CPS) as they affect the national employment statistics. The sample design considered in the profile does not include the supplementation for improved state estimates that will be used in obtaining the national estimates of employment as of January 1978. The purposes of this document are as follows: 1. To illustrate how an error profile is-created in an effort to encourage government statisticians to provide error profiles . for the major recurrent survey statistics; 2. To compile in a single document the sources of error and the information that is available about these sources of error and their impact; 3. To illustrate the need for controlled experiments to measure nonsampling errors because of the lack of knowledge of the impact of these errors; 4. To stimulate development of a mathematical model that will reflect the ways in which the errors from different sources interact. The Current Population Survey was selected to illustrate the creation of an error profile for many reasons. It is a survey with a long history, and subsequently much has been written about it. Additionally, a considerable amount of research has been done on the survey methods used. Finally it is noteworthy as a survey that produces data on important statistics. We have chosen to focus on "employment" to narrow the scope of the profile. Though we have tried to follow through the complete survey process, and find all relevant information either through written memoranda or discussions with persons responsible for certain operations, there are undoubtedly gaps 'in our information. A sincere vote of thanks goes to all those who helped us compile the information for this profile. Any errors in interpretation or any gaps in the reports are our responsibility. vii Members of the Federal Committee on Statistical Methodology, (September 1977) Maria Elena Gonzalez (Chair) Office of Federal Statistical Policy and Standards (Commerce) Barbara A. Bailar Bureau of the Census (Commerce) Norman D. Beller Economics, Statistics, and Cooperatives Service (Agriculture) Barbara A. Boyes Bureau of Labor Statistics (Labor) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Economic Analysis (Commerce) Marie D. Eldridge National Center for Education Statistics (HEW) Fred Frishman Internal Revenue Service (Treasury) Thomas B. Jabine Social Security Administration (HEW) Charles D. Jones Bureau of the Census (Commerce) Alfred D. McKeon Bureau of Labor Statistics (Labor) Harold Nisselson Bureau of the Census (Commerce) Monroe G. Sirken National Center for Health Statistics (HEW) Wray Smith Office of the Assistant Secretary for Planning and Evaluation (HEW) viii Table of Contents Page Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . .iv Authors' Preface . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Figures. . . . . . . . . . . . . . . . . . . . . . . . . . xii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Key Abbreviations. . . . . . . . . . . . . . . . . . . . . . . . .xvii Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . xviii Chapter I. Introduction. . . . . . . . . . . . . . . . . . . . . 1 A. Objectives of the Survey . . . . . . . . . . . . . 1 B. Specifications of the Survey . . . . . . . . . . . 1 Chapter II. Sampling Design and Implementation. . . . . . . . . . 3 A. The Frame. . . . . . . . . . . . . . . . . . . . . 3 The Basic Frame . . . . . . . . . . . . . . . . . 3 Limitations of the Frame . . . . . . . . . . . . . 4 1. Structure Undercoverage/Overcoverage. . . . . 4 a. Permit lag universe. . . . . . . . . . . 4 b. Time lag between permit issuance and chance of entering sample . . . . . . . . . 4 c. Nonpermit issuing TAR ED's . . . . . . . 4 d. Incompleteness of permit universe. . . . 4 e. Undercoverage of special places-mobile homes . . . . . . . . . . . . . . . . . . . 4 f. Other structure misses . . . . . . . . . 5 g. Structure overcoverage . . . . . . . . . 5 2. Within Household Misses . . . . . . . . . . . 5 3. The Effect of the CPS Undercoverage on the employment Statistics . . . . . . . . . . . . . . . . 5 Attempts to Strengthen the Frame . . . . . . . . . . . . . 7 B. Procedure for Sample Selection . . . . . . . . . . . . . . 7 The Sample Selection Procedure . . . . . . . . . . . . . . 7 1. The Selection of the Primary Sample Units (PSU's) . . . 7 2. Selection of Sample Households. . . . . . . . . . . . . 8 3. Rotation of the Sample. . . . . . . . . . . . . . . . . 9 4. Replacement of PSU's. . . . . . . . . . . . . . . . . . 9 Potential Sources of Error in CPS Sampling Procedure . . . 9 C. Quality Control of Sample Procedure. . . . . . . . . . . .11 Selection of Sample PSU's. . . . . . . . . . . . . . . . .11 Within PSU Sampling Process. . . . . . . . . . . . . . . .11 Chapter III. Observational Design and Implementation . . . . . . . .13 A. Basic Data Collection Procedure . . . . . . . . . . . .13 Listing and Sampling. . . . . . . . . . . . . . . . .13 ix Page 1. Listing, in Address, Permit, and Cen-Sup Segments or Groups of USU's. . . . . . . . . . . . . . . . . . . . . .13 2. Listing in Area Segments. . . . . . . . . . . . . . . .13 3. Listing in Special Place Segments . . . . . . . . . . .14 4. Updating. . . . . . . . . . . . . . . . . . . . . . . .14 Conducting the Interview. . . . . . . . . . . . . . . . . . . .14 1. General Procedure . . . . . . . . . . . . . . . . . . .14 2. Mode of Interview . . . . . . . . . . . . . . . . . . .15 3. Noninterviews . . . . . . . . . . . . . . . . . . . . .15 Potential Sources of Error in the CPS Data Collection Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . .15 1. Listing by Observation in Area Segments . . . . . . . .15 2. Problems with the Year Built Procedure. . . . . . . . .16 3. Determination of Race of Household Head for Type A Noninterview Units. . . . . . . . . . . . . . . . .16 4. Effect of Mode of Interviewing on the Employ- ment Estimate. . . . . . . . . . . . . . . . . . . . . . .16 5. The Effect of Proxy Respondents on the Data . . . . . .17 B. Questionnaire Design and Instructions. . . . . . . . . . .18 The Questionnaire and Interviewer Instructions . . . . . .18 Potential Sources of Error in the Questionnaire Design . .20 C. Data Collection Staff. . . . . . . . . . . . . . . . . . .21 Organization and Description of the Data Collection Staff. . . . . . . . . . . . . . . . . . . . . . . . . . .21 Potential Sources of Error Associated With Interviewers . . . . . . . . . . . . . . . . . . . . . . .22 D. Training of the CPS Interviewers . . . . . . . . . . . . .28 The Training Procedure . . . . . . . . . . . . . . . . . .28 1. Initial Training. . . . . . . . . . . . . . . . . . . .28 2. Refresher Training. . . . . . . . . . . . . . . . . . .28 3. Special Training. . . . . . . . . . . . . . . . . . . .28 4. Determining when an Interviewer is Trained. . . . . . .28 Limitations in the Training Procedure. . . . . . . . . . .28 E. Development of Quality Control Design. . . . . . . . . . .29 Coverage Check of Sampling Units . . . . . . . . . . . . .30 Questionnaire Checks . . . . . . . . . . . . . . . . . . .30 Interviewer Observation. . . . . . . . . . . . . . . . . .30 Reinterview. . . . . . . . . . . . . . . . . . . . . . . .30 1. The Basic Procedure . . . . . . . . . . . . . . . . . .30 2. Reconciliation. . . . . . . . . . . . . . . . . . . . .34 3. Results from Reinterview. . . . . . . . . . . . . . . .31 Limitations of the Quality Control Procedure. . . . . . . . . .32 1. Quality Control of Telephone Interviewing . . . . . . .32 2. The Reinterview Procedure . . . . . . . . . . . . . . .32 a.Independence. . . . . . . . . . . . . . . . . . . .33 b. Reinterview nonresponse rate . . . . . . . . . . .33 c. Coverage check . . . . . . . . . . . . . . . . . .33 d. 80 Percent vs. 20 ]Percent Sample. . . . . . . . .34 Chapter IV. Data Processing. . . . . . . . . . . . . . . . . . . . .37 A. Mcrofilming/FOSDIC . . . . . . . . . . . . . . . . . . . . .37 The FOSDIC/Mcrofilming Procedure. . . . . . . . . . . . . . . .37 x Page Potential Sources of Errors in the FOSDIC/Microfilming, Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . .37 1. Results from the Data Acceptance Runs . . . . . . . . .39 2. Results of Test Run of Blank CPS Questionnaires . . . .39 3. Variations in FOSDIC Scanning of Monthly Quality Control Samples. . . . . . . . . . . . . . . . . . . . . . . . . 4.1 4. CPS/FOSDIC Study. . . . . . . . . . . . . . . . . . . .41 B. Editing and Imputation . . . . . . . . . . . . . . . . . . .46 Procedure for Editing and Imputation for Missing Values . . . .46 Potential Sources of Errors Arising from the Editing and Imputation Procedure. . . . . . . . . . . . . . . . . . . . . .48 C. Quality Control. . . . . . . . . . . . . . . . . . . . . . .48 Quality Control of Microfilming/FOSDIC Procedure. . . . . . . .48 1 . Design of the Questionnaire. . . . . . . . . . . . . .50 2. Selection of Paper. . . . . . . . . . . . . . . . . . .50 3. Printing. . . . . . . . . . . . . . . . . . . . . . . .50 4. FOSDIC Scan . . . . . . . . . . . . . . . . . . . . . .51 5. Cameras . . . . . . . . . . . . . . . . . . . . . . . .51 6. Film. . . . . . . . . . . . . . . . . . . . . . . . . .51 7. Production Scanning (FOSDIC). . . . . . . . . . . . . .51 Quality Control of Editing and Imputation . . . . . . . . . . .51 Chapter V. Estimation. . . . . . . . . . . . . . . . . . . . . .53 A. Weighting Procedure. . . . . . . . . . . . . . . . . . . . .53 Specification of Weighting Procedure . . . . . . . . . . .53 Potential Sources of Error Caused by the Weighting Procedure. . . . . . . . . . . . . . . . . . . . . . . . .57 B. Specification of Estimation Procedure. . . . . . . . . . . .59 Composite Estimation . . . . . . . . . . . . . . . . . . .59 1. Description. . . . . . . . . . . . . . . . . . . .59 2. Potential Sources of Error in the Composite Estimation Procedure . . . . . . . . . . . . . .61 Seasonal Adjustment. . . . . . . . . . . . . . . . . . . .65 1. Description. . . . . . . . . . . . . . . . . . . .65 2. Source of error in Seasonal Adjustment . . . . . .66 a. Model Selection . . . . . . . . . . . . . . .67 b. Trend-Cycle Moving Average. . . . . . . . . .67 c. Final Seasonal Factor Moving Average Curve . . . . . . . . . . . . . . . . . . . .67 d. Forecasting Seasonal Factors for the Current Period . . . . . . . . . . . . . . . . . . . . .67 3. Estimating the Error in Seasonal Adjustment. . . .68 C. Specification of Estimation Procedures for Errors. . . . . .69 Estimation of Errors . . . . . . . . . . . . . . . . . . .69 1. The CPS Variance Estimation Procedure. . . . . . .69 2. Generalizing Estimated Sampling Errors . . . . . .70 3. Simple Response Variance and Response Bias . . . .70 Potential Problems in the Estimation of Errors. . . .72 D. Quality Control of the Estimation Procedure. . . . . . . . .76 Quality Control of the Weighting and Estimation Procedure . . . . . . . . . . . . . . . . . . . . . .76 xi Page Quality Control of the Keyfitz Estimate and Variance Program . . . . . . . . . . . . . . . . . . . . . . .76 Chapter VI. Analysis and Publication . . . . . . . . . . . . . . . .77 A. Description . . . . . . . . . . . . . . . . . . . . . .77 B. Potential Sources of Error in Analysis and Publication.77 Chapter VII. Conclusion . . . . . . . . . . . . . . . . . . . . . .79 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81 xii List of Figures Figure Page 1. CPS Rotation Chart. . . . . . . . . . . . . . . . . . . . . . . .10 2. Excerpt from CPS Questionnaire. . . . . . . . . . . . . . . . . .19 3. Excerpt from CPS Questionnaire Showing Index Marks and Marking Circles . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 4. Control Chart for Percentage of Film Associated Pre-Computer Edit Rejects (Computed from January 1974-December 1975 Rejects). . . . . . . . . . . . . . . . . . . . . . . . . . . .40 5. Control Chart for Percentage of Spurious Pickups Per Total Items Tested Prior to Processing (January 1976-December 1976) .43 6. Control Chart for Percentage of Spurious Pickups Per Total Items Tested during Processing (January 1976-December 1976) . . . . .44 7. CPS Noninterview Adjustment Cells . . . . . . . . . . . . . . . .54 8. Noninterview Adjustment in Mixed-Race Households. . . . . . . . .58 xiii List of Tables Table Page 1. CPS Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Labor Force Estimates Adjusted for CPS-Census Undercoverage Compared to March 1973 Estimates at Two Stages of Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Persons Employed as Percent of Labor Force for Two Stages of March 1973 Estimation Compared to Percentage Adjusted for Census-CPS Undercount. . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4. Effect of Omitted Persons on Labor Force Estimates, Comparability and Poverty Neighborhood Assumptions, 1967 Monthly Averages . . 7 5. Percent Telephone Interviews by Month in Sample-1976 Average. . .17 6. Percent Telephone Interviews by Month in Sample for Employed Persons-December 1973 . . . . . . . . . . . . . . . . . . . . .17 7. Total Employed as Measured by Three Procedures in Methods Test. .18 8. Percent Difference in Total Employed as Measured by Self and Proxy Respondent Procedure. . . . . . . . . . . . . . . . . . . . . .18 9. Frequency of Rewording of Labor Force Questions . . . . . . . . .20 10. Rotation Group Indices for Employment Items for Two Time Periods, 1968-69 (TI) and 1970-72 (T2). . . . . . . . . . . . . . . . . . . .21 11. Summary of Characteristics of CPS Interviewers, March 1975 . . .22 12. Number and Distribution of Total, Noninterview and Interview Units in A and C Design CPS Sample. . . . . . . . . . . . . . .23 13. Distribution of CPS Interviewer Type A Noninter-view Rates, Average July 1974-June 1975 . . . . . . . . . . . . . . . . . .23 14. Distribution of CPS Interviewer Questionnaire Edit Error Rates, Monthly Average July 1974 Through June 1975 . . . . . . . . . .24 15. Distribution of CPS Interviewer Production Ratios, Averages July 1974 Through June 1975. . . . . . . . . . . . . . . . . . . . .25 16. Distribution of Interviewers' Monthly Gross Coverage Error Rates in the Current Population Survey April 1973-December 1974. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 17. Distribution of Interviewers' Gross Content Error Rates in the Current Population Survey April 1973-December 1974. . . . . . .25 18. Responsibility for Differences Between Responses From CPS Original Interview and Reinterview: 1959 to 1966. . . . . . . .26 19. Estimates of the Average Ratio Over Six Months of the Correlated Component of Response Variance to the Sampling Variance for Employment Items, and the Estimated Standard Deviation of the Estimated Ratios . . . . . . . . . . . . . . .27 20. Tolerance Table of Acceptable Limits . . . . . . . . . . . . . .31 21. Summary of Percent of Persons Employed as Measured in the Original CPS Interview for the Reinterview Subsample and as Measured by the Reinterview After Reconciliation, 1956-1976 . .32 22. Annual Interviewer Error Rates . . . . . . . . . . . . . . . . .32 23. Noninterview Misclassification Rates. . . . . . . . . . . . . .32 24. Percent Net Change in Area and B Segments: April to September 1966, October 1966, November 1966 to May 1967, and June 1967. .33 25. Reinterview Classification of Units Originally Classified as Noninter-view . . . . . . . . . . . . . . . . . . . . . . . . .34 26. Percent Difference Between Reinterview and Original . . . . . .35 xiv Page 27. Distribution of Questionnaires Rejected by FOSDIC by Reason for Rejection, January 1976 to December 1976. . . . . . . . . .39 28. Number and Percentage of Spurious Pickups Detected on Print Sample Documents Tested on FOSDIC during CPS Processing (January 1976-December 1976). . . . . . . . . . . . . . . . . .42 29. Variation in the FOSDIC System . . . . . . . . . . . . . . . . .45 30. Some Results of the FOSDIC Experiment. . . . . . . . . . . . . .45 31. Measured Reflectance of Ten Questionnaire Positions. . . . . . .45 32. Results of Test on Opacity and Brightness of CPS Questionnaires. . . . . . . . . . . . . . . . . . . . . . . . .46 33. The Possible Item Combinations Used in Determining the Existence of A Legitimate Person's Record . . . . . . . . . . .47 34. Summary of Errors Due to Omissions or Inconsistencies by Item Number, February-December 1975. . . . . . . . . . . . . . . . .49 35. Average Frequency With Which A Value From the Same CPS Record is Used for Imputation for Item Nonresponse . . . . . . . . . .50 36. CPS First-Stage Ratio Adjustment Factors, November 1974-March 1975. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 37. Civilian Noninstitutional Population and Labor Force by Sex and Age, Using Old and New Method of Estimating Civilian Population; January 1974. . . . . . . . . . . . . . . . . . . .55 38. Civilian Noninstitutional Population and Labor Force for Negro and Other Races by Sex and Age Using Old and New Method of Estimating Civilian Population; January 1974 . . . . . . . .55 39. Employment Status by Sex and Age Using Old and New Method of Estimating Civilian Population, January 1974. . . . . . . . . .56 40. CPS Second Stage Ratio Adjustment Factors for Total Population by Age, Race and Sex. . . . . . . . . . . . . . . . . . . . . .57 41. Maximum, Minimum and Average Weights for Records in 13 Relationship Categories, March 1975 . . . . . . . . . . . . . .57 42. Mixed-Race Households in March 1975 CPS Sample . . . . . . . . .59 43. Separate Second Stage Ratio Adjustment Factors Applied to Persons of "Other Races" by Age' Sex, and Month in Sample, March 1975. . . . . . . . . . . . . . . . . . . . . . . . . . .60 44. 1975 Average of Unbiased, Ratio, and Composite Estimates of Selected Employment Items . . . . . . . . . . . . . . . . . . .61 45. Ratios of the Unbiased and Ratio Estimates to the Composite Estimate for Selected Employment Items. . . . . . . . . . . . .62 46. Rotation Group Bias Indexes in the Current Population Survey for Employment Status by Sex, Five-Year Averages, 1968-1972 . .63 47. Comparison of Estimated Mean Square Errors for Ratio and Composite Estimates for 1975. . . . . . . . . . . . . . . . . .64 48. Original, Current, and Historically Seasonally Adjusted Data for June,1974 . . . . . . . . . . . . . . . . . . . . . . . . . . .68 49. Components of Variance, CPS Composite Estimates of Level, Annual Average, 1975 . . . . . . . . . . . . . . . . . . . . . . . . .69 50. Variance of Composite Estimate of Level and Variance Ratio for Selected Estimators-Averages, 1975. . . . . . . . . . . . . . .70 51. Standard Errors for the CPS Estimates of Monthly Level- Employment and Unemployment . . . . . . . . . . . . . . . . . .71 52. Percent in Class (Reinterview Estimate After Reconciliation), Net Difference Rate (After Reconciliation) and the Index of Inconsistency (Before Reconciliation) by Quarter, 1974-1976 . .72 xv Page 53. REL-MSE of Estimates of Variance Based on Taylor Approximations. . . . . . . . . . . . . . . . . . . . . . . . .73 54. Serial.Correlation in Sampling Error . . . . . . . . . . . . . .73 55. Sampling Errors for CPS Seasonally Adjusted Series . . . . . . .73 56. Between PSU Variances for Selected Employment Items for 1975 . .75 xvi Key Abbreviations 1. AHS Annual Housing Survey 2. BLS Bureau of Labor Statistics 3. Cen-Sup sample Census Supplemental sample 4. CPS Current Population Survey 5. DPD Data Preparation Division 6. ED Enumeration District 7. FOSDIC Film Optical Sensing Device for Input to Computers 8. NSR nonself-representing 9. PSU primary sampling unit 10. SMSA standard metropolitan statistical area 11. SR self-representing 12. TAR tape address register 13. USU ultimate sampling unit xvii Executive Summary An objective of this error profile is to make a comprehensive list of the survey operations with the documentation of what is known about each survey operation as a potential source of nonsampling error. When possible, the magnitude of the error is given. When no data are available about the source of error and possible impact, this is also noted. Each stage of the survey from the choice of the sampling frame to the analysis of the data in a final publication is described. Associated with each stage of the survey is a description of the process and then a discussion of what is known about possible errors or biases arising from the process. The stages examined are as follows: 1. Sampling design a. frames b. sample selection c. quality control of sampling process 2. Observational design a. data collection procedure b. questionnaire design c. data collection staff d. interviewer training e. quality control of field work 3. Data preparation design a. data input operations b. cleaning, editing and imputation c. quality control of data processing 4. Production of estimates a. weighting procedure b. estimation procedure c. quality control of estimation procedure 5. Analysis and publication Certain areas of the survey operations have received a great deal of attention and much is known about the limitations inherent in these operations. One such area is the sample design and implementation. This area is discussed in the first section of the paper. The coverage bias resulting from the sampling frame not completely covering the entire universe of households is estimated to be of the order of 3 percent. However, in addition to missed households, there are missed persons within enumerated households. These missed persons are mo re likely to be young, male, and to be black or of races other than white. There is an adjustment procedure to take care of part of this bias. The full impact of the undercoverage in the 1970 census and the additional undercoverage in the CPS is not known. The next survey operation discussed is the actual fieldwork including the design of the data collection procedure, the design of the questionnaire, the data collection staff, the training program for interviewers, and the quality control program for the data collection. Several potential sources of error and their effects are discussed. For example, the effect of using proxy respondents in the collection of labor force data is discussed. Several years ago some experiments were conducted outside of the CPS to measure the effect of using proxy respondents. These measures are given, though they may not be strictly applicable to the CPS. Also, the impact of xviii interviewer variability is discussed, along with the limited amount of information available. A recent experiment conducted to evaluate the interviewer training procedures is described. As is true of almost every data collection process, there is the potential for several errors. However, the studies designed to measure the errors give fragmentary results. Although the fieldwork on this survey is probably studied more than any other government survey, the knowledge of the effects of nonsampling error arising from the data collection procedure is sketchy. Another major survey operation, that of data processing, includes the data input, and the cleaning, editing, and imputation procedures. In this area, extensive information is available about the data input process and an estimate of any errors of the process is made. The cleaning, editing, and imputation process is described. This process is intended to alleviate the impact of errors arising from the fieldwork. Therefore, one would not expect the biases arising from these errors to be additive with.the biases arising from fieldwork. Although these procedures may reduce the overall bias, they may also induce other biases. Some of the ways the cleaning, editing, and imputation impact on the final data are described, but no quantifiable information is available. Finally, the estimation procedure is discussed. Not only is the estimation procedure for employment reviewed, but also the estimation procedure for the variances of the employment statistics, and the estimation procedure for some of the nonsampling errors. Limited information is available on the impact of the estimation procedures. Nonsampling errors may also occur in the analysis and publication of the data. A brief discussion of this phase of the survey operation is included. Again, not much is known about the impact of errors in this procedure on the many uses made of these data. Though the Bureau makes frequent use of a mean-square error model to describe the combined effects of sampling and nonsampling errors on estimated means and totals, no research studies have been done that permit the estimation of the bias term. e do not know how the nonsampling errors arising from different sources fit together. In some cases, they are clearly additive; in other cases. they may not be. It is quite possible that some of the errors go in different directions so that some may have the effect of overstating employment while others have the effect of understating employment. Thus, at the end of the description of nonsampling errors, we are left with the important question: how do these errors interact and, what is the magnitude of the bias term in the mean-square error? This error profile, even with its limitations, can be used as a framework for a systematic approach to evaluate the different potential sources of error. It may also be used as an illustration of the need for controlled experiments to enable us to quantify the errors and learn more about their interaction. xix Introduction A. Objectives of the Survey Prior to the 1930's, there w ere no direct measures of the number of jobless persons. Because of the mass unemployment during the economic depression of the early 30's, the need for statistics became urgent, and widely conflicting estimates based on a variety of indirect techniques began to appear. As a result of dissatisfaction with these methods, experimentation with direct surveys of the population began. In March 1940, the Works Progress Administration initiated the Sample Survey of Unemployment on a monthly basis. This survey was the forerunner to the present program. The primary purpose of the Current Population Survey (CPS) is to obtain estimates on a monthly basis of employment, unemployment, and other characteristics of the general labor force, of the population as a whole, and of various subgroups of the population. This, report focuses on "employment". B. Specifications of the Survey The Current Population Survey is restricted to the civilian no- ninstitutional population age 14 and over. Although the official tabulations have been restricted since.1967 to data for persons 16 years of age and over, the CPS labor force questions, including those concerned with "employment", are asked of persons 14 years of age and over. The target population encompasses every person 14 years of age and over who is not institutionalized or in the military. The CPS utilizes a household sample which represents the universe of all households in the United States and also includes those persons living in nonhousehold units, such as dormitories, flophouses, budhouses, and the like. In using a household surveY, an implicit assumption is made that each person 14 years of age and over will be uniquely associated with either a household or one of the nonhousehold units mentioned. Because of the many uses of the employment data no single definition is appropriate for every situation. The criteria used in classifying persons as employed are as follows (see BLS Report No. 463-Series P-23): 1. All those people who worked, for at least one hour, as paid employees or in their own business, profession, or on their own farm; 2. All those people who worked 15 hours or more as unpaid workers in a family-operated enterprise; 3. All those persons who did not work but bad jobs or businesses from which they were temporarily absent because of illness, vacations, labor-management disputes or other reasons. No distinction is made whether or not they were paid by their employers for the time off and whether or not they were looking for other jobs. Employed persons, even if they hold more than one job, are counted only in the job at which they worked the greatest number of hours. Employed citizens of foreign countries, excluding those who live on the premises of an Embassy, are counted in the total. Excluded are persons whose only activity was work around their own homes or volunteer work. In addition to data on employment by age, sex, race, and marital status, the CPS provides data on many other characteristics. For example, separate data are provided for wage and salary workers, the self-employed, unpaid family workers, and total agricultural and nonagricultural employment. In this report, we shall limit the discussion to data on employment by age, sex, and race. We shall focus on estimates of the monthly level of employment. In,order to provide comparable data on a month-by-month basis, stringent guidelines must be followed. Although the survey provides a measure of employment for a given month, data are collected for a specific week. Each month, during the calendar week containing the 19th day, the CPS interviewers ask a series of standard questions on economic activity relating to the preceding week (the calendar week containing the 12th day of the month). All clerical and machine editing and computations are completed in the following two weeks, and the employment figures are released during the first week of the following month. Because of the importance of these estimates, the accuracy of the data must be 1 at a high level. The coefficient of variation on the level of employment has been approximately 0.2 percent for many years. The most pressing constraints on the survey are the timing and the need to present comparable statistics over time with great accuracy. 2 CHAPTER II Sampling Design and Implementation A. The Frame The Basic Frame The frame for this survey is derived from a variety of sources with the main source the 1970 Decennial Census. In the CPS, extensive use is made of the 229,000 enumeration districts (ED's) defined in ad- vance of the census; these are large geographic areas, each containing about 350 housing units on the average. There were three types of ED's in the census, each type identified by the manner of forming the address register. (An address register is a listing by address of each housing unit in the ED.) These are as follows: 1. Tape address register (TAR) ED's, approximately 95,000 in number, in which the address register was created from a computer tape copy of a commercial mailing list and cor- rected by the Post Office and enumerators. 2. Prelist ED's, approximately 26,000 in number, in which the address register was constructed by a listing procedure conducted in advance of the census and corrected by the Post Office and enumerators. 3. Conventional ED's, about 108,000 in number, in which the address register was prepared by the enumerator during the census enumeration. A 1970 census ED is referred to as an address or list ED if the conditions listed below are satisfied. 1. The ED is a TAR ED. 2. The ED is a prelist or conventional ED satisfying both "a" and "b". a. at least 90 percent of the 1970 census addresses within the ED are recorded with complete street name and house number; b. the ED is located in an area which issues building permits. In address ED's the CPS sample is selected from the census address registers' and the resulting sample is referred to as an address sample. All other 1970 census ED's are referred to as area ED's and the enumerator lists the structures in the sample area segments in these ED's about a month before the initial interview. The Census Supplemental sample, referred to as the Cen-Sup sample, is used to cover housing units in address ED's at addresses missed in the census or inadequately described in the address register. To obtain this sample, a sample of address ED's was selected, and a probability sample of blocks or areas within these ED's was selected and canvassed. The resulting addresses were then matched to the census. The complete addresses of unmatched units and those with inadequate addresses were stored in the computer and are sampled on a rotation basis for representation through the life of the current design. These units represent less than one percent of the CPS sample. The frame is further supplemented by the permit universe, consisting of jurisdictions where building permits were required as of January 1, 1970 and where suitable records are maintained. A self- weighting sample of permits issued as of January 1, 1970 or later is then selected from this universe on a periodic basis. Thus new construction is represented in address ED's and permit issuing area ED's by a sample of building permits from permit offices. In nonpermit issuing area ED's, new construction is covered by interviewer listing. The following table (ignoring frame deficiencies) shows the percent of the total CPS sample that comes from each part of the CPS frame as of 1976. Table 1. CPS Frame Percent of Part of Frame CPS Sample Census Address Sample including 2 percent special place1. . . . . . . . . . . . . . . . . . . . . . . . . .65 Area. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 Supplements to Census Permit Universe . . . . . . . . . . . . . . . . . . . . . . . .10 Cen-Sup Sample. . . . . . . . . . . . . . . . . . . . . . . . . 1 1 A special place ED is an ED consisting entirely of places such as transient hotels, boarding houses, and mobile home parks where the occupants have special living arrangement Special places are also within other ED's. 3 Limitations of the Frame It is known that the frame used for CPS sampling does not fully represent the target population. However, frame deficiencies (excluding within household coverage losses) represent less than 3 percent of the population, though it is concentrated in certain types of units. 1. Structure Undercoverage/Overcoverage a. Permit lag universe In ED's where building permits are issued, housing units completed after the census for which permits were issued before January 1, 1970 are not included in the CPS frame. These units are referred to as the permit lag universe. There is an estimated total of 598,000 units for which permits were issued prior to January 1, 1970 that were completed after April 1970 (MacKenzie, 1977). b. Time lag between permit issuance and chance of entering sample Because of sampling efficiency and data preparation time, there is at present a 5-month lag between the time a permit for new con- struction is issued and the time the unit has a chance of entering the CPS sample. Thus for a short period of time there are units in the new construction universe that may not be represented in the CPS sample. In a study by Linebarger (1975), it was estimated that ap- proximately 12 percent of the units for which building permits were issued were interviewable 4 months after date of issuance; however, this is a cumulative figure so most of these units would not have been interviewable for the entire 4 months. c. Nonpermit issuing TAR ED's A small number of the TAR ED's (approximately 47-50 or about 0.3 percent of all TAR ED's) are in nonpermit areas. Thus new construction which should be represented in the permit universe, is not represented in these ED's (Baer, 1973 and Boisen, 1971 d. Incompleteness of permit universe The permit universe used to select the sample of new construction units is not complete. One of the reasons for this incompleteness is illegal construction; i.e. construction for which the required building permits were not obtained. The undercoverage in the permit universe was estimated to be roughly 2.3 percent for 1976. e. Undercoverage of special places-mobile homes Mobile homes located in address segments are another potential source of coverage loss. Presently, in the CPS there is no general procedure for identifying or representing mobile homes in new mobile home parks, or new mobile homes at large in address ED's at addresses which were nonexistent in the 1970 Census; the permit universe includes regular housing units only. In addition to new mobile homes, the coverage problem of mobile homes in address ED's extends to those occupied at the time of the census but missed in the. census and to those vacant at the time of the census and not counted because of census rules. In area segment ED's in permit areas, new construction has a chance of selection from both the area segment and the permit uni- verse. Therefore, in these ED's new construction units are deleted from area segments by an unduplication procedure referred to as the "Year Built Procedure". However, mobile homes are not included in the permit universe. Therefore, special instructions are specified to include them in the area segment sample. A coverage problem occurs when CPS interviewers apply the "Year Built Procedure" to mobile homes. This can occur when a mobile home "looks like" a regular housing unit. The effect of this error has not been studied yet; however, it probably can be assumed to contribute only a small error to the CPS employment estimate. The coverage improvement program in the October 1976 Annual Housing Survey (AHS) located approximately 300,000 mobile homes previously missed for the period April 1970 October 1976. This improvement program has not yet been included in the CPS so these mobile homes which are eligible for inclusion in the CPS represented mobile homes missed by CPS (MacKenzie, 1977). Though concern is greater for mobile homes, other special places including transient hotels, boarding homes, etc. present some of the same problems as the mobile homes. 4 f. Other structure misses Other problems with the frame in address ED's that are not addressed by the -Census Supplemental sample include coverage of homes moved to a site that did not have an address in the 1970 Census and structures used for nonresidential purposes at the time of the Census and converted to residential use after the census. g. Structure overcoverage Coverage errors sometimes result in overcoverage also, but far less frequently. One example is a new structure built in the same location, with the same address as the old sample structure, and containing the same number of units as the old structure. If the interviewer fails to recognize that the unit is new and interviews at this address, then the units at this structure have two chances of coming into sample. Because it is a new structure, it is also represented in the permit universe in addition to being represented in the regular address sample. This, like many of the errors, occurs infrequently; in addition, the address is usually a one-unit structure rather than multi-unit, which further reduces the impact on the sample. 2. Within Household Misses Within household misses are believed to account for a large percentage of the total undercoverage; however, information on both the extent and the causes of this problem are limited. It is estimated that because of missed structures less than 3 percent of the target population is not included in the frame. Table 40, which shows the ratio of independent estimates of population to those of the Current Population estimates, indicates a coverage problem exceeding this amount. For white males and females the ratios are 1.049 and 1.023, respectively, while for males and females of black and other races the respective ratios are 1.155 and 1.075. (These ratios indicate an undercoverage of 4.9 percent, 2.3 percent, 15.5 percent, and 7.5 percent, respectively.) Further, the independent estimates of the population do not reflect the undercoverage of the census so the within household coverage problem is even greater than indicated by this analysis. Another study. indicated that an estimated 64 percent of blacks missed in the census and 42 percent of per- sons of white and other races missed in the census were missed within units enumerated as occupied in the census or in occupied units enumerated as vacant in the census (Jones and Blass, 1975). Though knowledge of the extent and causes of within household misses is admittedly very limited, there is some knowledge on the subject. Analysts of census undercounts and ethnographers cite that concealment and oversight are two reasons why respondents give incomplete, rosters to interviewers. Both reasons apply more to men than to women and more to poor persons than to nonpoor persons (Korns, 1977). Two small ethnographic studies conducted in the late 60's in (a) an Hispanic neighborhood and (b) a black and Hispanic neighborhood found that many of the households reported in surveys as female headed were actually male-headed. The survey respondents, generally- female, failed to report these men because of fear of loss of economic security. Further, illegal immigrants have strong incentives to conceal their presence in the household and persons loosely attached to households may be unintentionally left off household rosters by the respondents (Korns, 1977). In a Census Bureau study a sample of 710 young men 20-29, mostly black and in poor neighborhoods in an urban area were matched to the 1970 Census; the Bureau found that 23.5 percent of the men were missed or probably missed in the census. The report by Novoa (1971) explores the reasons for these misses which included both oversight and concealment. In addition to within household misses, persons with no attachment to any address present a coverage problem for CPS. 3. The Effect of the CPS Undercoverage on the Employment Statistics Each month CPS estimates of employment are weighted up to independent estimates of the population which are the result of census data carried forward to the present by considering births, deaths, etc. (see Section V.A.). In regard to CPS undercoverage, the accuracy of the CPS employment estimate is affected by the following problems: (a) the independent estimates of the population used to adjust the CPS estimates &f; employment are too low because they are not adjusted for the census undercount which varies by age-sex-race groups and because illegal aliens are not included in the count; 5 (b) the CPS sample misses people it is designed to cover (Korns, 1977). Korns (1977) has done extensive research into the effect of the CPS undercoverage on the employment estimate as part of his comparative study of the cyclical behavior of the CPS measure of employment and that of the Labor Department's payroll (or establishment) survey. He presents evidence where missed persons in enumerated households experience larger cyclical fluctuations in their employment ratios than covered persons of the same age, race, and sex; this suggests that missed persons can have a noticeable effect on employment estimates. A paper by Hirschberg, Yuskavage, and Scbeuren (1977) discusses the impact of alternative adjustment procedures on some selected labor force estimates obtained in the March 1973 CPS. Their three alternative survey estimates are presented in Tables 2 and 3: (a) initial-the March 1973 survey estimate before any adjustment for coverage; (b) standard-the published March 1973 estimate which has been weighted up to the independent estimates of the population; and (c) extended-the survey estimate which has been adjusted to correct for the March 1973 undercoverage. The adjustment procedure makes use of both demographic analysis and administrative records. Under the assumption that the contentions advanced by Hirschberg, et al, are plausible, Table 2 would indicate that the March 1973 published estimate (standard) of total employed persons was understated by 2.5 million. It also shows that the published estimate accounted for only 50 percent of the actual CPS undercoverage. (The difference between the unadjusted initial estimate and the extended estimate was 5.0 million.) Table 3 shows the effect of population undercoverage on the percent of persons employed for total persons and by sex and race. The published estimate of the percent of total persons employed is overestimated by 0.4 percent; Table 2. Labor Force Estimates Adjusted for CPS-Census Undercoverage Compared to March 1973 Estimates at Two Stages of Estimation1 (in millions) Persons Aged 16 Initial Standard Extended2 and Over (1) (2) (3) In the Labor Force. . . 84.7 87.3 90.3 Employed 80.3 82.8 85.3 Unemployed 4.5 4.5 5.0 1 Hirschberg, Yuskavage, and Scheuren, 1977. 2 Average of two methods of undercoverage adjustment demographic and administrative. Table 3. Persons Employed as Percentage Labor Force for Two Stages of March 1973 Estimation Compared to Percentage Adjusted for Census-CPS Undercount1 Persons Aged 16 and Over Initial Standard Extended2 Total 94.7 94.8 94.4 White males 95.6 95.6 95.3 White females 94.5 94.8 94.4 Black and other nonwhite males 92.1 92.0 91.2 Black and other nonwhite females 89.5 89.8 89.3 1Hirschberg, Yuskavage, and Scheuren, 1977. 2 Average of two methods of undercoverage adjustment demographic and administrative. for black and other minority race males the comparable figure is 0.8 percent. Johnston and Wetzel (1969) explored the effect of the 1960 Census undercount on the labor force estimates. The authors provided two alternative sets of "corrected" labor force estimates. In the first set, an assumption was made that the missed persons had the same labor force status as their peers (persons of the same age, sex, and race group). This is called the "comparability" assumption. In the second set omitted persons were assumed to have labor force status comparable to people of the same age, sex, and race but living in urban poverty areas. This is called the "poverty neighborhood' assumption. (These assumptions are at odds with those of Korns (1977) whose research supports the premise that the labor force status of missed persons differs from that of persons counted in the CPS of the same age, sex, and race.) Table 4 shows the effect of omitted persons on the labor force estimates of population coverage by using the Johnston and Wetzel study. This study focuses on the effect of the independent estimates of population used to adjust the CPS, for undercoverage. Though the official estimates of level of employment under the comparability and poverty neighborhood assumptions are understated by 2.8 million and 2.7 million, respectively, the employment rates under both assumptions are 96.2 percent as compared to an official estimate of 96.1 percent. Differences between the Hirschberg et al study and the Johnston and Wetzel study result from changes in the CPS over time, in the labor force, and in the undercoverage between the two time periods. However, a basic difference between the two studies results from the fact that Johnston and Wetzel in their study only looked at that portion of the CPS coverage problem which would be accounted for if 6 Click HERE for graphic. the independent estimates were corrected for census undercoverage (Hirschberg, Yuskavage, and Scheuren, 1977). Attempts to Strengthen the Frame Much research has gone into coping with these problems. Already in the Annual Housing Survey several supplemental universes are used in address ED's. These are listed below (Jones, 1976 and Montie and Schwanz, 1977): 1. The Woodall Universe derived from a private commercial list of mobile home parks created during the period January 1970- December 1974; 2. The Windshield Survey which consists of canvassing sample tracts by automobile to locate mobile home parks; 3. The Successor Check which provides coverage of certain types of mobile homes at large, houses moved to the present site, and structures converted from nonresidential use since the census; 4. Permit Lag universe, which provides coverage of new construction for which permits were issued prior to January 1, 1970, but for which construction was not completed until after April 1, 1970;' 5. Within structure check for SMSA's in address ED's. Theoretically, this is not needed in CPS since the interviewer is supposed to list/update units at the first interview and at intervals thereafter. (Of course, within structure misses can occur in CPS due to inadequate listing by the interviewers.) Another coverage improvement procedure still in the planning stage is the use of the records of mobile home dealers--Dealers' Survey--to obtain new mobile homes outside of parks. The Permit Lag universe, the Windshield Survey universe, and the Woodall universe created for AHS will be used to supplement the basic CPS sample, with implementation to begin in late spring/early summer 1978. Thus, many of the problems with the frame will be corrected in the future. Within household coverage losses are more of a dilemma since it is felt that at least some of this coverage loss results from a deliberate misrepresentation of the total household composition by the respondent. Since the studies regarding within household undercoverage were limited in scope, there is still some uncertainty about some of the reasons for this significant coverage problem and whether adequate solutions can be found to remedy it. B. Procedure for Sample Selection The Sample Selection Procedure After the establishment of the frame, the next step in the design of any survey is the selection of the sample. The selection of the sample in the CPS, which is a multi-stage cluster sample, involves the selection of the primary sampling units (PSU's) and the selection of the sample households within these units. A brief description of these procedures is described in this section. A more thorough ex- planation is presented in Technical Paper No. 40. 1. The Selection of the Printary Sampling Units (PSU's) 7 The CPS in which data for national estimates are collected from approximately 47,000 eligible. households, is redesigned after each decennial census in order to utilize the most recent census data available. These data from each county or county equivalent (independent cities, parishes, etc.) in the United States are used in the definition of the PSU's, restratification of these PSU's, and the selection of the ultimate sampling units. As of March 1973, the CPS was operating completely within the framework of the redesign based on the 1970 Census. In the 1970 redesign 461 PSU's, primarily counties or groups of counties, were selected from 376 strata' Of these 461 PSU's, 156 were designated self-representing (SR); i.e., the strata and PSU's are equivalent. The other 305 PSU's were selected from 220 strata with more than one PSU in each stratum and are referred to as nonself- representing PSU's. The CPS actually consists of two independent probability samples, designated the A and C samples. The nonself-representing (NSR) portion of the A sample consists of one PSU selected from each of 220 strata; the nonselfrepresenting (NSR) portion of the C design sample consists of one PSU selected independently of the A sample from each of 110 paired strata. The 156 self-representing PSU's are considered both A and C sample PSU's. The use of two independent samples in the CPS makes possible unbiased estimates of some of the survey errors and the use of one sample for surveys other than the CPS; e.g. the A sample is used for the Health Interview Survey (HIS) while a subset of the C sample is used for the Quarterly Household Survey (QHS). The nonself-representing PSU's were assigned probabilities of selection that maximized the retention of sample PSU's from the old design sample. The procedure was developed by Keyfitz (1951) and extended by Perkins (1970 and 1971). These probabilities were then used in a Goodman-Kish (1950) controlled selection procedure to arrive at the final selection of sample PSU'S. The controlled selection procedure, which is a probability sampling procedure' was used to control the number of sample PSUs selected from each State to approximately the number expected to, be selected from the State and to control the number of PSU's to be retained in the sample to the expected number. The controlled selection program was run independently for the A and C sample PSU's and for each region. 2. Selection of Sample Households The probability of selection of each household in the CPS national sample, essentially a self-weighting sample, is dependent on the predetermined total sample size. The rate is reduced at intervals in order to keep a constant sample size over the decade. At the present time (1977) the selection rate is approximately 1 in 1500. Basically, two stages were used to select the units to be enumerated in the CPS each month from the sample PSU's. First, a sample of ED's defined for the 1970 census was selected from each PSU. These ED's were selected systematically from a sort order of ED's approximating an additional stratification by city size and geography. The probability of selection of an ED was proportionate to its 1970 population. The second step in the selection process involved the selection of ultimate sampling units (USU's)--clusters of approximately four, usually contiguous, housing units within the designated sample ED's. The procedure for the selection of these USU's varied somewhat according to whether the sample ED was designated as an area ED or address ED. In sample area ED's each housing unit was located on a map and the ED was then divided into chunks or blocks containing approximately two to five USU'S. The sample USU's were designated with probability proportionate to 1970 population, and the chunk containing the sample USU was defined as the sample segment. When that segment is scheduled to enter the sample, the interviewer will visit and list all the units in the chunk. Within segment sampling instructions will then be applied to the segment to designate the sample household units. In TAR ED's already existing Census Tape Address Registers were corrected to incorporate changes the census enumerator had noted during the census interview procedures; in other address ED's, the tape address register was generated from the handwritten address registers. In an operation called "segmenting", 8 a computer' program was used to form the USU's from the census housing units listed on the revised tape address register, designate the sample USU, and prepare the Est of units for interview. 3. Rotation of the Sample USU's selected for sample do not remain in sample for the entire decade since it is felt that this would put too much of a burden on these household respondents. Thus several CPS samples must be generated for use during the decade. Each CPS sample consists of eight approximately equal systematically selected subsamples known as rotation groups. These rotation groups are introduced into the sample once a month for 8 months using a 4-8-4 rotation scheme; i.e., each sample USU is in sample 4 months, out 8 months, and then in 4 more months. Under this scheme each month there is a 75 percent month-to- month overlap and a 50 percent year-to-year overlap. An example of the CPS rotation schedule is given in Figure 1. 4. Replacement of PSU's The CPS sample PSU's are used for surveys other than the CPS, but households are in only one sample. Thus, in small PSU'S, the USU's can easily be exhausted before the decade is over. To handle this situation a system of replacement of PSU's was developed. If a stratum contained at least one PSU without enough USU's for survey needs during the decade, then the rotation clusters or groups of PSUs were formed such that each cluster had enough USU's to last the decade. The full sample for each replacement PSU is introduced in one month across all samples and rotation groups Potential Sources of Error in CPS Sampling Procedure In the development and implementation of the CPS sampling procedure, the faithfulness to the execution of the design was a major concern. However, as with any complex survey design it was, for all practical purposes, impossible for the process to be error-free. There can be nonsampling errors associated with the sampling procedure as with any other part of the survey. These nonsampling errors in the sampling process and its implementation are discussed below. Generally, the errors involve a very small percentage of the total sample households and were accepted because time and cost made it im- practical to do otherwise. 1. In area ED's, before the ED's were divided into chunks for sampling (discussed in Section II.B.), the location of each housing unit had to be determined. Because of time and cost considerations, it would have been impractical to have the field staff visit every ED with any problems in the location of housing units; an alternative was to set a limit on location problems. If the location of at least one-fourth of the units or 50 housing units (whichever was less) could not be determined, the field staff visited the ED and determined the location of all the current housing units. Thus, in a few ED's with housing units whose location could not be determined, some of the chunks or blocks might not have received their correct measure of size; these chunks would then have been selected with a slightly larger or smaller probability of selection than they should have been. However, this particular "problem" is reflected in the estimate of variance. 2. In the original segmenting program an error was detected after about half of the address ED's in Phase I areas had been segmented. (Phase I areas consisted of those areas which were self-representing in both the current and previous design and represented approximately 57 percent of the total population.) The error involved combining two addresses in a block with identical house numbers, but i different street names such that the units at the second address would not have a chance of being interviewed. An estimate of the total lost housing units in all affected ED's was 1.656. Because of the cost involved in resegmenting the affected ED's and the small percentage of units affected-an estimated 0.1 percent of the housing units in the 3,500 affected ED's it was decided to accept the segmentation in the ED's (Shapiro, 1972). 3. An in-house Bureau memorandum (Fasteau. 1973) summarized the evaluation of the overall quality of the final machine readable address file which was the output from the CPS segmenting process. The evaluation involved the selection of a sample of TAR, prelist, and 9 conventional ED's for analysis. Some of the findings were as follows: a. The percent of defective address units in the final file processed through the system and used as input for sample designation was estimated as 0.31 percent. b. Errors left in the records which could contribute to sampling bias or cause serious problems for the enumerators were estimated at only 0.09 percent or less than one-third of all errors. c. The total number of units from the census first count tapes was about 0.5 percent higher than the estimated total number of units used for sample selection. The overall quality of the segmenting process was considered very good. 4. The introduction of replacement PSU's into the sample is completed in one month; i.e. all households in the PSU are in sample for the first time the first,month a replacement PSU is in sample, with households in sample for the first or second time in the second month. (See rotation chart, Figure 1, Section II.B.) Thus the normal rotation pattern of 4-8-4 is not followed for the PSU's; in fact, for the first 15 months the replacement PSU is in sample, the time in sample of the rotation groups does not follow the regular pattern. Considering the effect of rotation group bias (the expected value of the response is not the same for each rotation group and appears related to the number of times the units in particular rotation groups have been in sample), this procedure introduces a bias into the sample (Boisen, 1973). However, the bias is expected to be minimal since there are only 18 NSR PSU's originally designated for sample scheduled for replacement in 1973-1983 and no more than four at any one time. C. Quality Control of Sampling Procedure Selection of Sample PSU's There was no formal quality control procedure attached to the selection of the sample PSU's; however, various checks were made to ensure that the procedures were performed correctly. Though there were a number of "rules" by which the PSU's were redefined and restratified, much of this procedure was subjective, and therefore it was difficult to designate any particular outcome as "correct". However, the stratification was reviewed, discussed, and revamped sufficiently to catch any gross "incorrect" procedures. In order to ensure that no PSU was included in more than one stratum or left out entirely, the stratum populations were totaled and compared with the total published counts for States, etc. The Keyfitz probabilities were reviewed by PSU by the clerical staff and differences reconciled. Several patterns in each run (region and sample A or C) in the controlled selection program were reviewed to see if the program was actually selecting patterns according to specified controls, and the probabilities adding to 1.000. Within PSU Sampling Process Before the within PSU sampling process was begun, a program was written which checked the census published counts for each county with those on the census first count tapes used in the sampling process. Any differences were reconciled. As previously mentioned, the sample ED's were selected by computer program and certain,hand checks were devised and compared to the computer output. The comparison usually involved, within each "work unit", such checks as the random starts and tallies of hits by CPS sample and rotation group. The total counts for the "work unit' were compared to those calculated. Since the SR PSU's had the same within PSU sampling rates, it was not considered necessary to check each PSU. Thus for SR PSU's there were hand checks at ED level by SMSA/non-SMSA and region for each work unit (several States sampled within one computer run). In addition, two to four PSU's were hand sampled and compared to computer output, record by record. (A record consists of the sample ED for each survey hit.) For the NSR PSU'S, which had different within PSU sampling rates, these checks were made for each PSU. After the ED's were screened clerically to determine whether they were area, address, permit or nonpermit, etc. ' these data were punched so that the ED's could be computer edited. Any ED that failed the edit was verified or investigated. Some of the edits included (1) the identification of impossible codes in the permit, area/address or special place code fields; (2) identification of an address type ED which was nonpermit; (3) an identification of a TAR ED in a nonpermit area; and (4) identification 11 of differences of housing and population counts from the address register and first count tape (Waksberg, 1971). The street address, block number, and house number for non-TAR address ED's were coded and punched in the Jeffersonville office twice independently and the records were computer matched and clerically reconciled at. the Bureau. This procedure was referred to as address keying. In TAR ED's additions, deletions, and changes needed to update the tape address register were keyed twice dependently and reconciled. In addition to this check, a stratified sample of TAR ED's and prelist and conventional ED's were selected for analysis. The analysis involved an evaluation of the coding process, clerical review and correction, and related computer processing utilized to correct and update the ED's. From the analysis of this sample of ED's, it was concluded that the overall quality of the addresses was good (Fasteau, 1973). The computer program used in the segmentation process in address ED's generated a summary of the segmenting results. It showed the number of units which came into the segmenting process and the number of USU's formed by size of USU. The number of USU's (measure of size assigned to the ED before the selection of the sample ED's) was multi- plied by four and compared to the number of units on the computer output of the segmentation process. If the counts differed by 10 percent or more, the ED was reviewed and the results reconciled. The sample area ED's were not prepared for sample selection by computer as were the address ED's. Before the ED was prepared for sampling by the procedure described in II.B above, the number of measures obtained from the census address register was compared to the number of measures on the area ED sample list generated from the then prepared ED summary records. Differences greater than 10 percent were reconciled. 12 Observational Design and Implementation This chapter reviews the entire data collection procedure and involves a discussion of the work of many divisions within the Bureau of the Census. The potential sources of nonsampling error are many, but little is known about the existence or size of any errors. Discussed in this chapter are the following elements: basic data collection procedure, the questionnaire design, the data collection staff, the training of CPS interviewers, and the quality control program for the data collection procedure. Each of these areas is reviewed in turn, potential sources of error are pointed out, and any pertinent material on the measurement of such errors is given. A. Basic Data Collection Procedure Listing and Sampling 1. Listing in Address, Permit, and Cen-Sup Segments or Groups of USU's Address segments which consist of regular addresses selected from the 1970 Decennial Census listings, permit segments, and Cen-Sup segments are listed basically by, the same procedure. For such a segment the interviewer is provided with a segment folder which contains the address listing sheets. The folder is used to record the segment identification and the survey procedures to be applied to the segment in a given month, record the transmittal of questionnaires and control cards for completed interviews and noninterviews, etc. Take-all addresses are those at which every unit is designated for interview and correspondingly non-take-all addresses are those at which only a sample are designated for interview. At non-take all addresses, after listing all the units at the address, the interviewer must verify the listing with the building superintendent, manager, etc. At take-all one-unit addresses the interviewer must verify with a household member that there are no more units at the address; at take-all multi-unit addresses' the interviewer must verify the listing with one of the respondents. If the number of units listed for an address is different from that reported, in the census (indicated on the address listing sheet), the interviewer is instructed to determine, if possible, the reason for the discrepancy in the number of units. Where the difference is excessive, specific instructions are provided to the interviewer who generally checks with the office before interviewing. Otherwise, the interviewer will interview each unit at take-all addresses and those units which fall on lines designated for the sample units during the computer sampling at non-take-all addresses. 2. Listing in Area Segments For each area segment the interviewer receives a segment folder which contains the area segment listing sheet (providing segment, PSU, and geographic identification), a segment map and, possibly, one or more grid maps. In listing area segments, the interviewer uses a grid and segment map to locate the segment and determine its boundaries. Within each segment the interviewer must list every structure in the segment, identifying (a) each housing unit, (b) each special place, and (c) each nonresidential structure. In general, the interviewer is required to list by observation. If a structure appears to contain more than one housing unit and the interviewer is unable to determine the number of units in the structure and their location by observation, he/she is to inquire. In apartment-houses with numbered or lettered apartments the interviewer is allowed to list by observation or inquiry. If the listing is by observation, the interviewer must verify the listing with a knowledgeable person such as the manager or, a longtime tenant. Regular housing units, including those constructed after April 1970 in area sample segments within jurisdictions which issued building permits as of January 1970, are represented in permit segments. If the area is one in which there is ..considerable" new construction activity, the interviewer must inquire at each listed unit at time of listing the year the structure was built; these regular units built after April 1970 will not be represented in the area sample. In areas of low new construction activity, inquiry is not made, and new construction units are identified later in completing the control card during the interview 13 and deleted from the sample as Type C noninterviews (see page 16). This procedure is referred to as the Year Built Procedure. The listing forms are sent to the Regional Office. where the sampling takes places. 3. Listing in Special Place Segments Take-all special places are listed at the time of interview on address listing sheets. For each nontake-all special segment, the interviewer, receives the special place listing sheet. Staff units in special place segments are houses, apartments, etc. occupied or intended for occupancy by resident employees and their families; unclassified units are houses, apartments, etc. occupied or intended for occupancy by persons in certain types of special places, e.g. guests in motels/hotels. The interviewer lists Only staff or unclassified units, using his/her instruction manual as a guide. If there is a usable register and more than 100 staff or unclassified units in the special place, the interviewer records the count on the listing sheet. For other special places, the interviewer uses the complete listing method; i.e., he/she lists each staff or unclassified unit on a separate line of the listing sheet. For take- au addresses, the interviewer interviews all units on the address listing sheet; for non-take-all special places the interviewer interviews all units that the office has transcribed to the listing sheet. 4. Updating Updating is the terminology used to describe the checking of the listing and, when required, the adding of new or missed units to the present listing sheet and the recording of changes in units which are already listed. 'ne rules for updating are as follows: Address, Cen- Sup, and take-all special places in special place segments are updated during the first and fifth month of interview if they have not been updated in the previous eight months. Area segments and non-take-all special places are updated the months prior to the first and fifth months of interview if they have not been updated in the previous eight months. Permit segments are not updated. Conducting the Interview 1. General procedure The week containing the 19th day of the month is designated as interview week and an views are supposed to be conducted on Monday through Saturday of that week. For households that are difficult to enumerate, interviews are conducted on the Monday or Tuesday of the following week. In all cases, the reference period .is the week containing the 12th day of the month, called "survey week". Thus, all questions on work activity refer specifically to work activity during the survey Week. Before the interviewer visits the households, an introductory letter describing the CPS and announcing the forthcoming visit is sent to the households scheduled for interview for the first and fifth time if they, have an address to which mail can be delivered. At the first and fifth time a household comes into sample, the interviewer must inquire whether the respondent received the respondent letter and, if not, the interviewer will furnish the respondent a copy. According to the provisions of the 1974 Privacy Act, the respondent must be told that the survey is voluntary; this is clearly stated in the letters. If requested, the interviewer must explain the provisions of the Privacy Act and give a limited explanation of the sampling procedure and uses of the data. Though almost any adult household member 14 years of age or older is eligible to act as respondent, the interviewer is encouraged to interview the most knowledgeable household member, usually the household. bead or spouse. As a last resort the interviewer is allowed to interview a nonhousehold member such as a neighbor, friend, or mother-in-law provided that (1) it is not the first or last time the household is in sample; (2) the individual is at least 14 years old; and (3) the individual is knowledgeable about the family, either having lived in the household during survey or interview week, or having spent a lot of time there. The interviewer is provided a field control card for each unit scheduled for interview. At the initial interview, the interviewer records on the control card the names of all persons at each household (including visitors, if they have at least spent the night before the interview there and are present at time of interview) and determines the usual residence and relationship to household head of each person listed., In addition he/she enters information on date of, birth and age, race, sex. etc. for each person who is determined to be a 4 household member by CPS definitions. At each subsequent visit the listing is updated. The questionnaire is completed for all household members 14 years old and older. Further, the interviewer is instructed to ask the questions exactly as worded on the questionnaire and in the prescribed order. (A facsimile of the questionnaire is shown in Figure 2.) The interviewer is instructed to check the completed questionnaires carefully before sending them to the Regional Office. All completed questionnaires are mailed to the Regional Office on a daily basis where they are reviewed. 2. Mode of Interview Personal interviews are required for households in sample for the first time and for the fifth time. Interviewers also are instructed to conduct personal interviews at households in sample for the second time; however, if the interviewer does not contact the household on the first visit, he/she is permitted to conduct the interview by telephone. At households which have telephones and where the householder has consented to be interviewed by telephone, telephone interviews are permitted all other times. Some telephone interviews are conducted from the Regional Office in sample areas containing the Regional Office and in other areas if the need arises. These interviews are conducted by the clerical staff or by interviewers for any households except those in sample for the first, second or fifth time. This interviewing is done only in spe- cial circumstances such as when an interviewer has an unusually large or difficult workload, when an interviewer is sick and cannot be replaced, or other times when it seems expedient to do so. Only about 300-400 interviews of this type are conducted each month. There is a reluctance to encourage interviewing from the Regional Offices because it tends to hurt the interviewer. Since in general those households that cooperate are more subject to be telephoned, it tends to decrease his/ her production rate. 3. Noninterviews The interviewer may encounter three types of noninterview situations: Type A-those households eligible for the survey for which the interviewer was unable to complete the interview; Type B-vacant units, vacant sites, or units occupied by persons ineligible for the survey; and Type C-units demolished, converted to permanent storage or business use, moved from site, or found to be in sample by mistake. Only the Type A noninterviews affect the reliability of the data. There are four types of Type A noninterview households-the "no one home", the "temporarily absent", "the refusal" and all "other" Type A noninterviews. The "no one home" Type A's are those whose household members cannot be found at home by the interviewer after repeated calls but who are not away for any extended period of time. The households which are "temporarily absent" are those whose household members are away on vacation, business trip, etc. and will not be available for interview during the survey period. "Refusal" households are those which are contacted but whose members refuse to respond. "Other" Type A households include those which could not be reached because of impassable roads, those with death in the family, and any other Type A's which cannot be classified into the other three categories. For Type A noninterviews, race of the household head and farm/nonfarm status are required for noninterview adjustment (dis- cussed in Section V.B.). The interviewer is instructed to acquire information needed for determination of farm/nonfarm status from neighbors if it is not previously recorded on the control card. The interviewer is instructed to "determine" race of the household head by observation or his/her knowledge of the neighborhood. Type B noninterview households are visited each month to determine if any have become eligible for interview. Type C noninterview units are not visited again. Potential Sources of Error in the CPS Data Collection Procedure The potential sources of error in the CPS data collection procedure are discussed below.,Some of these errors are systematically investigated as part of the CPS reinterview program. Some of the problems interviewers have in listing were discussed in Section II.A., entitled "The Frame"; others are discussed later in this chapter in Section III.C., entitled "Data Collection Staff". 1. Listing by Observation in Area Segments Interviewer listing in area segments is accomplished by observation. Inquiry is made only if the interviewer is uncertain about the number of 15 living quarters a structure contains. An alternative procedure to this method of listing in area segments is to knock on every door for address information. Most of the listing errors occur in area segments (Schreiner, 1977); the use of the "knock on every door" procedure could result in more accurate listing in these areas. But the cost could be prohibitive and could result in undue respondent burden. In the fall of 1975 a rural listing test was conducted in nine counties (two in Louisiana, three in Mississippi, and four in Arkansas) to investigate the feasibility of a mail census in rural areas. Three procedures were used: Procedure I in which the lister tried to obtain address information by observation or from neighbors, inquiring at a housing unit only when necessary. When he/she did stop at a unit, the lister tried to get address information for other units nearby. Procedure 2 in which the lister knocked at every door. If no one was home, address information was obtained from a neighbor or by observation. A single callback was allowed as a last resort. Procedure 3 in which the lister knocked at every door. If no one was home, return visits were made in order to obtain address information from the householder. Neighbors and observation were used as a last resort. Listing by Procedure I is closest to that used in area segments in the CPS at present. In Arkansas no coverage difference was detected between any of the listing procedures. However, in Louisiana and Mississippi both Procedures 2 and 3 achieved statistically significant coverage improvement over Procedure 1, but only Procedure 2. appeared to obtain enough additional coverage to offset the increased cost per net listing (Dinwiddie, 1977). These results could have implications for the CPS. 2. Problems with the Year Built Procedure It is difficult at times for a respondent to determine the year a structure was built, particularly when he/she was not the first owner of the housing unit or when the respondent is renting rather than buying. Thus the Year Built Procedure which is used to determine whether a unit was built after April 1970 in permit issuing area segments does not always perform its intended function. Units built after April 1970 that the respondent mistakenly believes were built before that time are represented in both the permit sample and the area sample; those units built before April 1970 that the respondent states were built after April 1970 are not represented at all. 3. Determination of Race of Household Head for Type A Noninterview Units The determination of race of household head of Type A noninterview households will not always be accurate. Those households which are in sample for the first time and are eligible for interview but for which interviews cannot be obtained are more likely to be subject to this error than households in sample for the other months. This error in the determination of race of the Type A noninterviewed households will a ct the noninterview adjustment factors discussed in Section V.A. 4. Effect of Mode of Interviewing on the Employment Estimate Are there differences in the data collected by means of personal interviews from data collected by telephone interviews? If there are differences, which mode of interviewing yields data that are more nearly correct? Before the use of the telephone in CPS was instituted, a test in a limited number of PSU's was conducted to determine its effect on the data. This test, conducted in the early 1950's, showed no appreciable difference in the labor force data obtained by the two methods of interviewing, personal visit and telephone (Hanson, 1976). However, the test conducted at the time was not a completely controlled experiment, the sample size was small, and the results for today's purposes are outdated. Not only has telephone interviewing increased, but the attitudes of respondents toward surveys have probably changed over the years. Because of the wide use of telephone interviewing in the CPS, there is growing concern about its possible effects on the data, and because of this, studies are now being planned to learn more about possible effects. Tables 5 and 6 present data on the amount of telephone interviewing in the CPS. Table 5 shows the average percent of interviewing by telephone by month in sample for 1976 and Table 6 presents the percent of telephone interviewing by month in sample for employed persons in December 1973. 16 Table 5. Percent Telephone Interviews by Month in Sample-1976 Average1 Percent Month in Sample of all Interviews 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44.5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10.3 3, 4, 6, 7, 8 . . . . . . . . . . . . . . . . . . . . . . . .76.0 1 Based on Bureau of the Census monthly CPS enumerator computer runs. Table 6. Percent Telephone interviews by Month in Sample For Employed Persons-December 1973 1 .2 Percent Month in Sample Telephone Interviews 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55.5 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81.5 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.5 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14.2 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78.9 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83.2 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82.8 Total. . . . . . . . . . . . . . . . . . . . . . . . . .60.2 1 Minutes of the BLS--Census Labor Force Committee Meeting of March 27, 1974. 2 The percent of interviews conducted by telephone may be somewhat higher in this month because of the energy crisis and because interviewers have traditionally conducted more interviews by telephone in December because of the Christmas holidays. It can be seen from the tables that telephone interviewing is extensive and occurs frequently even in months when. personal interviews are supposed to be conducted. Respondents in sample for the fifth time, coming back from a vacation from the survey of 8 months, are supposed to be interviewed personally and yet about 10 percent are interviewed by telephone. In months when telephone interviewing is permitted, over three-quarters of all interviews are conducted by telephone. The telephone interview rates are even higher for employed persons. At the present time there is no evidence that personal interviewing and telephone interviewing yield different results on employment questions. A recent study to assess differences it. response related to personal visit contrasted with telephone interviewing in the National Crime Survey showed differences in the number Of reported victimizations depending on the mode of interview (Woltman and Bushery, 1977). We have no evidence that the same thing would be true of employment reports. However, it is recognized that the use of the telephone may cause a different respondent to be interviewed. Data show that there is an increase in the number of "other relatives" who are respondents in later months in sample. To the extent that "other relatives" may not be as knowledgeable as the head and/or spouse of the household about the labor force status of all household members, the telephone data may not be as accurate. A well-designed study could provide answers to the questions raised. 5. The Effect of Proxy Respondents on the Data In the CPS the interviewer is instructed to interview the most knowledgeable household member, generally considered to be the household head or spouse. Technically, though, almost any adult household member 14 years of age or older is eligible to act as the respondent. Thus in the CPS, proxy respondents are frequently used. In the CPS only about 20 percent of the males and 55 percent of the females are interviewed for themselves. Groups which are largely responded for by proxies are working men and young men and women in school. Between February 1965 and June 1966 a Methods Test was conducted outside of the regular CPS with the purpose of testing new methods for the CPS. One thing studied in the test was the selection of best respondent for individual household members. Two different studies were made. In the first, three procedures were compared. The three procedures were as follows (Deighton and Zinter, 1965): Procedure 1: This procedure was similar to the present CPS procedure in that any responsible household member was accepted as the respondent for the entire household. However, unless it would have appeared odd to the respondent, the interviewer was requested to talk to only one respondent. Procedure 2: Each adult household member was to be interviewed for himself/herself, unless the interviewer judged that some person did not have the ability to give accurate responses. Procedure 3: An advance form containing important labor force questions was sent to each household in the test with a request that each adult household member fill the form personally. The interviewer was then to transcribe this information to the questionnaire and ask the household respondent the remaining questions about the household members 17 Table 7. Total Employed as Measured by Three Procedures in Methods Test1 Percentage of persons employed Procedure Total Males Female 1 (Household respondent)....... 55.5 74.2 39.7 2 (Self-respondent)............ 57.2 75.8 41.5 3 (Advance form)............... 57.3 75.8 40.9 1 Deighton, 1967. A comparison of the results was provided in a memorandum by Deighton (1967). Table 7 shows the results for the employed category. A rough approximation of sampling errors indicates that the difference in the percent of total persons employed as measured by the household respondent and the self-respondent procedures is significant. A second experiment reported by Williams (1969) also took place in which 4,500 households in three PSU's were to be interviewed in two consecutive months. In the first month, each person was to b e interviewed for himself/herself. In the second month, there were to be two respondents, each of whom would report for himself/herself and also for the rest of the household. The second month interviews also contained a retrospective interview. The experiment was actually conducted with about 70 percent of the originally selected households. About 12 percent of the households were one-person households and so were not eligible; another 12 percent were noninterview; and for 6 percent the inter- viewer had probably "curbstoned", i.e., contrived the information. By comparing the record for a person interviewed for himself/herself the first month with the retrospective interview from the second.month with a proxy respondent, an estimate of the joint effects of recall bias and proxy respondent was made. By comparing the second month interview of a person responding for himself/herself with the interview for the same time period provided by another household respondent, the effect of the proxy respondent was estimated. For the employment item, the results were as follows: Table 8. Percent Difference in Total Employed as Measured by Self and Proxy Respondent Procedure1 Procedure Effect All Males Females Joint effect, recall, proxy...... -4.1 -1.8 -8.4 Proxy alone...................... -1.2 - .8 -2.0 1 Williams, 1969 and Jones and Aquilino, 1970. The negative sign indicates that there was an understatement of the number of employed. Sampling errors given in Williams (1969) indicate that the -4.1 percent difference in total employed is sig- nificant. The smaller percentages are clearly not significant. The results above show the effect of a total proxy respondent procedure and thus are upper bounds on the effect of proxy respondents in the CPS procedure. B. Questionnaire Design and Instructions The Questionnaire and Interviewer Instructions The basic CPS questionnaire has been in use since 1961. However, there have been changes in the employment criteria over time, some that affected the questionnaire and vice versa. For example, before January 1967, persons who had jobs but did not work during the survey week and were looking for other jobs were classified as unemployed and the questionnaire reflected it; as of January 1967, persons with jobs were classified as employed even if absent from their work during the survey week and looking for jobs. The questionnaire in use as of January 1967 has a skip pattern within the labor force questions so that these individuals can be recorded as employed. The questionnaire effective as of February 1976 has further changes in the description of job or business and the classification of government employees into Federal, State, or local government employees. It should be noted that question 20 (see Figure 2 for portion of questionnaire containing labor force questions) has instructions for the interviewer to probe about unpaid work if the household contains a farm or business owner. Without the additional probe, the respondent might consider his/her contribution to family, business as "work around the house" and he/she would not be included in the employment count. Unpaid workers who worked on a farm or in a family business at least 15 hours a week are considered employed. Before January 1967, question 20 did not contain the added note to probe for this type of employment. Beginning in January 1970, the labor force supplement for non- labor force persons (item 24a-24e---see Figure 2) was asked in the fourth and eighth month panels; from January 1967 to December 1969 this set of questions had been asked in the first and fifth month panels. This change was made because 18 there was evidence that the use of this question in the first and fifth month panels added unemployed persons to these panels which already had higher numbers of unemployed than the average over all panels. The fourth and eighth month panels usually had the lowest numbers of unemployed. The interviewer's manual contains explanations of the purpose of the labor force questions and instructions for their execution. The labor force questions begin with question 19. In question 19, "What was . . . doing most of last week, working or something else?," the responses are arranged in a priority order, i.e. the interviewer is instructed to mark the first category which fits one of the respondent's responses. For example, if the respondent replies, "going to school and looking for work", the interviewer will. mark looking for Work. The following categories under "Working" are given in the Interviewer's Reference Manual (1976) but in somewhat more detail. 1. Paid work or wages or salary which includes cash wages, salary, commissions, tips, or meals, living quarters, etc. received in place of cash wages. 2. Work for profit or fees in the respondents own business or profession, or farm. 3. Work without pay in family business or family farm which actually contributed to the operation of a farm Or unincorporated business run by a member of, the same household related by blood, marriage, or adoption. 4. Exchange work or share work on farms. 5. Jury duty if the person is paid for jury duty. 6. National guard duty (not in the active Armed Forces by Presidential Order). There are two different methods used to tell the interviewer how to proceed from item to item directional arrows or italicized instructions in parentheses. The interviewer is instructed to proceed in numerical order (in the absence of either directional arrows or italicized instructions) and is cautioned not to skip questions unless told to do so. For cases in which the interviewer receives the answer to questions other than those he/she asks, the interviewer must always verify the responses to the additional questions when he/she encounters them. Potential Sources of Error in the Questionnaire Design Do the questions make sense to respondents? Do they know what is being asked of them? Do the interviewers feel comfortable in asking the questions as worded or do they frequently reword questions? Do other questions on the interview schedule impact on the classification of labor force status? Some limited data are available to answer these questions. In September and October 1969, Census staff members undertook some systematic observation of CPS interviewing as part of a questionnaire-research program. Members of the Washington staff were the observers. In January 1970 additional observation was undertaken, but with regular CPS observers. Specifically, the staff/regular observers observed the interviewers' wording of the questions, question explanations, use of probing, etc. as one indication of questions that could be sources of problems. An hypothesis was that if a question is rarely under,stood or replied to as initially asked, the interviewers would be likely to reword the questions to get replies. Such rewording could possibly alter the meaning of the question. Table 9 shows the results for three questions concerned with employment. The interviewer did not accept the first answer given for these three questions only 5 to 8 percent of the time; however, the percentage of times that these three questions were reworded was greater. Question 19 was reworded 8 percent of the time in the first period and 14 percent of the time in the second. Whether this rewording of the questions led .to less accurate answers was not within the scope of the study. In a Mock Interview Study conducted by the Response Research Staff, (see Section III.D.) further information was gathered on problems inter- Table 9. Frequency of Rewording of Labor Force Questions1 Percent of times Percent of times an interviewer question not asked did not accept as written first answer given CPS Question 1st 2nd 1st 2nd Period Period Period Period 19. What was... do- ing most of last week?......... 8 14 8 6 20. Did ... do any work at all last week, not counting work around the house?..... 13 14 7 6 21. Did ... have a job or business from which he was temporarily absent or on lay- off last week? ... 18 9 3 5 ______________________________________________________________________ 1 Rustemeyer and Rothwell, 1969 and 1971. 20 viewers have with specific questions. This information will be presented in a comprehensive study report not yet released by Rustemeyer. Additional questions on the questionnaire can affect the estimates of employment. The labor force supplement questions (24a- 24e) are asked of persons not in the labor force in part of the sample each month. In 1968-69, the questions were asked of persons in sample for the first and fifth times. In 1970 these questions were shifted to persons in sample for the fourth and eighth times. The questions may make a difference in the number of persons classified as employed, at least for women. The data are shown in Table 10. The rotation group indices shown are the average number of persons classified as employed in the particular month in sample divided by the average over all months in sample and then multiplied by 100. The indices arc higher for months one and five in the earlier time period and for months four and eight for the later time periods. The sampling errors of the differences of the indices is about .4 for mates and .7 for females. Though these differences are not statistically significant for the data shown, a third time period is now available which has data for 1970-1976. The differences for females are "significant". Thus, the data show that the additional questions on the interview schedule may result in more persons being classified as employed. These people should probably be classified as employed. Therefore, the questions as presently worded, without the additional questions, seem to result in a slight downward bias in the number of employed. Whether these additional questions should be included in the survey Table 10. Rotation Group Indices for Employment Items for Two Time Periods, 1968-69 (TI) and 1970-72 (T2)1 Month in Sample Characteristic All Persons 16 and over 1 4 5 8 Civilian labor force ... T1 102.3 99.5 100.8 99.0 Civilian labor force ... T2 101.6 100.3 100.0 100.0 Employed ............... T1 101.6 99.8 100.4 99.3 Employed ............... T2 101.1 100.3 99.9 100.1 Males Employed ............... T1 100.9 99.8 100.2 99.7 Employed ............... T2 100.7 100.2 99.9 100.2 Females Employed .............. T1 102.8 99.7 100.8 98.6 Employed .............. T2 101.9 100.3 100.0 100.1 1 Based on annual averages from gross change tables produced monthly, quarterly, and annually by the Bureau of the Census. every month can only be answered after further research and experimentation. Possibly the additional respondent burden of answering these questions each month could have a detrimental effect on the data. The CPS frequently has supplements to the basic labor force interview, It is possible that these supplements may affect the quality of the data. However, the labor force questions are asked first, so they are probably not changed because of more questions being asked. The March supplement, however, is very long and it has been noticed that the noninterview rate is higher in March. C. Data Collection Staff Organization and Description of the Data Collection Staff The data collection staff works from 12 Regional Offices under the overall supervision of the Chief of the Field Division. Regional- Offices were originally defined to equalize the workload over all programs. A Regional Director supervises each Regional Office. The CPS is the responsibility of the Demographic Program Coordinator who has a CPS Program Supervisor on his/her staff. There are three to five office clerks in each regional office who work essentially full time on the CPS. Each Regional Office has between 70 and 150 interviewers available for the CPS, or about one interviewer for every -55 households assigned for interview. Each office usually has 12 or more Supervisory Field Representatives who assist the CPS Program Supervisors in on-the-job training, observation and reinterview of the interviewer staff. The Supervisory Field Representatives also carry out CPS assignments as the need arises. Each interviewer is a part- time employee who works out of his/her home. About 30 percent of the interviewers leave the staff each year, though this is not evenly distributed throughout the country. As a result, the Re