| Federal
Committee on Statistical
Methodology Office of Management and Budget |
FCSM
Home ^ Methodology Reports ^ |
Statistical Policy Working Paper 13 - Federal Longitudinal Surveys
Click HERE for graphic.MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY Maria Elena Gonzalez (Chair) Daniel Kasprzyk Office of Information and Bureau of the Census Regulatory Affairs (OMB) (Commerce) Barbara A. Bailar William E. Kibler Bureau of the Census Statistical Reporting Service (Commerce) (Agriculture) Yvonne M. Bishop David Pierce Energy Information Federal Reserve Board Administration (Energy) Edwin J. Coleman Thomas Plewes Bureau of Economic Analysis Bureau of Labor Statistics (Commerce) (Labor) John E. Cremeans Jane Ross Business Analysis Social Security Administration (Commerce) (Health and Human Services) Zahava D. Doering Fritz Scheuren Defense Manpower Data Center Internal Revenue Service (Defense) (Treasury) Daniel H. Carnick Monroe G. Sirken Bureau of Economic Analysis National Center for Health (Commerce) Statistics (Health and Human Services) Terry Ireland Thomas G. Staple National Security Agency Social Security Administration (Defense) (Health and Human Services) Charles D. Jones Robert D. Tortora Bureau of the Census Statistical Reporting Service (Commerce) (Agriculture) PREFACE The Federal Committee on Statistical Methodology was organized by OMB in 1975 to investigate methodological issues in Federal statistics. Members of the committee, selected by OMB on the basis of their individual expertise and interest in statistical methods, serve in their personal capacity rather than as agency representative. The committee carries out its work through subcommittees that are organized to study particular issues and that are open to any federal employees who wish to participate in the studies. Working papers are prepared by the subcommittee members and reflect only their individual and collective views. This working paper of the Subcommittee on Federal Longitudinal Surveys discusses the goals, management, operations, sample designs, estimation methods, and analysis of longitudinal surveys. Conclusions are drawn about where to use longitudinal surveys, and the need to have an evaluation component in these surveys. The Appendices contain twelve case studies of recent longitudinal surveys. The report is intended primarily to be useful to Federal agencies in choosing to do, and then in designing, carrying out, and analyzing data from longitudinal surveys. The Federal Committee on Statistical Methodology intends to organize seminars to discuss the report with interested Federal agency staff members. The Subcommittee on Federal Longitudinal Surveys was co-chaired by Barbara A. Bailar and Daniel Kasprzyk, Bureau of Census, Department of Commerce. MEMBERS OF THE SUBCOMMITTEE ON FEDERAL LONGITUDINAL SURVEYS Barbara A. Bailar* (Co-chair) Lawrence Ernst Bureau of the Census (Commerce) Bureau of the Census (Commerce) Daniel Kasprzyk* (Co-chair) Marie E. Gonzalez* (ex officio) Bureau of the Census (Commerce) Office of the Information and Regulatory Affairs (OMB) Barry Bye Catherine Hines Social Security Administration Bureau of the Census (Commerce) (Health and Human Services) Dennis Carroll Curtis Jacobs Center for Statistics Bureau of Labor Statistics (Education) (Labor) Robert Casady Inderjit Kundra National Center for Health Energy Information Statistics Administration (Health and Human Services) (Energy) Steven B. Cohen Bruce Taylor National Center for Health Bureau of Justice Statistics Services Research (Health (Justice) and Human Services) ADDITIONAL CONTRIBUTOR TO THE REPORT Lawrence Corder Research Triangle Institute (Previously National Center for Health Statistics) *Member, Federal Committee on Statistical Methodology ACKNOWLEDGEMENTS This report is the result of collective work and many meetings of the Subcommittee on Federal Longitudinal Surveys. Each chapter had a principal author (or authors), as noted below, but the final report, particularly the introduction and summary sections, reflects contributions from all of the Subcommittee Many useful suggestions on content and organization were made by Maria Gonzales, chairperson of the Federal Committee on Methodology (FCSM). Barbara Bailar, Co-Chair of the Subcommittee, prepared the Introduction and the concluding Chapter, which embody the discussions held by the whole Subcommittee. All of the FCSM members reviewed several drafts and made many important suggestions. The Subcommittee in particular wishes to recognize the valuable contributions made by the primary reviewers: Zahava Doering, Fritz Scheuren and especially Monroe Sirken, who read and commented on two drafts of the complete report. The principal authors of each chapter of the report are: Chapter One Catherine Hines Chapter Two Lawrence Corder Chapter Three Bruce Taylor Chapter Four Daniel Kasprzyk and Lawrence Ernst Chapter Five Barry V. Bye The Subcommittee thanks also the following persons who were responsible for preparing the Case Studies that appear in the Appendix: Edith McArthur (SIPP), Curtis Jacobs (CPI), Steve Kaufman (ECI), Dennis Carroll (NLS-72, HS&B;), Catherine Hines (NLS), Barry V. Bye (RHS, WIE), Stephen B. Cohen (NMCES), Robert Casady (NMCUES), James L. Monahan (LED), John DiPaolo, Robert Wilson, and Peter J. Sailer (SOI). Catherine Hines edited the report. Joanne Watson (Bureau of the Census) prepared each of the drafts, and the Subcommittee thanks her for her patience and accuracy. iii GLOSSARY OF ABBREVIATIONS AHS American Housing Survey (Formerly Annual Housing Survey) CPI Consumer Price Index CPS Current Population Survey ECI Employment Cost Index HCFA Health Care Financing Administration HS&B; Longitudinal Survey of High School and Beyond ISDP Income Survey Development Program ISR Institute for Social Research (University of Michigan) NCES National Center for Education Statistics NCHS National Center for Health Statistics NCS National Crime Survey NLS National Longitudinal Surveys of Labor Market Experience NLS-72 National Longitudinal Study of the High School Class of 1972 NMCES National Medical Care Expenditure Survey NMCUES National Medical Care Utilization and Expenditure Survey OSIRIS Statistical Analysis software, Survey Research Center, U. Michigan PSID Panel Survey on Income Dynamics RAMIS Data base management system, Mathematical Research Inc., Princeton, N.J. RAPID Data base management system, Statistics Canada, Ottawa RHS Retirement History Study SAS Data base management system, SAS Institute, Cary, N.C. SSA Social Security Administration SIPP Survey of Income and Program Participation SIR Data base management system, SIR, Inc., Evanston, IL SOL Statistics of Income Program, IRS WIE Work Incentive Experiment, SSA iv TABLE OF CONTENTS Page GLOSSARY OF ABBREVIATIONS vi INTRODUCTION 1 Chapter I: The Goals of Longitudinal Research 5 Chapter II: Managing Longitudinal Surveys 11 Chapter III: Longitudinal Survey, Operations 19 Chapter IV: Sample Design and Estimation 35 Chapter V: Longitudinal Data Analysis 49 Chapter VI: Summary and Conclusions 63 APPENDIX: Case Study 1 Survey of Income and Program Participation 67 Case Study 2 Consumer Price Index 75 Case Study 3 Employment Cost Index 89 Case Study 4 National Longitudinal Study of the High School 97 Class of 1972 Case Study 5 High School and Beyond 101 Case Study 6 National Longitudinal Surveys of Labor Market 105 Experience Case Study 7 Social Security Administration's Retirement 111 History Study Case Study 8 Social Security Administration's Disability 115 Program Work Incentive Experiments Case Study 9 National Medical Care Expenditures Survey 123 Case Study 10 National Medical Care Utilization and Expendi 127 tures Survey Case Study 11 Longitudinal Establishment Data File 137 Case Study 12 Statistics of Income Data Program 147 REFERENCES 153 INTRODUCTION Since the 1960's, the Federal government has sponsored an increasing number of longitudinal surveys as vehicles for research on administrative and policy issues. The goal of the Federal Committee on Statistical Methodology's subcommittee on Federal Longitudinal Surveys is to identify the strengths and limitations of longitudinal surveys, and to propose some guidelines for using them most effectively. Beginning its work, the subcommittee found that there were multiple definitions of a longitudinal survey, so our first task was to define what this report would mean by the term. The difficulty arises because there are two facets to the definition, design and analysis. To be absolutely clear, one must distinguish between a longitudinally designed survey and a survey with longitudinal analysis. We have elected to put these components together in our definition. The distinguishing features of a longitudinal survey are: - repeated data collection for a sample of observational units over time; - the linkage of data records for different time periods to create a longitudinal record for each observational unit; and - the analysis is based on the longitudinal microdata and refers to data collected over time. The essential feature is that, from the beginning, there is a plan to elicit data from the future for each observational unit. This definition excludes some surveys with longitudinal elements, such as the Current Population Survey (CPS). The Survey of Income and Program Participation (SIPP) is included here as a longitudinal survey, although there are as yet no longitudinal analyses of SIPP. Federal agencies also conduct surveys of establishments that have longitudinal elements but these are not yet true longitudinal surveys either. There is an effort to create a longitudinal file for manufacturing firms at the Bureau of the Census. We included this program as a case study in this report because, although it does not meet our definition, it may be of interest to readers. Similarly, Federal agencies maintain longitudinal files of administrative records that do not meet our definition. Yet they may be used in ways that are similar to the analysis of longitudinal surveys, so we have included an example, the Statistics of Income Data Program, as a case study. 1 Rotating panel surveys* are often described as longitudinal surveys. They are not, but they may share many sampling, estimation, and analysis characteristics with longitudinal surveys. In addition, there is a tendency for ongoing rotating panel surveys to be changed to make longitudinal analysis possible. The National Crime Survey (NCS) is currently considering such a transition, and one possible result of the current redesign activities will be to create a longitudinal NCS data file if the cost is not prohibitive. There is interest in moving in the same direction with both CPS and the American Housing Survey (AHS, formerly the Annual Housing Survey). We should anticipate that eventually more rotating panel surveys will be modified, or designed from the beginning, to make longitudinal analysis possible. At this time, however, many rotating panels lack longitudinal data files, and many longitudinal surveys are designed without rotating panels. The subcommittee members examined in detail 12 recent longitudinal surveys sponsored by the Federal Government, as examples and illustrations. These are: (1) the Survey of Income and Program Participation (SIPP); (2) the Consumer Price Index (CPI); (3) the Employment Cost Index Survey (ECI); (4) the National Longitudinal Study of the High School Class of 1972 (NLS-72); (5) High School and Beyond (HS-B); (6) The National Longitudinal Surveys of Labor Market Experience (NLS); (7) the Social Security Administration's Retirement History Survey (RHS); (8) The Social Security Administration's Disability Program Work Incentive Experiments (WIE); (9) The National Medical Care Expenditure Survey (NMCES); (10) the National Medical Care Utilization and Expenditure Survey (NMCUES); (11) the Longitudinal Establishment Data File; and (12) the Statistics of Income Data Program (SOI). The surveys chosen for case study treatment were selected to represent a variety of sponsors, research questions and kinds of respondents. Each of the 12 case studies is described in the Appendix,,and they are frequently cited to illustrate important points throughout the text. We hope that the chapters of the text and the case studies in the Appendix will convince readers of four points that emerged from the subcommittee's review of longitudinal surveys. First, longitudinal survey designs are appropriate, and even required, for certain kinds of research. These include, but are not limited to, such topics as gross change, the causes of change, or the role of attitudes in change. However, many longitudinal surveys have not made full use of their longitudinal design in the analysis. Second, longitudinal survey design, operation, and analysis techniques are still evolving. There are a number of important design issues that are not yet explored or understood. An example is the optimal length of time between interviews, and the number of interviews to conduct to achieve research objectives. To some extent the variations in survey design ___________________________ * A panel is a sample of persons selected to participate at a particular point in the longitudinal sequence. In a rotating panel survey the sample units have a fixed duration. As they leave the sample, they are replaced by new units which are introduced at specific points in time. 2 reflect the wide and legitimate differences between the research goals that each survey was designed to accomplish. This does not explain, however, all the existing variation in methods . Decisions about sample design and attrition, about selecting the best respondent or analytical units, about the best estimation, imputation or weighting schemes, or about the impact of varying personal, mail or telephone interviews over the course of a longitudinal survey, have not always been consistent. Third, the important question of the costs of longitudinal surveys compared to cross-sectional surveys has yet to be answered. There are conflicting reports about the relative costs of the two types of survey. Costs are usually cited as higher for longitudinal surveys, but the costs being reported are confined to data collection costs and processing costs. This does not compare the full range of survey costs including quality costs, costs of analysis, and other such elements which could, in the long run, change the picture of the relative costs. The fourth and final point that emerged from the subcommittee's review was that the surest method for learning answers to design, operational, and analysis issues is to build an evaluation component into a longitudinal survey. By this means a record of comparative performance is created which benefits others. The case studies presented in this report, in particular, show how progress occurs when evaluation is built into survey operations, and how forethought and planning, far more than additional expense, are needed to increase our knowledge about longitudinal survey design. This report is presented in 6 chapters. The first chapter is a review of the kind of research question for which a longitudinal approach is appropriate, illustrated with examples. The second and third chapters describe some of the problems encountered in planning and managing longitudinal surveys. Chapter four discusses problems related to sample design and analytical units in longitudinal surveys, and special problems of estimation and weighting. Chapter five describes and evaluates major approaches to the analysis of longitudinal surveys. The final chapter, number six, summarizes some issues the subcommittee members recognized as important, and outlines the need for building an evaluation component into prospective longitudinal surveys; both to answer questions about the quality of data derived from each survey and to answer questions about optimal design for future longitudinal surveys. 3 CHAPTER 1 THE GOALS OF LONGITUDINAL RESEARCH There are at least five distinctive advantages to using a longitudinal survey rather than a cross-sectional survey some of these advantages are shared by rotating panel surveys. 1. A longitudinal sample reduces sampling variability in estimates of change. This is an advantage shared with rotating panel surveys such as CPS and NCS. 2. A matched longitudinal file provides a measure of individual gross change for each sample unit. This is an advantage shared to some extent by rotating panels, which can provide a measure of gross change, but not usually on an individual basis. 3. Longitudinal survey interviews usually have a shorter, bounded reference period that reduces recall bias in comparison to a retrospective interview with a long reference period. Rotating panels such as CPS and NCS also share this advantage. Longitudinal surveys with long intervals between interviews may lose this advantage. 4. Longitudinal data are collected in a time sequence that clarifies the direction as well as the magnitude of change among variables. 5. Longitudinal interviews reduce the respondent burden involved in creating a record that contains many variables. A single interview could not collect comparable detail without excessive respondent burden and fatigue. In addition, the quantity of data collected in a longitudinal survey is usually greater than that from several cross-sectional surveys because of the correlational structure of longitudinal data. There are also some distinct disadvantages to longitudinal surveys. Some of these are:. 1. The analysis of longitudinal surveys is dependent on the assembly of the microrecord data. The full advantage of compiling a detailed longitudinal record with many variables may not be available until years after the start of data collection. 2. Beginning refusal rates may be comparable to those of cross-sectional surveys, but the attrition suffered over time may create serious biases in the analysis. Principal Author: Catherine Hines 5 3. A longitudinal survey, including several data collections, is more costly than a single retrospective cross-sectional survey. A longitudinal survey may be less costly than a series of cross-sectional surveys. It is speculative whether a longitudinal survey is more costly than a rotating panel survey. 4. The estimates of gross change derived from longitudinal surveys tend to be inflated over time by simple response variance, The combined or net effect of such influences as simple response variance, response bias and time-in- sample bias effect on longitudinal estimates of gross change are still poorly measured. 5. Longitudinal surveys are often improperly analyzed, not taking into account longitudinal characteristics or attrition. For some research goals, the advantages clearly outweigh the disadvantages. For other research goals this may not be the case. Research goals that demand longitudinal surveys are described in this chapter. A. Measuring Change Both cross-sectional and longitudinal surveys can be used to measure change. The National monthly estimate of unemployment based on the CPS is always compared to the estimate for the previous month or the same month a year ago. Estimates of such things as crime victimizations, retail sales, housing starts, or health conditions are all compared to estimates from a previous time period. None of these data are currently based on longitudinal surveys. Which measures of change need a longitudinal file structure? One example is the components of individual change. These are measures of gross change for the observational units between points in time.* Longitudinal data are frequently displayed in a time- referenced table, showing the characteristics, attitudes, or beliefs of the sample at time 1; cross-tabulated by the same characteristics, attitudes, or beliefs at time 2. Another example is the average change for an observational unit. As pointed out by Duncan and Kalton (1985), if data are available for several time points for each observational unit, then a measure of average change or trend can be estimated. Finally, a longitudinal design permits the measurement of stability or lack of stability for each observational unit. Measures of gross change are of interest in several of the case studies described in this report. Respondents are followed through employment and unemployment (NLS), training and the labor force (NLS-72, HS&B;), into and out, of poverty (SIPP), or between health, treatment, and disability (NMCES, NMCUES, RHS, WIE). The focus is sometimes on movement across an arbitrary threshold (such as poverty, defined by household composition and income), and sometimes on a continuous measure. The observation periods in a longitudinal survey are commonly called waves. A wave describes one complete cycle of interviewing, from sampling to data collection, regardless of its duration. 6 In independent (i.e., cross-sectional) samples, sub- populations with very different gross-change patterns are indistinguishable if the sum of the changes is similar. This has been important to studies of employment. The NLS, for example, can distinguish a hypothetical population where 15% of the people are never employed, from a population where at each interview a different 15 % respondents report unemployment. A cross-sectional survey could not make the same distinction, which is vital to the development of intervention policies. Another example can be cited from the field of social indicators research. A series of variables, measured longitudinally, can be used to construct models for estimation to examine change over time with great elegance. (See Land, 1971, 1975.) Young adults in the years after full-time school are frequent longitudinal survey subjects (NLS Youth Cohorts, NLS-72, HS&B;) because individuals in these years are known to pass between statuses (employment and unemployment, school and training programs, in and out-of the armed services, between households) rapidly and irregularly. Cross-sectional studies would miss all the individual reversals and repetitive change. To develop detailed models of the causes of change in these fluid populations, longitudinal measures are needed to capture the record of individual and gross change. For example, cross-sectional studies of college enrollments have generally found relatively high stability over a number of years, whereas analysis of NLS-72 data identified frequent individual change occurring at a stable rate. A substantial percentage of the college students surveyed exhibited erratic enrollment patterns characterized by dropping out or transferring between 4-year and 2-year colleges. In light of these findings, student financial assistance (grants and loans) have changed. Legislation has shifted aid to channel the funds directly to the students, who choose the college they wish to attend -- rather than channelling the funds to college officials, who decide how the funds are doled out to enrolled students. Studying the relationship between attitudes and behavioral change poses particularly difficult problems in research design. The problems inherent in determining which variable in a pair changes first are present, and they are exacerbated by the problems encountered in surveys of subjective phenomena, such as attitudes. Using retrospective questions to ask respondents to reconstruct thoughts or feelings as they existed in the past has proved unreliable. Prospective longitudinal surveys provide the most reliable data on change in knowledge or attitudes, because longitudinal measures are collected while the subjective states actually exist. This appears to reduce the bias frequently caused by suppression or distortion of respondent recall. In addition, unlike retrospective measures of attitudes, contemporary measures can sometimes be probed or even verified. The longitudinal surveys of high school students (NLS-72 and HS&B;) demonstrate the method's power to collect data on changing subjective states, and to study causation. These surveys have measured attitudes and expectations about employment, and subsequent employment experiences and behavior. The data, which could not have been collected cross-sectionally, can be analyzed to understand the formation of attitudes, as well as to evaluate the effects that attitudes have on subsequent behavior. 7 When the research goal is to measure a component of individual change, longitudinal surveys have strong advantages. They are the only method available to collect data on a recent occurrence basis over a long period of time. Although a retrospective cross- sectional survey could be used to attempt the same thing, the recall bias may be a strong force against this decision. The bias from the attrition in a longitudinal survey has to be balanced against the bias or lack of information in a retrospective cross- sectional survey. The bias from attrition is usually preferred. Price and wage changes are measured in longitudinal surveys (i.e., the CPI and ECI) because the longitudinal sample design holds other variables constant. The assumption can be made that whatever unknown sampling bias exists in later waves was also present in earlier waves, and can be dismissed as a possible source of the changes being measured. B. Assembling Detailed Individual Records Longitudinal surveys generally provide researchers with more detailed records for each individual than is practicable through a cross-sectional design. In a longitudinal design, an extremely detailed record can be accumulated for each subject without making any single observation period (i.e., interview or wave) excessively burdensome. By 1982, for example, records for the original respondents in the NLS contained up to 1,000 data items for each sample case. To create a record of comparable detail complexity would have required a one-time questionnaire of extraordinary length. In addition, responses referring to earlier time periods would have been reconstructed from memory, reducing their reliability. In many instances, researchers are looking for cause- and-effect relationships that are more likely to be accurate if the data are compiled on a current rather than retrospective basis. C. Collecting Data That is Hard to Recall Some surveys ask questions that respondents have difficulty in answering precisely or objectively after much time has passed. These include questions that call for the kind of detail that people seldom recall clearly (such as complete records of expenditures, or health treatments), and questions that refer to events that respondents tend to telescope, embellish or suppress in their memories after time has passed (such as crime victimization, health problems, or visits to the doctor). Questions such as these have been used successfully in longitudinal surveys, in which the previous interview provides a clear marker to bound respondent recall, and which are constructed with short reference periods between interviews. For example, the Consumer Expenditure Survey, conducted as part of the CPI program, collects detailed records of household spending patterns through longitudinal interviews. (See Case Study no. 2 in the appendix.) A longitudinal survey with relatively short reference periods is one of the best methods for producing aggregated data for a longer time period, such as a year. For example, the primary goal of the NMCES and,NMCUES programs 8 was to develop estimates of medical expenditures for a calendar year. This was accomplished by obtaining medical expenditure data every 3 months and Compiling an annual total. A similar example is the new continuing Consumer Expenditure Survey, which covers all consumer expenditures. The SIPP program employs a similar design, using interviews at 4 month intervals to produce annual aggregates. The relatively short, bounded reference periods for these longitudinal surveys improve reporting by eliciting events closer to the time they occur. This increases the completeness of aggregated estimates and reduces error. D. Modelling Studies and Pilot Programs The detailed case histories built up in longitudinal surveys are important in analyzing the impact of alternative policies or intervention strategies. The complex individual case records accumulated in a longitudinal panel survey provide a microcosm in which the impact of changes can be simulated. Questions can be answered about the probable impact of changing a program's eligibility criteria, for example, or about the benefits which specified classes of respondents might anticipate under,various program changes. Intervention programs can be evaluated through longitudinal surveys to Study their effect on respondents with known characteristics. A sufficiently detailed record makes it possible to simulate alternative interventions, and predict a range of effects. (See Case Study 9 on the WIE, for example.) In some cases longitudinal surveys, pilot intervention programs and Federal policy experiments evolved together in the 1960's. Several longitudinal surveys authorized as components of pilot or experimental intervention programs to measure program effects and ensure that decision-making information would be available when it was needed. Longitudinal data collection components were built into pilot income maintenance programs, for example, administered temporarily in cities in New Jersey, Indiana, Colorado and Washington State. In conclusion, tho points about the periodicity of longitudinal research should be stressed. First, longitudinal data are never available immediately; any data that are based on the sequence of measures over time cannot be fully extracted until the final measures are collected. If information is needed at once, another research design has to be used which incorporates some alternative to a true longitudinal approach; such as retrospective measures, or the use of administrative records. Even if the quality of data from a longitudinal survey would be clearly superior, that would be irrelevant if the schedule outweighs these other considerations. Second, longitudinal data can be used cross-sectionally to provide immediate data as long as the research focus is not specifically on changing measures over time. Each wave of a longitudinal survey can also be analyzed as a cross-sectional survey. Thus some data can always be made available immediately. Record data from non-going longitudinal surveys can be analyzed quickly from a cross-sectional perspective to serve certain analytical purposes without delay. It is also possible to add questions to the current waves of a longitudinal survey to meet immediate data needs, using an existing longitudinal sample and base-line demographic data for maximum efficiency. In these ways a longitudinal design adds analytical strengths without sacrificing the potential for cross-sectional research. 9 CHAPTER 2 MANAGING LONGITUDINAL SURVEYS As described in the previous chapter, prospective longitudinal surveys have proved to be an important research approach, but certain limitations have also emerged that must be considered when these surveys are planned. The problems related to staff and management of longitudinal research differ in kind as well as degree from those encountered in cross-sectional research. The core of the problem in managing a longitudinal survey is a conflict between the need for long-term and for short-term resources. Plans and funding must be stable over many years, but the need for staff rises and falls over the course of a longitudinal survey. Most organizations sponsoring longitudinal surveys have solved the dilemma through some combination of permanent and temporary staff. Fluctuations in resources are less pronounced in longitudinal surveys that employ non-going rotating panels (such as SIPP or, to some extent, the CPI) than they are in fixed panel surveys in which interviews are conducted at longer intervals (such as NLS, NLS-72, or HS&B;). The major difficulty faced in planning and managing a longitudinal survey is in maintaining a core group dedicated to the project, and maintaining consensus between this group and senior agency staff. These groups tend to view long-term commitment of Staff and resources in different ways. The schedule, funding, and staff needs of a longitudinal survey are viewed differently by survey designers, by agency directors, and by those responsible for operations. It is a constant challenge to generate commitment to a long-term goal such as analysis of data, when senior staff with direct authority over the project often changes before the survey is completed. A. The Need for Long-Range Planning The need for long-range planning and organization for a longitudinal survey should be brought to the attention of senior staff very early with a planning document that outlines the workload, survey tasks, and anticipated products over time. The planning document should be prepared in conjunction with an analysis plan, and the design of the instruments and procedures will then follow once all groups are in agreement with the planning document. Long range planning is vitally important to a longitudinal survey, because it promotes enduring support at a senior agency level, it widens the pool of sponsors and supporters; and it begins the process of documentation that ensure continuity of operations. Principal Author: Lawrence Corder 11 A large-scale longitudinal Federal survey generally has at least nine principal management phases which may be briefly described as follows: 1. Budget Planning. Up to five years before data collection is to begin, a general plan must be conceived and provisions made to obtain continuing staff and funding resources throughout the longitudinal project. 2. Development of Position Papers. These are draft planning documents which discuss options, costs, and yields associated with various sampling plans, data collection designs, or questionnaires. These ensure widespread and enduring support for the longitudinal research. 3. Procuring outside assistance. If a contract is to be awarded, requests for proposals must be prepared, cleared and advertised, and responses must be evaluated before a contract is signed. This is a common approach to levelling out resource needs. 4. Final Research Plans. This stage includes final OMB clearance, conduct of field tests, revisions as necessary, and detailed agreements with any other cooperating agencies. 5. Data Collection. This refers to the full-scale field data collection. Longitudinal surveys (such as NLS) which have been extended beyond the original research period have repeated these 5 stages independently several times. 6 . File Preparation. Development of the system for data entry, data base design, processing, etc., may also require systems for optical scanning of questionnaires, machine/or manual edit steps, preparation of code books, the construction of composite variables, plans to preserve privacy in public data files, and numerous other activities. Each operation must be fully documented, to ensure comparability between waves. 7. Planning the Analysis. While the overall goals oft he analysis must be planned in the early stages, some details cannot be finalized until the data are available on computer files and code books are completed. Also, as policies shift, new analytical priorities must be met. In all cases, this process requires plans which may include in-house analyses and contracts for analyses. Contracts require a repetition of the procurement process described in phase 3. 8. Conduct of Analyses. These may go on for several years. Cross-sectional, analyses can be conducted as soon as one wave of interviews has taken place. Longitudinal analyses take place after some or all other waves are completed. 9. Publications. With in-house and professional peer reviews, these may continue for several years. 12 Each phase requires substantial time to complete, contains specific activities and results in the preparation of key documents. The final products of any longitudinal surveys are usually public-use data files and reports.* Ideally, these should be supplemented by rapid preparation of in-house documents as part of the policy-making process. Schedule milestones and due dates are part of any longitudinal survey, and the ultimate success of the project and even the usefulness of the analytical results may be judged against their timeliness. It is not unusual for a longitudinal survey to consume a decade or more from inception to completion of the publication plan. The NMCES and NMCUES Studies, for example, both took 8 to 10 years to complete. While field operations and the period for analysis vary with each survey's objectives and resources, the successful pre-field period is probably very similar in each case. The planning period should be dedicated to achieving consensus internally, then to producing instruments and obtaining clearances and approvals (for contracts as well as for questionnaires). A typical schedule for completing pre-field activities alone (excluding budget planning) would frequently require 12 to 18 months. Some of the most severe criticisms of longitudinal surveys have resulted from insufficient planning. It is not uncommon, for example, to omit thorough planning of the analysis. Then, at a production stage, it is discovered that people have different ideas on the tables and data to be produced and analyzed. It is also necessary to plan the linked files carefully so that the data needed for longitudinal analyses are readily available. Unfortunately, the planning of budgets and field work often takes precedence over the planning of processing and analysis, sometimes leading to delays, acrimony, and sometimes shifts in support. B. Funding Longitudinal Research The actual unit costs of doing longitudinal surveys may be no higher than for a series of cross-sectional surveys of comparable size and complexity (Wall & Williams:30). There is conflicting evidence on comparable costs, probably reflecting non-standard cost reporting on survey operations. Funds, however, must be committed over a number of fiscal years and budget plans are not easily altered. There is a trade-off to be made when errors are discovered or improvements can be implemented. Additional costs must be carefully considered, as well as the effect of changes in methodology on the longitudinal analysis. Errors, of course, should be corrected or, if too costly, an indication of their effects provided. Changes in methodology are different from changes necessitated by errors and must be thoroughly explored. Provision should be made to share information with analysts and data users on real change vs. methodologically-induced change. (The change to computer assisted telephone interviewing is one such change that needs careful exploration.) If errors or methodological changes result in higher costs, alternative methods of meeting those costs should be considered: higher funding, smaller sample size, more time between interviews, delayed processing, and so forth. Surveys of business or industrial establishments are often an exception to this rule, to protect the identity of large firms that dominate certain samples. 13 Inter-agency cooperation can help meet long-term funding needs. The Health Care Financing Agency (HCFA) and the National Center for Health Statistics (NCHS) chose this approach in conducting NMCUES. Inter-agency agreements frequently involve the Census Bureau for data collection and analysis, but they may also be used between other agencies with related research goals. Inter- agency Cooperation in longitudinal surveys could take the form of joint sponsorship of a new longitudinal survey, or it could be in the form of using an existing longitudinal sample as a vehicle for research to save the cost of starting a new longitudinal survey. The NLS-72 provides an example of a consortium approach: For the fifth follow-up interview in NLS-72, the National Science Foundation appended questions on math and science teachers, and the National Institute on Child Health and Human Development joined with the National Center for Education Statistics (NCES) to fund questions on child care and early childhood education issues. Longitudinal surveys are generally long term projects with significant start-up costs. If a survey can he constructed to serve more than one agency through an inter-agency agreement, start-up costs may be shared and several agencies will be bound to multiple-year funding commitments. When agencies select outside contractors to conduct longitudinal research, competitive procurement is required. The decision to use a contractor to conduct a survey increases the time needed to start a project, because approval of contracting plans must be added to other planning tasks. One advantage of contracting out the survey work is that it gives an agency access to additional staff support in cases where the agency has no authority to add permanent staff. Contracting for data collection by an outside agency may or may not be more expensive than employing a government organization for this purpose. In comparing costs, NCES found that the first NLS-72 follow-up, conducted by the Census Bureau, cost slightly more than the second follow-up, conducted by Research Triangle Institute (RTI), despite inflation. Other longitudinal surveys, including NMCES and NMCUES, have had just the opposite experience. The most cost-effective mode of operation appears to depend on the kind of survey, not on the agency conducting it. The duration of longitudinal surveys often requires periodic recompetition once a competitive award has been made. As a result, agencies have found themselves switching contractors part way through the data collection phase of a longitudinal survey. The competitive award of each data collection wave can, however, help control overall survey costs, because it provides contractors with an incentive to hold down their costs. The possibility of changing contractors over the life of a longitudinal survey requires a detailed documentation of methods that goes far beyond what is needed for any one-time survey. This level of documentation was not anticipated when the original contract to collect data for NLS-72 passed from the Educational Testing Service to RTI, and the change in contractors caused difficulties. Based on this experience, NCES now 14 builds a sub-contract to the previous contractor into any subsequent data collection awards. As a result, a later transfer of the NLS-72 contract from RTI to NORC was accomplished without problems. C. Staff Needs Staffing requirements for a longitudinal survey typically vary substantially, both by number and by type of staff throughout the history of the project. Staffing is much more controlled in rotating sample surveys, whether they are longitudinal or cross- sectional. Funding and staff needs for a longitudinal survey are much greater during the data collection period than during any other phase. However, some of the types of people needed for data collection, such as interviewers, are not needed in later phases. Staff monitors for field work and data processing are in high demand at early stages as well as intermediate stages. Because of sporadic needs, the use of a core group of survey professionals in combination with temporary staff, or interagency agreements or outside contracts, can be the best method to ensure adequate staffing for the entire effort. To distribute the costs of a contract more evenly over a longitudinal survey, NCES and NCHSR have used incrementally-funded contracts. During the longitudinal survey, separate contracts are awarded for each phase or wave. Each contract extends over two or more years. At any point, some survey tasks are being advertised for competition while others are being completed under contract. Looked at from the standpoint of each fiscal year, the total costs and level of effort remain more nearly constant. NCES has also found that giving agency survey analysts the responsibility for monitoring contract performance will help control variations in staffing patterns. By employing temporary peripheral groups in addition to permanent staff groups, two problems are solved: Research staff needs are met without adding permanent personnel to an agency; and peak workload needs are met without jeopardizing tight survey schedules. Inter-agency agreements or contracts not only bind parties to a specified set of research goals, but they also permit the level of staff effort to rise and fall as needed. D. Maintaining Core Staff The duration of longitudinal research projects creates another management problem (which has been called a Methuselah effect by Herbert Parnes). Each phase of a longitudinal study, such as planning, data collection, or analysis, is frequently carried out by different individuals, who may not even be part of the same organization. The relative inflexibility of a longitudinal study plan is an analytical necessity, but it could also prevent interim analysis or refinements in the design. For these reasons, it has been suggested that non-going longitudinal surveys may hold little interest for the calibre of professional staff that is needed for management or analysis (Wall & Williams: 35). NCES, however, has successfully attracted talented analysts to manage the agency's longitudinal surveys. To some extent this may be because NCES ensures that the Agency's staff have challenging responsibilities for program 15 analysis. Agencies which see only data collection as their primary mission may be more apt to encounter the staff problems recognized by Wall and Williams. in order to allow mid-course corrections and modifications of the survey plan, NCES uses a multi-phase sampling design (as in HS+B). This, too, contributes to the flexibility of the NCES longitudinal survey program. E. Data Collection and Processing Schedules Longitudinal surveys have become notorious for developing serious backlogs because data collection takes precedence over all other tasks. The schedule for observations is usually the least flexible aspect of the design, because each subject must have an identical record structure. As data collection continues, it creates an ever-growing backlog of other procedures, such as analysis. Uncompleted tasks tend to accumulate, becoming increasingly difficult to finish. To prevent backlogs and delays, a longitudinal survey must be well-organized and planned so that analysis and data release keep pace with data collection. Data collection schedules are not the only factor in backlogs. Another factor is data processing, including file linkage. Survey organizations that are more accustomed to doing cross-sectional surveys or other non-longitudinal surveys often have difficulty recognizing the special processing needs of longitudinal surveys. Databases need specification, key variables,need identification, and a policy on imputation needs to be thought through. Ideally, all this needs to be done when the survey questionnaire is designed, but this ideal is seldom, if ever, met. F. Data Analysis Data analysis is often looked on as the rewarding part of the job after the difficulties of data collection and data processing. Analytical interests often go beyond the agency conducting the study. Some agencies include analysis contracts in their contracting for services. Usually some analysis is done by agency personnel. One possibility to counter some,of the delay caused by the time it takes to complete a longitudinal survey is to analyze each wave as if it were from a cross-sectional survey. This not only provides timely data, but raises questions to be answered at later stages, and generally whets the appetite for more data and more analysis. Recent data from non-going longitudinal programs can be analyzed relatively quickly to serve some analytical purposes without delay. It is also possible to add questions to the current data collections of a longitudinal survey to meet immediate data needs. G. Release of Data A principal goal of any longitudinal survey should be to produce public use data tapes and analytical reports rapidly, both for policy-makers and the interested public. If public use files are to be created, then procedures to 16 protect confidentiality must be worked out in advance, File structure and documentation need to be readily available. Variance estimation must be provided for those using the file. The permanent survey staff should maintain a role in the preparation of files and reports, so that their expertise and interest are not lost. In conclusion, longitudinal surveys, sometimes taking 5 years or more to complete, inevitably encounter staff changes. Two management approaches can minimize the loss of institutional memory. First, it is vital that every survey activity be documented. Interview instructions, edit specifications, variable definitions, file layouts, sampling, weighting and imputation methodologies, all instruments and procedures should be recorded and readily available. This task is very labor-intensive and, unfortunately, apt to be slighted when staff time is short. Second, inter-agency agreements or contracts may clearly lay out both the procedures to be used and the final products. It is also wise to specify key contractor staff persons who cannot be replaced without sponsor approval. These actions are important to minimize the effect of staff changes and to prevent errors and delays. 17 18 CHAPTER 3 LONGITUDINAL SURVEY OPERATIONS The principal differences between field and processing operations in one-time surveys and in longitudinal surveys are created by the use of time as a significant factor in research. Longitudinal surveys typically encounter changing conditions, and survey designers have developed and evaluated a variety of methods for controlling the problems that can be caused by change in the sample or changes in the design or administration of the survey. A. Sample change over time The composition of the sample may be expected to change across waves for a variety of reasons. Respondents may refuse to participate, they may die, they may move and cannot be found, or they may leave the sampling frame (e.g., by entering an institutional population or by moving abroad). The danger is that the sample becomes increasingly less representative of the target population as time passes. To minimize the effects of these problems, new observational units are routinely introduced into the samples of some continuing surveys as time passes. 1. Selection of new units into sample For some longitudinal surveys, they are a number of concerns related to the length of time respondents are kept in sample. Respondent burden across several interviews may produce a decline in the quality of data gathered or may result in increasing refusal rates. Respondents may also leave the sampling frame, move and cannot be tracked, or die, thereby affecting the representativeness of the sample. for these reasons, it may be desirable to institute a rotating panel design, which regularly moves new respondents into the sample and retires other respondents after a fixed number of interviews or period of time. The Survey of Income and Program Participation (SIPP), the National Crime Survey (NCS), the new Consumer Expenditure Survey (CE), and the Consumer Price Index (CPI) have all adopted rotating panels. SIPP introduces new respondents annually and retains them for 2-« years (7 or 8 interviews) before rotating them out; NCS introduces new respondents monthly and interviews them for 3-« years (7 interviews). The CE Survey introduces respondents monthly and interviews them five times on a quarterly basis, while the CPI introduces new respondents once every five years and interviews monthly or bimonthly. Fienberg and Tanur (1983) note that rotating panel designs may create some problems of inference, according to conventional sample survey theory, in that random selections of respondents occur at different times for different respondents. The argue, however, that this is only important when date of selection is related to temporal changes in the phenomena the survey was designed to measure. The inferential Principal Author: Bruce Taylor 19 difficulties which might result from a rotating panel design must be balanced against the reduction of attrition-related bias, which is the alternative. 2. Movers Some respondents may be expected to move from originally sampled housing locations (or telephone numbers) during their time in sample. Depending on the purpose of the survey and procedures adopted to track movers, respondent mobility has varying implications for the representativeness of the sample over time. a number of factors may enter into decisions regarding whether, or how, to follow movers. A crucial consideration is to determine the most important unit of observation for the survey. A longitudinal survey of persons may be designed to follow sample individuals or households, if the substantive goals of the survey would be served by retaining as many of the originally sampled respondents as possible. A number of surveys, such as SIPP and NLS, focus on individual and household economic data, which continue to be relevant to the purposes of the survey regardless of respondent mobility. Consequently, following movers is an appropriate means to maintain data quality over time for such surveys. Following movers may create other problems, however. For instance, if there are ecological correlates for the phenomena of interest, such as crime or quality of housing, then following mobile respondents may result in deterioration of the geographic representativeness of the original sample, with a consequent potential for bias in some measures for later waves. A rotating panel design may minimize this problem, because newer respondents are more likely to reside in the originally sampled housing location. Another reason for following movers is that respondents may move for reasons related to the substantive goals of the survey. This makes it important to know why they move. If this is the only reason for following movers, then collecting data for only one wave after a move may be enough. In NCS, for example, some respondents may move from a high-crime area to a safer neighborhood, and it would be important to determine the proportion of moves which were related to crime victimization can be measured, but not the future consequences of victimizations for such movers. The SIPP is attempting to follow all individual movers. Because living arrangements vary according to economic circumstance --and affect eligibility for social welfare programs -- a change in residence can be related to changes in income and program participation. Thus, for SIPP it is crucial not to lose data on movers. The CPI, on the other hand, follows only those movers who provide services, such as doctors or lawyers, since their expertise is the item being purchased. When a commodity outlet changes location, this move is considered a unit "death" and the CPI record is terminated. The actual procedures developed for following movers are likely to reflect the field procedures of the organization conducting the survey, the collection mode used, the distance involved, and the costs associated with tracking movers. If the organization conducting the survey uses decentralized collection procedures, a respondent moving from the jurisdiction of one regional office to another may be more difficult and more expensive to track. Also, the costs of following movers may be greater if a face-to-face collection mode is used, rather than a telephone design, where tracking procedures may 20 be limited to obtaining a new telephone number. Depending on the cost, administrative difficulty, and proportion of respondents who move far enough to create problems, it may not be desirable to follow all movers or to rely on standard collection modes. SIPP field procedures, for instance, indicate that personal interviews need not be administered if the respondent has moved beyond 100 miles from any sample PSU, and rules also differ for respondents younger than fifteen years of age. If survey procedures allow telephone interviews in lieu of face-to-face interviews, a phone contact may be a desirable alternative for movers who are difficult to reach. The type of sample involved may also affect the ease with which movers may be located. For instance, it is usually easier to find a mover through neighbors or subsequent occupants of a sample housing unit if an area sample has been adopted rather than with a random digit dial sample. Asking respondents to notify the field office with pre-printed cards when they move can be a partial solution, but this option relies heavily on the respondent's cooperation. 3. Attrition When projected across waves of a longitudinal survey, manageable levels of non-response in a cross-sectional survey can become significant sample attrition. The potential for attrition in a longitudinal survey sometimes limits sample definition. Tracing mobile respondents generally accounts for a large proportion of field problems as well as costs, and refusal rates are likely to grow over the life of the survey. Incomplete records and missing interviews create analytical complexities that are unparalleled in cross-sectional research. Attrition is most dangerous when it is correlated with the objectives of the survey. For example, there is evidence that sample attrition may be related to victim status in the NCS. To the extent that the sample loses victims at a faster rate than non-victims, estimates from later waves will be biased. Also, Fienberg and Tanur(p.17) note than in social experiments disproportionate loss of respondents for different treatments may be a problem, because treatments often vary in their attractiveness to participants. Sample attrition between observation periods may create the illusion of change when means are compared between waves, without adjusting for non-response. In study focused on identifying change, there is a risk that changes are spurious, due to sample attrition. In addition, respondent participation that varies from panel to panel could produce the appearance of change even when aggregate non-response is stable. The estimates of central tendency (Cook & Alexander: 191). Mean test results from longitudinal panels of students taking ETS exams were compared to mean test results derived from a cross-sectional survey of the same population. The means were significantly different, which the analysts attributed to selective attrition in the longitudinal sample. Effects of attrition in demographic surveys have been harder to predict. Attrition does not necessarily created unmanageable bias in a longitudinal survey: The NLS was still contacting 92 percent of living respondents 3 years after the original contact, and still contacting 80 percent of eligible respondents 12 years after the study began (U.S. Department of Commerce:321). In the ISDP panels of 1978 and 1979, attrition did not climb steadily over the five or six interviews administered to respondents. Instead, it leveled off and then declined slightly over all waves (Ycas:150). Nonetheless, a combination of attrition and varying participation from wave to wave can create serious 21 problems in creating complete records. In the 1979 ISDP panel, for instance, only two thirds of the original sample persons had complete interview records (Ycas:150). Calculating the response rate in longitudinal surveys is itself difficult. The measures used in cross-sectional research are often not adequate for measuring non-response in complex records, as they do not reflect cumulative non-response across waves and do not take into account changes in the size of the eligible sample due to births, deaths, and the addition of new household members. The illustrate, non-response for entire housing units in the NCS is sometimes reported at 4 percent. However, when records for housing locations are linked to form a longitudinal file, it has been found that over half of the originally sampled housing units are missing at least one interview. This discrepancy is due to the fact that the former figure is a cross-sectional measure of unit non-response in a particular wave and does not account for the approximately 10% of sample housing units unoccupied at the time of interview (Fienberg & Tanur:14). This figure also dies not cumulate non-response over time. While the lower figure is an appropriate measure for many cross-sectional uses of NCS data, it clearly is inadequate for reflecting the completeness of linked housing unit records. The methods that have been developed for tracing respondents in longitudinal surveys have been successful, but they have also proven to be expensive. The Census Bureau has estimated that the cost of contacting each wave of an ISDP research panel increase by 8 percent over the previous wave, due to the costs of following movers and interviewing additional households (Fienberg & Tanur:11- 12, White & Huang). However, NCES also found that per-unit tracing costs for the High School and Beyond (HS&B;) Survey were approximately 20% less than the cost of base year sampling, which illustrates the economies which can be realized by mounting a longitudinal study, rather than separate cross-sectional studies. To control costs, as well as potential bias, each longitudinal survey must investigate the characteristics of respondents who move. Depending on empirical evidence about how atypical non- respondents are, a judgment can be made about the proper balance between the costs of tracing respondents and an acceptable level of non-response. Sample definition offers another approach to limiting unscheduled attrition. The probability of becoming a non- respondent is not randomly distributed among the population. In longitudinal samples such factors as rural resident, interval since contact, and region of the U.S. affect the probability of maintaining contact (Artzrouni:21-24). Some longitudinal designs have therefore sought to minimize attrition by avoiding the respondent classes that are most susceptible to attrition. Setting aside respondent classes to control attrition can conflict with attaining a sample that truly represents the reference population. However, a sample chosen without regard to eventual tracing difficulties may also gradually lose its representative power through attrition. Only empirical evidence can indicate the extent to which characteristics that predict attrition co-vary with the characteristics that the study is designed to investigate. A sampling design which sets aside respondent classes with potential attrition problems should be undertaken only after careful consideration of the relative magnitude of bias which could be introduced by such a strategy and other alternatives, such as imputation for missing data or performing analysis on the remaining sample cases of an initially representative sample. In cohort or panel studies, which require measurement to begin and end at the same time for all respondents, implementation of a rotating panel design, which reduces the impact of attrition by replacing respondents over time, will clearly not serve the goals of the survey. One possible strategy for dealing with attrition in such studies is to impute. 22 missing data, based either on statistical models or on complete data from prior waves or from respondents with similar characteristics. Another possibility is to reweight the sample for each wave to reflect non-response for various demographic groups in the sample. (See Chapter 4.) Duncan, Juster, and Morgan (1982) model such a procedure for the Panel Study of Income Dynamics (PSID), conducted by the Institute for Social Research (ISR) at the University of Michigan. They compare results for data gathered with persistent efforts to pursue respondents and for the data set which would have resulted if less intensive respondent contact strategies had been adopted. When the latter is reweighted to adjust for missing cases and compared with the first data set, there are minimal differences in outcome measures. While this procedure has promise for minimizing bias resulting from non-response across waves, it may also allow some relaxation in pursuing respondents, allowing cost reductions in survey administration. The authors do note, however, that reweighting entails some risk of covariation-related bias in multivariate estimates, especially for models that are not well specified, and that maintaining an adequate number of respondents in some key subsamples may remain a problem. A reasonable precaution to minimize the deleterious effects of sample attrition is to minimize respondent burden, which has been variously described as the amount of time which an interview entails or as the complexity of the task required of respondents for successful completion of an interview. Under the Paperwork Reduction Act of 1980, each Federal statistical program is restricted to a limited number of hours available for data collection in a fiscal year, thereby encouraging reduction of the burden placed on respondents. In addition to the statutory reasons for limiting the length of Federally sponsored surveys, controlling respondent burden may also improve data quality for longitudinal surveys in a number of ways. An important aspect of this data quality enhancement is that, respondent participation may be encouraged by reducing interview tedium, thereby reducing refusal rates and enhancing the representativeness of the sample over time. Respondent burden hours may be reduced by a careful evaluation of the utility of collecting information in every wave. The SIPP, for example, minimizes respondent burden by dividing the survey into a core questionnaire ad ministered at each interview, plus "topical modules" to collect data not required as regularly. Sometimes only a subsample of respondents should answer certain topics. Finally, lengthening and/or varying the intervals between waves should also be considered as a means for reducing respondent burden. The CPS, while not a longitudinal survey, adopts this strategy of varying tim e between interviews. Respondents are interviewed for four months in succession, not contacted for the following eight months, and then interviewed for a final four months. 4. Changes in Units of Observation A slightly different sample of respondents participates in each wave of a longitudinal survey. Such changes in sample may result from scheduled introduction or retirement of sample units in a rotating panel design, from attrition, or from introducing new respondents when household composition changes. This variation causes difficulties related to defining the correct reference population, in weighting for item non-response, and in weighting respondents who enter and leave the sample. In addition, the changing sample of respondents and aggregate units creates unique difficulties in analyzing data above the person level A variety of approaches has been used to define units of analysis in longitudinal research, and each has specific problems and strengths. These are discussed in detail in Chapter 4. 23 It should be noted here, however, that all weighting adjustments should be planned simultaneously. The problem of adjusting for non-response is the converse of problems created by persons entering the sample, and the adjustments for entrants and non-coverage, once selected, can be accomplished in a single operation. Split and merged households present particular problems for sample comparability across waves. Such recomposition of households creates obvious difficulties for longitudinal matching, which will be discussed below. However, changes in household membership also raise questions about how to treat new members of split households who were not members of the originally sampled household but who came into sample because of their associations with original sample persons. Rules developed by the ISDP offer one method which seems generally applicable to a number of surveys: New household members were added to the sample, but if they left the household, or if this household subsequently split, only those members who were selected for the original sample were followed. This procedure avoids excessive growth of the panel, thus minimizing artifactual changes in aggregate panel statistics, but still collects relevant household data which correspond to data from "stable" households. Whether a change in a household constitutes the birth or death of the sample unit depends on the goals of the survey. If the survey samples households and does not follow movers, then a complete turnover in the household occupants would indicate the birth of a new unit. If housing locations are sampled, then such a turnover would not constitute a death as long as the hosing unit remains occupied. The death of a member of the household, or event he head, does not constitute death of the unit for a household- based sample, but a divorce or separation often will be defined as termination of the unit. If an individual respondent leaves the sample, the reason for the departure should be determined. If the respondent has died, then the individual record should be terminated. However, if the respondent leaves the sampling frame for other reasons (e.g., entering the military or moving abroad), it is possible that he or she may return during the life of the panel, and the record should be retained. Often the death of a unit can be determined by observation. For instance, when a housing unit is vacant or destroyed and the sample is location-based, termination of the record may be indicated. However, in other cases respondents must be queried regarding the status of the unit. If the unit of measurement is the household, occupants of the sample location must be asked whether they lived at the current address when the previous interview took place to determine whether they should be considered part of the sample. (Rules for this decision will vary between surveys.) If only part of the household has moved since the previous visit, it may be necessary to determine the reason for the departure to ascertain whether the movers remain in the sampling frame. In designs which do follow movers and which allow the formation of new households during the life of the sample, permanent departure of individuals to form new households will indicate the need to establish new household records. (See Chapter 4 for a fuller discussion of these issues.) B. Changes Related to Respondents' Time in Sample Varying sample participation is not the only change over time which complicates inference from longitudinal data. A number of factors related to the time respondents remain in sample may produce changes in survey measures which are independent of any substantive changes in the phenomena under investigation. These factors include variation over time in the rules for interviewing particular respondents and changes in 24 respondents' approach to the interview based on increased experience with the survey instrument as the sample matures. 1. Response Variability Due to Changes in Respondent The manner in which a survey is administered may vary from respondent to respondent. "Proxy" interviews may be administered, in which adult household members complete interviews on behalf of younger respondents, or in which available household members supply data for other individuals in the household. (In some cases such proxies are restricted to household members who are not present, but, in other instances, one household member will supply personal data for all individuals in the household.) Respondent rules are also frequently needed for collecting household information if there is more than one respondent per household. A number of possibilities exist for respondent rules. For example, one respondent in a household may be selected to provide household data, while personal data is requested from each respondent individually. Alternatively, all respondents may be asked for household data. In the latter case, inconsistencies might be reconciled in the field, for instance, when respondents report conflicting details regarding a household crime incident. A computer edit, or a postweighting algorithm might also adjust for differences in reporting, when household measures are simply the sum of individual measures. Respondent rules can affect longitudinal data over tim e. For instance, during a longitudinal survey, younger respondents may become eligible to complete an interview without proxy, and may begin to report information of which previous proxies are unaware. There is also evidence that household-respondent status may affect the manner in which personal data are reported, particularly if the two types of information requested are related. Biderman, Cantor, and Reiss (1982), for example, find that respondents who report household data also report higher levels of personal crime victimization than do respondents who do not report household data. They also find that, if the household respondent changed between interviews, levels of personal victimization for the affected persons would also change. The authors hypothesize that the initial battery of household victimization items serves as a warm- up for personal items and aids recall for household respondents. If the household respondent is allowed to change across waves, then two effects should be anticipated. First, the quality of personal data reported by a given respondent is likely to change over time, depending on whether he or she serves as the household respondent. Second, different household members will vary in their knowledge of the relevant data, so the quality of household data may also be expected to change over time and thereby bias transition estimates. There are some obvious remedies for these problems. First, proxy interviews should be minimized, recognizing that obtaining certain information directly from younger respondents may be inappropriate or that there maybe no other way to collect data for some respondents. Surveys vary in their reliance on data collected by proxy (eg., about 60% for NCS, 40% for SIPP), and such a policy is likely to produce an improvement in data quality proportionate to the fraction of data currently collected in this manner. Second, care should be ta ken in assigning responsibility for answering questions about the household over time, either by consistently assigning this responsibility to the same respondent or by requesting these data of all respondents. The latter procedure minimizes the effect of an unavoidable change in household respondent and makes any respondent effect consistent across all waves however, due to mandated 25 ceilings on response burden for federally sponsored data collections, the additional precision realized may not justify the substantial number of redundant questions which are required. It should also be noted that the reconciliation procedures or post- weighting that would be required may make such a strategy very difficult to use. 2. Panel Bias A number of factors associated with respondents' time in sample may produce changes in survey measures over time and thereby complicate explanation. The impact of these factors has been described as a history effect, secular effect, maturation effect, rotation group bias, time-in-sample bias, or Heisenberg effect. These factors include the reactivity of respondents to survey measures, changes in the performance of the respondent role, the "conditioning" effect of multiple administrations of the survey instrument, the aging of the panel, interaction between interviewers and respondents, interviewers' perceptions of their role, and the correlation between variables of interest and the probability of response. Changes in survey measures due to such effects present a danger for bias in longitudinal estimation. Consequently it is important to consider the influence of such factors when designing a longitudinal survey and to minimize the potential for such changes. This is a difficult task, because the reasons for the phenomenon are not clearly understood. Ideally, the process of measurement should itself produce no change in the phenomenon under investigation. Research methodology in experimental psychology, for example, often involves disguising the purposes of research, so that the subject will produce the behavior under investigation with minimal "contamination" by the research procedure. In survey research, however, the respondent must not only understand the measures being collected but also must be led to appreciate the purposes and value of the research if response rates are to remain high. This is particularly important for longitudinal surveys, where retaining sample is a crucial goal Consequently the danger of reactivity between survey interviewing and the phenomena under investigation is a particular problem. Researchers studying labor market experience, for example, have speculated that repeated interviews asking about job mobility might cause some of the mobility reported (Parnes:15). Questions about mobility may in fact cause subjects to consider the possibility and act upon it. National Crime Survey data also indicate that proportionately fewer crime incidents are reported in successive waves. This finding may stem from respondents' heightened awareness of vulnerability to crime, caused by participation in the NCS, which results in increased precautions taken against crime victimization. It has been suggested that respondents in a longitudinal sample might exhibit non-typical behavior Simply because repeated questioning regarding a topic may alter respondents' perceptions of the subject under investigation and change their behavior or attitudes accordingly. For respondents no remain in sample, their responses can change over tim e solely as a function of longevity in the panel These temporal variations in response have implications for the quality of longitudinal data which are often unpredictable. In some cases, the quality of data may improve over time. Respondents may understand the respondent role better with repeated interviewing or pay greater attention on a day-today basis to the experiences being measured, with a consequent improvement in the richness or accuracy of the data gathered. Alternatively, if respondents or interviewers find the interview tedious or burdensome, they may become less enthusiastic about the 26 task over successive waves and avoid or give incomplete responses to survey items. One aspect of such a decline in data quality is the possibility that respondents may be "conditioned" by their participation over several waves to provide answers which produce artifactual changes over time. For instance, respondents may learn that a particular response will trigger a long battery of questions, which they may prefer to avoid in the future. This is one alternative explanation for the decline in the rate of crime victimization reported in the NCS over successive waves. Respondents may learn that reporting a crime incident leads to an additional series of items for each incident reported, which results in a substantially longer interview. The Census Bureau's Current Population Survey (CPS), which is not strictly a longitudinal panel survey but which has many of the attributes of a longitudinal survey, exhibits a similar trend. Reporting unemployment triggers a battery of questions dealing with reasons for unemployment and activities directed towards looking for work. Reported unemployment invariably falls between the first and second waves of interviews in the CPS. This phenomenon in CPS could be related to several factors. One has to do with repeated interviewing and attrition. Williams and Mallows showed that, if the probability of response in a given save of interviewing was correlated with variables of interest, then, even with no change in the variables, a spurious change would occur. The passage of time can also produce unintended change between observations because of gradual shifts in the meaning of questions and answers. Even when questionnaires are not changed, there may be evolution In the way respondents perceive or answer questions, which produces the appearance of movement (Parnes:14). This might be caused by events (including the survey itself), by maturation in the sample, or by non-response. It is very difficult to determine whether a change across waves is real change or spurious change. Continuing validation research is necessary to identify panel bias in longitudinal data. Panel bias may be studied by comparing data collected in subsequent waves of a longitudinal survey to data collected in cross-sectional surveys (as in Cook & Alexander). Although some conditioning or panel effects may be inevitable, several tactics can be used to minimize their impact. One option is to implement a rotating panel design to replace respondents after a predetermined number of interviews. This procedure affords two primary benefits. First, those respondents who have been in sample the longest are replaced with more "inexperienced" respondents. Second, the temporal overlap of old and new sample facilitates studies of time in sample effects. All respondents are administered the same instrument under the same conditions at the same time, which serves to test alternative hypotheses about panel effects. Another possible means to attenuate or postpone the effects of panel bias is to minimize the respondent burden imposed by the interview. Careful construction of the instrument to minimize tedium and encourage respondent rapport should be central concerns in planning any survey but take on added importance in longitudinal data Collections, because of the need to sustain the active participation of respondents overepeated interviews. The overall length of the instrument may play a role in the respondents willingness to participate fully in successive contacts. However, design of the instrument to minimize tasks which the respondent is likely to find either tedious or particularly difficult is also an important consideration. Use of long follow-up batteries should also be minimized, to attenuate the effects of respondent conditioning. 27 C. Operations Change Over Time Changes in the administration of a continuing survey are almost inevitable. Revisions to the instrument, redesign of the sample, introduction of new collection modes, and transfer of data collection responsibilities to another organization can all introduce changes in the data and compromise the validity of longitudinal comparisons. While a consistent time series may be difficult to maintain under such circumstances, means exist which allow the analyst to deal with the effects of such changes. Eventually in most longitudinal research there is a pressure to change the survey measures in response to changing hypotheses. In addition, later findings frequently indicate a need for measures of new variables. Particularly when longitudinal research is exploratory and designed to identify significant correlates of change, researchers may be inclined to correct large a mounts of data to minimize future requirements for change in the questionnaire design. This aspect of longitudinal research may be costly, but it is an understandable precaution given the tendency for research hypotheses and/or policy-aims to change over time. To accommodate changing methods, a survey may be run under old and new procedures simultaneously for a period of time, to allow comparisons between data collected before and after the change. Ideally, both old and new designs should be implemented at full sample, in effect twice the usual sample size, but budget constraints will often make this impractical The CPS has adopted this double-sample strategy to phase in new samples based on the 1980 Census. The CPI also used both old and new sample designs simultaneously for a six- month period in 1978, when the survey was revised. Another strategy to consider when a questionnaire item is rewritten or a derived variable in a file is altered is to make changes in such a way that analysts may record the revised variable to correspond to the original variable (and vice versa), or to retain old questionnaire items in the revised instrument for some time. NCES adopted the latter strategy for the HS&B; survey when it adopted an "event history" approach to gathering employment and education data. In addition to the new items, the previous "Point in time" activity item was continued, allowing calibration of new items to the old and providing a degree of comparability between versions. To reduce field costs, many sponsor agencies have approved designs which permit data collection by telephone after the first visit. NMCES and MNCUES, for example, used phone contacts for follow-up interviews. The available evidence suggests that such changes in mode may not produce uncontrollable fluctuations in the measures obtained. Benus (1975) notes that data collected by telephone and by personal visit for the Panel Survey of Income Dynamics (PSID) are quite similar. Groves and Kahn (1979) found overall that univariate distributions and bivariate relationships were not significantly different for 200 questions ad ministered by telephone and in person. However, they note that telephone interviews elicited more rounded financial figures, less detailed responses to open-ended questions and narrower distributions on some attitude items. They also indicate that respondents tend to perceive telephone interviews as longer than personal interviews of the same length. Findings that telephone respondents tend to give more "don't know" answers to filter questions triggering other questions may be related to this difference in perception of length. Telephone respondents may be more eager to bring the interview to a close. Consequently minimizing respondent burden seem s particularly crucial for interviews conducted by telephone. 28 While the research literature on the effects of interviewing mode on survey response is generally encouraging, there are enough examples of differences in respondent behavior to indicate that a mixed mode design should not be implemented without adequate pretesting and analysis of the effects. One danger is that a particular questionnaire design or questions about a certain subject area might trigger mode-related differences in respondent behavior. To facilitate measurement of such mode-related response variability, it is desirable to design shifts in mode of data collection so that the changes across waves are systematic, making the effects measurable. It is also important in surveys which do not require interviews with all household members to ensure that interviews are obtained from the same household members when the interviewing mode varies across waves, as respondent availability may vary by mode. In conclusion, prospective longitudinal surveys require administrative and operational features that are different in kind as well as degree from those in cross sectional research. The long-term analytical goals of the survey must be considered in planning every aspect of sample definition and weighting. Provisions should be made for validation studies to evaluate such factors as attrition and panel bias. Finally, changes in format, operations and staff must be anticipated and managed in ways that ensure the comparability of measures from wave to wave. In practice it is worth noting that there are only a limited number of organizations which handle nearly all large-scale longitudinal surveys. Due to their experience, these organizations have a high level of expertise, and the continuity of experience contributes to successful planning and implementation. However, the concentration of longitudinal research in such a small number of organizations increases the impact that any errors, such as limitations in the sampling frames most commonly used, would have on the representativeness of longitudinal research. D. Processing While the measures collected in longitudinal research may be similar, to those collected in cross-sectional studies, there are special problems in controlling and interpreting them. The sheer size of the data files created in national longitudinal surveys creates special problems in processing and analysis. The massive files can be difficult, expensive, and slow to process, which has often limited their use to organizations with the staff, equipment, and often complex software capable of handling complex data sets. As a result, data analysis has typically lagged behind the accumulation of data (Kalachek:17). Fortunately, this situation is changing with the advent of public use files for multivariate analysis and with the dissemination of m ore user-friendly "statistical data base" packages to facilitate data management and analysis. In processing data from longitudinal surveys, difficulties are encountered related to cross-wave case matching, cross-wave data revisions, and preparation of data files for analysis. Often there is no single "best" procedure for processing, because ease of processing and analytical requirements are not always compatible goals. Errors in individual record files can cause multiple problems. Often items which should remain consistent across waves (e.g., race and sex) or which should change only in predictable ways (like age and marital status) will exhibit changes due to respondent confusion, transcription error by interviewers, or keypunching errors by processing staff. Detecting these errors is important, not only because such items often define key 29 demographic variables for analysis, but because such items are frequently needed to match cases. Errors are also inevitably introduced when imputations are made for missing data. Several procedures are possible to minimize errors. For SIPP, the field office staff immediately checks completed interviews to reconcile discrepancies, avoiding more costly correction of data after they have been keyed. Another possible procedure is to build computer edits into the processing system to detect inconsistencies between current and prior interviews. NLS-72 and HS&B; use machine edits to identify and resolve inconsistencies for about thirty critical items. Another option, utilized by CPI, is to create a machine-generated control card, which avoids errors in transcription and which provides interviewers with prior-wave data necessary to reconcile discrepancies in the field. This latter procedure, however, can also lead to reduced reporting of actual change. 1. Cross-Wave Matching In order to link data across waves, variables must be created to match records at the desired unit of analysis. A number of data management issues must be addressed, including the consistency of linking variables across waves, providing for longitudinal matching at multiple levels of analysis, and rules for matching merged and split households. If longitudinal records are not matched correctly between waves, the effects can be similar to sample attrition or non- response. The records of one or more observations will be missing from a respondent's longitudinal file, giving the appearance of missing interviews. One possible consequence of matching errors is error in analysis, either because incomplete records are deleted, or because missing data are imputed. If records are linked incorrectly, longitudinal data are also likely to produce flawed results by showing false changes in status. Even cross-sectional analyses may be in error, if control card information or data from previous interviews are carried over onto the improperly matched record by the processing system. A number of procedures are possible for linking units accurately from wave to wave, including matching of household and individual line numbers, or matching independent person and/or household identification numbers. Economy in the number of variables used for a match is generally a virtue, because the opportunity for mismatches due to transcription or coding errors increases with the number of variables used. So does the likelihood of missing data, which often results in the computer assigning a missing data code, which hampers matching. Limited redundancy in linking variables can, however, provide some protection against false matches, in that such cases are more likely to be flagged in the matching process. Validation procedures to detect longitudinal mismatches should be incorporated into the processing system and can often rely on demographic variables which either should not change over time (e.g., race, sex, or date of birth) or which can be expected to change in predictable fashion (e.g., marital status or age). Such methods are particularly useful when person-level matching is performed using the assigned line number of respondents within household. It is also useful to imbed check digits in key linkage numbers, to detect miskeying. In addition to careful design of validation variables, immediate error checking by the field office of items important for matching and validation is likely to reduce the number of mismatches significantly. 30 Often, person records are linked across waves by matching on household ID and on the line number of an individual within the household record. This is usually cumbersome, and it makes linking individual data across waves extremely difficult if an individual moves out of the sampled household, if the household dissolves, or if the household merges with another household, all of which render the previously assigned household ID obsolete. Consequently, for surveys which are intended to follow individuals, regardless of the duration of their association with a sampled household or household location, assignment of an independent person ID is highly desirable. This is not to argue that ID is at other levels of observation are not useful, as longitudinal analysis at household, person, or event level is often needed. The important consideration is that linking variables be designed so that changes in sample composition do not prevent record matches. SIPP has implemented an ID which, while complex, illustrates the sort of linkage which is often desirable. (Cf Jean & McArthur, 1984). The ID consists of: PSU number - 3 digits Segment number - 4 digits Serial number - 2 digits Address ID - 2 digits Entry address ID - 2 digits Person number - 2 digits Household ID consists of address ID, PSU, segment, and serial numbers. The latter three numbers are fixed once assigned. The entry address ID also does not change. The first digit of the address ID indicates the wave at which the household was interviewed at that address. The second digit sequentially numbers, by address, households resulting from a split into two or more households by original sample persons. The first digit of the person number indicates the wave at which the respondent entered the sample, and the second two digits sequentially number persons within the household. This ID also remains fixed. Linking households or individuals with the SIPP system is fairly straightforward. Households whose composition does not change require the household ID, and individuals require the household ID and person number to provide a match. The inclusion of a fixed entry address ID also facilitates matching records for individuals or households who move, and for split households. Combining the person number and the entry address ID provides a person number which remains constant regardless of changes in address and household composition. This provides a link to data collected for an individual across all waves, allows a match to the initial household, and permits the analyst to filter data for only the original survey respondents, if desired. This system remains adequate for multiple movers or for households which split a number of times. In 1979 two waves of interviews from an ISDP panel were merged into a single longitudinal file using personal identification variables. Mismatching between records proved to be a significant problem, and there was evidence that additional matching errors were undetected (Kalton & Lepkowski:26). A second file was created using ID numbers rather than personal characteristics. This file had significantly fewer discrepancies during edit checks for such items as sex and age, indicating that fewer matching errors occurred with the use of the ID number for linking. 31 Sometimes the potential of longitudinal data has not been exploited because of the complexities involved in updating data with information collected in subsequent waves. For instance, a respondent may report a crime victimization or a health problem, but information on insurance coverage will remain incomplete, because the claim had not been settled at the time of the interview. It is frequently desirable to revise or add data during a later interview and to create an automated control system which would allow revision of the original record. One possibility is to provide a check item on the instrument for information which is frequently incomplete. The control system could then flag incomplete data during processing and direct the interviewer to follow up on this question in a later wave. Similar procedures were used in N M C E S and N M C U E S, which allowed validation of data collected on health care payments and insurance coverage during later interviews. Revising files obviously creates some complications, and there are trade-offs between ease of processing and ease of analyzing the revised records. One of the simplest procedures for processing is to reserve a field for follow-up data in the interview along with an incident or event ID which allows a match to the original record. This procedure unfortunately would make the analyst's task considerably more difficult, in that several files would have to be scanned to locate all updated material. The required matching and file restructuring routines would also be rather cumbersome and expensive to run, unless the data were released in a form compatible with a statistical data base which performed the matching. These complexities create potential for data management errors, particularly for inexperienced users accessing public use files. The alternative is to correct the original records based on followup data and to release the updated files. A disadvantage of this procedure is that several versions of the same, file would be in circulation.* Nonetheless this procedure appears to have greater potential for facilitating straightforward analysis and management of the data, particularly if early versions of a file are labeled as "preliminary." 2. Data Structures to Facilitate Analysis A number of strategies may be used to create longitudinal data files. One is to create, a separate fixed length record for each case at the smallest unit of analysis, with separate fields devoted to repeated measures of the same variable. Often this is not feasible, because this procedure entails a thorough revision of the file every time a new wave is completed. It is often preferable to produce a separate file for each completed wave or even more frequently if data collection extends over a lengthy period and to include in the files a number of linking variables which remain constant for each case across waves. Other than the size of the files produced, the main difference between these two approaches then is in the processing system adopted: The former produces Integrated longitudinal files, while the latter produces files resembling crow-sectional data sets which allow the analyst to link the records later. Producing a file which uses the smallest unit of observation as the basis for a record is often not the most efficient structure for a data set. A number of surveys ________________________________ *This is not as serious a problem for longitudinal files, the latest version of which can more easily be identified, as it is for cross-sectional files created from a particular wave. 32 collect data on households, individuals within households, and discrete events experienced by the household in aggregate or by individual members. Given the implicit "nesting" of such data, creating a file based on the smallest unit will result in much redundant information for higher level units. The number of events recorded and the number of household members may also be expected to vary between households, and variable length records will result, necessitating extensive "padding" to create a rectangular file. A more efficient strategy in such cases is to produce hierarchical files with the data pertaining to each level of observation appearing in separate records and with variables appearing in more than one type of record to allow for linkage across levels. A number of software packages such as SAS and OSIRIS now exist which can process and analyze such files. In addition, a number of "statistical data base" packages are available, such as SIR, Canada's RAPID, and Mathematical Policy Research's R A MIS, which provide sophisticated capabilities for matching across waves and levels, and which thereby simplify the analyst's data management tasks in working with longitudinal files. Decisions regarding the optimum structure for a longitudinal file also need to take into account the expected size of files. Limits on the number of records many soft ware packages can process may be exceeded by the size of large federal data collections. Consequently, file structure options for facilitating analysis of longitudinal data may be constrained. Sponsors may find it necessary either to forego compatibility with some otherwise useful software packages or to release subsets of their data to provide compatibility with a wider range of software packages. 3. Confidentiality Processing operations and data structures for analysis cannot be designed solely to reduce costs, complexity, or bias. They must also protect respondent privacy as far as possible. This is sometimes not compatible with maximum efficiency. Procedures for protecting confidentiality of paper records and of tape records must be thought through carefully. The problem of maintaining respondent confidentiality is more difficult in longitudinal surveys than in cross-sectional surveys. In cross-sectional research, the confidentiality of a response can be protected by stripping responses of identifiers at an early stage in processing. In longitudinal surveys, response records must be linked to personal identifiers, sometimes for decades, until data collection and analysis are complete. Longitudinal records commonly contain multiple identifiers in order to facilitate tracing and to ensure that records can be matched after each wave, regardless of missing data. Name, address and Social Security number are often augmented with the name and address of family, neighbors, or friends who are to be contacted in tracing respondents who have moved. The large number of identifiers, plus their dispersion across records and across time, makes protecting confidentiality in a longitudinal survey far more difficult than in cross-sectional research. However, most research organizations have learned over the years how to protect paper records. An illustration of one solution to problem is that adopted by N C ES for the NLS-72 and HS & B: Identifiers are stripped from the tape prepared by the contractor before it is turned over to the sponsor agency. These data are maintained by the contractor but may only be used with the explicit approval of the sponsor. The procedure provides a complicated, layered procedure which inhibits any unauthorized access by sponsor, contractor, or public users and provides protection similar to that of a cross- 33 sectional study. This example illustrates a number of the basic safeguards which should be integrated into any longitudinal data collection effort. First, identifiers should be used only to maintain the quality of the data, e.g., for tracing respondents or for matching purposes. Second, only staff performing these functions should be allowed access. Hardcopy media containing identifiable data should be stored in a secured area to limit access. Electronic files should be similarly secured and, when in use, access should be restricted by the operating system to authorized processing personnel only. Third, all privacy- relevant data should be stripped from public use tapes before release. Ideally, the collection agency should separate identifiers during processing and store them on a file separate from the substantive data. Finally, when data Section is complete, all copies of identifiers should be destroyed. Even when such measures are taken, agencies and research organizations must consider the possibility of confidentiality breaks. The quantity of information available about respondents creates the possibility that a series of rare responses can identify respondents. Current research in confidentiality is addressing this problem and should provide useful guidelines for enhanced security measures in the near future. 34 CHAPTER 4 SAMPLE DESIGN AND ESTIMATION There are many issues in the design and estimation strategies for longitudinal surveys that are identical to those for cross- sectional surveys. Some issues, however, such as weighting and compensating for nonresponse become more complicated with a longitudinal survey. Usually the complications arise because of the changing nature of the population, as discussed in Chapter 3. In this chapter, we discuss some of the major design and estimation problems, many of which need more research. A. Defining a Longitudinal Universe Defining the initial study universe for a longitudinal survey is no more complicated than defining the universe for a cross- sectional study, The initial universe is fixed at a specific point in time and is explicitly d fined. Sample units can be selected and the only difficulties are related to the sampling frame itself. Time, however, gradually complicates the problem of defining a longitudinal universe. The study universe usually does not remain constant over the period of the longitudinal survey, as was discussed earlier., The universe of individuals, households, families, or establishments changes over time. If a universe changes slowly along the critical dimensions of the survey, the problem of a longitudinal universe definition may be ignored. However, if changes in the universe over time are not trivial, a static universe definition may not be sufficient. The choice of definition for the longitudinal universe will have a direct effect on data collection and analysis. Judkins et al (1984) describe three methods for defining a longitudinal universe. These ideas are generalizable to any longitudinal study of persons or other units. One method for defining a longitudinal universe is to select a specific time during the course of the study as the point that defines the universe. If the universe is defined at the time of sample selection, it is called a cohort study. Units in the sample are defined at the time of the first interview. At later waves of interviewing, data need be collected only from these units. All inferences and estimates refer only to the universe in existence at the time of the first interview. For example, for the CPI commodities and service sector, the universe is a set of cohort samples with attrition due to deaths. Births are introduced only when an entire cohort is replaced with a new sample. Principal Authors: Daniel Kasprzyk and Lawrence R. Ernst 35 The longitudinal universe may also be defined at a time other than the time of sample selection. Under both scenarios, statistical, operational and methodological problems may arise because the sample was selected at one point in time and the analyses of the study universe reflect a different point in time. It is possible that elements of the study universe at the time of sample selection are no longer part of the longitudinal universe; it is also probable that elements of the longitudinal universe which exist at the time of definition were not in existence at the time the sample was drawn. This creates an operational problem -- whether to collect data from these "entrants" to the longitudinal universe -- and it creates a statistical issue, the development of estimation methods for this universe. For example, in the SIPP universe (the non-institutional population, and members of the military not living in barracks) individuals may leave the universe by moving outside the United States, to an institution, to military barracks, or by dying. At any time during the study period persons may enter the SIPP universe by returning from overseas, institutions, or military barracks, or through birth. A second method of defining a longitudinal universe extends the first method by looking at more than one time point. Several time points are selected, each one defining a universe at that time. Then the entire set of units -defined by these diffe
(November 1985)