| Federal
Committee on Statistical
Methodology Office of Management and Budget |
FCSM
Home ^ Methodology Reports ^ |
Statistical Policy Working Paper 10 - Approaches to Developing Questionnaires
Click HERE for graphic. MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY (November 1983) Maria Elena Gonzalez (Chair) Office of Information and Regulatory Affairs (OMB) Barbara A. Bailar Bureau of the Census (Commerce) Norman O. Beller National Center for Education Statistics (Education) Yvonne M. Bishop Energy Information Administration (Energy) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Industrial Economics (Commerce) Zahava D. Doering Defense Manpower Data Center (Defense) Maria D. Eldridge National Center for Education Statistics (Education) Daniel H. Garnick Bureau of Economic Analysis (Commerce) Charles D. ]ones Bureau of the Census (Commerce) Daniel Kasprzyk Bureau of the Census (Commerce) William E. Kibler Statistical Reporting Service (Agriculture) Thomas Plewes Bureau of Labor Statistics (Labor) Fritz J. Scheuren Internal Revenue Service (Treasury) Monroe G. Sirken National Center for Health Statistics (Health and Human Services) Thomas G. Staples Social Security Administration (Health and Human Services) Statistical Policy Working Paper 10 Approaches to Developing Questionnaires Prepared by Subcommittee on Questionnaire Design Federal Committee on Statistical Methodology Edited by Theresa J. DeMaio Bureau of the Census MEMBERS OF THE SUBCOMMITTEE ON QUESTIONNAIRE DESIGN Dawn D. Nelson (Chair) Maria E. Gonzalez* (ex officio) Bureau of the Census (Commerce) Office of Information and Regulatory Affairs (OMB) Deborah H. Bercini Janice Olson National Center for Health Social Security Administration Statistics (HHS) (HHS) Theresa J. DeMaio Anitra Rustemeyer Streett Bureau of the Census (Commerce) Energy Information Administration (Energy) Richard W. Dodge Ronny Schaul Bureau of the Census (Commerce) Bureau of Labor Statistics (Labor) Gemma M. Furno Margaret Weidenhamer Bureau of the Census (Commerce) Statistical Reporting Service (Agriculture) Additional Contributors to the Report on Approaches to Developing Questionnaires Catherine J. Baca Carol M. Utter Bureau of the Census (Commerce) Bureau of Labor Statistics (Labor) *Member, Federal Committee on Statistical Methodology ii OFFICE OF INFORMATION AND REGULATORY AFFAIRS Christopher DeMuth, Administrator Thomas D. Hopkins, Deputy Administrator for Regulatory and Statistical Analysis Dorothy M. Tella, Chief Statistician Maria E. Gonzalez, Chairperson Federal Committee on Statistical Methodology PREFACE The Subcommittee on Questionnaire Design was formed by the Federal Committee on Statistical Methodology to address the general topic of questionnaire design. The Subcommittee focused on a review of methods used in developing questionnaires. The working paper discusses approaches to devising questionnaires in three broad areas: tools for developing questions, procedures for testing the questionnaire draft, and techniques for evaluating the ques- tionnaire. While the report is intended primarily to be useful to Federal agencies that develop questionnaires, a broader audience may also find the report of interest. Seminars and meetings will be organized to discuss the topics addressed by this subcommittee with Federal agency personnel. The Subcommittee was chaired by Dawn D. Nelson, Bureau of the Census, Department of Commerce. As a subcommittee report, this document does not necessarily represent the views of the Office of Management and Budget. iii ACKNOWLEDGMENTS This report represents the collective effort of the Subcommittee on Questionnaire Design. Although all members of the subcommittee reviewed and commented on the entire report, individual members were responsible for preparing various chapters. Chapters 4 and 11, however, were prepared by persons who were not members of the subcommittee. The names of the authors of the respective chapters appear below. Chapter Author 1 Theresa J. DeMaio 2 Anitra Rustemeyer Streett 3 Margaret Weidenhamer 4 Catherine J. Baca 5 Dawn D. Nelson 6, Section I Gemma M. Furno 6, Section II Janice Olson 7 Anitra Rustemeyer Streett 8 Deborah H. Bercini 9 Theresa J. DeMaio 10 Richard W. Dodge 11 Carol M. Utter The following persons also deserve special recognition for their role in assisting the work of the subcommittee. Our work was initially guided by Naomi Rothwell who served as the chair from November 1980 until her retirement from government service in March 1981. Maria Gonzalez worked with the subcommittee throughout the development of the report and provided a link with the Federal Committee on Statistical Methodology (FCSM). Various members of the FCSM provided advice on our work at different stages, and Barbara Bailar and Zahava Doering supplied comments on the complete report draft. Much of the report material was also reviewed by Thomas Jabine who provided encouragement for our work. We especially appreciate Theresa DeMaio's contribution in sharpening the focus of the report by effectively organizing and editing the material. Also, we are grateful to Laura Taylor, Vicki Horton, Debbie Barnett, and Cathleen Tyson of the Bureau of the Census for typing and assembling the many drafts of this report. The Bureau of the Census also provided the funding for publication preparation and printing. iv CONTENTS Part I Chapter 1: Overview. . . . . . . . . . . . . . . . . . . 3 Part II: Tools for Developing Questions. . . . . . . . . . . . .11 Chapter 2: Unstructured Individual Interviewing. . . . .13 Chapter 3: Qualitative Group Interviews. . . . . . . . .21 Chapter 4: Participant Observation29 Part III: Procedures for Testing the Questionnaire Draft. . . . .41 Chapter 5: Informal Testing. . . . . . . . . . . . . . .45 Chapter 6: Formal Testing. . . . . . . . . . . . . . . .57 Section I. Pilot Studies. . . . . . . . . . . . . .57 Section II. Split Sample Testing . . . . . . . . . .70 Part IV: Techniques for Evaluating the Questionnaire Draft . . .89 Chapter 7: Investigating Respondent's Interpretations of Survey Questions. . . . . . . . . . . . . . .93 Chapter 8: Observation and Monitoring of Interviews. . 101 Chapter 9: Learning From Interviewers. . . . . . . . . 119 Section I. Interviewer Debriefing . . . . . . . . 119 Section II. Structured Post-Interview Evaluation . 124 Chapter 10: Using Record Checks . . . . . . . . . . . . 137 Chapter 11: Response Analysis Surveys . . . . . . . . . 151 References. . . . . . . . . . . . . . . . . . . . . . . . . . . 159 v Part I 1 Chapter 1 Overview I. INTRODUCTION Formulating a series of questions to obtain the answers to a set of data needs may appear to be a relatively simple task; however, constructing a questionnaire that will elicit accurate information from most respondents interviewed is more complicated than it may seem. For example, a seemingly simple question concerning vehicle ownership--How many cars do you own?--may appear to convey all the information necessary for respondents to answer it and to mean the same thing to respondents, survey designers, and data users alike.1 However, upon reflection, such a question is not as clear as it seems. The word "car" may or may not be intended to include such vehicles as vans, campers, motorcycles, tractors, and snowmobiles; "you" may or may not refer to household or family members as well; "own" may or may not include vehicles which are leased or are in the process of being bought. Questionnaire designers need to consider many factors during the process of creating a questionnaire. For example, will every question be interpreted in the same way by most respondents? If not, the data might not provide the information required by the questionnaire designer. Or, for another example, can respondents remember whether or not events of interest to the questionnaire designer have occurred within a given time frame, and if so, can they recall the details of those events accurately? Some generally accepted rules exist for wording, sequencing, and formatting questionnaires and can be used to guide a questionnaire designer in constructing an initial draft of a questionnaire. Yet the development of any particular questionnaire is unique. Refinement is necessary to ensure that any questionnaire used in the field will produce sufficiently accurate results. In the example described above, for instance, testing of the question would reveal the ambiguities inherent in it and lead to the development of a question more likely to meet the data requirements. The purpose of this report is to present a series of tools and tests which are useful in the initial drafting and subsequent refinement of a survey questionnaire, to explain their applicability to questionnaire design, and to describe the mechanics of implementing them. Numerous examples of these techniques are also provided to illustrate the points made. Although the _________________________ 1This example was adapted from one described by Biderman et al. (1982). Principal Contributor: Theresa J. DeMaio 3 4 focus is on survey questionnaires, many of the techniques are applicable to the development of data collection forms for administrative and other purposes. Many of these techniques are relatively simple, inexpensive ways to improve the quality of a questionnaire. For the most part they are appropriate for developing survey questionnaires regardless of the type of information being collected (e.g., factual, behavioral, opinion, or knowledge), the method used to obtain it (e.g., mail, telephone, personal visit, or a combination), or the type of reporting unit (e.g., households, individuals, farms, or estab- lishments). Used appropriately, these techniques should result in more efficient use of resources, reduced respondent burden and nonsampling error, and better realization of a survey's objectives. Maximum effort is justified during the developmental stage, because once a questionnaire is in use, problems are costly or impossible to correct. The time and money spent in developing a questionnaire should be repaid by collection of more relevant, better quality data. II. AUDIENCE FOR THE REPORT This report was written primarily for questionnaire designers in Federal agencies. While this does not limit the report's use by others, it may explain the focus and choice of materials for illustration. It i s hoped that those who have relatively little experience in this area will benefit from exposure to the techniques available for questionnaire development and how to use them. Even more experienced questionnaire designers may not be familiar with all the techniques and they may find the report useful as a reference. The report may also be helpful to persons who do not design questionnaires themselves, but who work in agencies that sponsor surveys to be conducted by private contractors or other government agencies. It is hoped that circulation of this report will promote increased familiarity with some of the less frequently used approaches and encourage use of all the techniques, thereby improving the relevance and quality of the data collected by the Federal Government. III. ORGANIZATION OF THE REPORT The approaches described in this report are divided into three sections: tools used to develop questionnaires, tests conducted to examine questionnaires, and techniques for evaluating questionnaires during the testing and developmental work. The order of presentation does not imply that the tools and tests must be used in a step-by-step order to develop a good questionnaire. It would be too costly, time-consuming, and inefficient to use every technique in the development of a single questionnaire. Moreover, each technique has strengths and weaknesses (in terms of cost, time, and resource requirements, and questionnaire design issues for which it is relevant) that render it appropriate or inappropriate for a given purpose. Within each chapter, an attempt is made to clarify when and how the technique can be used most appropriately. The topics discussed in each chapter follow the same general outline: I. Introduction; II. Method--A. Personnel and Skill Requirements; B. Selection of Respondents, C. Preparation; 5 D. Operation; E. Time Considerations; F. Cost Considerations; G. Mode of Data Collection; and III. Examples. Those who use this report should also be aware of the data collection requirements imposed on agencies by the Paperwork Reduction Act of 1980 or any Federal regulations which supersede this Act. A discussion of the current requirements monitored by OMB is contained in Statistical Policy Working Paper 9, "Contracting for Surveys" (Office of Management and Budget, 1983). IV. BACKGROUND: OVERVIEW OF THE QUESTIONNAIRE DESIGN ISSUES The focus of this report is the development and evaluation of questionnaires rather than the drafting and design of the questionnaire itself. However, in order to provide a framework for understanding the relationship between the development and evaluation process and questionnaire design issues, a brief description of the general issues is presented here. This is intended to provide the reader with some understanding of why the techniques that are the subject of this report are important components of the questionnaire development process. (However, since it is only an overview, readers unfamiliar with the topic should refer to other sources for a more detailed treatment of the issues--e.g., Payne, 1951; Sudman and Bradburn, 1974, 1982; Dillman, 1978; Bradburn and Sudman, 1979; Schuman and Presser, 1981; Turner and Martin, 1984.) In the chapters that follow, connections will be made between the techniques being described and the questionnaire design issues they are suited to address. The primary questionnaire design issues addressed in this report are content, question wording, question sequencing and flow, and questionnaire administration. Each of these is described briefly below; several other issues of secondary importance are described following this section. A. Content Decisions concerning what to include and exclude from a questionnaire and still meet the survey objectives are crucial . The analysts and data users should be consulted as early as possible in the process of specifying the subject matter. If an aspect of the problem is overlooked entirely, questions which would allow a fuller understanding of the subject of the inquiry may be omitted. For example, a questionnaire about child care arrangements could provide inaccurate information if the designer assumed that all parents make explicit and formal arrangements for such care when informal arrangements also exist. Alternatively, the content of a questionnaire can be limited by the type of data that can be collected--respondents may not be sufficiently knowledgeable to provide accurate responses to all questions. For example, in a survey of housing quality, a measure of floor space may be required; however, respondents may not be able to provide that information accurately. Some respondents may admit that they cannot answer the question, but others will provide inaccurate responses rather than acknowledge their ignorance. The extent to which people are able to answer the questions presented to them affects the quality of the data that are collected. 6 One type of request frequently made of survey respondents, about which knowledge and accurate recall are particularly problematic, concerns the recollection of whether specific types of events occurred and, if so, when they occurred. Survey researchers (or, for that matter, cognitive psychologists) have little information about the process, limits, etc., of human memory and how people place events in time when they are asked to recall the occurrence of a particular event. Asking respondents about events which occurred during reference periods of different lengths, and including examples which provide memory triggers as part of the question, have been used to increase the accuracy of recall data. To elicit the most accurate information possible, careful attention should he given to formulating such questions. In addition to not being able to answer questions they are asked, respondents, for various reasons, may not want to answer some of the questions included in a survey. They may feel that the information being solicited is sensitive--that some harm will come to them if they report some fact (e.g., use of illegal drugs), that they will be embarrassed by divulging certain information to an interviewer (e.g., inability to read), or that certain information is private and should not be disclosed to strangers (e.g., income). Effects on responses due to sensitive subject matter may be mini- mized through the sequencing of the questionnaire (see section on question sequencing and flow); however, a questionnaire designer must realize that the subject may be sensitive and take precautions to minimize response error or item nonresponse. B. Question Wording To provide comparable data from every unit in the sample, the survey questions must, as nearly as possible, present the same stimulus to all respondents. Several questionnaire design issues relate to this requirement. Generally speaking, the vocabulary used in each question should be familiar to respondents and mean the same thing to most respondents. Regional variations in the meaning of certain words would make them inappropriate for use in a national survey--for example, "soda," "soda pop," "pop," and "soda water" all refer to carbonated beverages, but any one of them would be interpreted differently in different parts of the country. A respondent who does not know what a word means to the researchers will not be able to provide an accurate response to a question that contains it. Respondents may provide answers to questions, but those answers may not reflect the reality intended by the questionnaire designer. A similar situation occurs with regard to the meaning that questionnaire designers and respondents attach to particular concepts used in survey questions. If a respondent, or some limited subgroup of respondents, does not interpret a question in the same way as it was intended by the questionnaire designer, then the answer will not be a valid measure of the survey designer's construct. Over the course of administering the question to the entire sample of respondents, ambiguity is introduced into the results, leaving the investigator uncertain as to what those results really mean. For example, take a situation in which respondents are asked to rate the seriousness of crime in their neighborhood. Even respondents living next 7 door to each other (may have different concepts of the boundaries of their neighborhood, which might affect their ratings. If the neighborhood were defined for them, differences in the ratings would reflect factors unrelated to conceptual differences in the geographic area covered. Investigators who are unaware of differences between their own "frame of reference" and the variety of "frames of reference" existing among respondents may interpret the results in ways that do riot reflect reality. Another question wording issue involves the response categories that are presented to respondents in fixed alternative or multiple choice questions-how many options should be offered, how they should be ordered, whether they should be presented in a forced- choice or open-ended format, whether a "don't, know" option is presented. Decisions made in this area may affect the quality of the data obtained in the survey, since the answers provided by respondents will be distributed differently, depending on the alternative response categories offered. The length of the questions in the interview is another issue related to question wording. An extended introduction to a question may afford the respondent time to think about the issues involved before giving an answer, thereby potentially providing more carefully considered and more accurate results. On the other hand, longer questions add to the length of the interview and may contribute to respondent fatigue, inattention, or confusion. Different types of respondents may react in different ways to long questions, introducing a systematic bias into the results. C. Question Sequencing and Flow Another set of issues in questionnaire construction concerns the order in which the questions are presented to respondents. Even if questions are worded so they mean the same thing to all respondents, response biases or problems in administration of the questionnaire may result from the way the questions are sequenced. One such problem involves the context imposed by the previous question or perhaps a set of questions contained earlier in the questionnaire. Such questions may invoke a particular mind set in the respondent's consciousness which may not reflect the way he or she thinks about a certain topic in other settings. A respondent may thus answer survey questions differently, depending on the order in which they are presented. For example, a person may respond one way when asked to evaluate the overall quality of the neighborhood if a question has just been asked about the street lights in the area. The rating of the neighborhood may be influenced by an opinion of the street lights, even if street lights are not an important criterion for determining neighborhood quality. If the two questions were reversed, however, the rating of neighborhood quality might be different. Another consideration is the location of so-called sensitive questions-questions considered intrusive or damaging to respondent self-esteem. Placing these questions late in the interview so they are asked after some degree of respondent confidence has been established may minimize refusals and response problems introduced by the nature of the subject matter. On the other hand, leaving such questions until the end may risk superficial 8 answers due to respondent fatigue. At the very least, such questions should be located where they fit logically in the flow of the questionnaire and, if necessary, be approached gradually through related, but less threatening, questions. The overall flow of the questionnaire deserves attention in the questionnaire design process, since it too may affect the quality of the data that are collected. If too many items are included in a list, the amount of thought given to each response may decline towards the end because of respondent fatigue. Excessive consecutive questions with the same type of format [nay condition the respondent to "acquiesce" unthinkingly with the same answer to each question (e.g., to yes/no type questions). If questions about the same topic are included in several different places in the questionnaire, a respondent may become confused by perceived redundancy or hostile because of perceived carelessness and treat the survey interview with less seriousness than the investigator would like. Thus, for many reasons, the flow of the questionnaire is an important element of questionnaire design. D. Ease of Questionnaire Administration In designing a questionnaire, the ease with which the questionnaire can be used by the interviewer/respondent is an important consideration. One aspect of questionnaire construction involves the placement of instructions. The extent to which interviewers/respondents are required to flip through the questionnaire, refer to previous answers that are not readily accessible, etc., should be minimized. The harder it is for the interviewers to determine the flow of the interview, the more chances for introducing interviewer error, item nonresponse, and respondent frustration. E. Other Design Issues Several other elements in the design of the survey may be relevant to the construction of the questionnaire. These issues, which are related to procedural decisions and format of questionnaires, are described in this section; however, they are of secondary importance in this report. External constraints imposed by cost, time, or OMB respondent burden requirements may dictate the length of the survey interview, thus limiting the amount of information that can be obtained in the questionnaire. This may affect the number of questions that can be included and, therefore, the number of topics included in the questionnaire or the amount of detail obtained about particular topics. To some extent, the types of questions that are included can also be affected. For example, time-consuming techniques such as randomized response or card sorting might have to be eliminated if they add too much time to the length of the interview. The criteria for selecting survey respondents should also be considered in designing a questionnaire. A survey involving responses from every eligible household member may require questions to be worded differently than for a survey in which a single respondent answers questions about each household member. In addition, the sequence of questions may have to be altered slightly to accommodate different types of respondents. 9 The method of data collection may also influence the design of the questionnaire. Questions employing visual aids, which are helpful in face-to-face interviews, are obviously not feasible for use in telephone interviews. The differences between modes of interviewing in the dynamics of interaction between interviewer and respondent may also suggest alterations in the types of questions that are used to obtain data of comparable quality in different interviewing modes. For example, mail questionnaires might be more successful in obtaining sensitive information than either mode involving direct interaction with an interviewer, who might be perceived as judgmental of respondents' answers. The unit of analysis for the data also affects the structure of the questionnaire or the type of information that is collected. If the objective of the survey is to compile data on families, information need not be collected about unrelated household members. If, however, data for households are required, information about unrelated household members would be needed. The provision of data for individuals, for specific population subgroups such as food stamp users, or for a combination of different units, may require alterations in the order or wording of questions. For surveys in which each unit is interviewed more than once, the number of interviews in the sequence for each sample unit and the length of time between interviews may influence some aspects of the questionnaire. For example, the amount of elapsed time between contacts affects the length of the reference period used in asking respondents to recall events. If data are collected in a series of interviews within a specified time period, the number of interviews conducted within that time period may affect the length of each interview. That is, the same amount of information that is collected quarterly could be obtained in only three longer interviews per year. Format is also an important issue in the design of the questionnaire. Concerns in this area relate to the appearance of the questionnaire--color or kind of paper, size or style of type, method of data processing, method of questionnaire administration, etc. These issues may affect the quality of the data by influencing how well respondents or interviewers are able to follow the instructions and answer the questions. Concerns about the relationship between appearance and answering a questionnaire correctly are, however, in a different realm than the previously described issues which relate to the meaning of the questions and respondents' ability to answer them accurately. The topic of format is not addressed directly in this report, although some of the techniques described here can be useful in this regard. V. SUMMARY The words "may" and "might" have deliberately been used throughout this description of questionnaire design issues. Many of the issues that have been raised here may, but do not necessarily, cause problems for questionnaire designers. Although progress has been made in the last several years in identifying sources of nonsampling error and in measuring its extent, guidelines for eliminating its existence through systematic rules for questionnaire design have not been forthcoming. Efforts to construct guidelines involve evidence based on individual cases and the extent that these 10 guidelines can be applied to questionnaires involving different subject matters, respondent populations, or survey designs (e.g., one-time vs. repetitive surveys) is not clear. Some issues are more clear-cut than others. For example, in the area of question wording, it is generally accepted that questions which "lead" the respondent in one direction or another should be avoided. Even in this instance, though, the determination of whether a particular question "leads" the respondent may be a subjective one. Moreover, a questionnaire designer may deliberately use "leading" questions to meet such objectives as measuring the effectiveness of alterna- tive advertisements or appeals. In addition, sometimes guidelines that are generally accepted may be mutually inconsistent for a particular questionnaire. For example: sensitive questions often produce better data if placed near the end of a questionnaire; and, it is generally recommended that important questions be placed near the beginning of a questionnaire to ensure obtaining that information even if a breakoff should occur. However, there may be questions that are both sensitive and important, and their placement is not addressed by these guidelines. For these reasons, guidelines are not always applicable, even in areas where they exist.2 To construct a questionnaire that causes the fewest problems when used in the field, questionnaire development should be a multistage process during which problems are systematically identified and either eliminated or minimized. The approaches described here can be useful components of this process. _________________________ 2For more extensive discussion of reasons for the inadequacy of proceduralizing guidelines for the design of forms, see Wright (1981) and Duffy (1981). These discussions are also applicable to the design of questionnaires. Part II Tools for Developing Questions This part of the report describes three tools to obtain information that will be useful in the task of actually drafting the questions and assembling them into a questionnaire for a proposed survey: (1) unstructured individual interviewing, (2) qualitative group interviews, and (3) participant observation. In some instances, these same techniques are used later in the process, i.e., during testing or the survey itself, to provide information that will aid in the interpretation of the test or survey results. However, the emphasis in this section is on the aspects of these techniques that contribute to the initial development of the content of a questionnaire. It is assumed here that a determination has been made that certain information is needed to address a problem and that a survey is the best way to provide this information. Obviously, this determination should be made only after it has been ascertained that the information is not already available elsewhere (e.g., from existing survey data, other records or research studies) or more easily obtained by another method such as the use of administrative records. To make a determination, the problem should be clearly stated, including its possible causes and the potential solutions. The temptation to start drafting a questionnaire before this is done should be avoided. Without a thorough analysis of the problem, the resulting survey may not provide the right information or enough information to solve the problem. The objectives of the survey, including what data should be 11 12 collected and how it will be used, need to be directly related to the solution of the problem. It may be possible to examine a problem and develop survey objectives by researching literature on the topic and through discussions with experts in the problem area. However, information or experts may not be available, particularly if a survey on the topic has never been conducted before. In that case, the techniques described here may be useful in obtaining the necessary background information. Since, in our information-rich society, the necessary information is usually available from other sources, these techniques are not used as frequently to develop questionnaires as some of the other methods described in this report. However, they are included here to ensure that questionnaire designers are aware of their possible uses. Each of these techniques is briefly described below. Unstructured individual interviewing, described in Chapter 2, is a discussion of the proposed survey topics between an individual member of the group to be surveyed and the questionnaire designer. It is guided by a topic outline rather than a set of specific questions. This technique is used primarily to gain insights into the best way to structure the questionnaire. Qualitative group interviews, the subject of Chapter 3, are informal discussions of selected topics between participants chosen from the population of interest and someone who is knowledgeable about group interviewing techniques and the purpose of the survey. The information from qualitative group interview sessions can aid in developing the conceptual framework and data specifications for a statistical survey and evaluating draft questionnaires. Qualitative group interviews are occasionally used after a survey has been conducted to help the analysts interpret the data. The last of these three techniques is participant observation research, described in Chapter 4. While it is not used frequently in designing questionnaires, it can be particularly useful when a survey is to be conducted among people whose language, values, or experiences are very different from those of the questionnaire designers. Information obtained through participant observation can be used to ensure that the content of the questionnaire will provide enough information to satisfy the survey's objectives and to help phrase questions that can be understood by the respondents. The information can also be used to help understand the meaning of respondents' answers to survey questions. Chapter 2 Unstructured Individual Interviewing I. INTRODUCTION Sometimes a questionnaire designer is required to develop a questionnaire on a topic which (s)he knows little about, and about which little information related to questionnaire design exists from previous surveys. In this situation, the development of a questionnaire can benefit from the use of unstructured interviews with members of the intended respondent universe. The term "unstructured interview" is used here to describe a discussion of the proposed survey topics between a member of the target survey population and the questionnaire designer.1 The discussion is guided by a topic outline rather than a set of specific questions. When sufficient numbers of such interviews are conducted with respondents who are fairly representative of the target population, the technique can provide ideas and insights about how best to structure the questionnaire before the first draft is written. It is a particularly valuable technique when there are many divergent interests in a survey. When there is more than one sponsor, initial disagreement can exist about what kinds of information can and should be obtained. This technique transfers the questionnaire design decisions from dependence on the tastes or preferences of the survey sponsors to reliance on the results of the field processes employed. Several of the questionnaire design issues described in Chapter 1 can be addressed by using this technique. The specific uses of unstructured interviewing include the following: (1) Topics previously thought to be important for inclusion can be discarded as unnecessary or irrelevant, and topics which had previously been neglected can be identified as important in fulfilling the objectives of the survey. (2) A determination can be made as to whether the information requested in the survey is readily available to respondents and whether particular kinds of questions can be asked. (3) An evaluation can be made of which topics might be especially sensitive to respondents. _________________________ 1This technique was initiated and has been used extensively by survey researchers in England. Researchers in this country were introduced to the technique by Jean Atkinson of the Social Surveys Division in England; it is described in Atkinson (1968) and Hoinville et al. (1978). Principal Contributor: Anitra Rustemeyer Streett 13 14 (4) Assistance can be provided to determine how to phrase particular questions so that the vocabulary is familiar to respondents and the words mean the same thing to all respondents. (5) Decisions can he made concerning the preferability of open- vs. closed-ended questions to obtain particular types of information, and a range of answer categories for closed-ended questions can be specified. (6) An identification can be made concerning who in a household or business is in a position to respond most accurately to questions on the survey topics and, therefore, would make the best respondent. (7) Suggestions can be made concerning the optimal order of questions or survey topics. (8) Insights about which aspect of a topic appeals most to people may be used to determine the best way to approach respondents in order to encourage their cooperation. II. METHOD A. Personnel and Skill Requirements A key concept in the successful use of unstructured interviewing is flexibility. The questionnaire designer functions as a researcher during this process, and must keep the objectives of the study firmly in mind while dismissing any fixed ideas about how to structure the questionnaire. Best results are achieved when several people, including one who serves as a team leader/questionnaire designer, work together as a team. The team should include interviewers as well as data processing and subject matter specialists. This allows diverse ideas and insights to be used in the refinement of the survey instrument. Persons selected to conduct unstructured interviews should be experienced interviewers and be capable of understanding the broad perspective of the research project for which the questionnaire will be designed. This type of interviewing requires skills different from those for structured interviewing (i.e., interviewing in which questions are read verbatim from a question- naire), and only some interviewers on a regular field staff are likely to possess those skills. Interviewers selected for this type of assignment should feel comfortable "thinking on their feet" as they will not have a questionnaire script to use as a crutch; if they are easily flustered or confused, they give respondents the impression that they are incompetent or that the study is unimportant. Members of the interviewing team need sufficient experience in unstructured interviewing to be sensitive to the effects of wording changes and to recognize responses that indicate potential problems with question wording or order. In addition, interviewers should be able to tolerate long pauses while the respondent thinks or looks for answers, have the ability to probe nondirectively to get the respondent's ideas, and have a thorough understanding of potential problems in questionnaire design which can affect the achievement of the survey objectives. Some of the interviewers should be highly knowledgeable and skilled in structured interviewing techniques. This allows informed judgments to be made concerning the kinds of things which can be asked in a closed-ended format 15 and what topics respondents can be expected to respond to within the discipline of a structured interview. Unstructured interviewing is actually a combined data collection and analysis process In addition to the interviewing skill necessary for successful results, a "coder" who is capable of making independent judgments is an essential part of the process. This person should be able to analyze and tabulate results of the previous day's work while the interviewers are in the field conducting additional interviews and then meet with them to explain how and where they are failing to meet survey objectives. The simultaneous conduct of these two tasks speeds up the questionnaire refinement process. Finally, sponsors or subject matter specialists can provide valuable insights in the frequent meetings held to charter the course of the work. B. Selection of Respondents Respondent selection for unstructured interviews generally involves purposive rather than systematic sampling. Although rigorous scientific selection procedures are not necessary, respondents should be members of the population to be surveyed and should be fairly representative of that population. The characteristics of people asked to be respondents for unstructured interviews may depend on the survey topic. For example, in developing a questionnaire dealing with saving habits to be administered to a national cross-sectional sample, the initial round of developmental work may include interviews with people from a variety of demographic population subgroups. During additional interviews, however, different classifications of saving habits may emerge, and it may be necessary to locate and interview persons who are members of specific categories. Thus, the "sampling" of respondents is an iterative process, too--as is the questioning of those respondents. Respondents may be located by contacting community or business organizations, or by selecting residential areas. C. Preparation Before embarking on this phase of a questionnaire design project, the team leader should become familiar with the objectives of the study and make a list of the data elements which are considered necessary to meet those objectives. These data elements include topics and concepts which are particularly vital to the quality of the study, or are otherwise thought to be related to the survey objectives. Prior to the first discussion with a respondent, the team leader should prepare some alternative orders in which the topics [night be discussed, as well as any specific words or phrases to be used in relation to any particular topic. The team leader's next task is to develop the work sheets to be used by the interviewers and coders. Those serving as interviewers should review the materials and meet with the team leader to discuss study concepts and objectives. The interviewers need to be provided guidance, so they will not go beyond the scope of the project. 16 D. Operation Interviewers may begin each interview by explaining that they are working on a very early phase of preparing a new survey. They should emphasize the reasons for and importance of talking to people before a questionnaire is prepared. During this type of interviewing, the interviewer should follow up on answers or comments that seem to have a bearing on how a concept is interpreted by the respondent or how a sequence of questions should be ordered. The interview should have a conversational flavor rather than the question-and-answer format of a formal interview. Interviewers should understand that their objective is not to collect data in the usual sense--rather, it is to become aware of any difficulties that are likely to arise when the survey is conducted. Throughout this process, extensive note-taking is valuable, so that insights gained during an interview are not lost or confused with other interviews. Verbatim recording, by shorthand or speedwriting, is ideal for this purpose; however, such a skill is not within the repertoire of every skilled interviewer. Even very abbreviated note-taking can make it possible for an interviewer to return to statements made earlier by the respondent. Following up immediately on some statements could take the interviewer off the topic being pursued; but "passing remarks" and apparently contradictory statements by the respondent can provide additional insights on how to phrase survey questions. During each unstructured interview, the interviewer should record how each key inquiry was phrased, as well as the wording used by the respondent in answering the question. (Since interviewers often respond to the answers of respondents with idiosyncratic or instinctive phrases of their own, it may be more difficult to remember their own words than those of the respondents.) Notes should be made (during or immediately after the interview) concerning the ordering of the inquiries (if different from the outline), how one topic relates to the next, if and how they overlap, what effect topic order has on the flow of the interview, the respondent's reaction to specific questions of interest, and the apparent level of difficulty of the inquiry for the respondent. Tape recording, with the respondent's permission, can be useful as long as time is available to listen to the tapes and extract information from them. Ideally, the team leader, team members who are conducting the unstructured interviews, and coders should meet frequently to discuss what they have learned to date. The reason for frequent meetings is to allow all interviewers to gain insights from the experiences of the others and to help one another interpret respondents' comments. Under the guidance of the team leader, changes to the topic outline should be made to refine ideas on how to present topics and sections of the questionnaire, and the order in which to present them. As experience using the topic outline is gained, interviewers will develop their preferred question wording for topics. They should exchange those wordings during their meetings and then try the wordings used by others in successive iterations of interviewing. 17 The input of the coder is beneficial in noting ambiguities or superficiality in the responses obtained in previous interviews which require further clarification before the response can be coded. Also, the relative frequency of responses to open-ended questions, the range of conditions imposed by respondents on their answers (e.g., "it depends on..."), and potential response sets can be obtained from the coders' tallies. The coders' analyses and the interviewers' annotated transcripts are discussed among team members, patterns are identified, and suggestions are made concerning potential question formatting, sequencing, etc. No set number of completed unstructured interviews or days of unstructured interviewing can guarantee a good questionnaire. Perhaps the best indicator that enough unstructured interviewing has been done is the lack of new insights and ideas on question wording and order by team members. The responsible researcher (i.e., the team leader) must judge whether the team has fulfilled its mission, and when the process of putting together the first draft of the questionnaire should be undertaken. E. Time Considerations The process outlined here may take longer to complete than drafting a questionnaire without any field work. On the other hand, when the questionnaire is drafted after these procedures have been followed, it is likely to require far less modification; therefore, time required for unstructured interviewing may be wholly or partly recovered later. The exact amount of time involved depends on the number of people who are available to conduct interviews, the number of interviews completed daily by each interviewer, and the iterations of the topic outline, question wordings, etc., required before members of the questionnaire design team are confident to construct a questionnaire. In general , when the use of unstructured interviewing is incorporated into the development process, 2 to 6 weeks should be allowed in the time schedule. This includes the preparation time for the team leader as well as the interviewing time itself. It does not include completion of the initial questionnaire draft, which would be required regardless of whether or not this technique is used. However, drafting the questionnaire should be much less time-consuming, because the knowledge gained from the unstructured interviews will clarify concepts and resolve most of the issues that are typically debated; e.g., which words to use and which to avoid, how much detail to request of respondents, and the order in which to present topics. F. Cost Considerations The monetary costs associated with the use of unstructured interviews are essentially limited to the salaries of the personnel who are members of the team. Depending on the number of people involved, the number of interviews conducted, and the amount of time spent in analyzing the interviews, these costs could vary considerably. In addition, other expenditures may be necessary for travel if the interviewing site is not located near the duty station of the people working on the project. One other "cost" should be mentioned here: the burden on the public. Although unstructured interviewing places some response burden on the public, 18 this investment may be more than repaid later if the unstructured interviewing results in a more efficient questionnaire than would be prepared without this type of field work. G. Mode of Data Collection Regardless of whether the final survey will be conducted face-to- face, on the telephone, or by mail, the use of unstructured face- to-face interviewing can provide valuable insights on how people respond to the topics of the survey. Benefits accruing from establishing the relevance of specific topics to the survey objectives, defining key concepts, and identifying words which have similar meaning for all types of respondents will be equally pertinent for surveys conducted through any method. Some of the other insights gained through use of this technique, such as the specification of question order, may be unique to the mode in which the data are collected. If the final survey is intended to be conducted exclusively on the telephone, unstructured telephone interviewing could conceivably be conducted. III. EXAMPLE: NATIONAL FIRE SURVEY In 1973 the Bureau of the Census was asked to determine the incidence and characteristics of household fires in the United States. It was decided that a few "screener" questions should be added to the (monthly) Current Population Survey to determine if a fire had occurred in the household within the preceding few months. If a fire had occurred, a separate questionnaire would be administered to gather more detailed information, including extent of damage, death or injury to household members, and financial loss attributable to the fire. The study directors and sponsors agreed to unstructured field interviews as a means of drafting a questionnaire, because they needed answers to several questions, including what definition of fire should be used and whether people would call things like the following a fire: a grease fire while cooking, a smoldering mattress caused by a cigarette, a small fire ignited by a child, a fire in an automobile engine, a chimney fire. They also wanted to know if questions about injuries, loss of life, and whether the fire was caused by carelessness were feasible and, if so, how to word them and where to place them in the interview. Another area of uncertainty dealt with economic loss and who paid to restore the damage: did people know the dollar value of the losses due to fire and to what extent were they covered by insurance, other family members, charity, etc.? Since household fires are fairly rare events in the general population, the households selected for the unstructured interviews were chosen from fire department records so that between one-third and one-half were known to have reported a fire within the preceding 6 to 9 months. (See Chapter 10 for a discussion of using record checks.) The other households were selected because they were within two or three blocks of the households identified in the records of a fire department. Choosing nearby households allowed the interviewers to conduct more interviews with less driving time. Interviewers were not told which households had reported a fire to the fire department. 19 The team leader was a senior member of the survey methods research group; others on the interviewing team were junior professionals from the research and operations offices who would work on the final survey. The team of five worked singly and in pairs and, with permission of the respondents, tape recorded some interviews. The team began with a list of topics to be covered and a thorough briefing on and discussion of the survey objectives. They met daily to share with the group what they had learned. After 3 days (approximately 4 interviews per day by each team member), patterns of questioning respondents had developed and these were discussed. Agreement was reached on two draft questionnaires. These draft versions were used by all team members during the next 2 days of interviewing. At the end of 5 days of unstructured interviewing it was fairly easy to draft a questionnaire that could be endorsed by all team members as suitable to meet the study objectives and workable with respondents. A definition of a fire was developed which included short lists of things to include and exclude, based on ambiguous areas encountered during the unstructured interviews. The questionnaire was used in an informal test and was judged to work very well. (See Chapter 5 for a description of the objectives and procedures of informal tests.) Chapter 3 Qualitative Group Interviews I. INTRODUCTION By qualitative group interviews we mean open, informal discussions of selected topics by participants chosen from the population of interest, or a subset of that population, led by someone who is knowledgeable about group interview techniques and the purpose of the discussion. Many other terms are used to describe this approach, such as group depth interviews, intensive interviews, focused discussion groups, and focused group interviews. This approach is similar in some respects to unstructured individual interviewing (discussed in Chapter 2) except that it involves a group of participants. The rationale for conducting qualitative group interviews is that information can be brought out through interaction of the participants which would not surface if each of them were interviewed separately. Qualitative group interviews allow closer contact between researchers and respondents than is normally possible in large-scale traditional survey research approaches and permit flexible exploration of research issues from the respondents' points of view. Qualitative group interviews are an appropriate vehicle for developing insights and hypotheses and for exploring the range of pertinent attitudes, opinions, concerns, experiences, and suggestions of the participants. They can be a helpful preliminary step in developing the conceptual framework, data specifications and question wording or evaluating draft questionnaires for a quantitative survey which will use structured questionnaires among a representative sample of respondents. In the example provided at the end of this chapter, the technique was used to evaluate proposed revisions to an existing administrative form. Qualitative group interviews are also sometimes undertaken solely to provide general information or to help determine whether quantitative research on a subject is feasible; occasionally, they are employed after a survey has been conducted to help the analysts interpret the data that were collected. II. METHOD A. Personnel and Skill Requirements Qualitative group interviews require the services of personnel with specific types of expertise. A discussion leader should be skilled in guiding the group interview within the topical area limits, covering all germane areas, Principal Contributor: Margaret Weidenhamer 21 22 probing for the meaning of comments which are not self-explanatory, yet remaining as unobtrusive as possible to avoid "leading" participants. It is his or her function to initiate discussions among group members and encourage all to join in the discussion, to subtly direct the discussion to the pertinent issues, to prevent domination of the group by any of the participants, and to bring the discussion back into focus whenever it digresses into irrel- evant areas. More than one discussion leader may be used depending on the number of groups to be interviewed. The discussion leaders usually summarize the results of the discussions. For this part of the task, analytical skills are required. B. Selection of Respondents The participants in qualitative group interviews are members of the population of potential respondents to the planned survey, but they may not be representative of that group. Usually, a relatively homogeneous group of people, such as middle income city dwellers or suburban homemakers with school children, are invited to participate in a given session. They are chosen by whatever nonprobability techniques may be convenient. During this phase of survey development, a number of group sessions are generally conducted. The total number of sessions conducted for a particular survey varies considerably. Normally at least four to six group interviews are conducted, and many more may be desirable for complex projects. When multiple sessions are held, different types of people in the target population may be recruited for different sessions. For example, in the development of a national survey on some topic, some qualitative group interviews may be conducted with young black males, others with middle-aged white females. It is also advisable to conduct sessions in several different geographic locations to reduce regional biases. Participants are usually paid a set fee or a donation is made to an organization of their choice in recognition of the time they spend and the incidental expenses they incur in attending the session. (The need for payment as an inducement to participate must be satisfactorily demonstrated to obtain OMB approval for compensating respondents in Federally-funded surveys.) C. Preparation An outline of topics to be covered is usually prepared in advance; it is likely to start with fairly general topics and gradually focus more on details of the subject matter of interest. The outline may be revised between sessions, as the scope of the research becomes more focused. D. Operation Generally qualitative group interviews are held in a central location which is convenient for participants, and are scheduled to run for about 2 hours. From 8 to 12 persons are suggested for participation in a given session; some additional invitations may be extended to allow for attrition. When conducting qualitative group interviews, the discussion leader's first task is to create an informal setting that encourages a frank, open discussion 23 among all the participants and to start the conversation off in the right direction. The approach used must not he so structured that the participants cannot engage in spontaneous discussions which would shed light on their views--particularly views which may not have been anticipated in the topic outline. The outline is usually used as a guide by the leader, but (s)he should allow the discussion to follow its natural course, unless it strays too far from the purpose of the session. Projective techniques and self-administered forms may be used during the session, and questionnaires or other exhibits may be displayed. Sometimes a series of two or more sessions is held with the same participants, perhaps with a homework assignment in- between. Also, follow-up individual interviews may be conducted with participants. The sessions are usually tape-recorded, and occasionally video-taped, to permit detailed study of the contents. A good deal of subjective judgment is involved in the analysis Of Such sessions, and the results must be interpreted with caution. The reports are often written by the discussion leaders who conducted the group interviews. Results should be presented in narrative form, not in terms of proportions or percentages, to avoid suggesting spurious interpretations. E. Time Considerations Group interview sessions can he planned, conducted, and analyzed in approximately 2 to 4 months. The time will vary depending on the number of sessions conducted and their locations. F. Cost Considerations Qualitative group interviews are a relatively inexpensive way to collect background information for use in developing a questionnaire. The major expenses are the salaries of the recruiters and discussion leader(s)/analyst(s) and the fees paid to participants or donations made on their behalf. Travel expenses and belated costs such as rental of conference rooms and taping equipment may increase the cost considerably if multiple sessions in various geographic locations are held. G. Mode of Data Collection A survey employing any mode of data collection--face-to-face, telephone, or self-administered--can potentially he improved through the use of qualitative group interviews in the early stages Of questionnaire development. III.EXAMPLE: EVALUATION OF A PROPOSED REVISION OF AN APPLICATION FORM The Social Security Administration of the U.S. Department of Health and Human Services employed group interviews to assess a proposed revision of the application form for a social security number.1 The proposed form contained ___________________________ 1The information presented here is selected from reports by Bayton (1978) and Scherr (1980). 24 three pages of instructions and other relevant information, and a one-page application. The application part of the form is shown in figure 1. Ten group interviews were conducted in a 1-month period in the spring of 1978. They are described below. Number of-- _____________________ Location Respondents Groups Respondents Washington, D.C. Male teenagers; black; 2 10,12 low socioeconomic status (SES*) Washington, D.C. Female teenagers; black; low SES 1 7 Glen Burnie, Md. Male and female teenagers; white; lower and middle class SES 2 14,14 Glen Burnie, Md. Male and female adults; black and white; lower and middle class SES 1 15 Los Angeles, Calif. Spanish-language background male and female adults; low SES 2 11,12 Los Angeles, Calif. Spanish-language backgrounds; male and female teenagers; two Asians in one of the groups; low and middle class SES 2 14,15 *NOTE: SES is used here as a proxy for expected level of functional literacy. Each session was tape-recorded and lasted approximately 1-1/2 hours. Each adult respondent was paid $15; each teenage respondent was paid $10. The teenage group sessions did not last as long as those with adults. One or two researchers involved with the project observed each of the sessions; other Social Security Administration staff members also attended some of the sessions. The group session topic outline followed this sequence: 1. Introduction--purpose of the project. 2. Why should a person apply for a social security number? 3. When should a person apply for a social security number? 4. How can a person go about applying for a social security number? 5. What information does Social Security want from applicants? 6. What documents are needed and why? 26 7 . Completion of the proposed form. Respondents filled out the form as though applying for a social security number. They were requested not to interrupt for questions or comments but to wait until the entire group had finished. 8. Open-ended inquiry. Questions or comments initiated by respondents after all members of the group had finished completing the form. 9. Directed inquiry. Item-by-item probing by the discussion leader of respondents' reactions to the form. Some of the findings from this study were-- The most consistently salient problem was with the race/origin part of item 3. In most groups, this was the first matter raised by the participants. The problems included not understanding what "origin" meant and what "Hispanic" meant. Another salient problem had to do with item 2b-Other name(s) used. Respondents who raised this issue asked whether nicknames were to be included. Father's name (item 4) was mentioned as a problem by some; did this mean "real" father or stepfather? The instructions for item 4 mention stepfather, but it is not clear as to which should be used, if there is a choice. Among teenagers, item 7-Telephone number where you can be reached during the day--was a problem. Did this mean that they should give the number of the school being attended? Upon receiving the proposed form to fill out, only in rare instances did the respondents read the instructions on the page facing the application or turn the form over and read what was on the back, despite the request to do so which was printed on the application. The more usual use of the instructions came when an individual stopped working on the application and referred to the instructions for an item on an as-needed basis. When asked why they did not read the instructions initially, typical comments were: "I've filled out application forms before;" "I didn't read them; it looked easy," and "If you look over it and you understand most of the questions, you don't need to read the directions..." When the discussion leader went through the entire form section-by-section, additional difficulties surfaced. For example, the statement "For statistical purposes only" appears over the part of item 3 that contains the race/origin information; many of the teenagers and foreign language background respondents did not understand the intended meaning of the phrase. Some of the respondents associated this term with the Government keeping a "record." Others said the term referred to the fact that the information asked would not be used in relation to particular individuals. The problem with item 5, Have you ever applied for a social security number before?, was the interpretation to be placed upon the word "you" Several teenagers reported that their mothers had obtained social security numbers for them when they were much younger. If the "you" were to be taken literally, these respondents would check "No." The instruction for this item does not address this problem. 27 The Social Security Administration used the information obtained through the qualitative group interviews in developing further revisions of the proposed new application package. Controlled testing of specific alternatives was then conducted with larger samples of actual applicants under operating conditions. Chapter 4 Participant Observation I. INTRODUCTION Participant observation research techniques have traditionally been used by anthropologists to study other cultures. By living among people and studying them as unobtrusively as possible, anthropologists have learned much about societies that were relatively unknown. Participant observation research can also be used as a preliminary stage in the design of certain questionnaires. It can be particularly useful in planning a survey among people whose language, values, or experiences are very different from those of the questionnaire designers, or about whom very little is known. Understanding the culture of potential respondents through participant observation research contributes to questionnaire design in several important ways. First, it increases the likelihood that meaningful inferences can be drawn from respondents' answers. A questionnaire designer who is familiar with the values and experiences of a population is in a better position to write questions which make sense to respondents and to which they will respond more willingly. In addition, a researcher who is familiar with the population suggested for study knows how to contact individuals with a greater probability of being knowledgeable about the survey topic. Participant observation research helps a questionnaire designer distinguish significant categories of people within the respondent community, and helps in identifying characteristics that may be associated with response. If participant observation indicates systematic differences among age groups, or occupations, or backgrounds in the way topics are conceptualized, the designer may find that a complex questionnaire design is necessary with several paths within a single questionnaire, or even use of multiple questionnaires. This may be necessary to ensure meaningful questions to respondents from different age groups, occupations, or backgrounds. Surveys of the population of the United States run into frequent problems with respondents who have difficulty understanding questions written in English. In November 1979, the Current Population Survey estimated that nearly 18 million Americans (almost 8 percent of the population) used a language other than English at home. In addition to the potential language problems in a national sample, there are many subpopulations where a much larger proportion of respondents need special questionnaire designs. For Principal Contributor: Catherine J. Baca 29 30 example, the population of Puerto Rico is routinely the subject of Federal surveys dealing with employment, the labor force, or food assistance. Recent Asian and Caribbean immigrant populations have been asked to respond to Federal surveys about immigration, literacy, and public assistance. These respondents, along with many employment and income groups that use special vocabularies or share distinct cultural outlooks, require questionnaires that are written specifically for them. II. METHOD A. Personnel and Skill Requirements There are three different ways in which participant observers can take part in the design of a questionnaire. First, when little or nothing is known about the respondent universe, participant observation data can be collected by a field researcher selected for this purpose. Such a person (or persons) might be recruited through university graduate, departments of anthropology or through national professional organizations1 that maintain records of their members' professional experience and research skills. Second, fieldworkers who have previously conducted participant observation research among the potential respondents can be involved in the questionnaire drafting phase of survey development (either on a full-time or consulting basis). In this way, insights into potential difficulties in respondent understanding and/or interpretation of the questions, respondent perceptions of the subject matter, etc., can be incorporated into the survey instrument. The example presented at the end of this chapter describes this use of the technique. Third, published data based on participant observation research can be used by questionnaire designers for certain projects. It is not likely that published monographs can be used to find solutions for specific questionnaire issues, except for very large populations which have been the subject of extensive research. But some combination of the second and third techniques will provide needed design assistance for most smaller populations. B. Selection of Respondents It is simple to state that the design or purpose of a survey dictates the selection of a respondent universe for participant observation. The task is far more difficult in practice. There is a complex literature on how to define an appropriate community for specific ethnographic research goals. If a survey is contemplated in a residential community, the universe for participant observation is easy to define geographically. But in the United States it is more common to conduct a survey among respondents defined by some characteristic besides residence. Again, the general rule is that the respondent universe for participant observation is bounded by the goals of the survey. In practice, the purpose of participant observation research _________________________ 1These include the American Anthropological Association, the Society for Applied Anthropology, and the Washington Association of Professional Anthropologists. Each of these organizations is headquartered in Washington, D.C. 31 may actually be to learn the boundaries and significant characteristics of the respondent universe. Many times it is up to the participant observer to discover who the potential respondents should be, if the goals of a survey are to be achieved. The role of individual respondents is discussed in more detail in part D, below. But it should be noted here that the results of participant observation depend upon the representativeness of the informants. Since much of the information is collected from a limited number of people, it is possible to make errors related to population variability. To avoid this, participant observers should make an effort to ensure that they observe and interview a variety of people. Whenever possible, more than one researcher should be involved in conducting the fieldwork, to reduce the likelihood of significant errors. C. Preparation The selection of participant observation field researchers can be done in consultation with one of the professional sources described under II-A, above, or through examination of published literature on the respondent community. Some researchers require formal introduction to "the field." This could be through personal letters of introduction to members of the respondent community or through temporary association with an institution with which respondents are connected in some way (for example, as employees or clients). In other cases, participant observation begins simply when the fieldworker travels to the location where respondents are to be found. D. Operation Participant observation is distinguished by four characteristics: Use of the respondents' own language; residence or participation in the respondents' community; key informants; and unstructured interviews. 1. Using the Respondents' Language The importance of conducting research in the respondents' own language may be easier to understand if "language" is thought of in the widest sense. A difference in "language" may be a regional dialect or a professional jargon. Two groups who speak the same "language", such as English, may have regional or cultural differences that cause them to infer very different meanings from the same words or arrangements of words. First, using a translator or bilingual interviewers will not solve the fundamental problem of assigning valid meanings to the answers of non-English speaking respondents. If respondents make the translations needed for answering questions, their decisions about what to include or exclude in the meaning of words may be far different from what the designer intended. It is the designer's responsibility to ensure that there will be no differences between what (s)he means and what respondents mean when each uses the ques- tionnaire. 32 Second, the period spent in learning the respondents' language has a value of its own in the research process. The participant observer's obvious effort to learn *the local language makes him/her more acceptable to potential respondents and reduces the disruption that an outside observer causes. As a result, valid observations can be made sooner than with more intrusive tech- niques. In addition, because language embodies culture, an observer learns much more than language. The frequency with which certain words, phrases, and concepts are used has often been a vital clue to researchers. In sum, the effort to learn a local language improves a researcher's efficiency and ensures that (s)he can recognize any potential failure to communicate. 2. Living Among Respondents A participant observer can gather information about a community in a variety of ways. The traditional approach involves living among the people being studied. However, participant observer research methods are also used to learn about groups that come together only at limited times or places, for example, ethnic groups or employee groups such as nurses. To study nonresidential, scattered communities such as these, participant observers spend as much time as they can with their subjects, over weeks or months, whenever the group is together. By spending relatively long periods of time among respondents, a participant observer accomplishes three things that cannot be accomplished as efficiently by any other means. First, there is an opportunity to study the variety of activities and people in the community without prejudging which is most significant. Second, a participant observer learns about the values of the community because to some extent the members' experiences are shared with them. Third, by acknowledging the research role and by seeking respondents' opinions, a participant observer earns the trust of the respondents. As a result, many respondents develop an interest in the research and even look for ways to assist. 3. Key Informants Participant observers find that much of their information is collected from key informants. Key informants are individuals who are willing to talk at length with the researcher, or who serve as an entree to many further contacts, or who reveal extraordinary knowledge about some topic. They provide richly detailed information about people and institutions. For example, elderly people are sometimes key sources of historical or genealogical data. A researcher who uses key informants does run the risk of collecting data from an individual who is not representative or who tries to mislead. These risks can be minimized through checking what is learned with a variety of other informants and through observation. There is also no reason why participant observation research cannot incorporate the principle of randomization of informants at some stage, as a check on key informants, or to counteract the fieldworker's own potential bias. But at other stages of the research, such as entry to the field, participant observation succeeds precisely because informants are allowed to volunteer. 33 4. Unstructured Interviews In addition to residence among the respondents, the participant aspect of the research involves unstructured interviews. This technique, which is described in Chapter 2, is particularly suited to the study of groups about which little is initially known. Unstructured interviews permit hypotheses about survey content and questionnaire construction to be tested and rejected very quickly. For this reason they are particularly appropriate to the beginning stage of questionnaire design. In developing a questionnaire for a respondent group that is not well known, however, methods such as unstructured interviews that yield consistent results do not necessarily yield meaningful results. If someone unfamiliar with a culture asks a limited number of questions, (s)he can get consistent responses, yet err in the meaning attributed to the responses. This is so for three reasons. First, interviews are artificial situations in which respondents may tailor answers based on their perceptions of what the questionnaire designer wants to achieve. Second, the questionnaire designer does not necessarily understand patterns of bias among respondents. For example, are there distinctive respondent strata represented? Third, the questionnaire designer cannot easily cross-check the results of these techniques. Results should be compared to responses derived in other situations, at other times, and from other respondents. Without alternative sources of information about the respondent population, interviews do not preclude major errors of interpretation. Consistency of results from such techniques may mean only that a "structured misunderstanding" is occurring. This phrase has been used recently to describe consistent and self-perpetuating mutual misunderstanding between U.S. census takers and members of a minority subculture (Hainer, 1979). In some cases when dealing with respondents from another culture, failure to communicate is recognizable. But misinterpretations might also go unrecognized. This is similar to a translation problem; if a phrase in language A is translated into language B, the words might make sense without it being in any way the sense intended. If the translation satisfies the expectations of those who speak B, no one will suspect a mistake. Complementary misunderstandings such as those described by Hainer can even permit groups to appear to cooperate. There is no single research technique that will uncover such mutual misunderstanding. But the multiplicity of methods used by a participant observer makes such an occurrence very unlikely. 5. Variations of the Method: How Much Participation? Participant observation research methods are on a continuum from unobtrusive observation to total immersion in a community as a member or actor. The optimum combination of methods depends on the characteristics of the researcher, the topic being studied, and the characteristics of the research subjects. As an example of the range of personal research styles, consider two studies of social organization among low income urban Black communities in the United States. One participant observer moved her household, including her children, to live among the families she was studying (Stack, 1974). Another participant observer was able to cultivate personal relationships in a similar community without leaving his own home. Every day he visited the 34 neighborhood he was studying and spent the day with his informants (Liebow, 1967). At the "observation" extreme of the continuum are studies of groups that could not be conducted by a resident fieldworker. If the research subjects do not live together, for example, it is impossible for a researcher to live among them. Participant observers have studied longshoremen, vagrants, and ethnic communities, for example, by visiting the subjects at the times when and places where they come together. Studies which focus on institutions, such as hospitals, factories or schools, are conducted primarily on site, at the times when respondents are willing to talk to a researcher. At the opposite end of the continuum are data collected while the researcher is a member of the subject community. The researcher might join a community to collect data, or might analyze an organization or group to which (s)he already belongs. Clearly this end of the continuum gives a researcher maximum access to insiders' values and behavior, but it is not always preferable. Not only does it create ethical dilemmas (i.e., subjects may not be aware of the researcher's intent), it often reduces the observer's capacity to interpret the observations. An insider lacks the outsider's awareness of alternatives, which is the first step to analyzing existing cultural elements. For the purposes of designing a questionnaire, participant observation research would generally tend toward the formal observer end of the continuum, as opposed to the member/participant end. Participant observers can present themselves in a variety of roles, ranging from the potentially unsettling identity of an outsider with no familiar attributes, to roles known to respondents such as student, government agent, adopted family member, etc. The purpose of selecting a role from among those available (or changing roles) is to minimize the obtrusiveness of the participant observer's presence while maximizing the likelihood of situations that provide useful observations. An experienced researcher balances obtrusive- ness and its potential adverse effect on data quality against the benefits of taking active steps to elicit certain kinds of response. 6. Variations of the Method: How Much Fieldwork? There are at least two kinds of research questions that can only be answered by spending a relatively long period in the field. The first kind deals with sensitive topics, information that people do not want to reveal. Informants who cannot expect anonymity will only discuss these topics when they trust the researcher, and that trust is developed gradually. The second kind of research question that [nay require relatively prolonged fieldwork deals with matters of which the respondents are unaware. People are seldom able to answer questions accurately about the relationships between variables in their own society. The problem is made more difficult when generalizations must be made about another society. Observations over time, however, are likely to provide a participant observer with hypotheses about the magnitude and direction of relationships between variables which can be tested through survey research. If a questionnaire deals with topics which most members of a community are familiar with, and willing to talk about, then the questionnaire designer's 35 job is relatively easy. In these cases, a questionnaire might be drafted after unstructured interviews are conducted and agreement is reached among most respondents as to the identity and meaning of important topics and concepts. If, on the other hand, a survey will deal with a topic that people are reluctant to talk about, or it is intended as a measure of variables of which respondents have only indirect knowledge, then the participant observation phase of the questionnaire design process is likely to be longer. E. Time Considerations Incorporating a participant observation research program into the development of a survey questionnaire may require a substantial amount of time. The specific duration of the research would vary with the survey topic and the type of respondent. For populations about which little or nothing is known, a year in the field might be necessary to obtain useful results. There are ways to shorten the time required for field research. A search of existing ethnographic literature may locate reports of previous field research which contain useful background information about the survey topic or the respondent universe. This literature can be used as a substitute for extended participant observation, or as a supplement to it. Another way of using the results of such research is to take more direct advantage of an expert's knowledge of the survey population. This can be accomplished by using consultants with relevant fieldwork experience during the questionnaire design process. F. Cost Considerations The cost of conducting participant observation research, consisting as it does of support for one or more researchers to live in the field, is usually a very small part of the cost of developing and conducting any survey with a large sample of respondents. If a survey should require extended original participant observation fieldwork, the total cost would be that of keeping the researcher in the field for about a year. That would include the costs of the researcher's travel and subsistence at the local level; supplies such as paper, pens, maps, film or magnetic tape; and equipment such as a camera, tape recorder, and typewriter. Sometimes a researcher also pays a research assistant a part-time local wage, and sometimes a participant observer needs a supply of such commodities as tobacco, medicine, or food as gifts to informants. The direct costs of participant observation are generally so low that they are outweighed by overhead costs incurred when a researcher is affiliated with an institution such as a university. The costs of incorporating the results of existing participant observation literature into questionnaire development are even lower; the only costs for this are salaries for personnel involved in locating, reading, and interpreting the reports of previous fieldwork. The cost of employing a knowledgeable researcher who has already studied a potential respondent population is also relatively low, consisting of charges for professional consultation during the questionnaire design process. 36 G. Mode of Data Collection Participant observation is an appropriate too] for the development of any type of questionnaire. Regardless of whether the method of administration is by face-to-face interview, telephone interview, or mail questionnaire, the knowledge gained through the use of this technique can improve the quality of the survey data. III. EXAMPLE: 1980 CENSUS OF POPULATION AND HOUSING Participant observation research was vital to the design of the 1980 Census of Population and Housing as it was carried out in the Outlying Areas of the Pacific. These areas include American Samoa, Guam, the Northern Marianas and the Trust Territory of the Pacific Islands. The traditions, languages, and environment of these islands are so different from those of the United States that the Census Bureau contracted for an anthropologist to serve as a con- sultant in the design of the questionnaire and the procedures for the 1980 census of the Pacific Islands. The anthropologist who served as consultant had spent most of the previous decade becoming familiar with the culture and languages of the Pacific area. He had conducted participant observation fieldwork on two atolls in Micronesia, and in American Samoa. His research had required fluency in several native languages, and he conducted censuses of individual communities and islands for research which included genealogical, demographic, and socio- economic analyses of island populations. The questionnaire and procedures used in the 1980 Census of Population and Housing in the Outlying Areas of the Pacific Islands were modified from those of the 1980 U.S. census in three ways. First, there were a large number of changes which reflected the unique characteristics of the Pacific Islands, including differences in environment, technology, and material culture. These were changes in labelling (of names, definitions, or response categories) which made questions and answers more comprehensible to local respondents. The second category of changes included questions where the content of a question or answer had to be changed as well as the labels. The data collected in the Pacific were, as a result, not exactly like the data collected in the United States. However, the questions used in the Pacific elicited data that could be used in building inferences comparable to those based on responses from the United States. The third category of changes consists of questions which were added because of the participant observer's knowledge about the culture of the Pacific Island communities. Some questions were ultimately added to the census of the Pacific Islands because anthropological analysis documented their significance to communities in the Pacific. A. Category 1, Label Changes Anyone familiar with the characteristics of the people and environment of the Pacific Islands would point out that many definitions and response categories used in the U.S. census were inappropriate for use in the islands. 37 The answer categories on ethnicity, for example, had to be modified to match the probable responses in the Pacific Islands. Parallel changes were made to the question on place of birth. The answer categories for these questions were selected to represent the most likely patterns of inter-island migration, given the level of specificity permitted in a census. Local land tenure patterns are reflected -in housing questions H29a and H29b. The question on the value of property (H11) which was used in the United States was only appropriate in Guam. In the other Pacific territories, where traditional land tenure is communal, individuals have no precedent for gauging the value of the property upon which their dwellings are built, so the question covered only the value of the dwelling. ASK H29a IN AMERICAN SAMOA,COMMONWEALTH OF THE NORTHERN MARIANA ISLANDS, AND THE TRUST TERRITORY OF THE PACIFIC ISLANDS ONLY. H29a. If this is a one-family house (or condominium unit) which is owned or is being bought What is the value of this house, that is, how much do you think it would sell for if it were for Sale? Do not include the value of the land. ASK H29b IN GUAM ONLY H29b. If this is a one-family house (or condominium unit) which is owned or being bought What is the value of this property, that is, how much do you think this property (house and lot or condominium unit) would sell for if it were for sale? H11. If you live in a one-family house or a condominium - unit which you own or are buying What is the value of this property, that is, how much do you think this property (house and lot or condominium unit) would sell for if it were for sale? Other examples of relatively simple label changes are found in the Questionnaire Reference Book (QRB) and the enumerators manual prepared for the Pacific Islands. The instructions for recording respondent names, for example, describe the procedures for dealing with hereditary local titles. Samoa: Reference to matai title ... when a person uses his title as the last name, the people who "belong" to this title may also take this name. For example, a person whose real last name is Talofa might report his name as John Samoa (the name of his title), and his children might have either Talofa or Samoa reported as the last name, print the last name as reported. [SIC] Additions were made to the enumerator's manual and QRB to deal with the special characteristics of housing in the Pacific. In the census of the Pacific, respondents were asked what material was used to build the walls and roof of their dwelling. One of the answer categories added was "thatch," 38 which is defined as "palm or pandanus thatch, palm leaves, straw, etc." (QRB, p. 97). Enumerators in the Pacific were taught to calculate a household's annual fuel costs if respondents said that charcoal was purchased by the bag or kerosene by the can (QRB, pp. 112-113). The participant observer knew such replies would be common in the small islands and atolls that predominate in the Pacific Islands. Finally, the simpler technology of the Pacific territories is reflected in changes made to questions about kitchen and bathroom facilities. A summary question was used in the United States where complete facilities are virtually taken for granted. But positive responses to a summary question would be so rare in the Pacific Islands that separate questions had to be asked about such facilities as hot and cold running water and bathtubs. B. Category 2, Content Changes The definitions of a number of questions were changed so that they would generate data comparable to data collected in the U.S. census. These changes were more subtle than the changes discussed in the preceding section. They were based on the participant observer's knowledge about the meaning cultural traits have for respondents. This knowledge was derived primarily from the participant aspect of research, in which the anthropologist became familiar with what respondents think and feel, the language they use, and the relationships of cultural traits to one another. The meaning that certain cultural traits have for respondents and the relationships between traits were reflected in questions about fertility, migration, language and work asked in the census of the Pacific Islands. Questions on fertility were redesigned to allow demographers to use the data from the Pacific Islands to make analyses and estimates parallel to those calculated for the United States and other places. Earlier attempts to measure individual fertility in the Pacific territories based on questions used in the U.S. census were complicated because there is a higher rate of adoption among households in certain islands. In addition, indirect measures of fertility were needed because vital registration was incomplete in the Pacific territories. To analyze individual fertility, it was necessary first to match children to their biological mothers, regardless of their current residence (e.g., adoption). In the Pacific census, three questions that had no counterparts in the U. S. census were asked for children: Is the biological mother living in the household? Is she still living? And, if she appears on the questionnaire but the relationship is not acknowledged, what is her person number?. Questions concerning children ever born were also expanded in the census of the Pacific to provide better estimates of fertility and mortality. The participant observer's experience indicated that cultural attitudes toward vital registration of such events as infant mortality and adoption made it necessary to ask these additional questions to make data comparable with the U.S. data. So the instructions pointed out that adopted children were not to be reported among children ever born, and, in addition, women were asked how 39 many of their children were still living, and if any were born alive in the last 6 months. Migration was a second subject for which a whole series of questions were modified to improve data from the Pacific. Despite the vast distances between the islands, it was not unusual for a significant number of people to be living (for work, school, or other reasons) far from the island of their birth. There was a traditional pattern of temporary migration for most young men on the islands before European contact. Today, young people of both sexes are encouraged to travel to distant education centers in the Pacific territories, or to the United States, for schooling. As a result, many adults live far from their place of birth. Migration patterns are significant in the Pacific for many reasons. Perhaps the most critical is that, on small islands, population growth can very quickly get out of balance with limited ecological resources, and the greatest source of population shifts in Pacific Islands in this century has been migration. In the Pacific territories, migration data are also significant because they are relevant to public policies concerned with education and labor. Programs are limited by the willingness of the population to migrate. Analysts need data to measure the potential effect of these policies; for example, is there resistance to migration? What is the rate? What factors cause return migration? Who migrates, and what happens to those who are left behind? To answer these questions within the limits of the census, each respondent was asked about place of birth (if it was not the place of enumeration), mother's and father's place of birth, and any lengthy period of residence or activity in the United States. The question on residence 5 years ago which was used in the United States was retained as well. The participant observer was able to predict that questions about language used in the U.S. census would cause problems in the Pacific. Few native residents of the Pacific Island territories use English as their primary language in the home. For the non- European population in the Pacific territories, native languages (and even multiple native languages) would be reported far more frequently than English as the language used at home. The question used in the U.S. census to measure fluency in English (How well does this person speak English?) would be of little use among a population with a majority of non-English speakers. In the Pacific census, therefore, a question for all respondents was designed to identify actual language practice. Major changes were also made in the questions dealing with the "work" that adults reported doing in the Pacific. Economic activity in the Pacific Islands is very different from that in the United States. A high proportion of adults in the islands derive support from indigenous noncash-related subsistence activities. This includes producing food or goods for home consumption, with little or nothing exchanged for cash or other goods. In the Pacific, subsistence activity was provided as an alternative response in the question on activity last week. It was also incorporated into the 40 series of questions on income, partly as a check to ensure that only activities distinct from the cash economy were being reported. These questions identified persons involved exclusively in subsistence activity and distinguished them from persons in the cash labor force who were not working. They also allowed subsistence activity to be reported as a distinct activity pursued along with participation in the cash economy in some form. C. Category 3, Questions Significant in the Pacific Territories The final category includes questions that were unique to the Pacific census. These were included because they dealt with topics that are significant in the Pacific Islands and which merit collection of data in an enterprise as costly as the decennial census. The anthropologist was able to help the Bureau evaluate the relative importance of potential census data to the people of the Pacific Islands. The simplest example is literacy. Citing patterns of native language use and English fluency, the participant observer documented the need for a question on literacy. His experience suggested that data from this question would be important in analysis of programs related to education, training, and employment. In the enumerator instructions literacy was defined as the ability to read or write a personal letter providing an explanation comprehensible both to native enumerators and respondents. A second major illustration is found in the example of the questions dealing with migration. Of the 31 population items covered in the Pacific census, 9 were directly related to analysis of migration patterns. Because of the immense significance of migration phenomena to the interpretation of a variety of related social processes in the Pacific territories, as described in part B, a lengthy series of questions on migration was eventually included in the questionnaire. In conclusion, the census in the Pacific territories differed from the U.S. census in many ways. Some of the differences appeared superficial. Other changes allowed the answers from the Pacific to serve the same analytical purposes as answers from the United States. These changes required familiarity with local culture, including knowledge of native languages and native use of English words and categories. Finally, the most fundamental differences in the Pacific questionnaire are reflected in the topics chosen. The anthropologist, serving as consultant, helped the Bureau select the questions which were most valuable for use within the limits of a decennial census. Part III Procedures for Testing the Questionnaire Draft The three previous chapters identified tools that can be used to obtain background information to assist in developing the first draft of a questionnaire; i.e., before any specific survey questions are written. However, other means are more commonly used to obtain such information. For example, the questionnaire designer can review available literature on the topic and questionnaires from other surveys, if there are any, that also addressed the identified data requirements . If another questionnaire exists, persons involved in that survey, if available, and reports on the results should also be consulted as possible sources for learning more about developing a similar questionnaire. Often, unless one's own research indicates otherwise, specific wording of a question can be adopted from another survey. In addition to having wording that has been "tested," it might allow the data to be compared with another source. Even if other questionnaires on the proposed subject of the survey do not exist, there are several reference sources the designer might use for guidance in writing questions. Since many household surveys include questions on respondent characteristics for categorization into analytic groupings, several attempts have been made to gain acceptance for standard wording of these types of questions. Two such attempts are Basic Background Items for 41 42 U.S. Household Surveys (Social Science Research Council , 1975) and Social Concepts Directory for Statistical Surveys (Statistics Canada, 1980) These reports, or others like them, may be useful in determining how to word questions on age, marital status, education, income, etc. Although there is still some debate on the possibility and desirability of standardizing questions, it is generally agreed that even small differences in the wording of a question may affect the resulting data. In addition, many books have been written on how to design questionnaires. Works such as The Art of Asking Questions (Payne, 1951), Designing Forms for Demographic Surveys ( Sirken, 1972) and Asking Questions (Sudman and Bradburn, 1982) are valuable sources of general advice on how to write questions and on other aspects of designing questionnaires. Finally, before the first attempt is made to draft questions, there are some other basic issues which need to be considered. These include such things as the number of interviews with each respondent (more than one may be necessary), the frequency of the interviews, the data collection mode, and the type of respondent. (See Chapter 1 for further discussion of these issues.) The overall structure of the questionnaire should also be established showing the organization and relationship of the various components, pieces, or sections making up the entire questionnaire. For example, a questionnaire may have separate sections or even physically separate documents for different topics covered in the survey and/or for different persons within the household who are to be interviewed. Once the overall structure of the questionnaire is determined, it can serve as a guide for developing the individual questions. Writing the questions is a critical step because the results of the survey depend on the answers given to each question. The question wording must be clear and comprehensible to most respondents to minimize biasing of the survey results. In addition to writing the questions, the designer must sequence them in a natural order that will flow smoothly from one topic to another. The flow may be improved by using screening questions and skip patterns. Screening questions are specifically designed to determine whether certain questions should be asked of a particular respondent. For example,, respondents might be asked if they have any children before they are asked a series of questions about their children; respondents without children would be "skipped over" (i.e., not asked) these questions. Skip patterns are used in the same way to avoid inapplicable questions depending on the respondent's answer to a previous question. When the first draft of the questionnaire has been prepared, it should be subjected to extensive review. The reviewers should include the analysts and other staff members working on the survey and, whenever possible, other persons outside the staff who are familiar with the topic of the questionnaire or uses of the data. The review process should ensure that the data requirements or objectives of the survey are being met. The draft can also be administered to friends and/or coworkers to check for problems such as skip pattern errors or awkward wording. Sometimes questions which look good on paper sound stiff or verbose when read aloud. The responses to the draft at this point might indicate how respondents selected for the survey will react to the questions. After considering the comments and suggestions received during the review, another draft of the questionnaire will probably 43 need to be prepared to incorporate revisions. Several iterations of the questionnaire and review process may be necessary before the designers are satisfied with the product. At this stage, it is imperative that the draft questionnaire, be tested with the population under study. This part of the report discusses various ways of testing the questionnaire under field conditions. Field testing is particularly appropriate for questionnaires administered by interviewers in person or by telephone. It also may be used for self-administered questionnaires which are usually mailed to respondents. Another type of testing which is more useful for self-administered questionnaires is laboratory or classroom testing. In this type of testing, a subjective evaluation is made of the questionnaire under controlled or semicontrolled conditions. This is done by having participants complete the draft questionnaire, in a group setting or individually, and then talk with the questionnaire designers about problems encountered. However, only field testing is covered in this report. This report divides field testing into two broad categories: informal and formal. The main distinctions between tests in these categories are in the size and the sophistication of their sample design and the completeness of their objectives. Informal testing relies primarily on subjective evaluations of the questionnaire; whereas, formal testing relies on statistical evaluations. As the word "informal" implies, less control is necessary in choosing the sample and conducting the interviews for such testing. The next chapter, Chapter 5, describes informal testing in more detail; formal testing is described in Chapter 6 with emphasis on two variations: pilot studies and split sample tests. These chapters describe the circumstances and factors that should be considered in determining the type of testing to be undertaken in preparation for a survey. Chapter 5 Informal Testing I. INTRODUCTION Once the initial version of the questionnaire has been drafted, several types of field tests can he conducted to refine the questionnaire. One type is the informal test. In this report, informal testing refers to a questionnaire field test involving a relatively small number of interviews in the kind of setting chosen for the final survey (i.e., home, work, etc.) as opposed to a laboratory setting. In this type of testing, the detection and correction of errors or weaknesses in the questionnaire draft depends mainly upon subjective information provided by interviewers and observers. The test is not designed to be evaluated on a rigorous statistical basis. If a series of tests is planned in the questionnaire development process, an informal test is frequently a first step, with formal tests involving more sophisticated types of evaluation coming later in the refinement process. Or, it may be the last step in the process to ensure that the revisions made as a result of previous formal tests work well together. If time and money permit only a single test, the relative speed and low cost of an informal test (in comparison with a formal test) may make it a logical choice. In terms of the questionnaire design issues outlined in Chapter 1, informal tests are particularly appropriate and useful in discovering poor question wording or ordering, errors in questionnaire layout or instructions, and negative response effects caused by the length of the interview or a respondent's inability or unwillingness to answer the questions. In addition, they can be used to a lesser extent to assess the feasibility of using a parti- cular concept in a questionnaire, to determine if the questions seem to elicit appropriate responses, and to suggest additional questions or response categories which can be precoded on the questionnaire. Other relevant objective information which might affect the final questionnaire design can also be obtained in an informal test-- e.g., a preliminary indication of the interview length (called respondent burden by OMB), the refusal rate, and field costs. Principal Contributor: Dawn D. Nelson 45 46 II. METHOD A. Personnel and Skill Requirements Several types of Skills are necessary to conduct an informal test, some of which may be combined in a single person. However, it is usually necessary to have a team of persons or several different groups of people. If a team of persons is used, someone must coordinate all the activities involved. These include selecting the test site, selecting the sample, selection an