For decades professional, academic, and trade associations, as well as government agencies, and individual academics and practitioners have been discussing survey standards. Debates have centered around 1) whether surveys should meet certain standards and, if so, what those standards should be and 2) how various features of surveys should be defined and what those definitions should be. Within the general debate over standards and definitions, there has been considerable discussion over the issue of survey nonresponse. This paper reviews the current position of professional, academic, and trade associations on the matter of survey standards for final disposition codes and outcome rates.¹

Professional, Academic, and Trade Association Codes

To ascertain the current situation regarding nonresponse standards the codes and official positions of the 14 major professional, academic, and trade associations involved in survey and marketing research were examined. The core professional associations on the survey research side are the American Association for Public Opinion Research (AAPOR) and the World Association for Public Opinion Research (WAPOR). Then the European Society for Opinion and Marketing Research (ESOMAR) is a bridge between survey research and market research and the main market research associations are the American Marketing Association (AMA), Marketing Research Association (MRA), and the Association for Consumer Research (ACR). The main statistical groups are the American Statistical Association (ASA), the International Association of Survey Statisticians (IASS), and the International Statistical Institute (ISI). The main trade associations or associations of organizations are the Council of American Survey Research Organizations (CASRO), the Council for Marketing and Opinion Research (CMOR), the National Council of Public Polls (NCPP), and, on the market research side, the Advertising Research Foundation (ARF). Finally, there is an association of associations, the Research Industry Coalition (RIC), which includes among its members AAPOR, ACR, AMA, ARF, CASRO, MRA, and NCPP, plus the American Psychology Association, the Newspaper Association of America, the National Association of Broadcasters, the Professional Marketing Research Society of Canada, the Qualitative Research Consultants Association, and the Travel and Tourism Research Association.²

a. AAPOR

 The AAPOR Code of Professional Ethics and Practices (AAPOR, 1986) in its Standards of Minimal Disclosure states:

At a minimum, the following items should be disclosed:

5. Size of sample and, if applicable, completion rates and information on eligibility criteria and screening procedures.

 AAPOR also has official statements on nonresponse beyond its code provisions. In its Best Practices booklet (AAPOR, 1997) AAPOR cites the following:

9. Maximize cooperation or response rates with the limits of ethical treatment of human subject.

12. Disclose all methods of the survey to permit evaluation and replication....
A comprehensive list of the elements proposed for disclosure...includes:

> size of samples and sample disposition - the results of sample implementation, including a full accounting of the final outcome of all sample cases: e.g. total number of sample elements contacted, those not assigned or reached, refusals, terminations, non-eligibles, and completed interviews or questionnaires;

> documentation and full description, if applicable, of any response or completion rates cited (for quota designs, the number of refusals), and (whenever available) information on how nonrespondents differ from respondents;

 In "Standard Definitions" (AAPOR, 1998) AAPOR provides a detailed set of final disposition of case codes for telephone and in-person household surveys, a set of formulas using these codes to calculate response and other outcome rates, and guidance on various related matters such as the definition of completed cases and how substitution should be reported. The Standard Definitions have in turn been adopted by AAPOR's official journal, Public Opinion Quarterly (Price, 1999 and "Notice," 1999).³

b. WAPOR

The WAPOR Code of Professional Ethics and Practices (WAPOR, n.d.) states:

Every complete report on a survey should contain an adequate explanation of the following points:

f) the degree of success in actually carrying out the [sample] design, including the rate of non-response and a comparison of the size and characteristics of the actual and anticipated samples;

WAPOR's journal, the International Journal of Public Opinion Research, has no guidelines on nonresponse.

c. CASRO

The CASRO Code of Standards and Ethics for Survey Research (CASRO, 1997) states:

A Research Organization's report to a Client or the Public should contain, or the Research Organizations should be ready to supply to a Client or the Public on short notice, the following information about the survey:

6. A description of results of sample implementation including (a) a total number of sample elements contacted, (b) the number not reached, (c) the number of refusals, (d) the number of terminations, (e) the number of non-eligibles, (f) the number of completed interviews.

7. The basis for any specific "completion rate" percentages should be fully documented and described.

However, the information in points 6 and 7 is not required among that which must be included in "any general public releases of survey findings."

Details of the definition and calculation of response and completion rates are given in a report of the CASRO Task Force on Completion Rates (CASRO, 1982; Frankel, 1983).

d. NCPP

The NCPP in its Principles of Disclosure (NCPP, 1998a) does not include anything relating to response rates in what must be automatically provided, but does provide for reporting under the following circumstances:

It is reasonable to require disclosure of pertinent information regarding methods when questions are raised about survey results or how a survey has been conducted.

The Code then describes a procedure by which a survey methods inquiry may be pursued. It indicates that the NCPP Committee on Disclosure may require the release on certain information including (but not limited to):

Sample design: sample frame, stages, number of sampling points, clustering, respondent selection, number of callbacks, refusal rate, substitution rules (if any), and non-contact rate of sample units;

In addition in its "20 Questions for Journalists" (NCPP, 1998b) which lists "questions for the journalist to ask the pollster before reporting poll results" the NCPP instructs:

You ought to know how many people refused to answer the survey or were never contacted. The nonresponse rate is the percentage of people contacted who should have been interviewed, but were not.

e. CMOR

CMOR has no code of ethics or standards. One of its two main missions is "promoting internal research practices to encourage respondent cooperation and self-regulation." It refers to as a benefit of membership its "efforts to increase respondent cooperation," and as one of its "top ten reasons to join CMOR" says "7. CMOR is committing resources to improve respondent cooperation (CMOR, 1998)."

CMOR also coordinated a study with eight survey firms using an experimental design to measures aspects of survey design associated with nonresponse ("Respondent Cooperation," 1996).

f. RIC

RIC (RIC, 1999) takes the following relevant positions on research:

Describe how the research was done in enough detail that a skilled researcher could repeat the study.

Explain the applicability and limitations of the research and provide information users need to judge for themselves the usefulness of the research and its quality.

g. ESOMAR

ESOMAR in its International Code of Marketing and Social Research Practice (ESOMAR, 1986) provides:

25. The Researcher must provide the Client with all appropriate technical details of any research project carried out for the Client.

In its notes to Rule 25 ESOMAR elaborates:

The Client is entitled to the following information about any marketing research project to which he has subscribed:
2. Sample - a description of the intended and actual universe covered.
- the size, nature, and geographic distribution of the sample (both planned and achieved); and where relevant, the extent to which any of the data collected were obtained from only part of the sample.
- details of the sampling method and any weighting methods used.
- where technically relevant, a statement of response rates and a discussion of any possible bias due to non-response.

The ESOMAR code has in turn been adopted by a number of national organizations such as the Market Research Societies of Great Britain and Australia.

ESOMAR official journal, Marketing and Research Today, has no guidelines on nonresponse.

h. AMA

The AMA in its Code of Ethics refers (AMA, 1999) to:

Maintaining research integrity by avoiding misrepresentation and omission of pertinent research data.

In its official journal, the Journal of Market Research, authors using data are required to make available to the editors "exact information regarding their procedures, materials (excluding data), and stimuli..." They are also told that "empirical research should be reported in sufficient detail that readers can evaluate and replicate the methodology." As part of the "acceptance criteria" authors are expected to abide by the following:

When data collection is discussed, consider the relevance of the sample to the subject matter. Carefully chosen sample groups are preferable to haphazardly chosen subjects who have little knowledge of or relevance to the subject being studied.

Give as much information as possible about the characteristics of the sample and its representativeness of the population being studied.

Do not ignore the nonrespondents. They might have different characteristics than the respondents.

i. MRA

MRA in its Code of Data Collection Standards (MRA, 1999) requires that its members "will report research results accurately and honestly."

j. ACR

ACR's mission is "to advance consumer research and facilitate the exchange of scholarly information..." It does not have any code and "does not monitor, evaluate or adjudicate ethical behavior." It further states that "Everything ACR members do in advancing consumer research is based on the belief that each other's research is honestly designed, analyzed, and reported (ACR, 1999)."

k. ARF

ARF does not have any code of ethics or standards. It does have as its mission "Promoting the highest quality business and consumer marketing, advertising and media research by developing guidelines and standards, and by providing objective and impartial technical advice and expertise." Research activities are organized by Research Councils and these include the Research Quality, Methods, and Practices Council which aims "to support the development, application, and use of sound research methods and practices conductive to better business decision making for advertising, marketing, and media." Among the "key ARF research projects" is one on "The Effect of Interview Attempts on Survey Results." However, unlike most other key research papers it does not put forward either guidelines or standards (ARF, 1999).

ARF's official journal, the Journal of Advertising Research, requires contributors to "make clear what the sampling frame was, why it was selected, and what the response rate was (ARF, 1999)."

l. American Statistical Association

The ASA's Ethical Guidelines for Statistical Practice (ASA, 1998) under Reporting Responsibilities requires:

5. Account for all data considered in a study and explain the sample(s) actually used.
6. Report the sources and assessed adequacy of the data.
8. Report the analytic methodology and its relation to the assumptions.
10. Report the limits of statistical inference of the study and possible sources of error, both random and systematic.

The Ethical Guidelines for Statistical Practice of the Survey Research Methods Section (ASA-SRM, 1998) states:

Statistical work must be visible and open to assessment with respect to quality and appropriateness in order to advance knowledge, and that such assessment may involve an explanation of assumptions, methodology, and data processing used...
[Statisticians should] be prepared to document sources used in an inquiry.

Also, the Section on Survey Research Methods in its series "What is a Survey?" in its booklet on "How to Collect Survey Data?" urges following up on nonrespondents to reduce the nonresponse rate and nonresponse bias (ASA-SRM, 1995a; 1995b; 1995c). In addition, the ASA and ASA-SRM have participated in various research efforts and conferences on nonresponse (Smith, 1999; "Report," 1974).

The official journal of the ASA, the Journal of the American Statistical Association, has one requirement for authors that indirectly touches on nonresponse:

Whenever a data set is used, its source should be fully documented. When it is not practical to include the whole data set in the paper, the paper should state how the complete data set can be obtained. Unless conditions of security or confidentiality intervene, availability of the data on which the paper is based is a requirement of publication.

m. ISI

The ISI Declaration on Professional Ethics (ISI, 1985) states:

2.4 Statisticians are frequently furnished with information by the funder or employer who may legitimately require it be kept confidential. Statistical methods and procedures that have been utilised to produce published data should not, however, be kept confidential....
One of the responsibilities of the statistician's professional citizenship, for instance, is to be open about methods in order that the statistical community at large can assess, and benefit from, their application. Thus, insofar as it is practical, methodological components of inquires should be free from confidentiality restrictions so that they can form part of the common intellectual property of the profession.

3.2 Within the limits of confidentiality requirements, statisticians should provide adequate information to colleagues to permit their methods, procedures, techniques, and findings to be assessed.

The ISI's official journal, International Statistical Review, has no guidelines on nonresponse.

n. IASS

IASS is a section of ISI. It does not have its own code of standards/ethics and does not appear to have any relevant official positions. The IASS' official journal, Survey Statistician, has no guidelines on nonresponse.

Summary

Of the 14 professional, academic, and trade organizations examined, three have no codes nor any relevant official statements (CMOR, ACR, and IASS - but IASS is covered by the code of ISI). Another three organizations have only brief general statements about doing good, honest research (AMA, ARF, MRA). Yet another three have general pronouncements about being open about methods and sharing technical information with others, but no details on what should be documented (ASA, ISI, RIC). Then, there are five that have some requirement regarding nonresponse (AAPOR, CASRO, ESOMAR, NCPP, and WAPOR).

Of the eight organizations that have an official journal (AAPOR, WAPOR, ESOMAR, AMA, ARF, ISI, IASS, ASA), one (AAPOR-POQ) has a definite standard about reporting and calculating response rates, two have some general pronouncements that mention nonresponse bias or the response rate (AMA-JMR and ARF-JAR), and one has a marginally relevant standard on data sharing (ASA-JASA).⁴

Of the five referring to nonresponse in their codes and statements, all require that response rates (or some related outcome rate) be reported. AAPOR and WAPOR put nonresponse among those facets of survey methodology that must be reported automatically and routinely while CASRO, ESOMAR, and NCPP have less comprehensive reporting rules. Three organizations provide at least some definition of response and/or related outcome rates, but only in non-binding documents and statements and not as part of their codes (AAPOR, CASRO, NCPP) and two provide no definitions (ESOMAR and WAPOR). Two organizations deal with the issues of nonresponse bias. The WAPOR code, right after requiring the reporting of the nonresponse rate, calls for information on the "comparison of the size and characteristics of the actual and anticipated samples." AAPOR in its "Best Practices," but not its code, urges that nonresponse bias be reported. Finally, AAPOR, as part of "Best Practices," but not its code, indicates that survey researchers should try to maximize response rates.

In addition, professional, academic, and trade organizations have advanced the cause of standards by their general promotion and dissemination of research methods. For example, as Hollander (1992) has observed, "the annual AAPOR conference was recognized early on, together with POQ, which is older still, as a means of advancing standards..." Specific AAPOR-POQ examples include (Kviz, 1975; the Huffington discussion in Smith, 1999; and Frankovich, 1999).

Conclusion

In brief, only the professional, academic, and trade organizations at the core of survey research take up nonresponse in their codes, official statements, and organizational journals. Market research and statistical organizations do not explicitly deal with nonresponse issues in their codes and standards and only marginally address these in the guidelines of their official journals. Even among the organizations that do address the matter of nonresponse the proclaimed standards are minimal. Some, but not automatic, reporting is required by all of the core organizations. However, definitions are provided by only three of the five and none include them as part of their codes. Other aspects such as nonresponse bias and performance standards are barely touched upon. Thus, even among those organizations that consider nonresponse, reporting standards are incomplete, technical standards are lacking and/or regulated to less official status, and performance standards are non-existent.

Endnotes

¹On the general reasons that professions and other groups adopt standards see Wilensky, 1964.

²Trade or industry associations are those in which organizations rather than individuals belong and consist of CASRO, CMOR, NCPP, ARF, and RIC. Professional and academic associations have individuals as members.

³For a history of AAPOR's development of its standards see Smith, 1999.

⁴On the editorial policies of other journals see Smith, 1999; Presser, 1980; and "Instructions," 1998.

References

Advertising Research Foundation, www.arfsite.org, 1999.

American Association for Public Opinion Research, Best Practices for Survey and Public Opinion Research and Survey Practices that AAPOR Condemns. May, 1997.

American Association for Public Opinion Research, Code of Professional Ethics and Practices. n.p.: AAPOR, 1986.

American Association for Public Opinion Research, Standard Definitions: Final Disposition of Case Codes and Outcome Rates for RDD Telephone Surveys and In-Person Household Surveys. Ann Arbor, MI: AAPOR, 1998.

American Marketing Association, www.ama.org, 1999.

American Statistical Association, Ethical Guidelines for Statistical Practice. draft, 1998.

American Statistical Association, Survey Research Methods Section, What is a Survey? Alexandria, VA: ASA, 1995a.

American Statistical Association, Survey Research Methods Section, How to Plan a Survey? Alexandria, VA: ASA, 1995b.

American Statistical Association, Survey Research Methods Section, How to Collect Survey Data? Alexandria, VA: ASA, 1995c.

American Statistical Association, Survey Research Methods Section, Ethical Guidelines for Statistical Practice. Alexandria: VA:ASA, 1998.

Association for Consumer Research, www.acrweb.org, 1999.

Council for Marketing and Opinion Research, www.cmor.org, 1999.

Council of American Survey Research Organizations, "On the Definition of Response Rates," Special Report, 1982.

Council of American Survey Research Organizations, The Rules We Live By...Code of Standard and Ethics for Survey Research. Port Jefferson, NY: CASRO, 1997.

European Society for Opinion and Marketing Research, ESOMAR Guidelines. n.p.: ESOMAR, 1986.

Frankel, Lester R., "The Report of the CASRO Task Force on Response Rates," in Improving Data Quality in a Sample Survey, edited by Frederick Wiseman. Cambridge, MA: Marketing Science Institute, 1983.

Frankovic, Kathy, "AAPOR Addresses Standard Issues," AAPOR News, (Winter, 1998), 5.

Hollander, Sidney, "Survey Standards," in A Meeting Place: The History of the American Association for Public Opinion Research, edited by Paul B. Sheatsley and Warren J. Mitofsky. n.p.: American Association for Public Opinion Research, 1992.

"Instructions to Contributors," American Political Science Review, 92 (June, 1998), viii.

International Statistical Institute, Declaration of Professional Ethics. Voorburg, The Netherlands: ISI, 1985.

Kviz, Frederick J., "Toward a Standard Definition of Response Rate," Public Opinion Quarterly, 41 (Summer, 1977), 265-267.

Market Research Association, www.mra-net.org, 1999.

National Council of Public Polls, Principles of Disclosure. 11/5/1998a.

National Council of Public Polls, Twenty Questions. 11/5/1998b.

"Notice to Contributors," Public Opinion Quarterly, 63 (Spring, 1999), 159-161.

Presser, Stanley, "Comment on Nelson and on Baloyra," American Political Science Review, 74 (June, 1980), 459-460.

Price, Vincent, "Editorial Note," Public Opinion Quarterly, 63 (Spring, 1999), i-ii.

"Report of the ASA Conference on Surveys of Human Populations," American Statistician, 28 (February, 1974), 30-34.

Research Industry Coalition, www.researchindustry.org, 1999.

Respondent Cooperation and Industry Image Survey. Port Jefferson, NY: Council for Marketing and Opinion Research, 1996.

Smith, Tom W., "Developing Nonresponse Standards," NORC Report, August, 1999.

Wilensky, Harold L., "The Professionalization of Everyone?" American Journal of Sociology, 70 (Sept., 1964), 137-158.

Wiseman, Frederick, "Editor's Preface," in Proceedings of Marketing Science Institute Workshop, edited by Frederick Wiseman. Cambridge, MA: Marketing Science Institute, 1983.

Wiseman, Frederick and McDonald, Philip, The Nonresponse Problem in Consumer Telephone Surveys. Report No. 78-116. Cambridge, MA: Marketing Science Institute, 1978.

Wiseman, Frederick and McDonald, Philip, Towards the Development of Industry Standards of Response and Nonresponse Rates. Report 80-101. Cambridge, MA: Marketing Science Institute, 1980.

World Association for Public Opinion Research, Code of Professional Ethics and Practices. Chapel Hill, NC: WAPOR, n.d.