| Federal
Committee on Statistical
Methodology Office of Management and Budget |
FCSM
Home ^ Methodology Reports ^ |
Statistical Policy Working Paper 9 - Contracting for Surveys
Click HERE for graphic.MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY (June 1983) Maria Elena Gonzalez (Chair) Office of Information and Regulatory Affairs (OMB) Barbara A. Bailer Bureau of the Census (Commerce) Norman D. Beller National Center for Education Statistics (Education) Yvonne M. Bishop Energy Information Administration (Energy) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Industrial Economics (Commerce) Zahava D. Doering Defense Manpower Data Center (Defense) Marie D. Eldridge National Center for Education Statistics (Education) Daniel H. Garnick Bureau of Economic Analysis (Commerce) Charles, D. Jones Bureau of the Census (Commerce) Daniel Kasprzyk Bureau of the Census (Commerce) William E. Kibler Statistical Reporting Service (Agriculture) Thomas Plewes Bureau of Labor Statistics Raymond C. Sansing Internal Revenue Service (Treasury) Fritz J. Scheuren Internal Revenue Service (Treasury) Monroe G. Sirken National Center for Health Statistics (Health and Human Services) Wray Smith Energy Information Administration Energy Thomas G. Staples Social Security Administration (Health and Human Services) OFFICE OF INFORMATION AND REGULATORY AFFAIRS Christopher DeMuth, Administrator Thomas D. Hopkins, Deputy Administrator for Regulatory and Statistical Analysis Maria E. Gonzalez, Chairperson Federal Committee on Statistical Methodology PREFACE The Subcommittee on Contracting for Statistical Surveys was formed to review current practices by agencies in contracting for surveys and to make recommendations for their improvement. This working paper, the report of that Subcommittee, addresses the following: technical expertise needed in statistical contracting, preparation of the Request for Proposal, contractor selection, and post-award activities. While the report is intended to be useful to agencies that may utilize contracting for surveys, a broader audience may find the report of interest. Some" of the material should be useful as a supplement to the existing training provided, agency project officers. Seminars and meetings will be organized to discuss the findings of this subcommittee with Federal agency personnel involved with contracting for surveys. The working paper was prepared by the members of the Subcommittee on Contracting for Statistical Surveys, Federal Committee on Statistical Methodology. The Subcommittee was chaired by Thomas G. Staples, Social Security Administration, Department of Health and Human Services. As a subcommittee report, this document does not necessarily represent the views of the Office of Management and Budget. MEMBERS OF THE SUBCOMMITTEE ON CONTRACTING FOR STATISTICAL SURVEYS Thomas G. Staples, Chairperson Social Security Administration Department of Health and Human Services Henry P. Brehm Social Security Administration Department of Health and Human Services David W. Chapman Bureau of the Census Department of Commerce Maria E. Gonazalez Office of Information and Regulatory Affairs Office of Management and Budget Nancy D. Pearce National Center for Health Statistics Department of Health and Human Services Richard J. Pratt Bureau of Labor Statistics Department of Labor Margaret Weidenhamer Statistical Reporting Service Department of Agriculture (ii) ACKNOWLEDGEMENTS This report represents the collective effort of the Subcommittee on Contracting for Statistical Surveys. Although the subcommittee members reviewed and commented on all parts of the report, the following specific individuals were responsible for preparing the chapters: Chapter Authors I Henry P. Brehm II David W. Chapman III Margaret Weidenhamer/Richard J. Pratt IV Henry P. Brehm V Nancy D. Pearce/Thomas G. Staples Appendix I J. Timothy Sprehe II Nancy D. Pearce In addition to the subcommittee members, special recognition goes to the Federal Committee on Statistical Methodology, whose membership identified the need for this report. Portions of several committee meetings were devoted to feedback to the subcommittee as the report was developing; individual members of the committee provided detailed oral and written comments. In addition, special recognition goes (1) to the contractor and Federal agency personnel who reviewed draft copies of the report and (2) to the contractor and Federal agency representatives who participated in the surveys of sponsoring agencies and contractors. Also, several individuals who made significant contributions to the report participated in the early stages of the subcommittee's work. These include Thomas B. Jabine, Leon Gilford, Barbara A. Bailar, Bette S. Mahoney, Henrietta Hyatt, David B. Orr, Irene C. Montie, J. Timothy Sprehe, and Corrinne B. Lennox. Jacob Deutch and David Arnaudo participated in the surveys of agency and contractor representatives. Among the subcommittee members special recognition goes to Nancy D. Pearce for organizing the draft and final versions of the report and to Nancy D. Pearce and Henry P. Brehm, for pulling together a compact version of the report for the 1982 proceedings of the American Statistical Association. Maria E. Gonzalez, on numerous occasions, chaired meetings and performed other responsibilities when the chairperson was absent. Thomas G. Staples (iii) TABLE OF CONTENTS Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . 1 Background and Purpose . . . . . . . . . . . . . . . . . . . 1 The Chapters to Follow . . . . . . . . . . . . . . . . . . . 4 Chapter 2. Technical Skills Required in Contracting for Surveys. . . . . . . . . . . . . . . . . . . . . . 7 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 7 Basic Types of Skills Required . . . . . . . . . . . . . . . 8 Technical Input at Various Stages of Contracting for a Survey. . . . . . . . . . . . . . . . . . . . . . . . 12 Some Technical Aspects Associated with Current Practice. . .14 Chapter 3. Considerations in the Preparation of Requests for Proposal . . . . . . . . . . . . . . . . . . . . .17 Introduction . . . . . . . . . . . . . . . . . . . . . . . .17 Initial Considerations . . . . . . . . . . . . . . . . . . .17 Possible Contents of Requests for Proposal . . . . . . . . .20 Role of the Project Officer. . . . . . . . . . . . . . . . .28 Evaluation Criteria. . . . . . . . . . . . . . . . . . . . .29 Chapter 4. Contractor Selection Process . . . . . . . . . . .31 Introduction . . . . . . . . . . . . . . . . . . . . . . . .31 Components of the Selection Process. . . . . . . . . . . . .31 Technical Evaluation of Content and Capability . . . . . . .33 Discussions with Offerors in the Competitive Range . . . . .34 The Final Award Process. . . . . . . . . . . . . . . . . . .35 Chapter 5. Post-Award Activities. . . . . . . . . . . . . . .37 Introduction . . . . . . . . . . . . . . . . . . . . . . . .37 General Techniques for Monitoring Contractor Performance . .38 Administrative Responsibilities of the Project Officer . . .38 Quality Assurance Responsibilities of the Project Officer. .40 Appendix I. Informal Survey of Agencies and Contractors Concerning Contracting for Surveys . . . . . . . .47 Introduction . . . . . . . . . . . . . . . . . . . . . . . .47 Federal Agency Responses . . . . . . . . . . . . . . . . . .48 Contractor Responses . . . . . . . . . . . . . . . . . . . .50 Appendix II. Office of Management and Budget Requirements: Information Collection Budget and Clearance. . . .55 Introduction . . . . . . . . . . . . . . . . . . . . . . . .55 Information Collection Budget. . . . . . . . . . . . . . . .56 Clearance. . . . . . . . . . . . . . . . . . . . . . . . . .56 After Receipt of Approval. . . . . . . . . . . . . . . . . .58 (v) Chapter 1. INTRODUCTION BACKGROUND AND PURPOSE Surveys are a major source of data for many agencies of the Federal Government. Surveys deal with a multitude of subjects, for example, demographic, economic, psychological, sociological, and medical topics, spanning a multiplicity of diverse governmental concerns for the general welfare. Included are surveys on the effectiveness of government programs, the impact of regulated and unregulated industrial activities, and problems for which population groups are currently or may in the future be at risk or in need of government assistance in some form. Although these topics are highly diverse. the issues in the use and conduct of surveys to obtain data on them are far more amenable to systematic treatment with a unified focus. For the Federal Government to conduct a survey, there are the three following primary options: - do the work in-house; - enter into an interagency agreement; or - contract the work out. These options can be used in combination with each other, and in actual practice this is often the case. An agency may have the data collection, reduction, and tabulation phases of a survey conducted under a contract or an interagency agreement.1 It may choose to design the survey and conduct the analysis in-house. The Annual Survey of Manufactures is an example of a statistical survey carried out by in-house staff. Planning and preparation, data collection and tabulation, and presentation of the data are done by or under the immediate supervision of Bureau of the Census employees. Additional staff needed to satisfy the demands of high volume data collection and handling are obtained on a limited individual employment basis. Because of its capacity to conduct statistical surveys, the Bureau of the Census is one of the agencies often used by Government agencies to conduct surveys. For example, the Current Population Survey conducted by the Bureau of the Census for the Bureau of Labor Statistics is a statistical survey carried out through an interagency agreement. However, agencies that do reimbursable work have limited subject matter expertise beyond their own missions. Additionally, their facilities are in demand and are by no means unlimited. For many survey efforts, then, it becomes necessary that agencies of government utilize the services of outside organizations. A range of such services is available under contract from non-federal government ___________________________ 1Statistical Policy Working Paper 8, "Statistical Interagency Agreements" (October 1982; Regulatory and Statistical Analysis Division, Office of Information and Regulatory Affairs, Office of Management and Budget), prepared by the Subcommittee on Statistical Interagency Agreements, presents data on the prevalence of interagency agreements and summarizes the prevailing practices and policies of Federal agencies with respect to their use. 1 sources. Depending on the needs of the individual agency or survey, these may include carefully delineated and defined services related to data collection and processing or more general involvement in all phases of a survey from planning to final report. In some cases the suggestion of the subject matter area in which a survey is needed may originate outside the government and result in a proposal for the entire effort, including the definition of the area for study. The overall functions of the Office of Management and Budget (OMB) as related to review and approval of agency budgets, program plans, and data collection plans set limits and provide guidance for the use of statistical surveys. For the most part, the issues of concern center on the need for the data for policy and program activities, the reasonableness of the data collection plan and questionnaires, the coordination among agencies with overlapping interests to avoid duplication of effort, and the concern for the burden on potential respondents. From another perspective, the Federal Procurement Regulations and the unit within each agency responsible for the administrative aspects of formal procurement actions, in accordance with these general regulations, provide standards and requirements for a contract as a legal document independent of what is being purchased. OMB Circular A-76 sets restrictions and requirements on developing in-house capacity versus using nonfederal contracting sources to obtain various services. This affects the justification for moving work in-house that has been done previously by contractors and contracting work out for which an in-house capacity exists. The implications of Circular A-76 for contracting for a survey should be reviewed with the procurement office. However, beyond the general standards there is no specific guidance on the use by Federal agencies of contractors for the conduct of surveys. They do not define when and how the procurement mechanism should be used for such a purpose nor do they indicate what the relationships should be among an agency's need for survey data and analysis, its staff resources, and the specification of contract requirements and monitoring of contractor performance. The Subcommittee on Contracting for Statistical Surveys of the Federal Committee on Statistical Methodology was organized to suggest ways to improve the survey contracting process in the U.S. Federal Government in the future. The Subcommittee's work was done under the general mandate of OMB. Because of its unique responsibilities, OMB was able to call upon the experience of a variety of individuals with backgrounds in diverse aspects of the conduct of surveys and the use of contracts. This report represents the collective effort of the Subcommittee. The opinions expressed here reflect the collective judgment of the Subcommittee and do not necessarily reflect those of OMB. It is not possible within any given presentation of this nature to cover all the issues related to the use and conduct of surveys. The following questions are some that need to be answered: What is the problem on which the study should focus? Should a survey be conducted as opposed to the use of some alternative research strategy or data collection approach? If a survey is to be used, should it be done totally in-house, totally under an interagency agreement, totally under a contract, or using some combination of these three approaches? The Subcommittee focused on what should be done after the decision is made to contract out the survey, and not on the factors involved in selecting this option as against an interagency agreement or in-house conduct of the survey. 2 As related to survey design and conduct, issues are: What should be the universe for study? What is the most appropriate sample design? Should the data be collected by mail, by telephone, or by personal interview? What specific questions should be asked and how should the data be analyzed? These and many other issues must be addressed at some point in the course of deciding to use, planning and preparing for, and actually conducting a survey. While the need to address such issues is discussed, there is no effort to provide input into the decision making process on such matters. There are many references and resources available on survey design and content considerations so that it would be inappropriate to try to duplicate these here. This report is intended for circulation to Federal agencies and offices which may utilize contracting for surveys. The report tries to draw attention to the issues that need to be considered, how to use or obtain the necessary expertise in preparing for and using the contract mechanism, how to prepare specifications and carry out contractor selection, and how to maximize contractor performance. It should be pointed out that this report does not deal with surveys conducted or supported under grants or interagency agreements. The report focuses on contracts awarded through an open competitive negotiated procurement mechanism based on evaluation of the technical and business proposals submitted in response to generally available Requests for Proposal (RFPs). An RFP is a formally issued notice of a government agency's need to have work done by outside organizations where contractor selection cannot be based on price alone. The RFP is the publicly released notice which contains the internally approved program Request for Contract. The primary reason for focusing on this mechanism is that it offers the agency the maximum opportunity to set the requirements for the survey and the standards for contractor performance and to select the organization that offers the best combination of anticipated product quality and cost factors, without regard to issues unrelated to these concerns. Other contract approaches, while serving other purposes, do not necessarily promote selection of the best choice of contractor for conducting a given survey at a given point in time. Among the alternative contract approaches are sole source selection because of a unique capacity to perform or based on an unsolicited proposal, and set-aside programs for competition only among small businesses in general or minority-owned small businesses. The small business set-aside and the minority-owned business programs permit competition for prime contract work among firms which are qualified for these programs but totally exclude the larger organizations with greater resources and all nonprofit research units, including academically-based units. This is part of the Federal Government's efforts to promote the economic viability of small, and particularly minority-owned, businesses. However, on any given procurement, these restrictions limit the range of competition. Although excluded from the small business and minority-owned business programs, nonprofit organizations are eligible for sole source contracting. The mechanism totally restricts competition, either because only one organization is determined to have the capacity to perform or because the organization submitted an unsolicited proposal that represents a new and unique approach and opportunity to obtain a valuable product. In this latter situation, the organization submitting the proposal controls the product offered, that is, the particular survey to be performed, and is the only one to which the contract can be awarded. In line with the intention to provide information designed to improve the ability to use the contracting mechanism for procurement of survey services it was important to learn more about what the agencies were doing at that time. Since no useful data 3 existed on this process, Subcommittee members interviewed officials of several selected Federal agencies involved in contracting for surveys and some of the contractors with which the agencies dealt. Appendix I summarizes the data collected. The agencies selected for the survey of their contracting operations were asked to have representatives of both the program and the administrative aspects of contracting operations available. The da were collected by personal interview. The questions were directed toward contracting for surveys only. The questions traced the use of this mechanism from the factors in the decision to use different types of contracts (cost or fixed-price) and how often they were used, through developing RFPS, soliciting bids and evaluating proposals, to monitoring contracts. Similar information was collected from a limited number of contractors. This information has been analyzed and its implications assessed for developing suggestions designed to improve the use of the contract mechanism for conducting surveys. The various reviews of current practice that have been made have been interwoven into this presentation. THE CHAPTERS TO FOLLOW In structuring the material to be presented, an effort was made to follow a logical order of stages in the development and management of contracts for surveys. The sequence of events in this process was a guiding principle within the general format, but not an absolute basis for organizing the chapters. The different chapters deal with the development of requirements for RFPs and with issues in the selection and monitoring of a contractor after an open competitive RFP has been issued. It must be understood that no part of this process is independent but is closely linked to and an integral part of the entire process of defining a requirement to be done under contract and seeing the contract work through to completion. In this report, "technical skills" refers to survey and programmatic content skills as distinct from management and administrative aspects of the procurement process. Chapter 2 deals extensively with general issues related to the technical skills involved in survey planning and conduct and in the use of contractors for the performance of various tasks in the process. The remaining chapters follow an order of activities in the use of contracts. Chapter 3 contains a review technical survey design issues and other considerations in preparing RFPS. Chapters 4 and 5 deal with issues in the selection of contractors and the management and evaluation of contracts after award. Chapter 2 is a detailed discussion of the skills required in contracting for surveys and the types of individuals who may serve as technical advisors. This chapter also includes a discussion of the activities and concerns in the planning and preparation for a survey to be done under a contract, the competitive selection of a contractor, and the monitoring of performance as these relate to the skills required at various stages in this process. A variety of technical survey research skills, as well as content area competencies, is needed to deal with issues such as determining the overall research design to be followed, the design of the data collection instrument or instruments, the universe to be studied, the design of the sample, the conduct of the data collection effort itself, data editing, coding, and quality control. At each stage of a survey performed under a contract, there are various technical skills required to maintain a smooth operation. Differences of in-house technical capabilities of agencies and the possible need to supplement those skills are discussed. Both the use and misuse of technical advisors by agencies are considered. 4 Chapter 3, on considerations in the preparation of RFPS, contains a review of the tasks that may need to be performed in preparing a statement of work for competition among prospective offerors rather than the skills that need to be available to the agency in undertaking this effort. The chapter includes a discussion of issues to be considered in developing an RFP. The interaction between technical requirements and degree of specification and the contract reimbursement method (that is, fixed-price or cost-type contract) is also reviewed. Different types of contracts imply different assumptions about the ability to predict and specify activities, results, and costs and, therefore, impose different limitations on the ability to trade off between cost and performance. In Chapter 4, on the contractor selection process, the selection process for open market competitive RFPs is discussed. The various stages of the contractor selection and contract negotiation process and the roles played by the contracting officer, project officer and technical evaluation panel are described. This is the framework within which the technical merits of the work proposal are presented and evaluated, and in which a contract is awarded. The process of debriefing unsuccessful offerors is also discussed. The last of the substantive chapters deals with contract management and evaluation after award. This is the period of actual contractor performance in line with the terms and requirements of the contract. The project officer and the contracting officer are jointly involved in monitoring and controlling the contract effort. The discussion in the chapter centers on the roles of these individuals and contact that should be maintained between the project officer and the contractor in monitoring these activities. Depending on the specifics of the contract and the background and competence of the project officer, this person's input into the technical aspects of the survey will vary. There may be specific activities which require approval; data collection instruments usually require formal clearance. The project officer must certify invoices for payment and must, therefore, be aware of contractor progress and current and planned activities under the contract as these relate to the remaining time and funds available. In addition to these four substantive chapters, there are two appendixes. These are a review of the informal survey of agencies and contractors conducted by the Subcommittee, and a discussion of the OMB clearance process for surveys. 5 Chapter 2. TECHNICAL SKILLS REQUIRED IN CONTRACTING FOR SURVEYS NTRODUCTION A survey is generally a rather complex activity. Furthermore, the results to be derived from a survey are usually difficult to specify, for example, types of data and analyses desired. Consequently, the task of contracting for a survey is not straightforward. Specifically, a considerable amount of effort is required to provide complete and clearly written survey requirements in a Request for Proposal (RFP), select the best offeror to receive the contract, and adequately monitor and review the progress of the survey. Technical expertise is essential to the success of contracting for and monitoring the execution of a survey. In this chapter, technical skills that should be available when contracting for surveys are presented. Following are areas in which technical skills are required: - Developing project objectives and specifications, Subject matter knowledge, - Project cost and scheduling, - Questionnaire design and testing, - Sample design and selection, - Data collection, including follow-up procedures, - Data processing, including coding, editing and file creation, - Data analysis, - Quality control, and - Survey documentation and report preparation. The subject matter knowledge required for a particular survey will, of course, vary depending on the nature of the survey. As mentioned previously, surveys deal with many subjects, for example, demographic, economic, psychological, sociological, and medical topics. The other types of technical skills listed above are more generally applicable to all surveys. It is very unusual to find all the skills required for a particular survey in one individual, although it is often true that some people will possess more than one of these skills. Consequently, it is usually necessary that there be a Government project team consisting of the project officer and one or more technical advisors to supply the required technical skills. The number and types of advisors needed will vary according to the size and complexity of the survey being planned and the experience of the project officer. Smaller and less complex surveys will usually require fewer, and less specialized advisors than large, complex surveys. 7 It is important that the need for a project team and the types of members needed be recognized early in the development of a project. At least some of the advisors should participate with the project officer in the following major phases of the survey contracting process: (1) project planning and preparation of the RFP, (2) evaluating proposals and awarding the contract, (3) and monitoring the work done under the contract. In many cases the agency contracting for a survey will not have on its staff all the technical advisors needed. In such a case, the agency should use technical advisors as part of the project team. These advisors could be persons employed by other Federal agencies or nonfederal consultants. Nonfederal consultants who assist in development of the RFP are not eligible to respond to the RFP. If advisors are needed, they should be brought in at the beginning of the contracting process so that they can participate in planning the project, in preparing the RFP, and in evaluating the proposals. if advisors are brought into the project early, they can provide the guidance and assistance needed in designing and executing a survey. It is generally a mistake to rely solely on the contractor to provide the technical expertise required during the execution of the survey plan and the processing and analysis of the data. If this is done, the technical quality of the survey may be diminished because the contractor's technical expertise may not be fully adequate, or the contractor may decide that it is not in his or her best interest to take steps to ensure the highest technical quality. Even with the highest quality contractor, the Government should still provide technical review and guidance as part of the monitoring of the survey process; otherwise, the value of the project may be reduced. The technical aspects of some phases of a survey are often not fully appreciated. For example, in the design of the data collection activities of a survey, a reasonable time schedule cannot be formulated without regard to technical input on follow-up activities, such as the number and type of follow-up attempts required. The next section of this chapter contains a discussion of the required technical skills listed earlier. The following section covers the type of technical input required at the three major stages of the contracting process. The final section includes a discussion of some of the technical aspects of current practice in contracting for surveys. BASIC TYPES OF SKILLS REQUIRED Developing Project objectives and Specifications The formulation and definition of project goals and objectives is the most fundamental aspect of a project. Often goals and objectives are not defined adequately. In such cases there will be confusion regarding the type and amount of survey information to be collected, and hence the strong possibility that the project objectives will not be met. Sometimes a project may have multiple objectives that may be somewhat conflicting. For example, if both subgroup and total population estimates are important, a decision must be made on how to allocate the sample to these subgroups. The sample allocation that would provide equal precision, or some, other specified precision, for subgroup estimates will generally differ from the sample allocation that would maximize the precision for total population estimates. Priorities must be set for each established goal and objective at the beginning of the project in order to resolve any conflicts that may subsequently evolve. 8 In addition to stating the objectives clearly, an effort should be made to define the objectives to be as narrow or specific as possible. If the objectives are too broad or general, the survey might become too large in scope to manage adequately for the time and funding available. If the goals and objectives are defined adequately and if the study population has been defined, then appropriate technical specifications can be developed for instrument design, sample design, and data analysis. consequently, these technical aspects of the project should be kept in mind as the goals and objectives are developed. Subject Matter As mentioned previously, sample surveys cover many different subjects. In order to formulate and develop a project adequately, there should be a subject matter expert on the project team at the start of the project. Such subject matter expertise is required in order to insure that all data requirements. are delineated. The subject matter expert may not know the appropriate manner in which to phrase a question in order to elicit the desired response, but will know the fundamental issues that must be explored in order to satisfy the data requirements. For example, to develop a survey to investigate the accuracy of the amounts paid to railroad retirement beneficiaries, there should be someone on the project team with knowledge of the railroad retirement system. This subject matter expertise should allow for the formulation of the most appropriate survey questions and should enable the project to collect adequate information from respondents. Project Scheduling and Costing Scheduling and costing various phases of a survey are generally difficult. Typically, a gross cost for the project is set very early, particularly for one-time surveys. Agency personnel are generally limited by the initial gross cost allocated to the project. The amount of time required for planning and writing the RFP is often underestimated. This can delay the scheduled start of the survey which, in turn, can jeopardize the quality of the work. Sufficient time for execution of the survey by the contractor must be allowed. Since there are several aspects of a survey, such as printing, data collection, and data processing, that may require special clearances, applicable requirements should be determined as early as possible in development of a survey to be conducted under contract. The procurement office can provide a list of required clearances so that their applicability to a particular survey can be determined and provision for compliance can be made in the project schedule. In addition, there are several survey tasks which involve aspects for which the completion time and cost are typically hard to predict. For example, a survey may involve a large frame development phase in which a number of frame sources have to be investigated. This could require working with a number of computer files involving problems of incompatibility between various computer systems, different record layouts and file definitions, and matching elements in different files. These types of activities are typically difficult to assess in terms of time and cost. Other survey activities that are typically difficult to assess in terms of cost and time requirements are questionnaire design and testing, data collection, and data processing. Due to the potential for cost and scheduling problems as indicated above, it is important during the project development phase to include a person on the 9 project team who has experience in developing project schedules and cost estimates for surveys. Questionnaire Development and Testing Proper development of a survey questionnaire and other data collection instruments is a deceptively difficult phase of a project. The knowledge, time, and effort required to complete this phase adequately is often underrated. In addition to preparing the instruments, time must be allowed to adequately test them. Testing is important since it is generally impossible to anticipate all the problems that will arise when an instrument is used. It is important to include someone on the project team who has had considerable experience in questionnaire development and testing so that the common pitfalls of instrument design can be avoided and so that the instrument may be designed to facilitate data processing. If a questionnaire is used to collect the survey data and the questionnaire is not developed by someone with appropriate experience, it is likely that some questions will be unclear to the respondents, will be misinterpreted by the respondents, will confuse or upset some respondents, or will tend to influence or "lead" the respondents. The position of the question on the questionnaire may influence the response rate to the question. For example, sensitive questions, like income, are usually answered at a higher rate if they are asked after the respondent has become more comfortable with the interviewer. Also, the nature of the questions surrounding a question of interest may influence response proportions by several percentage points. Furthermore, the questionnaire may be difficult to administer and may not provide adequate survey information. In addition to operational difficulties, these problems can cause both response and nonresponse biases in survey estimates. Sample Design and Selection Even for a survey that seems straightforward in terms of the information to be gathered and the definition of the target population, there are usually some difficult questions that arise in designing the sample selection method, particularly if the survey has multiple objectives. Questions about sample size determination, frame development, stratification procedures, sample allocation to strata, or whether or not to use differential probabilities of selection arise in all surveys. These questions become particularly difficult to answer when there are conflicting objectives. For example, if the project team has not decided whether subgroup estimates are more important than overall estimates or vice versa, satisfactory decisions on stratification and allocation cannot be made. ,Many of the questions that arise are not easily answered by referring to standard textbooks. Consequently, the project team should include a person who not only has. the theoretical knowledge of sampling methods, but who has also had substantial experience in the practical application of sampling principles to the design of the type of sample survey involved in the contract. This type of experience would include, for example, knowledge of available frame sources and of data available to use for strata formation or for assigning differential probabilities of selection. Data Collection The data collection phase of a project contains several aspects. First, the mode of interviewing - personal visit, self-administered questionnaire (usually mail), or telephone - should be selected based on the type of questions to be asked, the population to be studied, and the time and resources available. Next, interviewers must be recruited and trained for the in-person or telephone administration of a 10 questionnaire. Even if a staff of experienced interviewers is available, some training for each new questionnaire is necessary. A budget and schedule for interviewing, including callback procedures, must be developed. For personal interviews, the schedule often includes a mailing or telephone call to the selected sample units prior to the personal interview. The number and timing of callback visits allowed in the schedule will have an important impact on the survey response rate, which, in turn, is a key factor in controlling nonresponse bias. Acceptable response rates need to be determined and follow-up procedures must be devised and implemented. Another aspect of data collection is questionnaire check-in. In addition to recording the receipt of the questionnaires, this involves the development of editing procedures to check for faulty or missing data. Procedures also have to be developed to follow-up on questionnaires that fail this edit. These procedures are important in order to improve data quality and reduce the response and nonresponse biases. Data Processing The data processing phase usually involves entering the survey data into a computer file so that it can be accessed and analyzed. The first step is to provide any coding and editing of questionnaire responses that are needed. The coding is needed to convert all the responses into numerical codes for entry into a computer file. For example, if occupation was a survey item, a numerical code would be assigned to each occupation to use in recording the responses on the file. Proficiency in this activity requires considerable practical experience in coding and editing of survey data and knowledge of computer systems. Editing is necessary to detect responses that are unreasonable or that are inconsistent with other responses. The editing may either be performed manually or by use of the computer. Developing editing rules is difficult and requires input from subject matter specialists. Responses that fail edit checks can either be followed up, replaced (imputed for) or discarded. All imputed data should be flagged on the file. The survey data are placed into a computer file by having the coded questionnaire responses keyed to a specified record format. It is important that the computer record layout be documented completely and accurately so that the data will be correctly accessed. Data Analysis An analytic plan and strategy should be developed in the earliest stages of the survey development process. Often included in the plan are specific table shells to be used in survey tabulations. This plan is the foundation for sample and questionnaire design and is essential for the survey to provide the results intended. Often the survey analysis phase includes the estimation of population and subpopulation characteristics (for example, means, totals, and proportions). In addition, there are many types of more complex statistical analyses that may be applied to survey data, depending on the objectives of the survey. Some examples of analyses performed are the following: hypothesis testing on population means or proportions, regression and correlation, analysis of variance, and log-linear analysis. Before statistical analyses are applied to survey data, imputed (or pseudo) responses are often inserted in the records of respondents that have missing or faulty responses. Setting up an imputation procedure for missing data is a complex process that 11 requires experience. Also, differential probabilities of selection should be taken into account in applying statistical analyses. This is often accomplished by assigning weights to survey respondents to reflect varying probabilities of selection. These weights are usually adjusted to account for eligible sample units that do not respond. Once the survey estimates and statistical analyses have been made, the results must be examined. Study conclusions, and perhaps recommendations, are then made. The data analysis phase is a very important part of the project and requires the assistance of appropriate subject matter experts and of a statistician who has had experience in analyzing data from surveys. Quality Control During virtually any survey operation there are many points when data collection or processing errors can be made that diminish the accuracy of survey data. These instances include sample selection, interviewing,and/or completing the survey instrument, check-in and field editing, coding and office editing, and keying the survey data. It is strongly recommended that quality control procedures be set up at each step to detect and correct as many of these survey operations errors as is feasible. At a minimum, the quality control program should cover interviewing, field editing, coding, and keying. These procedures should be set up by a person who has had experience with survey quality control procedures. Survey Documentation and Report Preparation It is important throughout the project to prepare enough project reports to adequately document the survey procedures, results, and conclusions. Poor documentation is a very common problem with survey projects. One reason for inadequate documentation is that project funds often run low and time becomes short near the end of a project. A conscientious effort should be made to allow adequate time in the project schedule for ongoing documentation of the survey procedures used. These procedures include definition of the target population and frame construction; interviewing procedures, including callback and follow-up procedures; sample design and selection; editing and coding procedures; weighting and nonresponse adjustment methods; estimation and variance estimation equations and statistical analyses used; and quality control methods applied. The final project report should include survey results, analyses, conclusions, recommendations based on the results, and suggestions for additional research. Project specialists, such as subject matter specialists and statisticians, should be involved in preparation of the final report. TECHNICAL INPUT AT VARIOUS STAGES OF CONTRACTING FOR A SURVEY As mentioned earlier, substantial technical input is needed at each of the following three major phases of contracting for a survey: project planning and preparation of the RFP; evaluating proposals and awarding the contract; and monitoring the contractor's performance. 12 Unfortunately, the importance and/or extent of technical input needed at these stages is often underestimated, thus diminishing the quality of the survey. The required inputs are discussed below for each of these major phases. Project Planning and Preparation of the RFP Right from the start of the process of contracting for a survey, it is important to have technical input and program and subject matter expertise. Subject matter experts and statisticians can help define specific project goals and objectives that will satisfy the purpose of the study and yet will be achievable, within the time and budget available for the project, by applying appropriate survey and statistical methods. A statistician should participate in the development of the RFP so that it will be complete and accurate in terms of the statistical requirements of the survey. In cooperation with the project staff, the statistician should prepare or assist with the preparation of the following portions of the RFP;1 - Sample size specifications, if any, for the target population and/or for population subgroups, - Precision requirements for one or more key survey estimates, including any requirements for population subgroups, - Response rate required and/or number of callbacks specified, - Requirements for nonresponse imputation procedures, - Quality control procedures, - Variance estimation requirements, and - Specification of particular population or subpopulation estimates required. If a statistician does not participate in the preparation of those portions of the RFP that address the above topics, it is virtually certain that some of these specifications will be omitted from the RFP, and some of those included will not be properly stated. In such a case, offerors would be confused and may prepare proposals that miss the real objectives or which are somewhat vague, incomplete, or misdirected in terms of important technical aspects of the proposed survey. This, in turn, would make it more difficult to compare the proposals and to accurately evaluate them in terms of technical content. Judging and Selecting Among Offerors Technical input is essential in the evaluation of the proposals. In addition to subject matter specialists, at least two persons with varying experience in the different aspects of a survey should be included on the proposal review panel. At least one of these persons should be a statistician. A major focus of the technical review of proposals is the offerors' responsiveness to the survey specifications, such as frame construction, sample design and selection, quality control plans, and estimation and analysis plans. Just as important as evaluating these technical responses is the assessment of the offerors' abilities to fulfill the statements and claims made in their proposals. ___________________________ 1For a general type of RFP the technical input discussed above is still needed. However, some of these items would not usually be specified. 13 For example, if they claim that they would achieve an 85 percent response rate, is there sufficient evidence to support this? The basis for this type of evaluation would primarily be the previous experience and perforMance of each offeror and the technical skill and experience of the proposed project staff. Of course, the survey plans in the proposal would also be used to help assess the offerors' potential to achieve the proposed statistical standards. Monitoring the Contractor's Performance After a contract is awarded and the project begins, technical input is still very important to the success of the project. The contractor will presumably be describing details of their survey plans at meetings and in written documents. Technical personnel representing the agency should attend all project meetings which involve technical issues and should review all project documents to ensure that specified technical standards are met. Even if the contractor's project staff appears to be technically sound, it is necessary for the agency to continue technical input. This should ensure that the standards of the contract are met and may avoid technical problems that might have appeared near the end of the project if technical input had been interrupted. Any problems on technical issues between the agency and the contractor should be discussed and resolved. If resolution of these problems requires changes in the specifications, the changes should be proposed to the contracting officer for decision and formal action. To facilitate resolution of problems, there should be an open line of communication between the contractor's statisticians and other technical staff personnel and a liaison for he agency's project staff. Good communication between contractor and agency staffs can prevent substantial misunderstandings on technical issues that may arise. Ideally, there should be enough discussion of technical issues between the agency staff and contractor staff so that when the contractor submits a technical document, such as a sampling or analysis plan, agency personnel will be aware of the major technical aspects of the document. If communication has not been good and major problems with technical issues arise, there may be substantial time delays in the project work. In addition to participating in meetings and reviewing documents, in some surveys it is useful for agency technical personnel to make site visits. The purpose of a site visit might be to observe and review the implementation of sampling procedures, interview activities, coding and editing, or quality control activities. SOME TECHNICAL ASPECTS ASSOCIATED WITH CURRENT PRACTICE Use and Misuse of Technical Advisors As expressed previously, the use of technical advisors in contracting for a survey is vital to the success of the project. Although project managers generally recognize this need, technical advisors are not always used properly. In the previous section it was pointed out that it is important to have technical input and program-specific expertise from the beginning of the process. Often the technical advisors are added to the project team too late to assist with the preparation of the RFP. In such a case, the specifications of the survey may be inadequately defined. For example, the precision requirements for the survey may be unclear, causing confusion and apprehension among offerors. As another example, the minimum acceptable response rate may have been omitted, which would probably cause problems in comparing proposals. 14 Technical advisors are generally called upon to review and judge the technical proposals submitted. However, in addition to rating proposals on technical aspects with which they have knowledge and or experience, members of the review panel are also asked to rate proposals with respect to technical aspects with which they are less "familiar, or even unfamiliar. For example, a statistician who specializes in sample design may be asked to rate the proposals with respect to questionnaire design and quality control procedures, in addition to sampling aspects. The reviewer's experience may be rather limited in areas other than sampling. As a result, ratings of these aspects of the proposals might be inappropriate. To try to minimize this problem, individual reviewers, during the discussion and review of ratings, may revise their original score on an item to reflect the opinion of another reviewer with particular expertise in the area. To allow for this type of rating adjustment, it is advisable for an agency to attempt to balance the members of the review panel with respect to the important technical aspects of the proposals that need to be rated. Use of Outside Help to Supplement in-house Technical skills For many surveys being contracted, the in-house technical skills needed to design the survey, to judge offerors, and to monitor survey progress, are not fully adequate. In such cases technical advisors from other Federal agencies or nonfederal consultants should be added to the agency's project team. For example, advisors specializing in sample" design, questionnaire design, or field operations may be needed. it is, of course, important that the advisors required are added to the project team early in the development of the survey. As was indicated earlier in this chapter, the agency should have its own technical advisors to provide the technical expertise in a particular area, rather than relying solely on the contractor's expertise. Even if outside consultation is needed, it is important to have the appropriate technical advisors on the agency's project staff to protect the agency's interests. There are a number of Federal agencies that have technical staff who should be able to provide assistance directly or to suggest others with appropriate skills. These agencies include the following: Bureau of Economic Analysis, Department of Commerce; Bureau of Labor Statistics, Department of Labor; Bureau of the Census, Department of Commerce; Bureau of Justice Statistics, Department of Justice; National Center for Education Statistics, Department of Education; National Center for Health Statistics, Department of Health and Human Services; Office of Research and Statistics, Social Security Administration, Department of Health and Human Services; Statistical Reporting Service, Department of Agriculture; Statistics Division, Internal Revenue Service, Department of the Treasury. 15 Chapter 3. CONSIDERATIONS IN THE PREPARATION OF RFPS INTRODUCTION This chapter covers some considerations in the preparation of a Request for Proposal (RFP) for a survey. These comments and suggestions are not all-inclusive; additional considerations will be warranted for some RFPS. For example, certain types of surveys - such as repetitive or longitudinal surveys - pose special problems that are not addressed here. All of the possibilities cannot be covered because of the wide range of purposes and survey designs encountered in surveys sponsored by the Federal Government. Emphasis is placed on elements which are included in many of the RFPs prepared in Federal Government agencies and on those portions of the RFP to which the project officer or other technical personnel are most likely to contribute.1 On the other hand, the comments and suggestions which are presented are not likely to be applicable to every RFP, either because of the nature of the proposed surveys or because of variations in regulations, policies or procedures among government agencies. INITIAL CONSIDERATIONS Because of the variations noted above, it is essential to contact the procurement off ice that services the sponsoring agency at the conceptual stage when a survey contract is contemplated. Early discussions with the contracting office will enable determination of the contracting options, if any, which are available, the input expected from the program staff, the steps which should be followed from development of the RFP and a list of potential offerors through award of the contract, and the amount of time each step is likely to take. Payment Provisions One option that might be available concerns the payment provisions of the proposed contract. A contract may stipulate a fixed-price which is agreed upon in advance by the sponsor and the contractor, or it may provide for reimbursement of allowable costs (up to a preset maximum) incurred by the contractor in performing the survey. The decision about which approach to use is usually made by the procurement office based on technical contracting considerations, not by the program staff. Some agencies use only one of these financial arrangements for all of their contracts; some agencies use one or the other, depending" on the nature of the survey to be undertaken. The RFP should indicate whether the proposed contract will stipulate a fixed price or will provide for reimbursement of costs incurred by the contractor. There are advantages and disadvantages to each of these payment mechanisms. For example, fixed-price contracts assure, in advance, that sufficient funds are available for completion of the work and keep the sponsor's administrative audit expenses to a ___________________________ 1A complete RFP contains a number of standard provisions or sections originating in the contracting office to which the program staff have little or no input. 17 minimum. However, some qualified potential offerors will not submit a proposal for a fixed-price contract and competition is therefore reduced. it is particularly important that fixed-price contracts contain comprehensive, detailed statements of the requirements for the work to be performed by the contractor to avoid misunderstandings about what the agreed-upon price actually covers. These requirements should be based on a combination or merging of the specifications in the RFP and the contents of the successful offeror's final proposal. If there are many unknown or undecided factors in the project plan that affect the predictability of costs, a cost-type contract may be preferable, provided adequate funding will be available. A "dollar cap" or maximum cost can be included in a cost-type contract. If a cap is specified, flexibility should be built into the survey design so that if avoiding cost over-runs, which jeopardize completion of the survey, is not possible, a limited but still useful product can be obtained. Phased Surveys If there are any serious doubts about the feasibility of key aspects of the survey plans because of unknown factors such as the ability or willingness of designated respondents to provide the information sought, it may be prudent to limit the scope Of the project to which the sponsor will be committed.2 One approach which might be appropriate in this situation is to issue an RFP solely for an evaluation of the proposed methodology, with the stated intention of issuing another RFP for a full-scale survey if the initial effort indicates the methodology is feasible. Obviously, this approach would consume more time because of the two RFPs involved. Another possible approach is to specify work segments which may be done during the course of the contract if the concerns are resolved in the earlier stages of the contract. If a phased survey is deemed advisable, early discussions with the Procurement office will indicate what contracting alternatives are available for consideration and their advantages and disadvantages. Survey Design Specification RFPs may state all, some, or none of the details of the survey design. The level of survey design specification found in RFPS, in practice, ranges from "This is exactly what we want, including the questionnaire to be useful" to "We have a problem and need information to help solve it; what do you suggest?" The choice of the level of specification is dictated by a number of considerations such as the nature of the survey to be conducted, the applicable regulations and policies, and the technical skills available among the sponsor's staff or advisors. Detailed survey design specifications may be appropriate if the sponsoring agency has highly qualified and experienced statistical and survey staff available to develop the RFP who are aware of potential contractors' capabilities and resources, and who know exactly how the agency wants the survey to be conducted. ___________________________ 2Although the general provisions incorporated in Federal survey contracts usually provide for termination of the contract for the convenience of the Government, exercising provision can be difficult, complex, and time consuming. 18 Such an RFP may be more difficult and time consuming to develop than one that calls for offerors to suggest survey designs. Maximal specification of the survey design will indicate the quality standards of the sponsor to potential offerors, and will also permit them to repeat the requirements in their proposals without demonstrating capability or understanding of the problem. It may be easier and quicker to evaluate the responses to a detailed RFP, but more difficult to determine the technical capabilities of the offerors. Selection is therefore more likely to be made primarily on price. Any unnecessary constraints imposed on the survey design would preclude desirable flexibility and innovation by offerors which could improve the quality of the survey or decrease its cost. Furthermore, some of the more competent survey research organizations dislike this approach and are reluctant to respond to this type of RFP. Less detailed RFPs which, invite offerors to make suggestions or develop survey designs, allow them to apply their knowledge and demonstrate their ingenuity or creativity. However, minimum design criteria or quality standards should be stated, when known, even if the offerors are asked to develop the survey design. Evaluating the proposals received in response to a less detailed RFP is likely to be more difficult and time consuming. When an RFP does not specify the details of the survey design, the successful offeror's explicit statement of the survey design specifications is commonly incorporated in the contract awarded. An alternative that incorporates many of the advantages of both the detailed and less detailed survey design specification approaches has been used effectively by some agencies. In this procedure, potential offerors are invited to submit proposals based on detailed specifications but are also encouraged to suggest modifications to the detailed plans which they believe will be advantageous to the sponsoring agency. Bidders' Conferences and Due Dates Some RFPs - usually only those for large or complex surveys - include provision for a Bidders' Conference on a specified date to which prospective offerors are invited. The conference should be held after prospective offerors have had the opportunity to study the RFP, but well before the due date for proposals. They may be asked to submit their questions beforehand so that answers can be prepared in advance for the conference. At the meeting, technical and procurement personnel from the sponsoring agency answer questions raised by potential offerors about the RFP or arrange to provide answers later for any issues which cannot be resolved during the conference. They may also review contractual administrative requirements. Discussions at a Bidders' Conference will benefit the sponsoring agency if they identify RFP specifications which are misunderstood or which appear to potential contractors to be inconsistent, inadvisable or unacceptable. Any additional information or material developed as a result of the Bidders' Conference should be mailed to all requesters of the RFP. It is very important in any RFP to allow a reasonable amount of time for preparation and submission of proposals - at least 30 calendar days after the RFPs are mailed and more than 30 days if the proposed survey is complex or offerors are asked to develop substantial portions of the survey design. The availability of an RFP is announced through the Commerce Business Daily. it is also advisable that the names and addresses on mailing lists of those to receive the RFP are current. 19 POSSIBLE CONTENTS OF RFP's When developing an RFP, it may be helpful to refer to RFPs for similar surveys conducted by the sponsor or by another Federal agency as sources of ideas for content and wording. However, these RFPs should be used together with comments or suggestions from persons familiar with those surveys about problems or misunderstandings which arose during the contractor selection process or the conduct of the study - particularly difficulties which might have been forestalled if the RFP had been written differently. The goals of the procurement process should be kept in mind when an RFP is written. Ideally, an RFP does the following: - Requests the development of or specifies a technically sound survey design which will meet the sponsor's information needs and is realistic, considering the time and funds available for the project and the capabilities and resources of potential contractors. - Conveys all the information needed by potential contractors to prepare a responsive proposal, including all the terms, conditions, and provisions that the sponsor intends to incorporate in the contract. - Asks for all the information that procurement and technical personnel will need to conduct an accurate and equitable evaluation of the proposals received. - Attracts responsive proposals from enough of the qualified potential contractors to provide adequate competition and to permit a suitable selection. Some agencies include in the RFP a table of contents for responses to be followed by all offerors and stipulate that staffing requirements should be specified in a common measure (for example, hours). This simplifies reading and comparison of proposals. Specific suggestions for developing an RFP that meets these criteria are presented in the remainder of this chapter. Statement of Survey Objectives Every RFP should contain a clear statement of the specific objectives of the survey to be conducted under the ensuing contract. Discussing the proposed research in the larger context of the mission of the agency may also be advisable, particularly if the project is part of an ongoing integrated program. The statement of the survey objectives should also indicate what groups (such as government agencies, congressional committees, state or local governments, trade associations, and business firms) will be the key users of the findings. If data from the survey or an interpretative report of the findings must be provided in time to meet a deadline, the date of and reason for the deadline should also be stated. Any germane reference material that would be helpful to potential offerors in planning the broad outline of the proposed survey or in estimating time, skill, or cost requirements should be cited and also included in the RFP package. If the material cannot be provided with the RFP, information on where and when it can be accessed or obtained should be supplied. Appropriate reference material might include such information as estimated numbers of eligible respondents, geographic distribution of eligible respondents, and the methodology and results of any related research that has been conducted by or is known to the sponsoring agency. However, materials which 20 would be of interest only to the successful offeror should not be included with the RFP. Although RFPs sometimes state anticipated level of effort in terms of person-years or similar proxies for dollar estimates, the Subcommittee's interviews with contractors indicated that these proxies are less helpful to them than dollar estimates would be because of wide variations from agency to agency in the dollar amount one person-year signifies. If agency procurement policies permit, it may be desirable to inform potential offerors of the approximate amount of funds available for the contract. Providing offerors with this information would help avert some of the difficulties that might otherwise be encountered in the contractor selection stage, such as the problems that arise when the cost estimates for all of the technically sound proposals received by the sponsoring agency substantially exceed the funds available for the contract. If some but not all of the technically sound proposals fall outside the anticipated price range, this reduction in competition may result in a real but less obvious disadvantage to the sponsoring agency. Request for Information About the Offeror The RFP should request the relevant information about the offeror. It is not sufficient for an RFP to simply define the requirements for the technical proposals. Before a contract is awarded, the reliability of the contractor should be established. When appropriate, the following specific requests should be responded to by the offerors before a contract is awarded. Description of experiences. The following list contains areas where it could be important to obtain specific descriptions of the offerors' experiences: - frame refinement processes, - probability sample designs, especially for large-scale surveys, - data collection methodology (including callback procedures), - questionnaire development and testing, - estimation procedures, including nonresponse adjustments, - variance computation procedures, especially for complex samples, - training and supervision of interviewers, coders, and others, - analysis of data from large-scale surveys, - data processing, - quality control, and - publications. Examples of previous relevant work. The examples of offerors' work should include questionnaires, sampling documentation, instruction manuals, and reports containing the results of the projects. The agency should request information on all previous work pertaining to the subject matter areas for a specified period of time. The time period should be such that enough material is received but that the offeror does not inundate the agency with paper. In most cases, five years should be a reasonable period for both the offerors and the agency. Included with this information should be the name and current telephone number of the sponsor's representative (or government project officer). At least a sample of these references should be verified by the agency before the award of any contract. 21 Staff experience. This should include the experience of the current staff. This could differ substantially from the company's experience. Their background in statistical methodology, previous contracts, and subject matter areas, especially the ones related to the RFP, should be stated. Key project staff. The names of key project staff members should be required along with the percent of time each one will work on the project. Their resumes should be included with the proposal. It may be desirable to have a key personnel clause stating which staff members cannot be replaced on the project without government approval. Letters of intent should be provided by key persons who are not members of the offeror's firm. Field staff. The interviewers' field operations background in the primary sampling units (PSUs) in the frame from which the sample will be drawn should be requested. Also their experience in handling nonrespondents should be stated. The background of supervisory and clerical field staff should also be requested. Conflicts of interest. The RFP should contain a section requesting information that will allow the agency to determine if any possible conflicts of interest exist. This section could include questions about the offeror regarding other clients, activities of staff members, subsidiaries, or parent companies. Facilities. In the RFP, the agency should request information about the potential offerors! facilities in order to make certain they are adequate to complete the contract. This is especially true of computer facilities. The offeror should have the computer capabilities to handle all data processing requirements. If necessary, this should include a requirement of compatibility with the agency's data processing facilities. Subcontracting. If an offeror plans to use a subcontractor in any aspect of performance under the contract, detailed information should be provided on their experience, resources, and facilities. If subcontracting is anticipated but there is no identified subcontractor, standards for subcontractor selection should be specified. Survey Design Checklist The purpose of this section is to present a descriptive checklist which contains topics that should be addressed or considered when writing the survey design portion of an RFP. As was stated previously, the amount of specification is dependent on many factors. Ideally the amount of detail that each item in the checklist requires will depend on the type of study to be conducted. For this reason, the following list should be used only as a guide and the appropriateness of each item should be judged separately for each RFP. However, if a design topic is not specified in detail, it should be mentioned as an item to be discussed by the offeror. If an item is not mentioned in the RFP, the agency might not be able to use it as a criterion for evaluating the proposals. In addition, the importance of consulting a qualified sampling statistician before proceeding with this section cannot be overemphasized. Design and selection Definition of the target population and key subgroups. The objectives of the survey should define the population that is intended to be covered. However, it may be necessary to eliminate certain segments of the population and thereby redefine the scope of the survey because of practical problems. Examples would be excluding Alaska and Hawaii because of the high cost of collecting information by personal visit, excluding nonrepriceable items in the production of a price 22 index, and excluding nontelephone households when telephone interviewing is the primary method of data collection. The survey results will apply only to the population sampled. However, it might be possible to obtain estimates for the excluded populations by using alternative collection methods or by using auxiliary data which are correlated to the information gathered in the survey. The RFP could require that proposals include a method of estimating for the excluded population. Any conclusions for these groups would have to be stated separately and their limitations would have to be noted. A description of any subpopulations for which estimates are desired should be included. Examples could be geographic and demographic detail. Sampling frame. Ideally the sampling frame should fully cover the target population. Every effort should be made to.ensure that the source is as current and as complete as possible. Consideration should be given to requiring procedures for updating the available source, including merging information from different records and removing duplicates. If more than one frame is available, a decision must be made on which one to use or whether to use multiple frames in the survey. If multiple frames are used, a method to identify and handle overlap in the frame must be developed. Sample selection. The RFP should specify that a probability sample be selected for the survey and that the sample design be adequately defined. This includes a description of the proposed sampling plan for each stage of sampling. The details of this description should include the following topics for each stage of selection: - Definition of the sampling unit. A sampling unit is a cluster of one or more elementary (or observation) units. Each elementary unit should belong to exactly one sampling unit. In some RFPs the sampling units for some of the stages of selection (for example, the primary sampling units or PSUS) are specified. Any sampling units that are not specified in the RFP should be clearly defined in each proposal. Examples of sampling units are counties, enumeration districts, households, persons, companies, and retail outlets. - Number (or expected number) of sampling units to be chosen. In many cases the RFP gives the target final stage sample size (for example, 5,000 completed interviews) and sometimes the first stage target sample size (for example, 100 PSUS). A proposal should indicate all selection stage sample sizes not specified in the RFP. - Specific method of selecting units from the frame. The method of selecting sampling units at a given stage (for example, simple random sampling, systematic sampling, or stratified sampling) should be clearly specified so that the probabilities of selection of the sampling units can be derived. In addition, any variation from strict probability sampling that an offeror feels is required should be clearly described and justified in the proposal. Examples of cases for which variations from probability sampling may be appropriate are unavoidable coverage deficiencies, or the selection of a major brand to represent a product line in a price survey. Offerors should base their choices of sampling units, sample sizes, and selection methods for the various stages of a sample design on optimization. considerations (that is, maximum precision per unit of cost). The RFP should require that proposals include a justification for all the sample design choices made. 23 It is essential that at least one sampling statistician with experience in the particular area of interest assist in the preparation of the sampling portions of the RFP. It should be noted that unless probability sampling is required in the RFP, a proposal cannot be considered technically unacceptable just because it allows for some form of nonprobability sampling. This could cause problems during the contractor selection procedure. In addition, the Office of Management and Budget (OMB) requires justification if probability sampling is not used. Sampling error. Two different approaches are commonly used in RFPs for sample surveys to specify the level of effort desired by sponsors to provide estimates that are sufficiently reliable for their purposes. One method is to specify the sample size; the other is to specify the precision (that is, desired minimum level of sampling error) for estimates of one or more key variables. It is not desirable or practical to specify both sample size and precision in the RFP, since the desired precision may be impossible to achieve with the specified sample size. Whichever is specified, the other should be an important factor in evaluating the proposal. The justification for using one method over the other is dependent on the study to be done and the resources available. The RFP often specifies the target sample size, usually in terms of the number of completed questionnaires or interviews. The derivation of a target sample size may be based on cost considerations or on the approximate precision levels desired. If a sample size is specified in the RFP, it is important to clearly define the final stage unit and whether or not the sample size is in terms of original selections, inscope selection, or completed interviews. Any required sample sizes for population subgroups should be clearly stated. Because of the different components of the sampling error, it may be necessary to require a minimum number of sampling units at each stage of selection. Otherwise the resulting sample, although meeting all the specified criteria, might not yield acceptable results. Finally, if the sample size requirements are in terms of completed interviews, the RFP should include a precise definition of what data must be collected before an interview is complete. If the sampling error is to be specified then a decision must be made whether it will apply to one key variable or more than one key variable. If it applies to more than one, will there be a controlling variable, that is, the design which meets requirements for that variable will exceed requirements for all other variables. In addition, it must be made clear whether the precision requirements are in terms of the standard error, the coefficient of variation (relative standard error), or 95 percent confidence levels (+ 2 standard errors). Any misinterpretation of these requirements can lead to gross differences in the proposed sample sizes. Also it must be made clear whether the required minimum sampling error will apply to specific subpopulations "or just the overall population under consideration. A qualified sampling statistician should be consulted to assist with the decisions made in these areas. Data collection, questionnaire development and testing This section of the RFP, more than any other, will require input from the.people who handle the agency's OMB clearance process. In addition to knowing the agency's procedures regarding data collection they are familiar with OMB's regulations. (See appendix 11 for a discussion of OMB's clearance and, Information Collection Budget 24 requirements.) Their assistance can help in making the clearance process go as smoothly as possible. When preparing an RFP, the following issues concerning data collection, questionnaire development, and testing, should be considered. While they may not apply in all cases, when they do they will have an effect on the contract price. Therefore, in an attempt to avoid any misinterpretation by potential offerors, these issues should be addressed when applicable. Depending on the situation, they can be either explicitly defined or stated in general terms and then used in the evaluation of the proposals. Method of data collection. The three most common types are personal (face-to-face) interview, telephone interview, and self-administered (primarily mail) questionnaires. Each has specific advantages and disadvantages which must be considered. They may also be used together in a survey. An example would be a self-administered questionnaire and a personal interview follow-up. It is important to make certain that the desired information can be collected successfully or more accurately and that the target response rates can be achieved by the proposed method or combination of methods. Mail questionnaires seldom produce adequate response rates without some form of telephone or field follow-up. Data specifications or information to be obtained. They should be listed in sufficient detail to indicate what the questionnaire content should be and what the amount of effort required in developing and testing the questionnaire will be. They should be complete and realistic. When the survey is covering a new topic, the sponsor should identify the variables for which data are needed, but should allow the contractor, once selected, to play a major role in developing the specific questions needed to get data on these variables. Pretest. The need for a pretest is paramount in any survey. Without it the chances of achieving good results are minimal. The three following primary issues arise in a pretest which must be considered in any survey: sample size, probability sampling, and the survey time schedule. Sample sizes should be sufficiently large to learn about significant problems that may be encountered. For this reason, specifying a pretest of nine respondents so that the pretest does not have to be cleared through OMB should be avoided. Furthermore, clearance at an early stage of development may identify possible overlap of the planned survey with work of other agencies or may bring to light experience in other surveys that will help to avoid repetition of past mistakes. It is better to determine the sample sizes needed without reference to clearance requirements and to ensure that the survey development schedule allows enough time for necessary clearances. While probability sampling at all levels of selection is desirable for nearly all surveys, it may not always be indicated for pretests. Purposive sample designs are frequently used in pretests to ensure inclusion of important subpopulations. Probability selection methods at the final stages (that is, households within block) are still desirable to avoid selecting the "easy to get" units. The survey time schedule should allow for adequate time and professional staff to both conduct and analyze the findings from the pretest and to use the findings effectively in subsequent stages of development. 25 The following is a minimum list of topics which should be addressed in the planning of a pretest: - Should exploratory interviews be conducted? The purposes might include identifying items that are difficult or sensitive to answer, determining if the data specifications are complete, or ascertaining what records, if any, are needed and if they are available. - Will the pretest be conducted in waves? That is, will the pretest sample be sent out in parts, each of which contains questionnaires modified based on the findings in the previous part? - Will the effect of the sequence of the questions be considered? - How long is the average time for pretest interviews likely to be? - Will the pretest determine the required experience level of the interviewers or is this one of the requirements in the RFP? - What records, if any, will the respondent need to refer to and will they be reasonably accessible? - What results are necessary before advancing to the next stage of the study? Respondents' obligation to reply. Any information about the respondents' obligation to reply that would help the offerors should be included in the RFP. Most surveys are voluntary. However, if the survey is mandatory, if a response is required in order to obtain or retain benefits, or if informed consent is required, this could greatly affect the proposals and should be included. Length of interviews. If the contractors are to develop the questionnaire, a ceiling on the interview time should be set. A respondent may agree to cooperate only if this time is kept to a minimum. Care should be taken to obtain the required information in this limited time. To do this, a determination of the minimum number of questions needed to accomplish the survey's purpose should be made. Confidentiality. Any questions regarding confidentiality or the Privacy Act system of records should be addressed in the RFP. If there is a promise of confidentiality in the survey, the contractor may have to take special provisions to allow for it. These could include locking up data and only allowing access by authorized personnel. Special provisions regarding the security of the questionnaires might also be needed. Payments to respondents. If there is to be financial compensation to respondents for completing a questionnaire, the offerors should not only be made aware of this fact, but also the reason for it. Following are several reasons (each of which could have different effects on the bids) are: studies have shown that the success of the project depends on the agency providing compensation, or the data collection requires a significant amount of work on the part of the respondents and they should be compensated for their time. In addition, OMB and many Government agencies require a description or justification of any remuneration to persons supplying information. If the success of the survey depends on these payments, approval should be obtained from both the Department and OMB before the contract is signed. Advance notice to respondents. Many surveys require advance notice to the respondents. This is so they can consult their records and documents, thereby 26 making the interview as useful as possible. If this time is needed, the offerors should be made aware of it so they can plan their resources accordingly. Dates for data collection. There may be reasons that specific beginning and ending dates for the data collection must be set. These reasons could be that the particular event being measured will only occur during those times or that the collection is not allowed while the decennial census is being conducted. Make certain the reasons are justified. Moving up the time schedule for a survey is usually not a sufficient justification. Training. It must be determined who will be responsible for training interviewers and preparing the necessary materials. In addition, what specific types of training requirements exist and will the sponsor participate in the training., Response rate. The required minimum response rate should be specified along with the method which will be used to calculate it. Included with this information should be the number of completed questionnaires required and the criteria for defining a completed questionnaire. The procedures for field follow-ups should be stated. These should include any time-of-day and day-of-week requirements along with the number of follow-ups and a requirement that the contractor complete all specified follow-ups, even if the required minimum response rate is met. In addition, there should be a provision that the contractor must exceed the minimum number of callbacks if they are required to meet the minimum response rate. Of course, any additional field follow-ups needed to meet the response rate will have a cost associated with them. This tradeoff should be considered while decisions on the survey requirements are being made. No attempt is made in this report to determine what the minimum response rate should be for every survey; each survey will have factors that will have an effect on this number. The best advice is to check current policy and consult a survey statistician before writing this section of the RFP. Quality control. The quality controls that will be required in the data collection process and the validation requirements must be addressed in the RFP. Who will be conducting these tests must also be specified. It is usually best to indicate that quality control is a necessary part of the survey and let the offerors indicate how they propose to control quality at each stage. Interviewer qualifications. If the interviewers must possess certain qualifications because the survey has special circumstances that require them, they must be specified in the RFP if they are to be used in the evaluation process. These could include expertise in the specific field of study, knowledge of a foreign language, previous data collection experience, or knowledge of survey sampling. Additional questionnaires. If there are to be different versions of the questionnaire , these requirements must be specified along with the reasons for needing them. They could be necessary due to the questionnaire must be tailored to the various regions of the country because of content or other considerations. Examples are asking about citrus fruit production only in the appropriate areas or having a foreign language version of the questionnaire in areas where English is not the predominant language. Tabulations and reports The final products and quality of the completed work should be monitored very closely. To facilitate the monitoring process, certain deliverables should be 27 specifically requested in the RFP. A schedule should be set up and rigidly followed. The following list contains several of the more important deliverables. Progress reports. Progress reports should be required according to an agreed upon plan. The p!an should specify content as well as time of delivery. Draft reports for agency review. The RFP should indicate that a draft of each required task report will have to be submitted for approval before the final report is written and the task is considered complete. Tabulations required. When it is feasible, the format and content of all required tables should be included in the proposals. The RFP should also state whether the contractor will be expected to perform analyses of the data and in what form they must be delivered. Data descriptions. All data supplied should be accompanied by sufficient documentation. For example, the formats of any data tapes must be required and the formula used to calculate any weights must be supplied with the weights. Survey documentation. The RFP should require a complete detailing of the methodology used. This report should include the contractor's impressions of the effectiveness of the survey procedures, suggestions for improved methods in future surveys, and all appropriate technical materials. Included would be the following: copies of all forms; documentation of all (especially sampling and data collection) procedures in all phases; all adjustments for nonresponse and missing data; survey estimates and estimates of sampling errors; full accounting of all data collection results, including response rates and interviewer validation results; coding error rates, along with all quality control procedures used; methodological findings; and a description of the procedures used in handling confidential data. Analytic reports. If the agency wants the contractor to produce one or more reports analyzing the findings of the survey, such a report should be requested in the RFP. Frequently this analysis and the resulting report are done in- house. If this is to be the case, it should be stated in the RFP. Analytic reports should contain a complete analysis of the survey data along with any appropriate conclusions and recommendations. ROLE OF THE PROJECT OFFICER Monitoring functions of the project officer that directly affect the performance of the contractor should be clearly discussed in the RFP. This allows the offerors to know in advance how the project officer intends to monitor the contract. Following are several important points which should be defined. The RFP should list all areas where reviews and approvals will be required. These could include sampling plans, questionnaires, training materials, and data processing requirements. If site visits are planned or meetings are to be scheduled, they should be mentioned in the RFP. The RFP should clearly specify the role of the Project Officer regarding attendance at any training that is planned. Detailing the role of the project officer in the RFP allows the offeror to plan for additional resource requirements and permits a smoother relationship in the performance of the contract. 28 EVALUATION CRITERIA Every RFP must include the criteria to be used in evaluating the technical merits of the offeror's proposal. These will be listed in descending order of points to be allocated to the item and may show the actual number of points. The technical evaluation criteria need to cover the critical elements to be used in determining the acceptability of the offeror and the proposal submitted for the contract survey. Differentiation between acceptable and unacceptable proposals and offerors is made on the basis of these criteria. If a vital element of approach or capacity is not reflected in the evaluation criteria, it may not be possible to distinguish among offerors or their ratings based on the issue. All too often program staff members concern themselves with technical aspects of the survey (that is, sampling, interviewer training, field work, editing and coding) and ignore the need to translate these specifications into the criteria that will be used to evaluate the proposals received. The criteria will be used in the selection process. The survey specifications are the basis on which the offeror provides the proposal that will be the subject for evaluation. Proposals can only be evaluated based on the published criteria. 29 Chapter 4. CONTRACTOR SELECTION PROCESS INTRODUCTION The work to be done under the contract will have been described in the Request for Proposal (RFP) as well as the material and presentation to be submitted by the offeror as a basis for evaluating the response. The criteria and scoring to be used in this evaluation are also included in the RFP, thus giving each offeror the same knowledge about evaluation of the technical proposal. It is in this sense that contractor selection starts when the decision is first made to issue a competitive RFP. By that time, there should have been consideration of the possibility of doing the work in-house or through another Federal agency, of contracting with an 8(a) firm, or of limiting competition to small business firms under the set aside program. The decision by the contracting officer to issue an open competitive RFP instead of taking one of these other options involves decisions and justifications within the agency as to the conduct of the work or some basic characteristics that potential contractors must have. These may relate to performance or capacity requirements or experience or reputation requirements of potential contractors. These characteristics should be part of the specifications included in the RFP. Since the agency has made the decision to go competitive based on specific requirements of the proposed contract as these relate to capacities of potential offerors, it is incumbent upon the writers of the RFP to make this information available to potential offeror organizations. It will be useful to them in making their decisions as to whether to respond to the RFP. The mechanism for doing this is the RFP itself. An adjunct to this that is sometimes used is a Bidders' Conference as discussed in Chapter 3. The thinking that resulted in a competitive RFP should provide the framework of offeror capacities and expected activities performance that will be used to evaluate responses. The remainder of this chapter is based on the premise that the RFP discussed what was being looked for in the contractor selection process. COMPONENTS OF THE SELECTION PROCESS The response to an RFP is submitted in two separate parts, the Technical Proposal and the Business and Cost Proposal. These are evaluated separately. Evaluation of the Business and Cost Proposal is the responsibility of the contracting officer and is done in terms of the appropriateness and documentation of costs relative to resources to be devoted to the effort. The technical proposal is evaluated in terms of the material on project conduct and management, the staff and other resources to be devoted to the project, and the qualifications of the staff. If the RFP was specific in the description of the work to be done, the capacity and experience of the organization and staff will probably be weighted more heavily in the evaluation criteria. If the RFP was more general in the description of the work to be done, the approach to conducting the project proposed by the offeror will probably be weighted more heavily. In any case, the RFP should 31 have specified what the offeror was to submit as a basis for technical evaluation and the criteria to be used. The contracting officer has the responsibility to ensure that the technical evaluation is conducted in a way that satisfies the Federal Procurement Regulations and agency requirements. The program office bears responsibility for the conduct of this review. The guidelines in the regulations governing technical review require establishing a selection process that will insure fair and impartial treatment of all offerors, the selection of sources whose performance is expected to meet the contract requirements at a reasonable cost or price, and avoidance of arbitrary or capricious behavior, inequitable treatment, or undue influence. It is essential that persons involved in the evaluation process have no conflict of interest with this function and that no information concerning the evaluations be released to another individual except one who is participating in the process. The project officer for the contract is responsible for proposing the names of individuals to be on the technical evaluation panel to his/her superiors within the program office. The project officer is expected to be a member of the panel and may serve as its chairperson. The panel is usually composed only of government personnel. Outside evaluators are used only under unusual circumstances, such as when required expertise for evaluation is not available within the government. The project officer should consult with the contracting officer, who bears official responsibility for the overall contracting process, on aspects of the technical review process and current standards and requirements of the Federal Procurement Regulations and agency regulations on technical evaluation. In general, project officers are given a significant amount of flexibility and latitude in establishing panels and review plans and in conducting technical evaluations. However, a list of persons to serve on the technical evaluation panel and a plan for the technical evaluation must be submitted to the contracting officer before the technical proposals are released to the project officer for panel evaluation. To the extent possible, members of the project team should be included on the panel. The contracting officer should attend the initial panel meeting to clarify the basic rules and considerations under which the technical review will take place. Beyond that, the prime requirements are that the process be equitable, that the basis for the evaluation of each proposal be documented, and that the evaluation be conducted in terms of the criteria published in the RFP for contractor selection and the evaluation plan. The purpose is to make the contractor selection process as fair and open to competition as possible. The process is designed so that all potential offerors have the same base of information on which to assess their qualifications for competition for award of the contract, to know the basis on which a selection will be made and to have the selection based on preset criteria. The goal of this approach is to eliminate the actuality and, to the extent possible, the appearance of bias and preselection. In assembling the technical review panel, the project officer should try to obtain a balance of areas of survey expertise and subject matter interest as related to the purpose of the contract and the needs of the agency. For example, this might include experts on survey methodology and sample design and experts representing interrelated content and program areas. Expertise and experience available in other agencies should be used as a resource. Representation of other agencies is also advantageous when there is overlapping program interest. A panel that is biased in composition toward a given expertise may distort the contractor selection in that direction independent of the balance built into the technical evaluation criteria and" plan. A minimum of three persons should be on the panel to allow for more effective 32 discussion and resolution of disagreements. The project officer should arrange for a meeting place where panel members can discuss the merits of each of the proposals in private. It is important that there be continuity over the full process of proposal review and contractor selection. Individuals selected for the panel should be available to review later modifications and additions as well as the initial responses to the RFP. TECHNICAL EVALUATION OF CONTENT AND CAPABILITY The technical evaluation process is designed to carry out the objective consideration of the technical merits of proposals submitted in response to an RFP in terms of the published evaluation criteria contained in the RFP. By the end of the technical review process each proposal will be identified as being either acceptable or unacceptable. A proposal can be determined at the outset to be technically unacceptable without detailed review if it fails to meet a specific expressed requirement of the RFP. As an example, if the RFP requires probability sampling for the planned statistical survey and an offeror proposes only an alternate sampling approach, the proposal can be considered an inappropriate response to the RFP and technically unacceptable. A technical evaluation plan should be prepared by the program office. It should include information for the panel members based on the specific criteria, as included in the RFP, to be used in rating proposals. The evaluation plan should provide more specific guidance for the panel on details to be looked for within an offeror's proposal in scoring it under various elements of the evaluation criteria. The plan will specify the number of points to be allocated to each part of the evaluation criteria. In general, these criteria should deal with the offeror's approach to the contract tasks and the organizational ability to perform those tasks. The evaluation criteria for ability or capacity to perform, which should have been built into the RFP, should include some forms of the following considerations: demonstrated organizational expertise in the subject area, staff availability and their expertise in the subject area, past experience in surveys under contract, ability to meet special requirements, and, availability of needed resources. The references for past surveys under contract should be checked. Before discussing the proposal in a group, each member of the evaluation panel should individually read each proposal and prepare a tentative indication of its strengths and weaknesses and a preliminary score on each evaluation criterion using the rating sheets in the technical evaluation plan. Adequate time should be allowed for this key aspect of the process. After this has been done, the panel members should meet to discuss the relative merits of each proposal and try to arrive at a consensus opinion on the strengths and weaknesses of each proposal and the possibilities for correcting the weaknesses. The evaluators should then individually rate each proposal. The individual scores of the panel members have to be combined into a single ranking of all of the proposals. While a numerical average of the panel members ratings for each proposal is generally used in developing a ranking, this is not a requirement and depends on the rating plan used. The panel then should identify each proposal as acceptable or unacceptable. However, this can not be done by the use of predetermined cutoff scores. The definition of a proposal as acceptable or unacceptable is based on technical consideration of the offeror's proposal in terms of the requirements of the RFP "as assessed, using the evaluation criteria published in the RFP. A determination of 33 unacceptability must be supported by concrete factual statements consistent with the evaluation criteria and indicate that the proposal's deficiencies are so major it could not be brought up to an acceptable level without the equivalent of a new proposal being submitted. The technical evaluation panel members will prepare and sign a report to the contracting officer showing the ranking of the proposals and identifying each as acceptable or unacceptable. The individual rating sheets should be included. A narrative assessment of each proposals strengths and weaknesses must be a part of the report. The report should also include any considerations related to the selection of sources for negotiation and award and any specific issues and questions for discussions. DISCUSSIONS WITH OFFERORS IN THE COMPETITIVE RANGE After the technical evaluation and analysis of the business proposals, the contracting officer will determine which proposals are in the competitive range as a basis for conducting written or oral discussions. Discussions are communications essential for determining a proposals acceptability or to provide the offeror with the opportunity to revise or modify its proposal based on deficiencies or weaknesses noted in the evaluation process. More than one round of discussions may be held by the contracting officer with offerors depending on the situation. The determination for inclusion in the competitive range will be based on price or cost and technical and other important factors as stated in the RFP. The competitive range will include all proposals that have a reasonable chance of being selected for award. In making this determination, the contracting officer may request the technical evaluation panel to review the cost or price data, or request comments from the project officer on labor and other resources needed to conduct the project. Discussions will be held with all offerors in the competitive range. The competitive range may be narrowed after initial discussions by dropping an offeror if a revised proposal is determined to no longer have a reasonable chance of being awarded. All discussions with offerors are under the control of the contracting officer. Offerors are given the opportunity to correct deficiencies, resolve uncertainties and errors, and to submit revised material based on these discussions. However, no information is to be given to offerors in these discussions on other offerors' proposals or the evaluations, or on competitive or estimated prices other than to indicate that the Government considers its price too high. No specific help should be given an offeror in rewriting the proposal. At the end of the discussions with all offerors in the competitive range, each offeror will be notified that the discussions have ended and that they have an opportunity to submit a "best and final" offer by a common cutoff date, including a revision of an earlier offer or a confirmation that a prior offer is a "best and final" offer. These "best and final" technical and business proposals are subject to evaluation on all significant factors by the contracting officer and project officer and, if necessary, a" full technical rescoring and reranking by the technical evaluation panel. At the end of this process, the contracting officer will select for award the proposal that offers the Government the greatest advantage, price, and other factors considered. While the final selection decision is the responsibility of the contracting officer, this' decision is usually made after consultation with the project officer. Since it will be 34 the project officer's responsibility to monitor technical performance, he or she should be part of the assessment of the relative importance of technical quality and price differences. The analysis of business management and price factors is the contracting officer's domain. Although price or cost is one factor in the final decision, whether certain items are allowable and the assessment of the basis for estimating cost is an administrative issue. Usually, whether a contract is to be a fixed-price or a cost-type is specified in the RFP. However, the final decision to use a fixed-price or a cost-type contract is a contracting officer decision and the ability to estimate certain costs is a factor in this decision. THE FINAL AWARD PROCESS .After the successful offeror has been selected, the contracting officer may hold limited negotiations with this offeror. However, these negotiations cannot relate to any factor which could have an effect on the Selection process. It is the Contracting Officer's responsibility to ensure that negotiations do not change the RFP,s requirements nor make any other changes that would impact on the selection process. If there is a material change in requirements because of negotiations, the competition must be reopened to all offerors who were in the competitive range. A significant increase in the offeror's cost proposal may also require reopening of the competition. After any negotiations are held, the final contract document is prepared. In preparing the contract document, including all the agreed upon terms and conditions, the option should be considered of specifically incorporating the offeror's final proposal to avoid any question as to what was offered and accepted. This can be of particular value when the RFP specified only an area of survey concern and not the specific approach to be used. After the contract has been awarded, the unsuccessful offerors will be notified. Unsuccessful offerors can, and often do, request debriefing. The project officer and possibly other members of the technical evaluation panel may be. called upon in a requested debriefing of an unsuccessful offeror. It is at this point that the documentation of the basis for technical evaluation can be of critical importance in explaining why the offeror's proposal was not selected for award and in establishing that the competitive process was appropriately conducted. Unsuccessful offerors may formally appeal a contract award. However, a well documented and unbiased basis for selection will reduce such appeals of the selection process. 35 Chapter 5. POST-AWARD ACTIVITIES INTRODUCTION Following award of a contract, both the contractor and the Government have responsibilities that must be fulfilled. Post- award activities are all activities that occur or should occur once the negotiated contract has been signed by both parties. it is the joint responsibility of the contractor and of the Government to see that work under the contract is accomplished in a timely and proper manner. Government policy and procedures designed to ensure that the contractor and the Government staff understand their mutual responsibilities after the contract is awarded are set forth in the Federal Procurement Regulations. The major elements of post-award contract administration include monitoring of performance (including review and approval of products, services, or reports), contract modifications, approval of subcontracts, property administration, and contract closeout. Aspects of these elements may be performed by various Government personnel, but the two principal persons are the project officer and the contracting officer. It should be made clear at the outset of the contract exactly what the respective roles of these two individuals are with respect to contract administration and monitoring. Typically, the contracting officer, the person who actually signed the contract on behalf of the Government, delegates to the project officer responsibility for monitoring technical aspects of work under the contract. In most instances, a contract identifies tasks, subtasks, schedules, review processes, criteria identifying skills, techniques and methodologies to be used, and a schedule of deliverables. These requirements are usually included in the Request for Proposal (RFP) and are addressed specifically in the contractor's technical proposal, which is often incorporated into the final negotiated contract. The project officer must monitor closely all aspects of work to ensure that it is carried out as specified. In addition to technical and procedural requirements for conduct of the work associated with the survey, the contract should also include performance requirements and a description of measures to be used in determining if certain requirements, such as the specified response rate, have been met. The contract is the basis for monitoring and assessing contractor performance. A project officer should have a complete copy of the final negotiated contract for a survey, should read it completely to ensure that it contains all that it should contain, and should reread it periodically to insure that all of its terms and specifications are being followed as specified. Most post-award responsibilities of the project officer relate directly or indirectly to technical aspects of the work. The role of the project officer is discussed in the following material, which emphasizes activities related to insuring that the product resulting from the contract is of the highest quality possible. 37 GENERAL TECHNIQUES FOR MONITORING CONTRACTOR PERFORMANCE Meetings with the Contractor To the extent possible, there should be regular meetings between the project officer and the contractor. For a large, complex contract such meetings may have been included in the statement of work and travel funds provided for contractor participation. If such meetings are not possible or are only possible on an infrequent basis, conference telephone calls may serve as a reasonable, though less satisfactory, substitute. These meetings provide a forum for structured, though somewhat informal, review of activities and can offer an early alert to technical problems or to slippage of the project schedule. Maintenance of Files Ideally, the same project officer will serve from the inception of the project through its completion. Experience shows, however, that this is often not the case and that project officers may change several times during the course of a project. Thus, it is essential that the project officer maintain complete, organized files pertaining to the project. Included in the files should be the following: all formal contract documents, including the negotiated, signed contract and any modifications to it; copies of all correspondence relating to the project between th