Cognitive Testing of Questions to Measure Family Violence

Anna Paletta⁽¹⁾ and Karen Mihorean⁽²⁾

Statistics Canada

Abstract: In 1998 the Questionnaire Design Resource Centre of Statistics Canada undertook cognitive testing of the questions on spousal violence and senior abuse for the 1999 General Social Survey on Victimization.The testing provided unique challenges in the design of the cognitive methods used due to the nature of the survey topic. The findings of this research demonstrate how cognitive testing can be used to test questions on sensitive issues. Moreover, the cognitive testing was instrumental in demonstrating that a national quantitative survey on victimization can collect information on spousal violence and senior abuse.

Key words: family violence, spousal violence, senior abuse, qualitative methods, cognitive interviews, focus groups, questionnaire testing, sensitive questions

Introduction

Beginning in the fall of 1997, Statistics Canada began planning the 1999 General Social Survey⁽³⁾ (GSS) on Victimization. This is the third cycle of the GSS that focuses on criminal victimization in Canada. The first two were conducted in 1988 and in 1993. The main objective of the survey is to collect national and provincial level information on the nature and extent of criminal victimization in Canada.

As part of the survey planning process, extensive consultations were held with key stakeholders.⁽⁴⁾ Through these consultations information gaps were identified. These included public perceptions of the prison and parole systems, perceptions toward alternatives to imprisonment, hate crime, spousal violence and senior abuse. While much of the survey remained similar to the 1988 and 1993 surveys, several significant changes were made to its content to address these gaps.

Key to ensuring the appropriateness of the new survey concepts and question wording, especially questions concerning spousal violence and senior abuse due to their sensitivity, extensive testing was necessary. The purpose of this paper is to demonstrate how cognitive testing is beneficial to the development of measures for sensitive survey concepts, such as issues of family violence.

The GSS on Victimization

Using Random Digit Dialing techniques, 25,000 people 15 years of age and older living in the 10 provinces of Canada, with the exception of full-time residents of institutions, are asked about their experiences of criminal victimization. Similar to previous cycles, this survey measures public fear of crime, precautions people take to ensure their safety, public perception of crime and the justice system, and household and personal victimization, including vandalism, personal, household and motor vehicle theft, break and enter, robbery, physical assault and sexual assault. All respondents who report being a victim of a crime are asked more detailed questions about each crime incident, including injury, use of medical and victim services, reporting to the police, location of the incident, etc. Consistent with all cycles of the GSS, a number of socio-demographic measures of respondents are included (e.g., income, education, health, main activity, etc.).

As indicated earlier, new content has been added to the survey on public perception of the prison and parole systems, hate crime, perpetrators' characteristics, and use of victim services, including interest in participating in victim/offender mediation programs. In addition, questions on public perception toward alternatives to imprisonment have been funded by the Solicitor General Canada, and under the Federal Family Violence Initiative, Health Canada, Statistics Canada, Justice Canada and Human Resources Development Canada have funded modules on spousal violence and senior abuse.

Measuring Spousal Violence and Senior Abuse

While victimization surveys such as the GSS are proficient at measuring property offences, crimes committed by strangers and people's perceptions of crime, they have not been designed to measure the more sensitive kinds of victimization that occur within the family. Therefore, the undertaking by Statistics Canada to measure spousal violence and senior abuse through detailed questions on a traditional victimization survey is considered somewhat unique.⁽⁵⁾

While measuring spousal violence and senior abuse through a general victimization survey is a unique undertaking by Statistics Canada, it is not the first time that the Agency has measured spousal violence. In 1993, through funding from the Federal Family Violence Initiative, Health Canada asked Statistics Canada to undertake a national survey on Violence Against Women (VAW). This survey captured information about sexual and physical assaults by men that women experienced since the age of 16. A key component of this survey was a module that collected data on the nature and extent of wife assault in Canada.

Based on the success of Statistics Canada's VAW Survey, questions measuring wife assault were incorporated into the 1999 GSS on Victimization. Similar to the VAW Survey, it was considered important to restrict measures of spousal violence and violence against seniors by children and caregivers to Criminal Code definitions of assault and sexual assault in order to capture "violence" as it is legally understood. Therefore, violence by current and previous spouses and common-law partners and violence against seniors by children and caregivers are measured through questions about a series of violent acts similar to some of those contained in the Conflict Tactics Scales (CTS) of Murray A. Straus and Richard J. Gelles (1990), ranging from threats of violence to threats or use of guns or other weapons. There is an additional question on sexual assault (seniors are not asked about sexual assault by children due to the rarity of such incidents and the extreme sensitivity).

The questions that were included in the cognitive testing included:

• threatened to hit you with his/her fist or anything else that could have hurt you;

• thrown anything at you that could have hurt you;

• pushed, grabbed or shoved you;

• slapped you;

• kicked you, bit you, or hit you with his/her fist;

• hit you with something that could have hurt you;

• beat you up;

• choked you;

• used or threatened to use a gun or knife on you; and

• forced you into any unwanted sexual activity by threatening you, holding you down, or hurting you in some way.

  Recognizing the links between physical and emotional abuse, questions related to emotionally and financially abusive behavior were also included. A total of ten questions were included in the qualitative testing, five of which were from the VAW Survey. Questions included:

• is jealous and doesn't want you to talk to other men/women (VAWS);

• insists on knowing who you are with and where you are at all times (VAWS);

• tries to limit your contact with family or friends (VAWS);

• calls you names to put you down or make you feel bad (VAWS);

• harms, or threatens to harm, someone close to you or your pets;

• damages or destroys your possessions or property;

• prevents you from knowing about or having access to the family income, even if you ask;

• tries to force you to relinquish control over your finances when you don't want to (VAWS);

• tries to force you to give up something of value when you don't want to; and

• tries to force you to change your Last Will and Testament when you don't want to.

  The following question was also asked of respondents 65 years of age and older in the case of their children and other caregivers:

• tie, or threaten to tie, you to your bed or a chair.

  Goals of the Cognitive Testing

  Cognitive testing was conducted during February and March 1998. While all the questions in the draft questionnaire were tested, the focus of the testing was on the spousal violence and senior abuse modules. The main goals of the testing of these questions, similar to the goals in testing questions and questionnaires in general, were to ensure that the final questionnaire would be relevant to the divergent experiences of all respondents, that it would be comprehensive, and that the questions could be easily understood and completed accurately. In addition, there were a number of special considerations specific to testing a national survey collecting information on spousal violence and senior abuse. Foremost was to test the impact of the survey on respondents and to ensure that respondents would disclose very personal experiences about spousal violence and senior abuse in a national crime survey. The sensitivity of both the questions and the survey process needed to be tested in order to gauge and minimize any emotional repercussions of being asked about potentially painful experiences while ensuring that respondents would disclose spousal violence or senior abuse in the context of a national survey on criminal victimization. In this regard, collecting data on sensitive issues such as spousal violence or senior abuse, response burden is not only in terms of the time and effort it takes to complete the questionnaire, but also in terms of the emotional or psychological discomfort or distress. The goals of the cognitive testing and the methodologies used to test the 1999 GSS reflected these concerns of sensitivity, impact and related necessary special considerations.

  Methodology

  At Statistics Canada cognitive methods, in-depth interviews and focus groups are extensively used in the development and testing of new questions and of questionnaires that have undergone extensive revisions.⁽⁶⁾ The testing in preparation for the 1999 GSS on Victimization provided unique challenges in the application of these methods to the testing of this questionnaire.

  The cognitive testing was undertaken by the Questionnaire Design Resource Centre (QDRC) in conjunction with other Statistics Canada personnel. The testing team included four consultants from the QDRC, four senior interviewers and a supervisor from the Operations and Integration Division (OID), and two subject matter experts in the field of victimization, spousal violence and senior abuse. Unique to this survey, an external professional psychologist who specializes in family violence was also part of the testing team.

  Due to the complexity of this subject matter, a variety of methods were used to evaluate the questionnaire. With some respondents testing was through a telephone interview followed by participation in a focus group, with others only face-to-face cognitive interviews were used. Use of a particular method was based on needs specific to the type of respondent participating in this stage of testing. In addition, interviewer debriefings were conducted to incorporate their experiences in administering the questionnaire.

  Cognitive interviews

  Two types of respondents were recruited to take part in these cognitive one-on-one interviews: men and women who had been victims of spousal violence; and seniors 65 to 79 years of age who had been victims of senior abuse. Recruiting women who had been victims of spousal violence was done through transition homes and second-stage housing.⁽⁷⁾ Men who had been victims of spousal abuse were recruited through men's support groups, while seniors who were victims of senior abuse were recruited from agencies and associations that had senior abuse programs. In order to recruit individual respondents, QDRC consultants contacted coordinators of the various facilities and/or programs. Once contacted, the purpose of the survey and testing, as well as the testing process were explained. Typically these coordinators and program leaders approached their past clients and then forwarded the names of people who had agreed to take part in the testing. This approach greatly facilitated the recruitment of these respondents.

  QDRC consultants administered the questionnaire face-to-face in concurrent think-aloud interviews with these respondents. A total of 19 cognitive, one-on-one interviews were completed. These included 6 interviews with female victims of spousal violence, 4 with male victims of emotional abuse,⁽⁸⁾ 6 with senior victims of abuse, and 3 with "over flow" from the focus groups.⁽⁹⁾ A number of interviews were conducted in transition homes, while others were conducted in respondents' homes. The choice of the location for the interview was left up to the respondent.

  The main objectives of the cognitive interviews with respondents who were known to have experienced abuse were to ensure that the survey concepts were reflective of their realities, especially in regards to men and seniors since Statistics Canada had never tested these questions with these populations, and to ensure that these types of questions were appropriate for a victimization survey.

  The strength of cognitive interviews as used during this testing was that they incorporated in-depth observation of the four stages of the response process, namely understanding the question, remembering or retrieving the information requested, evaluating and judging how much of the information to provide, and finally, providing an answer. Each of these stages was thoroughly explored in this test as difficulties resulting in inaccurate data were possible at each stage, especially in remembering and retrieving the information requested and evaluating how much of the information to provide. The use of one-on-one interviews with these respondents enabled direct and in-depth observation and probing as participants responded to the questions.

  Focus groups

  Respondents recruited from the general population took part in a two-step procedure, namely, a telephone interview followed a few days later by participation in a focus group. A total of 8 focus groups were conducted in 5 locations. Focus group compositions were:

  1 group with the general population respondents (in Calgary, Alberta);

  2 groups with male respondents only (1 in Ottawa, Ontario; 1 in Montreal, Quebec);

  3 groups with men and women aged 65 to 79 (2 in Ottawa, Ontario; 1 in Saskatoon, Saskatchewan);

  2 groups with Aboriginal respondents (1 in Saskatoon, Saskatchewan, with men and women aged 20 to 64; 1 in Prince Albert, Saskatchewan, with men and women aged 65 to 79)

  Sensitivity was no less important in the testing with respondents drawn from the general population than it was in testing with those who were known to be survivors of family violence, as these respondents might also be reporting incidents of victimization and spousal violence during the telephone interview.

  Statistics Canada OID interviewers first administered a paper-and-pencil version of the questionnaire over the telephone. Consultants of the QDRC conducted the focus groups with these respondents a few days following the telephone interview to discuss the questions and the interview process. The focus group moderators monitored the telephone interviews (with the full knowledge of the respondents) to allow them to better understand the experiences of the focus group participants and the follow-up discussion at the focus groups.

  Special Measures

  The use of cognitive one-on-one interviews with respondents who had been victims of violent or abusive relationships allowed for special considerations to be addressed due to the sensitivity of the questions. First, one-on-one interviews provided privacy, thereby protecting the integrity and vulnerability of the respondents. In addition, face-to-face interviews facilitated establishing a rapport between each respondent and the QDRC consultant. This rapport was important at this stage because of the requirement for in-depth questioning and follow-up probing of emotionally difficult experiences.

  In addition, the testing allowed an assessment of the proposed survey procedures aimed at minimizing respondent burden. To address this a number of steps were proposed. First, sensitivity training for QDRC consultants and OID interviewers was provided by a professional psychologist whose area of expertise is family violence and abuse. Specifically, training focused on sensitization to cues that would indicate when a respondent may be experiencing distress, and how to respond if this occurred. This training also focused on how to identify cues that indicated that the respondent was not ready or was unwilling to discuss any incident that had occurred or that may still be occurring by trying to minimize the violence they had experienced.

  A second procedure to facilitate respondent support was the compilation of a list of community services available to respondents in their community in the event that a respondent looked for advice from the interviewer, or if the respondent became upset during the interview because of disclosing violence. This was considered crucial as interviewers are not trained counsellors but may feel a need to provide access to support to a respondent.

  An additional procedure was put in place to provide respondent support in the focus group component of the test. In the event that respondents from the general population reported spousal violence or senior abuse in the telephone interview, a second QDRC consultant was to be present at each focus group to conduct the follow-up through a one-on-one interview. However, the reporting of at least one incident by a number of respondents in each group made this impossible. Indeed, where over-recruiting for focus groups is the common practice, one of the indicators of the success of this questionnaire and its testing was the high respondent participation rate. It was this high participation rate that dictated that the second consultant conduct 3 one-on-one interviews with "over flow" respondents as ten was considered the maximum acceptable focus group size given the topic at hand.

  Finally, surveys must be interviewer-friendly as well as respondent-friendly. Due to the subject matter it was considered especially important to ensure that interviewers were also well taken care of by providing them with special support throughout the survey process. Therefore, the psychologist retained for the duration of the testing was made available to the QDRC consultants and OID interviewers for debriefing following any particularly difficult interviews. This support allowed the interviewers and consultants to continue to maintain a professional approach at all times by providing a mechanism to aid them with any problems they may experience during the testing due to disclosures of abuse.

  Findings, Recommendations and Outcomes

  Although there were numerous findings on the details specific to the draft questionnaire, there were a number of findings that would be relevant to any national victimization survey that collects data on spousal violence and senior abuse. The following outlines what was learned through the cognitive testing and the changes that were made to both the survey instrument and survey approach.

  Just as with the VAW Survey, it was learned that special training of the interviewers is crucial to the success of a survey that incorporates sensitive subject matter. During the interviewer debriefing, interviewers indicated that they felt well-prepared to identify and respond to cues that the respondent might be distressed or concerned about being overheard by someone else in the household. Focus group participants also stated that they found the interviewers' tone of voice to be very sensitive and that the interviewers took their time asking the more sensitive questions. Respondents found that this helped them feel comfortable about talking about painful experiences.

  While some respondents indicated that the questions were about painful experiences, others welcomed the opportunity to provide information about their situations because they wanted to tell their story. Testing revealed that, for some respondents, it was the first time they told anyone about the violence, thereby emphasizing the acceptance of these questions on a crime victimization survey. Indeed, almost all respondents accepted a national crime survey that collected information on spousal violence and senior abuse. Moreover, most participants felt that Statistics Canada as a federal government agency should be gathering these data.

  The cognitive testing confirmed that personal suitability and the ability to cope with stress were also important criteria to be an interviewer on such a survey. To ensure that the interviewers and focus group moderators were coping well with the subject matter, debriefings took place with the psychologist. The psychologist was also available to interviewers on an individual basis throughout the interviewing period. Again, interviewers stated that it was comforting to know that the psychologist was available to them if needed, and that it was important to have an external resource to talk to directly so as not to appear weak or troubled in front of immediate supervisors.

  It was also learned that gaining the trust of the respondent is of particular importance in surveys on sensitive topics. This can be particularly challenging in the case of a telephone survey where face-to-face interaction is absent. Testing confirmed that there may be under-reporting among individuals currently living in abusive situations because of fear of repercussion.⁽¹⁰⁾ While it is mandatory at Statistics Canada to inform survey respondents that all information they provide will be kept strictly confidential, it was learned through the cognitive testing that the issue of confidentiality had to be repeated throughout the survey. This is especially important prior to asking questions related to spousal violence and senior abuse. Another step taken to increase respondent trust is to provide a 1-800 telephone number that the respondent can use to verify the legitimacy of the survey.

  Similarly, introductory statements were found to be of even greater importance on a survey such as this since they prepare respondents for upcoming questions. While designers of survey instruments often try to keep survey introductions short to reduce survey costs and maintain the interest of the survey respondent, it was learned through the cognitive testing that survey participants needed a smoother transition from the questions related to crimes "on the street" to questions concerning spousal violence and senior abuse. This was found to be a particular challenge, since many people do not consider violence in the home a crime, but rather a domestic issue. Therefore, respondents may think that this type of questioning does not belong on a survey related to criminal victimization. In order to make the transition smoother the following statement was added, "now I would like to ask you about some things concerning your spouse/partner." This is followed by a question that asks how long they have been together which is then followed by the statement, "I'm going to read a list of statements that some people have used to describe their spouse/partner. I'd like you to tell me whether or not each statement describes your spouse/partner."

  The approach taken in the questionnaire (i.e., having the series of violent acts itemized) in general worked very well since it facilitated the respondents' ability to report with a minimum of effort on their part. However, in almost each of the focus groups the issue of asking respondents the entire hierarchy of items (from less serious acts to more serious acts) in the spousal violence and senior abuse scale was raised by one of the participants. Participants raising this issue felt that a screening question asking respondents whether their partner had ever been violent towards them would be efficient to screen people either in or out of the fuller hierarchy of questions. While the groups recognised that this would lessen the burden on respondents who were not in a violent situation, many participants felt that this would lead to an undercounting of spousal violence and senior abuse because they felt that many respondents may not identify certain acts as violent. In order to address the issue of asking people in non-violent relationships about progressively more serious types of violence, the order of some of the questions in the scale was randomised. Specifically, the first two items that are considered fairly low level forms of violence are always presented first in order to ease respondents into the questions. Through the cognitive interviews, this was found to be especially important with respondents who are either currently living with violence or who have been in a violent relationship. The remaining 8 items were randomised.

  In measuring men's experiences of spousal abuse, it was found that questions specific to verbal and emotional abuse were of most relevance here.⁽¹¹⁾ Furthermore, in measuring female-to-male spousal abuse, it was necessary to modify two questions on the scale to be more reflective of men's experiences of physical violence. First, the question that asked respondents "has he/she pushed, grabbed or shoved you?" was modified to include "in a way that could hurt you." Even though the introduction to the series of questions on violence states that "it is important to hear from people themselves if we are to understand the serious problem of violence in the home," some men felt that simply asking about pushing, grabbing or shoving could be misconstrued to include playful acts. In combination with the section introduction that states "it is important to hear from people themselves if we are to understand the serious problem of violence in the home," the addition of "in a way that could hurt you" helps ensure that only violent acts are being captured.

  The question asking, "has he/she beaten you up?" was also modified. Men felt that males may unintentionally under report their experiences in this question because they argued that while a woman could "beat" a man, rarely could she "beat-up" a man because of the difference in their physical stature. The word "up" was therefore dropped from the question.

  No other modifications to the violence scale were necessary to be reflective of the experiences of both men and women. Following the cognitive testing, however, several changes were made to the questions concerning emotional and financial abuse. This included dropping a number of items about financial abuse in the emotional abuse by spouses or ex-spouses sections. While these questions were considered to be appropriate in the case of senior abuse, they were deemed to be inappropriate in the case of spouses. For example, issues of giving up property make unclear what are legitimate property settlements versus controlling behaviour. In addition, items asking seniors whether their children or caregivers "lock, or threaten to lock, you in some place" or "tie, or threaten to tie, you to your bed or a chair" were dropped. These questions did not elicit any disclosures of emotional abuse and were considered to be too sensitive to warrant jeopardising respondents answering further questions.

  Finally, some focus group participants were frustrated by the 12-month and 5-year time frames. They explained that having responded to the questions within these time frames made it appear as though they had lived a "rosy" life when in fact a number of years ago they had experienced serious types of criminal victimization or violence. While the survey did include a question asking about the most serious thing that has ever happened that could be considered a crime, it appeared much too early in the interview. This question has been moved to follow the general crime-related questions and the spousal violence and senior abuse questions. The benefits of this change are two-fold. First, respondents are given a chance to report incidents that they experienced in their lifetime. Second, the quality of the data is much richer because respondents, at this point in the interview, clearly know what is meant by crime and therefore are more likely to identify criminal incidents.

  Conclusion

  Overall, the rules of good survey and questionnaire design apply equally to the 1999 General Social Survey on Victimization. There are, however, unique survey design needs for such a survey because of the sensitivity of the data to be gathered. This was considered crucial in terms of ensuring that a general victimization survey can incorporate measures of spousal violence and senior abuse and that the survey concepts and approach used are sensitive and reflective of the realities of women and men in abusive situations.

  There were a variety of methods used in the testing of the survey and survey approach. These included cognitive one-on-one interviews with respondents who were known to have been in abusive relationships, telephone interviews followed by participation in a focus group with respondents drawn from the general population, and interviewer debriefings to incorporate their experiences in administering the survey. Together, these approaches provided rich insights into understanding how the questionnaire worked and would be better structured.

  This research showed that cognitive methods can be used to test questions on very sensitive topics such as spousal violence and senior abuse, provided that certain steps are followed. Moreover, the cognitive testing demonstrated that gathering information on spousal violence and senior abuse through a national crime survey is viable and is considered part of a statistical agency's mandate. It also demonstrated that the success of such a survey requires the addition of specific procedures to meet specific needs arising from the particular sensitivity of the data to be collected.

  Bibliography

  Gower, Allen R., Brenda Belanger, and Anna Paletta, (1998), "The Contribution of Qualitative Methods in Improving the Quality of Survey Data," Ottawa, Canada: Questionnaire Design Resource Centre, Statistics Canada. Paper presented at the Quality Criteria in Survey Research Seminar, Cadenabbia, Italy.

  Straus, Murray A, and Richard J. Gelles, (1990), Physical Violence in American Families: Risk Factors and Adaption to Violence in 8,145 Families, New Jersey: Transaction Publishers.

1. Questionnaire Design Resource Centre, Social Survey Methods Division, Statistics Canada, 15-J, R.H. Coats Building, Tunney's Pasture, Ottawa, Ontario, K1A 0T6, Canada.

2. Housing, Family and Social Statistics Division, Statistics Canada, 7-D2, Jean Talon Building, Tunney's Pasture, Ottawa, Ontario, K1A 0T6, Canada.

3. The two primary objectives of the GSS are to gather data on social trends in order to monitor temporal changes in the living conditions and well-being of Canadians, and to provide immediate information on specific social policy issues of current or emerging interest.

4. Consultations were conducted with Federal, Provincial and Territorial government representatives, academics, various non-government agencies and representatives from the police, courts and corrections communities.

5. The British Crime Survey (BCS) conducted in 1996 is the only other general crime survey that included a component designed to measure the extent of domestic violence in England and Wales. The major difference between the BCS and Canada's GSS on Victimization is that the BCS captured domestic violence information through a computerized self-completion questionnaire, while the GSS captures all information through computer- assisted telephone interviewing techniques.

6. See Gower, Belanger, and Paletta (1998) for more information on this.

7. Transition homes are also known as women's shelters or safe houses. They provide temporary housing, usually for no longer than two weeks, in times of family crisis due to wife assault. Second stage housing is housing provided to women and their children who are in the process of leaving abusive partners.

8. It was not possible to recruit men who had experienced physical violence by a spouse. Men who were recruited did, however, experience emotional abuse.

9. This is discussed further in the discussion under Special Measures.

10. A number of respondents who had reported spousal violence by an ex-spouse stated that, had they still been living with that ex-spouse, they would not have reported any violence in order to preserve their sense of safety.

11. It should be kept in mind that it was not possible to recruit men who had experienced physical violence by a spouse.