Success Rate Study Methodology
Clearwater conducted a computer-assisted telephone interview (CATI) survey of a random sample of 204 participants in the Jude Thaddeus Program™ and one corroborating friend or family member for each participant. Data were collected between the dates of May 31 and June 18, 2006. We provided BRI with assistance to adapt the questionnaire for CATI administration, a pretest of the programmed CATI questionnaire, data collection, data set preparation, and analysis. At the end of the project, we delivered a cleaned data set, a report documenting the survey design and implementation, frequency tables of the survey responses with statistical analysis, and other deliverables required by the RFP.
Survey Instrument Clearwater collaborated with BRI to adapt the existing interview protocol for use with our CATI system. The questionnaire was very short, essentially asking one question about the sobriety status of the program participant. We provided our standard review of the questionnaire to assess possible issues with question wording, item order, and the flow of the survey from start to finish. Our bid assumed that the average interview length would not exceed one to two minutes per respondent, would not include any open-ended items, and would be conducted in English only.
We tested the questionnaire programming thoroughly to ensure accuracy in data collection. Clearwater data collection and research staff closely monitored the interviewers initially to identify any issues that eluded detection during the testing. The final questionnaire is presented in Appendix A.
The average interview length was approximately 9½ minutes per case. This includes all of the time spent by an interviewer getting the question answered by both the guest and the corroborator. It exceeded our original estimate of two minutes per interviewee due to the amount of time spent contacting the individuals being called, the extra comments they wished to provide, and Supervisor discussion. The actual interview was much shorter in length.
Sampling Clearwater consulted with BRI regarding the specific information needed for fielding. The Sobriety Survey Protocol provided by BRI originally specified the random sample of participants be generated manually (physically drawing cases from a container.) However, after consultation it was mutually decided we would use Access to randomly choose names from a list provided to Clearwater by BRI. After the names were selected, Clearwater accessed BRI’s in-house database containing contact information for each participant in the Jude Thaddeus Program™ to gather specific information that was copied manully into a spreadsheet. The contact information was loaded into CATI and the interviewers began to contact designated individuals and complete interviews.
Out of a total of 475 records called, 204 resulted in completed interviews with both parties giving an overall response rate of 42.95%. Clearwater interviewers “chased” members of the sample who were not reachable at the household or telephone number provided by BRI when someone we contacted could provide us with an updated telephone number.
Data Collection For the Sobriety Follow-up project, Clearwater adhered to the instructions in the Sobriety Survey Protocol provided by BRI with the request for proposals. Data collection was orginally scheduled for May 30–June 18, 2006, but was extended through June 25, 2006. Clearwater collected data using our 120-station computer-aided telephone interview (CATI) system. Interviewers were able to see and record responses to questions on a computer screen. The software managed the telephone calling, controled distribution of sample, consolidated data, and tracked interviewer activity and productivity. Interviewers were thoroughly briefed prior to data collection and rehearsed the questionnaire before conducting actual interviews.
After the first week of calling, interviewers were briefed a second time immediately following changes in the programming to make the survey flow better. Monitoring staff listened to a sampling of interviews throughout the fielding period to maintain data quality. Hard copies of monitoring reports are included with the final deliverables.
Clearwater used computer-aided dialing, but notpredictive dialing. Predictive dialing has the potential to annoy respondents by introducing a delay in connections after respondents answer the telephone. This delay leads to higher hang-up and refusal rates and a correspondingly lower response rate for the survey.
Interviewers attempted each record a minimum of three times or until a final disposition (e.g., completed interview, refusal) was reached. All definite appointments were attempted regardless of the attempt number. Some records were attempted up to 14 times with an average of 5½ attempts per guest/corroborator pair to locate the correct individuals and make contact with them. To maximize the likelihood of reaching the sampled person, interviewers called each telephone number at a variety of times, including weekdays, weekday evenings, and weekends. Each attempt represents a different calling time, but could include dialing several different phone numbers. Detailed descriptions of each attempt can be found in the attached “Attempt File.” (See Appendix F for a description of the layout.)
Our interviewers are trained in techniques of refusal avoidance that are effective in maximizing response rates. We developed an interviewer manual for the Sobriety Follow-up study that provided the interviewer with information about the study to be used while relating with respondents to keep them on the phone. Answers to frequently asked questions and guidelines for dealing with certain situations were available during the interviews on hotkeys listed in the interviewer manual. The interviewer manual is presented in Appendix B. The text of the scripts are in the questionnaire document in Appendix A.
Data Preparation During data collection, interviewers documented any errors they encountered on data change forms. At the conclusion of data collection, these errors were corrected in preparing the data set for analysis. The data were converted and formatted for review in SPSS (a statistical analysis software package) and Microsoft Access.
Clearwater pays close attention to comprehensive and routine data inspections, data cleaning, and data set preparation. The completed survey data set was cleaned and prepared in an electronic file format in SPSS and Excel. Any data changes were documented on data change forms which are included with the final deliverables. For table production and statistical analysis, variable and value labels were added to the cleaned SPSS data set.
Calling on the RDD sample was considered complete at the point all records had reached their minimum attempts or a final disposition had been assigned. We exported the data from CATI, cleaned, and compiled them into a labeled SPSS datafile.
Clearwater calculated the final disposition for each sample record. The CATI call history tables recorded the interim or final disposition entered by the interviewer for each call attempt. Database programming referenced both the CATI call history tables and the data sets to determine the correct final disposition.
Call Outcomes Each call attempt was given an interim disposition depending on the outcome of the call. At the end of the field period, each record was assigned a final disposition for the study based on the history of interim dispositions and data collected for that case. Because there were multiple phone numbers on each record, the interim dispositions only captured the results of the last phone number attempted, so additional notes were made.
Table 1 shows the calculated final dispositions. Detailed descriptions of each disposition can be found in the interviewer manual in Appendix B, Table 2. The final dispositions shown represent the overall outcome of a guest/corroborator pair. Some individual refusals may be hidden within the partial complete counts.
Table 1 : Final Dispositions
| Final Disposition |
Description |
Count |
| 2 |
Final Refusal - By Guest |
4 |
| 23 |
Disconnect/Non-working |
23 |
| 27 |
Not Avail. during Int Period |
10 |
| 30 |
Wrong Number |
23 |
| 34 |
Final Refusal-By Corraborator |
13 |
| 35 |
Deceased |
5 |
| 36 |
Complete - both |
204 |
| 40 |
Do Not Call |
5 |
| 41 |
Partial Complete Guest Only |
32 |
| 42 |
Partial Complete Corroborator Only |
103 |
| 43 |
Refusal - Unknown |
3 |
| 44 |
Noncontact |
50 |
| 99 |
Insuffcient contact information or record states not to contact corroborator |
192 |
| |
TOTAL SAMPLE |
667 |
Response Rates We calculated response rates based on a break down of four final disposition categories, summarized in Table 2. The overall response rate was 47.55%. This rate reflects the percentage of completed interviews achieved after fully processing all attempted sample records according to the prescribed sample management rules. Table 2 also estimates the invalid numbers from the total number of sample.
Table 2 : Response Rate Formula Categories
| Symbol |
Description |
Dispositions |
Both |
Guest Only |
CRB Only |
| I |
Complete interviews by both |
36 |
204 |
204 |
204 |
| P |
Partial Complete |
41 or 42 |
- |
32 |
103 |
| IN |
Invalid |
99, 23, 30 |
238 |
|
|
| TS |
Total Sampled |
All |
667 |
667 |
667 |
| RR1 |
Response rate 1 - out of total valid records |
47.55% |
55.01% |
71.56% |
| RR2 |
Response rate 2 - out of total sampled records |
30.58% |
35.38% |
46.03% |
| INV |
Invalid rate |
|
35.68% |
|
|
The higher corroborator response rate reflects both a greater availability and willingness to answer among the corroborators than the guests.
The formulas for the response rates were designed specifically for this study, but were based upon the widely accepted response rate formulas published by the American Assciation of Public Opinion Research. The following formulas were applied to each column individiually:
Discussion
The success rate based on the answers of both the guest and corroroborator is 53.2%. The success rate based on just the guest answer is 61.8%. When both the guest and the corroborator were reached, 107 agreed the guest was sober and 94 said they were not. Based on just the guests’ response, 144 guests said they were sober, while 89 said they were not. According to the corroborators’ responses, 188 were sober and 116 guests were not.
Considering both guest and corroborator answers, the true percentage of all guests who are sober is 95% likely to lie in the range of 47.1% to 59.4%. More than six out of ten guests (61.8%) indicated they are currently sober. Based upon their answers alone, the true percentage of all guests who are sober is 95% likely to lie in the range of 56.4% to 67.2%, as shown in Table 3. Considering only the responses from corroborators, 61.8% of guests are currently sober. The true percentage of guests who are actually sober according to the corroborator is 95% likely to lie in the range of 57.3% to 66.4%.
|
