TUSD Non Responders Survey: Understanding Response Bias
Summary of Findings
The following survey attempted to understand the population which did not respond to four
previous surveys (New Teacher, 301 Site Fund, Smart Board, and/or School Council surveys).
The most common reasons cited for lack of survey participation are:
- I did not see the survey in my email account,
- The survey did not pertain to me/I didn't understand the questions,
- Lack of time,
- I thought the email was spam, phish, a virus, from a questionable individual, and
- I chose not to participate.
Based on results, the Accountability and Research Department can follow a set of steps that
might help to achieve higher response rates in the future: 1) keep the survey short, and more to
the point, 2) evaluate the survey population list prior to emailing invitations to ensure that
the right population is being targeted, 3) clearly mark who the survey is from, followed by a
brief title, 4) provide the participant with a detailed description of who the survey is
targeting and what the data will be used for, and 5) following closure of the survey, send a
final email inviting participants to review survey findings. With these steps, and the
suggestions provided by the current survey participants, it is possible to control non-response
rates and minimize the potential for non-response bias.
Introduction
The Accountability and Research Department conducted a survey to determine how to improve the
quality of future surveys and attempt to understand the population which does not respond. The
department's main goal was to improve survey administration, methodologies, and techniques for
the future. It is hoped that by improving survey methodologies, a wider survey population might
be captured.
Survey & Data Analysis Procedures
In order to accomplish this goal, a census of survey participants who were invited to contribute
to surveys by Accountability and Research during the 2005-2006 school year, but did not respond,
comprised the survey population. If an individual did not participate in two out of four surveys
(New Teacher, 301 Site Fund, Smart Board, and/or School Council surveys), they were contacted by
phone for the current voluntary survey (Table 1).
Phone calls were placed by staff from both the Accountability and Research Department and the
Educational Technology Department from May 10th to May 19th. Telephone interviews lasted anywhere
from two to eight minutes. A total of 224 individuals comprised the survey population. Survey
participants were not called for an interview more than three times. If participants did not
respond after three phone messages were left, their survey was completed with a “no” in question
one of the survey and recorded as “unable to contact”.
Responses to telephone interview questions were recorded on an anonymous on-line survey
[View Online Survey Procedures].
Survey participants were requested to answer
a series of questions regarding the four surveys they failed to participate in. These surveys
were the New Teacher, 301 Site Fund, Smart Board, and/or School Council surveys. Each of the
surveys were originally administered on-line [
View Completed Surveys]. All members of the survey population were emailed an invitation
requesting their participation, explanation of the survey topic, the link and access code to the
survey webpage, as well as the person to contact if the survey link and/or access code failed to
work. The survey participant was then emailed a second request to participate if the individual
did not respond to the first email. If the second invitation failed to solicit participation
as well, the individual was recorded in the survey population as a non-participant.
The current survey questions focused on the following:
- whether participants had technical issues with the on-line survey format, or survey design
itself,
- if enough background information was provided in order to complete the survey,
- general survey participation (if respondents participate in surveys in general, or plan to
participate in future surveys), and
- additional concerns/comments.
Once the survey was concluded, both qualitative and quantitative analyses of the data were
completed.
Data Findings (Statistics & Coded Comments)
Out of the 224 potential survey responders that did not participate in two or more previous
surveys, 121 were interviewed for the current survey (54% response rate). Of the respondents
that were not surveyed, 15 (7%) refused to participate, while 89 (40%) could not be contacted
by phone after leaving three messages, or were out on medical leave.
Online survey format
When survey participants were asked to comment on whether they had any technical issues with
the on-line survey format, 10.7% mentioned that they did not see the email survey invitation in
their email account (Figure 1). The second most cited answer was that the individual thought
the email was spam, phish, a virus, or from a questionable individual. After talking to several
survey participants, it became clear that a number of emails sent to those with TUSD accounts
are junk mail and hard to distinguish from legitimate messages. However, 30 participants out
of the 102 who provided additional comments to this question did not have any technical issues
with the on-line survey format. They simply did not have any time to participate in the survey
(N=10), thought they had already participated (N=10), or chose not to participate at all (N=7).
Survey design
The second question interviewers were asked was whether participants had technical issues
with the survey design, such as if the survey was too long, the questions were too difficult,
they didn't understand the survey questions, or whether the survey scales were appropriate for
the questions. While issues with the survey scales did not surface, 3.1% of the population did
not know anything about the survey questions (Figure 2.). When respondents were pressed further,
18 out of 85 participants mentioned that the reason why they did not participate was because
they thought the survey did not pertain to them. For example, the New Teacher survey was sent
to a population of teachers that had worked at a particular school site for three or fewer years.
Teachers that were employed in a school for two or three years did not feel like they should be
included in the same group of teachers who had worked for only one year
[View New Teacher Summary Statistics
]. Therefore, these individuals chose not to participate in the survey.
Individuals who did not to participate in the Smart Board survey stated that they did not
have a Smart board, the board did not work, or that they gave it to another faculty member.
Since the survey population attempted to capture only those that were assigned a Smart Board,
this reveals that the initial survey population list needed evaluation prior to being finalized
[View Smart Board Survey Summary Report
].
Those that chose not to participate in the 301 Site Fund surveys thought that they did not
have access, or had not received 301 stipends. The reason for this is unclear, for the
population for the survey was based on those that had received or would be receiving a 301
stipend for the 2005-2006 school year. This small subset of people was either misguided or did
not receive sufficient communication about this particular program to know they were eligible
for the stipend [View Site Fund (301)
Survey Summary Statistics].
In summary, reasons for lack of participation are not due to survey length, difficulty of
survey questions or inappropriate survey scales, but rather because surveys did not pertained
to them.
Background information
Traditionally, one of the most common reasons why participants do not choose to respond to
surveys in general, is because they did not have enough background information. For example,
participants needed to know who was conducting the survey, how the data was going to be used,
who would have access to the data, or that they had questions about anonymity. All of these
reasons, however, comprised a very small percent of the participants surveyed for the current
survey (Figure 3).
The majority of reasons why individuals did not participate in the New Teacher, 301 Site Fund,
Smart Board, or School Council surveys, was because they felt the surveys did not pertain to
them. Specifically, 14.7 % said the School Council survey did not apply, 13.8% mentioned that
the Smart Board survey did not pertain to them, 10.7 % commented that the New Teacher survey did
not apply, while 2.2 % stated that the 301 Site Fund survey did not pertain to them. Other
comments for not participating included lack of time, or that they simply chose not to
participate.
General survey participation
The last area the survey attempted to explore was whether the current survey population of non
responders participates in surveys in general, and anticipates participating in the future. The
majority (8%) of those asked if they participate in surveys in general said no (Figure 4). Other
reasons for not participating mentioned that they were requested to participate in too many
surveys (3.6%), and because if they did participate, their answers wouldn't make any difference
(1.3%).
When survey participants were asked if they anticipated responding to surveys in the future,
81.5% said they did. Only 18.5% commented that they will not participate in future surveys.
However, final suggestions mentioned by survey respondents stated that surveys should be kept
short (around ten questions or so), and that a clear email subject heading should be provided
in order to distinguish the email from other junk mail they receive (Table 2).
Discussion and Recommendations
In summary, many survey participants of the current survey revealed numerous reasons for not
participating in the New Teacher, 301 Site Fund, Smart Board and/or School Council surveys.
However, the most common reasons cited include: I did not see the survey in my email account,
the survey did not pertain to me/I didn’t understand the questions, lack of time, I thought the
email was spam, phish, a virus, from a questionable individual, or chose not to participate.
Based on the above responses, the Accountability and Research Department can take several steps
which might help to improve the non-response rate for future surveys. First, when designing the
survey itself, keep in mind the number of questions that are included on the survey. One of the
more cited excuses for not participating in surveys was due to a lack of time. Teachers are
inundated with countless email requests, meetings, professional development opportunities, etc.
They are also requested to respond in a number of surveys throughout the school year. By keeping
the survey shorter and more to the point, this might increase the number of participants in the
future.
The second step that might improve non-response would be to re-evaluate the survey population
list prior to emailing invitations to ensure that the right population is being targeted. In some
cases, however, because the survey is sponsored and designed by a department outside
Accountability and Research, checking the original participant roster is difficult. However,
Accountability and Research staff can make strides to provide more clear definitions as to who
the survey is targeting.
Third, because teachers and staff receive a lot of email messages and/or junk mail on a daily
basis, it is difficult to distinguish TUSD sponsored surveys from other email messages. Clearly
marking who the survey is from in the header (such as from TUSD), followed by a brief title
(marking it as a survey) may make it more clear to the email recipient.
Once the survey email is opened, the fourth step that might improve non-response would be to
provide the participant with a detailed description of who the survey is targeting and what the
data will be used for. This description might also include that the department who requested the
survey, will be referring to survey results to contribute to discussions regarding future decision
making, program developments, etc. While very few respondents in the current survey mentioned a
lack of background information as the reason they did not participate, this is one step the
department can take to improve survey administration.
One of the benefits to administering surveys on-line is the efficiency in which the
Accountability and Research Department can have quantitative data available for survey
participants. Following the closure of the survey a final email inviting survey participants to
review quantitative survey results could be automatically generated. This would provide the
respondents with instant feedback to survey questions, as well as where their responses stand
within the survey population. While qualitative data analysis is not an automatically generated
on-line process, Accountability and Research staff can make the data available on-line as soon
as it is analyzed. With these steps, and the suggestions provided by the current survey
participants, it is hoped that non-response rates might improve in the future.
