Abstract
Postal
mail surveys have been the mainstream choice of survey research for decades (Mehta
& Sivadas; 1995). However, with the advent of computer networks and the
World Wide Web, Internet-based surveys have recently become popular.
Internet-based surveys are a new development in the counseling profession and
have varying favorability among researchers. Therefore, this article will
present specific guidelines for conducting research utilizing both modes of
methodologies simultaneously.
Due to the modern advent of
computers, computer networks, and the Internet, research methodologies are
continually changing. Since the mid 1990’s e-mail and the Internet have been
used to administer survey methodologies (Sheehan,
2001). However, such modern
technology-based methodologies have not been thoroughly researched in their
application to the counseling profession. Even with this lack of research,
“Internet-based data collection is becoming an increasingly popular method
of conducting psychological research” (Epstein,
Klinkenberg, Wiley, &
McKinley, 2001; p. 339).
With approximately 600
million individuals on-line world wide (NUA,
2002), the Internet has presented
new and unique opportunities for data collection. Though, according to Best,
Krueger, Hubbard, and Smith (2001), the current use of Internet-based survey
methodologies “only permit the generation of diverse, not representative,
samples” (p. 131).
According to Bosnjak and
Tuten (2001) Internet-based surveys are superior to traditional survey models
because Internet-based surveys can automatically track the response process of
participants; therefore providing “insight into the sequencing and
completeness of responses”. In addition, many believe Internet-based surveys
have surpassed postal mail surveys in terms of response speed and cost
efficiency (Cook, Heath, Thompson, & Thompson,
2001; Sheehan, 2001).
However, there is currently
no evidence to show that new methodological technologies such as e-mail and
Internet-based surveys produce higher response rates than postal mail surveys
(Sheehan & Hoy,
2001). Many authors have concluded that conducting survey
research via postal mailing was superior to e-mail in terms of response rate (Crawford,
Couper, & Lamias, 2001; Kittleson,
1995). What has been found
is that the response rate of Internet-based survey methods are highly
connected with higher awareness of technology and greater use of e-mail (Ranchhod
& Zhou, 2000).
Many studies that used
Internet-based data collection had methodological limitations and
inappropriate data analysis strategies. More specifically, such inappropriate
strategies included non-random sampling and assignment to non-equivalent
samples, and various limitations to statistical analysis in the comparison of
samples (Epstein,
Klinkenberg, Wiley, & McKinley, 2001).
In
an analysis of over thirty studies utilizing Internet-based and postal mail
surveys, most of the surveys consisted of different sampling practices,
follow-up procedures, and research designs. In Table 1, eleven of the most
drastically different studies show divergence in response rate between postal
mail surveys and Internet-based surveys.
|
Table
1
|
|
Study
|
Mail
|
Internet
/ E-mail
|
|
Bauchman,
Elfrink & Vazzana (1996)
|
52.5%
|
65.6%
|
|
Guterbock,
Meekins, Weaver, & Fries (2000)
|
48%
|
37%
|
|
Kiesler
& Sproull (1986)
|
67%
|
75%
|
|
Kittleson
(1995)
|
76.5%
|
28.1%
|
|
Kwak
& Radler (2000)
|
42%
|
27%
|
|
Medlin,
Roy, & Ham Chai (1999)
|
47%
|
28%
|
|
Mehta
& Sivadas (1995)
|
56.5%
|
54.3%
|
|
Paolo,
Bonaminio, Gibson, Partridge, & Kallail (2000)
|
41%
|
24%
|
|
Ranchhod
& Zhou (2000)
|
20%
|
6%
|
|
Schaefer
& Dillman (1998)
|
57.5%
|
58%
|
|
Tse,
et. al. (1995)
|
27%
|
6%
|
Closer
analysis of these surveys lead to a better understanding of the response
rates. Many studies that indicated similar response rates between postal mail
and Internet-based methodologies did not use equivalent forms of the survey or
had small sample sizes that portrays Internet-based surveys with a greater
percentage.
Even though Internet-based
survey methodology has shown conflicting results and the superiority of
Internet-based surveys to traditional postal mailing surveys are unknown,
counselors may be forced to begin using such methodologies. Currently,
Counselors and Counselor Educators have the increasing responsibility to
utilize technological advances in clinical practice and in the training of
counseling students. The newest revision of the Council for Accreditation of
Counseling and Related Educational Programs (CACREP,
2001) standards has
modified core curriculum standards and specialty standards to include a
variety of technological requirements. In addition, the Association for
Counselor Education and Supervision (ACES) Technological Interest Network (1999) developed technical competencies for counselor education students upon
graduation. Such competencies include the following: develop web-pages, use
computerized statistical packages and tests, use e-mail and listservs, search
the Internet, evaluate "the quality of Internet information", etc.
The author of this article
posed that the utilization of solely Internet-based research is not always the
best practice due to sample size, complexity and diversity of populations,
disputing consensus on response rates, and the inability to thoroughly sample
a population. However, Internet-based survey research has found to be more
beneficial to postal mailing surveys in terms of data collection, data entry,
and data analysis. The current state of research methodology shows compounding
benefits and limitations for both Internet-based and postal mailing
methodologies. Therefore, the purpose of this article is to describe specific
guidelines on conducting research utilizing both Internet-based and postal
mailing survey methodologies in the counseling field. The guidelines presented
in this article were derived from a doctoral dissertation in Counselor
Education and Supervision that utilized equivalent forms of postal mail and
Internet-based surveys, various follow-up procedures, and a national sample of
professional counselors (McGlothlin, 2001).
Step-by-Step
Methodological Guidelines for Conducting
Internet-Based
and Postal Mailing Surveys
McGlothlin (2001) conducted a
study utilizing a 72-item survey that was distributed both via the Internet
and via postal mailing. The study consisted of a proportional randomized
sample of 1,022 counselors (School Counselors, Mental Health Counselors, and
Counselor Educators). Within a four-week period, 641 individuals responded,
thus creating a 62.7% response rate. 431 (67.2%) responded by utilizing the
e-mail format of the survey, while 210 (32.8%) responded via postal mailing
(See Table 2).
|
Table
2
Summary
of McGlothlin (2001) Instrument Response
(E-mail
and Postal Mailing)
|
|
Variable
|
N
|
%
|
|
Total
Study
|
|
|
|
E-mail Response
|
431
|
67.2
|
|
Postal
Mailing Response
|
210
|
32.8
|
|
Counselor
Educators
|
|
|
|
E-mail Response
|
112
|
78.3
|
|
Postal
Mailing Response
|
31
|
21.7
|
|
Mental
Health Counselors
|
|
|
|
E-mail Response
|
164
|
67.8
|
|
Postal
Mailing Response
|
78
|
32.2
|
|
School
Counselors
|
|
|
|
E-mail Response
|
155
|
60.5
|
|
Postal
Mailing Response
|
101
|
39.5
|
The
methodological procedures utilized by McGlothlin are detailed below.
Step One: Preliminary Decision Making
The first step in developing
the methodology for a study utilizing Internet-based and postal mailing
surveys is preliminary decision making. This step ensures ethical research
practices and the assurance of addressing appropriate research questions.
The
type of instrumentation that is employed in studies utilizing Internet-based
methodological practices places a new dilemma for researchers. First, any
assessment that constitutes the use of tactile assessment devices or the
manipulation of objects (e.g., Weschler Adult Intelligence Scale-III) will not
be appropriate for a study over the Internet because the respondent could not
physically complete these tasks.
Second, the use of published
instruments and copyrighted instruments needs additional documentation to
ensure ethical practices. Written consent (stating consent to utilize a
specific instrument) is always needed from the publisher or author of an
instrument when conducting a traditional study. At this time, copies of the
instrument would be distributed to the sample population. However, additional
consent is needed when conducting Internet-based surveys. Additional consent
must include the author’s or publisher’s consent to reproduce the
instrument in web format. It is important to note that consent to publish
widely published instruments (e.g., Minnesota Multiphasic Personality
Inventory II) may be problematic due to copyright and reliability issues.
Third, researchers must
obtain appropriate technology services to ethically distribute Internet-based
surveys. Ethical standards suggest that surveys and survey results must be
secure (NBCC,
1997). Therefore, researchers are encouraged to use a
non-commercial Internet server (an Internet server physically located at the
researcher's institution, school, or agency). If this is not accomplished, the
survey and results may not truly be secure and are possibly open to ethical
breach.
Lastly, researchers must
become skilled in on-line survey and web page development. It may be
convenient to obtain outside technical support due to the technical aspect of
web page development. However, it is suggested that the researcher become
proficient in such technology to ensure appropriate test development
practices.
Step Two: Participant Data Gathering
After
researchers crystallize the research question(s) and ensures appropriate
technical and ethical practices, data must be gathered on all participants. To
ensure thoroughness of sampling, both e-mail and postal mailing addresses must
be obtained for the individuals being surveyed. Depending on the sample
population, this may be problematic because some populations may not have
e-mail addresses (Step Four: Initial Survey Distribution, accounts for this problem).
Step Three: Survey Development
Once the survey questions are
developed and the participants are identified, the format of the overall
survey needs to be delineated. The main goal of survey development is to
ensure that both surveys (Internet-based and postal mailing) contain identical
information and formatted in the same order. If Likert-scale questions are
used in the postal mailing survey, they must be used in the Internet-based
survey (the same stands for open-ended questions, lists, and all other
variables). Once that is accomplished, esthetics and ease of response is of
key concern. The esthetic value and ease of response of postal mailing mainly
consists of using font size large enough so that the respondents do not have
to strain to view the survey, leaving enough space between items to ensure
lack of confusion, and making sure that all directions are clearly printed at
the top of each page.
Esthetics and ease of
response is more troublesome in Internet-based surveys. Researchers must be
aware of the different types of forms (or option buttons) that can be utilized
in an Internet-based survey (See Appendix A). There are essentially four types
of options that researchers can choose from when developing Internet-based
surveys.
1)
Check Box: Respondents can check a certain item or
multiple items at one time. Check boxes are ideal for representing items
in a list where a respondent could pick multiple choices.
2)
Drop Down Box or List Box: Respondent can pick from a determined
set of choices that appears (or drops down) when the form is clicked. Drop
Down Boxes (or List Boxes) can be used for presenting demographic data.
3)
Option Button: Respondents can check a certain item; however,
multiple items cannot be checked. Option Buttons are ideal for Likert-scale
questions.
4)
Text Box: Respondents can enter text. Text Boxes are ideal for
qualitative responses or obtaining feedback on the overall survey.
An Internet-based survey must utilize the
above options to best match the types of questions and esthetic value of the
postal mailing survey. In addition, choices must be made to the ease of
responses. A major goal in creating an easily completed Internet-based survey
is to make the respondent use the least amount of mouse clicks as possible
(e.g., Option Buttons only require one mouse click, while Drop Down Boxes or
List Boxes require at least two clicks).
The development of
Internet-based surveys provides a greater arena for visual effects compared to
postal mailing surveys. In Internet-based surveys, visual animation, scrolling
text, digital video and audio, and a vast array of background colors and
designs could be used. However, it is recommended that such visual effects are
kept to a minimum or not used at all. There needs to be a minimal amount of
distraction provided to respondents completing a survey. Furthermore,
Internet-based surveys also must be as similar to the postal mailing survey as
possible.
It is also recommended that a
pilot study be conducted during the survey development stage. There are a
variety of purposes of conducting pilot studies in research that mainly relate
to improving reliability and validity issues (for the purposes of this
methodology, alternate forms reliability is a necessary measure to observe).
However, an additional reason to conduct pilot studies with Internet-based
surveys is noted; to ensure accurate administration, completion, and analysis
of the results.
Step
Four: Initial Survey Distribution
As stated in “Step
Two: Participant Data Gathering” e-mail and postal mailing addresses need to
be collected. To establish a baseline of who has usable e-mail
addresses, researchers must send participants an initial e-mail. All e-mails
include a short description of the overall study typed in the subject line
(e.g., "Counselor Training"). This gives participants an
understanding of what the scope or focus of the e-mail. Also, e-mails are
personally addressed by typing the individual name of the participant in the
body of the e-mail (e.g., Dear Susan,). This may prevent individuals from
believing that the e-mail is part of a mass mailing or “junk mail and
possibly deleting the e-mail because it is not personally addressed. Therefore, listservs and long multiple e-mailings are not
utilized.
The e-mail (See Appendix A)
includes a request for individuals to participate in the survey, an individual
password to enter the survey, and a hyperlink to the web page (See Appendix
B)
containing informed consent and an Internet-based version of the survey. The
contents of the e-mail and the web page must be properly written to ensure
ethical informed consent (See Appendix B and C).
E-mail
addresses that are discontinued or unusable are returned to the researcher
within 48-hours. The e-mail addresses that are returned are then identified
and matched to the individual's postal mailing address. The researcher must
then send a cover letter with informed consent (See Appendix
D), a copy of the
postal mailing version of the instrument (with a tracking code), and an
enclosed postage paid self-addressed envelope to all individuals that had a
discontinued, unknown, or unusable e-mail address. In the cover letter, these
subjects are given an option to complete the enclosed survey or complete the
survey on-line; an individual password and Internet address to the survey is
provided. Therefore, all
participants in the study have been contacted.
Step Five: Participant Organization
To promote cost
effectiveness, efficiency, and security, each person is assigned an individual
password in order to gain access to the Internet on-line version of the
survey. This password is not to be used for identifying respondent’s data;
rather it is to be used for tracking purposes. When respondents enter their
password in the Internet-based format of the survey, that password is
extinguished. When a written format is returned, that assigned password is
also extinguished. The purpose of this is threefold. First, deletion of
passwords prevents individuals from completing the survey more than one time.
Second, a password also prevents individuals, outside the selected sample,
from completing the survey. Third, when a password is used and deleted, that
person's e-mail is cross-correlated with the overall mailing list. At that
time, their name and e-mail is deleted and that individual did not receive
further requests for participation in this study. It is important that a
respondent’s name or e-mail address is not associated with their return
survey to ensure confidentiality and anonymity.
As surveys are e-mailed and
postal mailed it is important for the researcher to organize who is
participating and who is not. Researchers are encouraged to keep separate
lists of the passwords used in the Internet-based survey results and the
tracking codes used in the postal mailing survey results.
Step Six: Follow-Up Survey Distribution
Two weeks after the initial
survey distribution, a follow-up e-mail request is sent to those participants
with working e-mail addresses. The follow-up e-mail request has the exact
content as the initial e-mail request. At this time, requests to participate
in the study are also sent via postal mailing to those who have not
participated and do not have a working e-mail address. In the postal mailing
reminder, a postcard was sent reminding them of the study and encouraging them
to either send in their paper-pencil copy of the study or complete the
instrument on-line (the participant’s individual password is also provided
on the postcard).
Step
Seven: Organization and Data Entry
The last step in conducting a
study utilizing Internet-based surveys and postal mailing surveys is
organizing your surveys and entering data. It is recommended that the
researcher make copies of all surveys submitted via e-mail to ensure proper
record keeping in times of computer problems. In addition, if the researcher
is utilizing a computer statistical package to analyze the data, it is
recommended that data form from the Internet-based surveys are directly cut
and pasted into the statistical software (some Internet-based surveys can be
imported directly into statistical software packages); this prevents typing
errors upon data entry. Lastly, researchers need to keep track of what results
were submitted via Internet on-line surveys and via postal mailings.
Statistical analysis needs to be conducted on the differences and similarities
between the two forms of the instrument.
Summary
There are many discrepancies
of the feasibility of using Internet-based survey methodology. However, many
organizations (CACREP and ACES) are requiring counselors to utilize
technology. Therefore, this article detailed specific guidelines on conducting
a study utilizing both Internet-based and postal mailing survey methodologies
simultaneously. The author has hopes that as time progresses, Internet-based
surveys will become more accepted due to further analysis of such methodology.
Association
for Counselor Education and Supervision Technology Interest Network. (1999).
Technical competencies for counselor education students: Recommended
guidelines for program development. Retrieved May 22, 2002, from
http://filebox.vt.edu/users/thohen/competencies.htm
Bachman,
D., Elfrink, & Vazzana, G. (1996). Tracking the progress of e-mail vs.
snail mail. Marketing Research, 8(2), 30-35.
Best,
S. J., Krueger, B., Hubbard, C., & Smith, A. (2001). An assessment of the
generalizability of Internet surveys. Social Sciences
Computer Review, 19 (2), 131-145.
Bosnjak,
M. & Tuten, T. L. (2001, April). Classifying response behaviors in
web-based surveys. Journal of Computer-Mediated Communication, 6 (3).
Retrieved March 11, 2002, from http://www.ascusc.org/jcmc/vol6/issue3/boznjak.html
Cook,
C., Heath, F., Thompson, R. L., & Thompson, B. (2001, Aug.). Score
reliability in web- or Internet-based surveys: Unnumbered graphic rating
scales versus likert-type scales. Educational and Psychological Measurement,
61 (4), 697-706.
Council
for Accreditation of Counseling and Related Educational Programs. (2001).
Accreditation standards and procedures manual. Alexandria, VA: American
Counseling Association.
Crawford,
S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of
burden. Social Sciences Computer Review, 19 (2), 146-162.
Epstein,
J., Klinkenberg, W. D., Wiley, D., & McKinley, L. (2001, May). Insuring
sample equivalence across Internet and paper-and-pencil assessments. Computers
in Human Behavior, 17 (3), 339-346.
Guterbock,
T. M., Meekins, B. J., Weaver, A. C., & Fries, J. C. (2000, May). Web
versus paper: A mode experiment in a survey of university computing. Paper
presented at the annual meeting of the American Association of Public Opinion
Research, Portland, OR.
Kiesler,
S., & Sproull, L. S. (1986). Response effects in the electronic survey.
Public Opinion Quarterly, 50, 402-413.
Kittleson,
M. J. (1995, Mar.-Apr.). Assessment of the response rate via the postal
service and e-mail. Health Values, 19 (2), 27-39.
Kwak,
N., & Radler, B.T. (2000, May). Using the Web for public opinion research:
A comparative analysis between data collected via mail and the Web. Paper
presented at the annual meeting of the American Association of Public Opinion
Research, Portland, OR.
McGlothlin,
J. (2001). Council for Accreditation of Counseling and Related Educational
Programs: An evaluation of the perceived benefit of core curriculum standards
to professional practice. Digital Dissertations International. (AAT 3022998).
Medin,
C., Roy, S., & Ham Chai, T. (1999). World Wide Web versus mail surveys: A
comparison and report. Paper presented at ANZMAC99 Conference: Marketing in
the Third Millennium, Sydney, Australia.
Mehta,
R., & Sivada, E. (1995). Comparing response rates and response content in
mail versus electronic mail surveys. Journal of the Market Research Society,
17(4), 429-440.
Mehta,
R., & Sivadas, E.. (1995, Oct.). Comparing response rates and response
content in mail versus electronic mail surveys. Journal of the market Research
Society, 37 (4), 429-439.
National
Board of Certified Counselors, Inc. (1997). Standards for the ethical practice
of webcounseling. Retrieved August 8, 2002, from http://www.nbcc.org/wcstandards.htm.
NUA
Internet Surveys. (2002). How many online? Retrieved August 8, 2002, from
http://www.nua.ie/surveys/how_many_online/index.html
Paolo,
A. M., Bonaminio, G. A., Gibson, C., Partridge, T., & Kallail, K. (2000).
Response rate comparisons of e-mail- and mail-distributed student evaluations.
Teaching and Learning in Medicine, 12 (2), 81-84.
Ranchhod,
A., & Zhou, F. (2000). Comparing respondents of e-mail and mail surveys:
Understanding the implications of technology. Marketing Intelligence and
Planning, 19 (4), 254-262.
Schaefer,
Dr. R., & Dillman, D. A. (1998). Development of a standard e-mail
methodology: Results of an experiment. Public Opinion Quarterly, 62(3),
378-397.
Sheehan,
K. (2001, Jan.). E-mail survey response rates: A review. Journal of
Computer-Mediated Communication, 6 (2). Retrieved March 11, 2002, from http://www.ascusc.org/jcmc/vol6/issue2/sheehan.html
Sheehan,
K. B. & Hoy, M. G. (2001, Jan.). Using E-mail to survey Internet users in
the United States: Methodology and assessment. Journal of Computer-Mediated
Communication, 6 (2). Retrieved March 11, 2002, from http://www.ascusc.org/jcmc/vol4/issue3/sheehan.html
Tse, A.,
Tse, K. C., Yin, C. H., Ying, C. B., Yi, K. W., Yee, K. P.,
& Hong, W. C. (1995). Comparing two methodologies of sending out
questionnaires: E-mail versus mail. Journal of
the Market Research Society, 17(4), 441-445.
Jason
McGlothlin is an Assistant Professor
of Counseling at Kent State University in Counseling
and Human Development Services. Dr. McGlothlin serves as the coordinator of the
Community Counseling and School Counseling programs. Contact Dr. McGlothlin at: jmcgloth@kent.edu
