Lecture 5: Constructing a Survey Questionnaire [PPTX]

hurriedtinkleAI and Robotics

Nov 15, 2013 (3 years and 6 months ago)

98 views

Survey Contents

Creating an Effective
Questionnaire

Survey Contents


When creating a questionnaire, remember:


1.
G
et it right at first

one cannot
change
questions
in mid
-
survey!


2.
The goal
is to
measure concepts as accurately as possible
(validity
and
reliability).


3.
R
espondents
will be

Lazy Thinkers.


Survey Contents


Always
consult other surveys first


They g
ive good
ideas
for measuring concepts


They may provide
valid ways to
ask questions


Avoids

reinventing the wheel




Always
pre
-
test
the
instrument


Survey Contents


When creating questions, consider the
effects of the following elements:


Type

Constraints, if any, placed on responses


Wording

Clarity of questions and response options


Order

Relative placement of
items
in the instrument


Topics

Issues the
items
cover



Question Types

Constraints, if any, placed on responses


Open
-
ended
avoid

A survey question to which the respondent replies in
his or her own word, either by writing or by talking



Closed
-
ended or fixed
-
choice
preferred

A survey question that provides preformatted response
choices for the respondent to circle, check, mark, etc.


Question Types

Fixed
-
choice



Closed
-
ended or fixed
-
choice examples
(not exhaustive):



T
wo choices:

Are you afraid to walk around your neighborhood at night?


___ Yes ___No



(Level: Dichotomous)


Question Types

Fixed
-
choice


Rating Scales,
Asking
respondents to rate
something (e.g., an
attitude, belief)
on a numbered scale,
often a
Likert

scale
(lick
-
urt
)



Likert

Item
, or “
Likert

Scale” (common
in
GSS):

The police in my community are mostly courteous.


1 2 3 4 5

Strongly Agree Neither Disagree Strongly


Agree A nor D Disagree



Semantic Differential



a
juvenile delinquent
is


Bad





Good

Awful


















Nice


(Level: Ordinal used as Interval
-
Ratio)

Question Types

Fixed
-
choice


Ranking Scale, Asking respondent
s
to put a set
of items
into
order

Place
the following
crimes
in order of
severity:


___Armed Robbery

___Home Invasion

___Stalking

(Level: ordinal)


List

In which type of community do you live? Pick one:

__Urban __Suburban __Rural __ Other (please specify)


(Level: nominal)


Please indicate whether you were the victim of a crime last year
(Mark all that apply)

__Assault __Auto Theft __Robbery __Other (specify)


(Level: series of dichotomous)

Question Types

Lazy Thinkers


Consider “neutral” or

no opinion


a valid option, but do not
allow if
fence sitters
will
likely hide their real opinions


Example:

Avoid

neutral for attitudes such as racial, sexual orientation, religious . . .



Questions
about past behaviors that ask for specific details
or rely on good
memory

are typically useless.


Example:

Avoid:
How many glasses of wine did you consume last month? _____

Use:
About how many glasses of wine do you drink in a typical week?

a. None b. 1 or 2 c. 3 to 6 d. 7 to 14 e. more than 14



Bad
memory leads to:


Forward
(backward) telescoping:
reporting
that events occurred
more recently
(further back) than
in
reality


Salient

events
being
over
-
reported


Mundane
events
being
underreported



Habitual


filling
in for lost information.




Question Types

Lazy Thinkers


Because of
agreement bias
, use negative
and

positive
statements, such as those in the self
-
esteem scale.



R

s
may tire or lose track in a series of questions and
choose salient options (such as first in series).


Example: Response set
problem, where R

s quickly
check the same

response
for all questions
.



Survey Contents


When creating questions, consider the
effects of the following elements:


Type

Constraints, if any, placed on responses


Wording

Clarity of questions and response options


Order

Relative placement of
items
in the instrument


Topics

Issues the
items
cover



Wording

Clarity of questions and response options



Respondents should understand
questionnaire items


Guidelines for good
phrasing:

1.
Be direct

2.
Maintain simplicity

3.
Be specific

4.
Take the role
of the
respondent

The
following points elaborate on these four main
themes.


Wording

Clarity of questions and response options

Some
things to do to make questions
less

u
nclear and
less
incomprehensible to

reduce inaccuracy:


Avoid
Double
-
barreled questions
. They contain two
questions in one.


Example
Avoid:


Do you think that students and
Professors should be given discounts on sports tickets?




Avoid
Double
-
negative questions
.


Example
Avoid:


Do you disagree that professors should
not be required to help students outside of class?



Wording

Clarity of questions and response options


Avoid
complex rhetoric, syntax, or disciplinary
jargon
.


Example
Avoid
:
Please rate your impressions of the
interactional sequences with your subordinate at the biennial
review.



Avoid teaching a new word or information.


Example
Avoid
:
In this questionnaire,
deflection

is the
discrepancy between fundamental cultural sentiments and
transient, situated impressions in the relevant semantic space
.



Sometimes, however, one must establish a particular
definition


Example
OK
: In this questionnaire,
Binge
Drinking
refers to
5 or more drinks in an hour for
men or 4 or more drinks in an
hour for women.


Wording

Clarity of questions and response options


Avoid ambiguity.


Example
Avoid
:


Do
you teach your children to effectively function
?



Do your customers
complain normally?




Avoid
biased questions:


E
motionally leading
terms
:


Example
Avoid
: Should political prisoners

who are
suffering inhumane conditions be rescued by American
forces?



Making a particular response more appealing:


Example
Avoid
: Given the success of three strikes laws,
do you believe that California should expand this law to
cover other crimes?


Wording

Clarity of questions and response options


Avoid biased questions (continued)


Avoid
unbalanced
response
options
.


Example
Avoid
:

Rate your professor’s friendliness?

Poor . . . Satisfactory . . . Quite Warm . . . Exceptional


Example From
the
GSS
Avoid
:


What
is your opinion about a married person having sexual
relations with someone other than the marriage partner

is it
always wrong, almost always wrong, wrong only some times,
or not wrong at all
?


Always wrong



Almost always wrong



Wrong only sometimes



Not wrong at all





Survey Contents


When creating questions, consider the
effects of the following elements:


Type

Constraints, if any, placed on responses


Wording

Clarity of questions and response options


Order

Relative placement of
items
in the instrument


Topics

Issues the
items
cover



Order


Relative
placement of items in the instrument

C
entral goal, to maintain respondents’ engagement.


Opening questions: simple, introduce survey topic.



More important
questions
come earlier, avoiding
“missing data” if R

s fatigue while taking survey.



Sensitive
questions
are in middle,
never

at
the
beginning.
Explain their necessity.



Ending questions: routine
,
demographic.



Avoid, but if necessary, put open
-
ended questions
toward the end.


Ever
notice
how much having to type an answer makes you want to
leave a consumer survey, and if required, ensures that?



Try not to mix
topics. So put
like
questions
into
sets.


Example: 10 item self
-
esteem scale is 10 contiguous items.



May
need
transition statements between
types of
questions, or may need transitory
questions or
directions.


Example: “In this next set we ask about personal
thoughts…”




Order


Relative placement of items in the instrument

Questionnaire Outline

1.
Introduction, topical questions

Transition

2.
Central research questions

1.
Set of related questions

Transition

2.
Set of related questions

Transition

3.
Sensitive questions, justification

Transition


4.
Routine, demographic questions

Order

Context Effects



Avoid
context effects
, framing subsequent questions with
ideas about topics potentially related to them
. Doing so may
invoke stereotypes, bias, or otherwise influence R’s thoughts.



Example
Avoid
:

1.
Those who use crack should be given longer prison terms.



_____Yes _____No

2.
The government should spend more money on improving the
lives of the urban poor. _____Yes _____No


Order

Fatigue Effects


Avoid fatigue effects.


Use only as many questions as necessary


Vary
question complexity or response options


Use
an appealing

format or
interface.


Do not ask R’s to navigate complex format questions
(e.g.,

skipping


or

go to next section

).


Paper
-
and
-
pencil imposes limits


Electronic questionnaires can, however, create seamless
navigation

Survey Contents


When creating questions, consider the
effects of the following elements:


Type

Constraints, if any, placed on responses


Wording

Clarity of questions and response options


Order

Relative placement of
items
in the instrument


Topics

Issues
the
items
cover



Topics

Issues the items cover



Make survey topic(s) clear
.



No surprises with
questions about unrelated issues
.


Example: The GSS explains that it is an “omnibus” survey and will
have a variety of questions



Consider Self Bias and Belonging.


People (Lazy Thinkers):


assert
a
particular vision
of themselves


are
the center of their
universe


view themselves as good


d
o not want to appear to not belong



Explain
that there are no right or wrong answers, only
honest or not. Honest answers will be helpful to others.




Topics

Issues the items
cover


Focus
on
factual
information, avoid impressions


Example:

Avoid:
Does your boss sexually harass you?

Better:

Has your boss ever told you a sexual joke? Has your boss
ever touched your body in a way that was too familiar? and so
forth



Remove social stigma when asking questions
about criminal or deviant behavior


I
nclude
normalizing
statements

Example
: “Many people use drugs for a variety of reasons.
Please indicate which of the following you have
you ever
used.”



Topics

Issues the items
cover



For personally sensitive questions:


P
rovide
a rationale for asking them.


Avoid
phrasing
that seems personal
or
directed at
respondent.


Example
Avoid
:

Do you abuse your kids?



Example
Better
:

Have you ever spanked or hit your child with
enough



force to leave a bruise or mark on the skin?




Do not open or close a survey with personal questions.

Example
Avoid
: Now the final question: Have you ever been so sad
that you attempted to commit suicide?

_____ Yes _____ NO

Survey Contents


When creating questions, consider the effects of
the following elements:


Type

Constraints, if any, placed on responses


Wording

Clarity of questions and response options


Order

Relative placement of
items
in the instrument


Topics

Issues
the
items
cover


THE END!