|
home
|
Twisting rating scales: Horizontal versus vertical visual analogue scales versus categorical scales in Web-based research
Frederik Funke & Ulf-Dietrich Reips
Paper presented at the 3rd conference of the
European Survey Research Association (ESRA)
June 29 - July 3, 2009 in Warsaw (PL)
Abstract
Changing the alignment or type of rating scales is a simple way of expanding the possibilities for designing questionnaires. Flexibility in design is appreciated in Web questionnaires, for example to be able to optimize questionnaires according to the respondent's screen format (especially in mobile Web surveys to avoid bias that occurs when respondents have to scroll to see all response options). But there is a danger that changes in response scales can impair data quality.
We manipulated the available rating scale in a 2 (alignment) x 2 (type of scale) factorial between-subjects design. Alignment was either horizontal or vertical. For type of scale we used discrete five-point categorical scales and continuous visual analogue scales (VASs). VASs were implemented as plain lines with 250 possible values and both ends verbally anchored. VASs are appreciated, e.g. in medical research, because small differences can be detected within- or between-subjects. However, they are rarely used in the social and behavioral sciences, most probably because the analysis of data from non-computerized VASs is burdensome and because of a considerable lack of methodological research.
Method and Results
392 students participated in a Web experiment with a questionnaire consisting of 40 items on the Big Five dimensions of personality. Neither type of scale nor alignment mattered regarding dropout and lurking (i.e. participants rating not a single item at all). Missing data occurred more often with 5-point scales than with VASs, χ2(2, N = 391) = 5.24, p < .05, alignment did not matter. Neither alignment nor scale type had a statistically significant effect on means of the five dimensions of personality.
Thanks to computerized data collection we were able collect two paradata: response time and frequency of modified answers. Neither type of scale nor alignment showed an influence on response time. With VASs ratings were modified more often than with 5-point scales, F(1, 390) = 49.83, p < .001, eta2 = .114. Alignment had no influence on frequency of modified answers. There were no interactions.
Five items (one from every dimension of personality) were repeated at the end of the questionnaire. We found test-retest reliability (Cronbach's alpha) to be a little higher with 5-point scales (M = .92) than with VASs (M = .90). Horizontal alignment led to higher scores (M = .92) than vertical alignment (M = .90). Consequently, horizontal 5-point scales led to the highest reliability scores (M = .93) and vertical VASs performed worse (M = .88).
Overall, alignment can apparently be used as a means of design with five-point categorical scales as well as with VASs without running the risk of seriously biasing data. However, it should be kept in mind that scales of short to medium length were used in this study. Further research with longer scales is needed.
|