eLmuse - establishing Longitudinal research methods in the usability evaluation toolbox

In the field of Human-Computer Interaction and Information Visualization, the majority of the evaluation methods applied collect data at a single point in time, e.g. in controlled experiments. Although such experiments discover valuable findings, certain drawbacks of the research method have been discovered. For example it is questionable whether an appropriate amount of training can be given to control learning effects or whether usability flaws might disappear over time, since users adjust their strategies. Longitudinal research methods, collecting data at several points in time, try to accommodate for these drawbacks, but have rarely been applied in human-computer studies although they are commonly used in other partially related fields such as social sciences and psychology. This project aims at building up a research framework for such longitudinal research methods by defining research questions, measurements and data analysis methods. Therefore certain studies will be carried out in different domains such as input devices, interaction techniques and information visualizations. Such a comprehensive longitudinal research framework should serve as a decision help for researchers and practitioners to choose the appropriate and tailor-made methods and procedures when dealing with a specific research question. In the long-term we hope to establish longitudinal research methods as a common practice in the usability evaluation toolbox.

Within the social sciences, which have served quite often as the methodological basis for human-computer evaluation methods, collecting and analyzing longitudinal data has emerged during the last 30 years as an important and indispensable research approach. So for example, longitudinal data allows researchers to study how the perceived quality of human relationships with and without children changes over the years or in market research to study how people change their level of consumption of certain products.In research literature this is expressed by an increasing number of publications in this field [1, p. vii]. Longitudinal data could be defined as follows: "basically, longitudinal data present information about what happened to a set of research units [in our case the participants of a study] during a series of time points. In contrast, cross-sectional data refer to the situation at one particular point in time" [1, p.1]. This definition also shows that longitudinal data could be obtained with one single retrospective study, where participants are asked about e.g. their attitudes or behaviour at several distinct time points in the past. Furthermore, although it is quite common it is not necessary to obtain longitudinal data in the field. A set of laboratory based experiments, inviting the same participants again and again can have certain benefits compared to a cross-sectional study as well as compared to a longitudinal field study.

In Figure 1 we present a classification of evaluation methods by means of four different dimensions: type of measurement, type of task, place, and duration of the study. We think that, although this classification can’t cover all dimensions relevant for evaluation methods, it allows us to create a huge design space for longitudinal research methods and specifically points out that such methods aren’t limited to field studies with real tasks and qualitative measures.

Figure 1: Classification scheme for evaluation methods

Current studies:

In our current studies we apply different methods that can be used in longitudinal research, such as diary techniques, iterated lab experiments, logging analysis, interviews, and combinations of these. The goal is to not only apply and report the findings of these longitudinal methods but also point out certain drawbacks to find out whether applying longitudinal research methods is worth the effort. Detailed results will be reported in a short time.

Laserpointer Performance over time:

In this study we examined the influence of learning effects when using novel input devices - in our case a laserpointer. We conducted a series of lab experiments - each participant took part in 5 consecutive sessions during one week. Our main research objectives were:

• Analyze whether learning effects can be accounted for during one single lab session.

• Estimate the performance potential of laserpointer interaction.

• Analyze ways to motivate participants to constantly try to improve themselves

HyperScatter vs. HyperGrid in the long run:

• Which of the two visualizations is better suited for which kind of task?

• Is it possible to compare two visualizations in such a study design?

• Is a duration of two weeks in total sufficiant to receive valuable results?

• What parts of the method (interviews, diaries, log files) contribute most to the results?

MedioVis over the years:

Our MedioVis system is constantly used by in the library of the University of Konstanz. We will examine how usage has changed over the years by analyzing log files and conducting interviews with regular MedioVis users. This study is still running.

Learning a visual requirements specification tool - Inspector:

In this study we will further improve the method of combining logging with interviews and diaries. Our participants will work on a student assignment for 2-3 month, allowing us to analyze the scalability of the method. The study will take place in May-July 2008.

Literature: