usability, design, interaction design, information, web usability, user interface design, usability engineering,user testing, usability inspection, heuristic evaluation, webdesign, UI, GUI, HCI, user-centered design,human-computer interaction, mensch computer interaktion,universität konstanz, usability test, benutzerfreundlichkeit, mensch-maschine-interaktion, mensch-computer-interaktion, human-computer-interaction, usable, website usability, ergonomie, ergonomisch, psychologie, gestalt principles, scenario, information visualization,informationsvisualisierung, User Interface Evaluation, Usability testing, Human Factors, Erogonomic, Ergonomics, User Interfaces, User Interface, Interfaces, informations visualisierung, visual information seeking system, visual search interface, Mensch & Computer 2012Harald Reiterer

Research

eLmuse establishing Longitudinal research methods in the usability evaluation toolbox

eLmuse establishing Longitudinal research methods in the usability evaluation toolbox (Flyer)

Status

Members

Gerken, Reiterer, Dierdorf, Leifert, König, Schmid, Gebhardt

Address

http://hci.uni-konstanz.de/eLmuse

Flyer

http://hci.uni-konstanz.de//elmuse/elmuse_30.04.08.pdf (Download as PDF)

Description

eLmuse - establishing Longitudinal research methods in the usability evaluation toolbox


In the field of Human-Computer Interaction and Information Visualization, the majority of the evaluation methods applied collect data at a single point in time, e.g. in controlled experiments. Although such experiments discover valuable findings, certain drawbacks of the research method have been discovered. For example it is questionable whether an appropriate amount of training can be given to control learning effects or whether usability flaws might disappear over time, since users adjust their strategies. Longitudinal research methods, collecting data at several points in time, try to accommodate for these drawbacks, but have rarely been applied in human-computer studies although they are commonly used in other partially related fields such as social sciences and psychology. This project aims at building up a research framework for such longitudinal research methods by defining research questions, measurements and data analysis methods. Therefore certain studies will be carried out in different domains such as input devices, interaction techniques and information visualizations. Such a comprehensive longitudinal research framework should serve as a decision help for researchers and practitioners to choose the appropriate and tailor-made methods and procedures when dealing with a specific research question. In the long-term we hope to establish longitudinal research methods as a common practice in the usability evaluation toolbox.


Within the social sciences, which have served quite often as the methodological basis for human-computer evaluation methods, collecting and analyzing longitudinal data has emerged during the last 30 years as an important and indispensable research approach. So for example, longitudinal data allows researchers to study how the perceived quality of human relationships with and without children changes over the years or in market research to study how people change their level of consumption of certain products.In research literature this is expressed by an increasing number of publications in this field [1, p. vii]. Longitudinal data could be defined as follows: "basically, longitudinal data present information about what happened to a set of research units [in our case the participants of a study] during a series of time points. In contrast, cross-sectional data refer to the situation at one particular point in time" [1, p.1]. This definition also shows that longitudinal data could be obtained with one single retrospective study, where participants are asked about e.g. their attitudes or behaviour at several distinct time points in the past. Furthermore, although it is quite common it is not necessary to obtain longitudinal data in the field. A set of laboratory based experiments, inviting the same participants again and again can have certain benefits compared to a cross-sectional study as well as compared to a longitudinal field study.


In Figure 1 we present a classification of evaluation methods by means of four different dimensions: type of measurement, type of task, place, and duration of the study. We think that, although this classification can’t cover all dimensions relevant for evaluation methods, it allows us to create a huge design space for longitudinal research methods and specifically points out that such methods aren’t limited to field studies with real tasks and qualitative measures.




Figure 1: Classification scheme for evaluation methods

Current studies:

In our current studies we apply different methods that can be used in longitudinal research, such as diary techniques, iterated lab experiments, logging analysis, interviews, and combinations of these. The goal is to not only apply and report the findings of these longitudinal methods but also point out certain drawbacks to find out whether applying longitudinal research methods is worth the effort. Detailed results will be reported in a short time.

Laserpointer Performance over time:

In this study we examined the influence of learning effects when using novel input devices - in our case a laserpointer. We conducted a series of lab experiments - each participant took part in 5 consecutive sessions during one week. Our main research objectives were:

  • Analyze whether learning effects can be accounted for during one single lab session.

  • Estimate the performance potential of laserpointer interaction.

  • Analyze ways to motivate participants to constantly try to improve themselves

HyperScatter vs. HyperGrid in the long run:

In this study we compared two visualizations of search results in a movie database in a longitudinal field study. Participants could use each of the visualizations for one week at home. Usage was observed directly via log files and indirectly via diaries and retrospective interviews at the end of each week. The main research objectives were:
  • Which of the two visualizations is better suited for which kind of task?

  • Is it possible to compare two visualizations in such a study design?

  • Is a duration of two weeks in total sufficiant to receive valuable results?

  • What parts of the method (interviews, diaries, log files) contribute most to the results?

MedioVis over the years:

Our MedioVis system is constantly used by in the library of the University of Konstanz. We will examine how usage has changed over the years by analyzing log files and conducting interviews with regular MedioVis users. This study is still running.

Learning a visual requirements specification tool - Inspector:

In this study we will further improve the method of combining logging with interviews and diaries. Our participants will work on a student assignment for 2-3 month, allowing us to analyze the scalability of the method. The study will take place in May-July 2008.

Literature:

[1] Toon W. Taris, „A Primer in longitudinal data analysis“, London: SAGE Publications, 2000.

Publications
& Theses

Display

sorted by

in

order



2011

The Concept Maps Method as a Tool to Evaluate the Usability of APIs  [main author webpage]  [Bibtex]

Gerken, Jens; Jetter, Hans-Christian; Zöllner, Martin; Mader, Michael; Reiterer, Harald

CHI'11: Proceedings of the 29th international conference on Human factors in computing systems, ACM Press, p. 3373-3382, May 2011

@inproceedings{Gerken20118609456,
author = {Jens Gerken and Hans-Christian Jetter and Martin Zöllner and Michael Mader and Harald Reiterer},
title = {The Concept Maps Method as a Tool to Evaluate the Usability of APIs},
booktitle = {CHI'11: Proceedings of the 29th international conference on Human factors in computing systems},
year = {2011},
type = {inproceedings},
pages = {3373-3382},
month = {May},
publisher = {ACM Press}

}
Download PDF Video

2010

Can "touch" get annoying?  [main author webpage]  [Bibtex]

Gerken, Jens; Jetter, Hans-Christian; Schmidt, Toni; Reiterer, Harald

Proceedings of ITS 2010: The ACM International Conference on Interactive Tabletops and Surfaces 2010, Poster Session, ACM Press, p. 257-258, Nov 2010

@inproceedings{Gerken20108609144,
author = {Jens Gerken and Hans-Christian Jetter and Toni Schmidt and Harald Reiterer},
title = {Can "touch" get annoying?},
booktitle = {Proceedings of ITS 2010: The ACM International Conference on Interactive Tabletops and Surfaces 2010, Poster Session},
year = {2010},
type = {inproceedings},
pages = {257-258},
month = {Nov},
publisher = {ACM Press}

}
Download PDF Download PowerPoint

Pocket Bee - a multi-modal diary for field research  [main author webpage]  [Bibtex]

Gerken, Jens; Dierdorf, Stefan; Schmid, Patric; Sautner, Alexandra; Reiterer, Harald

Proc. of nordiCHI 2010: Extending Boundaries, ACM, Oct 2010

@inproceedings{Gerken20108609445,
author = {Jens Gerken and Stefan Dierdorf and Patric Schmid and Alexandra Sautner and Harald Reiterer},
title = {Pocket Bee - a multi-modal diary for field research},
booktitle = {Proc. of nordiCHI 2010: Extending Boundaries},
year = {2010},
month = {Oct},
publisher = {ACM}

}
Download PDF

Dynamic Force Fields zur Präzisionserhöhung von Zeigegeräten  [main author webpage]  [Bibtex]

Specht, Marcus; Söter, Andrea; Gerken, Jens; Jetter, Hans-Christian; Bohrer, Lorenz; Reiterer, Harald

Mensch und Computer 2010 - Interaktive Kulturen, Oldenbourg Verlag, München, p. 27-36, Sep 2010

@inproceedings{Specht20108609140,
author = {Marcus Specht and Andrea Söter and Jens Gerken and Hans-Christian Jetter and Lorenz Bohrer and Harald Reiterer},
title = {Dynamic Force Fields zur Präzisionserhöhung von Zeigegeräten},
booktitle = {Mensch und Computer 2010 - Interaktive Kulturen},
year = {2010},
type = {inproceedings},
pages = {27-36},
month = {Sep},
publisher = {Oldenbourg Verlag}

}
Download PDF

Using Concept Maps to Evaluate the Usability of APIs  [main author webpage]  [Bibtex]

Gerken, Jens; Jetter, Hans-Christian; Reiterer, Harald

CHI 2010 Extended Abstracts on Human Factors in Computing Systems, Work-In-Progress Session, ACM Press, p. 3937-3942, Apr 2010

@inproceedings{Gerken20108609438,
author = {Jens Gerken and Hans-Christian Jetter and Harald Reiterer},
title = {Using Concept Maps to Evaluate the Usability of APIs},
booktitle = {CHI 2010 Extended Abstracts on Human Factors in Computing Systems, Work-In-Progress Session},
year = {2010},
type = {inproceedings},
pages = {3937-3942},
month = {Apr},
publisher = {ACM Press}

}
Download PDF

2009

Theorie und Praxis von Benutzerbefragungen, Benutzungsstudien und Benutzbarkeitstests in der Automobilindustrie  [main author webpage]  [Bibtex]

Söter, Andrea

Masters-Thesis, University of Konstanz, Konstanz, Germany, Dec 2009

@phdthesis{Söter20098609450,
author = {Andrea Söter},
title = {Theorie und Praxis von Benutzerbefragungen, Benutzungsstudien und Benutzbarkeitstests in der Automobilindustrie},
booktitle = {Masters-Thesis},
year = {2009},
type = {mastersthesis},
month = {Dec},
school = {University of Konstanz},
publisher = {University of Konstanz},
note = {Master Arbeit}

}
Download PDF

Eine Taxonomie für Längsschnittstudien in der MCI  [main author webpage]  [Bibtex]

Gerken, Jens; Reiterer, Harald

Mensch & Computer 2009: Grenzenlos frei?, 9. Konferenz für interaktive und kooperative Medien, Oldenbourg Verlag, Oldenbourg Verlag, München, Sep 2009

@inproceedings{Gerken20098609139,
author = {Jens Gerken and Harald Reiterer},
title = {Eine Taxonomie für Längsschnittstudien in der MCI},
booktitle = {Mensch & Computer 2009: Grenzenlos frei?, 9. Konferenz für interaktive und kooperative Medien, Oldenbourg Verlag},
year = {2009},
month = {Sep},
institution = {University of Konstanz},
publisher = {Oldenbourg Verlag, München}

}
Download PDF

Investigating longitudinal approaches for pointing device evaluation  [main author webpage]  [Bibtex]

Gerken, Jens; Bieg, Hans-Joachim; Reiterer, Harald

CHI 2009 Workshop - Best Practices in Longitudinal Research, Boston, USA, Apr 2009

@inproceedings{Gerken20098609390,
author = {Jens Gerken and Hans-Joachim Bieg and Harald Reiterer},
title = {Investigating longitudinal approaches for pointing device evaluation},
booktitle = {CHI 2009 Workshop - Best Practices in Longitudinal Research},
year = {2009},
type = {inproceedings},
month = {Apr}

}
Download PDF

Enhancing Input Device Evaluation: Longitudinal Approaches  [main author webpage]  [Bibtex]

Gerken, Jens; Bieg, Hans-Joachim; Dierdorf, Stefan; Reiterer, Harald

CHI 2009 Extended Abstracts on Human Factors in Computing Systems, Work-In-Progress Session, ACM Press, New York, NY, USA, Apr 2009

@inproceedings{Gerken20098609398,
author = {Jens Gerken and Hans-Joachim Bieg and Stefan Dierdorf and Harald Reiterer},
title = {Enhancing Input Device Evaluation: Longitudinal Approaches},
booktitle = {CHI 2009 Extended Abstracts on Human Factors in Computing Systems, Work-In-Progress Session},
year = {2009},
type = {inproceedings},
month = {Apr},
publisher = {ACM Press}

}
Download PDF Download PowerPoint

Novel Input Devices: Technophobia, Practice, and Acceptance  [Bibtex]

Rieger, Martin

Master-Thesis, University of Konstanz, Konstanz, Germany, Jan 2009

@mastersthesis{Rieger20098609405,
author = {Martin Rieger},
title = {Neuartige Eingabegeräte: Technophobie, Lerneffekte und Akzeptanz},
booktitle = {Master-Thesis},
year = {2009},
type = {mastersthesis},
month = {Jan},
school = {University of Konstanz},
publisher = {University of Konstanz},
note = {Master-Thesis}

}
Download PDF

2008

HyperScatter – Modellierungs- und Zoomtechniken für Punktdiagramme  [main author webpage]  [Bibtex]

Gerken, Jens; Demarmels, Mischa; Dierdorf, Stefan; Reiterer, Harald

Mensch & Computer 2008: Viel mehr Interaktion, 8. Konferenz für interaktive und kooperative Medien, Oldenbourg Verlag, in: Herczeg, Kindsmueller, Oldenbourg Verlag, München, Sep 2008

@inproceedings{Gerken20088609379,
author = {Jens Gerken and Mischa Demarmels and Stefan Dierdorf and Harald Reiterer},
title = {HyperScatter – Modellierungs- und Zoomtechniken für Punktdiagramme},
booktitle = {Mensch & Computer 2008: Viel mehr Interaktion, 8. Konferenz für interaktive und kooperative Medien, Oldenbourg Verlag},
year = {2008},
month = {Sep},
institution = {University of Konstanz},
publisher = {Oldenbourg Verlag, München}

}
Download PDF

How to use interaction logs effectively for usability evaluation  [main author webpage]  [Bibtex]

Gerken, Jens; Bak, Peter; Jetter, Hans-Christian; Klinkhammer, Daniel; Reiterer, Harald

BELIV'08: Beyond time and errors - novel evaLuation methods for Information Visualization (a CHI 2008 Workshop), Apr 2008

@inproceedings{Gerken20088609343,
author = {Jens Gerken and Peter Bak and Hans-Christian Jetter and Daniel Klinkhammer and Harald Reiterer},
title = {How to use interaction logs effectively for usability evaluation},
booktitle = {BELIV'08: Beyond time and errors - novel evaLuation methods for Information Visualization (a CHI 2008 Workshop)},
year = {2008},
type = {inproceedings},
month = {Apr},
institution = {Univeristy of Konstanz},
note = {Position Paper presented @ CHI 2008 Workshop BELIV'08}

}
Download PDF

2007

Longitudinal Evaluation Methods in Human-Computer Studies and Visual Analytics   [main author webpage]  [Bibtex]

Gerken, Jens; Bak, Peter; Reiterer, Harald

Metrics for the Evaluation of Visual Analytics (an InfoVIS 2007 Workshop), Oct 2007

@inproceedings{Gerken20078609344,
author = {Jens Gerken and Peter Bak and Harald Reiterer},
title = {Longitudinal Evaluation Methods in Human-Computer Studies and Visual Analytics },
booktitle = {Metrics for the Evaluation of Visual Analytics (an InfoVIS 2007 Workshop)},
year = {2007},
type = {inproceedings},
month = {Oct},
institution = {University of Konstanz},
note = {Position Paper presented @ InfoVis'07 Workshop}

}
Download PDF