Abstract
Provenance tools can help capture and represent the history of analytic processes. In addition to supporting analytic performance, provenance tools can be used to support memory of the process and communication of the steps to others. Objective evaluation methods are needed to evaluate how well provenance tools support analysts' memory and communication of analytic processes. In this paper, we present several methods for the evaluation of process memory, and we discuss the advantages and limitations of each. We discuss methods for determining a baseline process for comparison, and we describe various methods that can be used to elicit memory of an analysis for evaluation. Additionally, we discuss methods for conducting quantitative and qualitative analyses of process memory. We discuss the methodology in the context of a case study in using the evaluation methods for a user study. By organizing possible memory evaluation methods and providing a meta-analysis of the potential benefits and drawbacks of different approaches, this paper can inform study design and encourage objective evaluation of process memory and communication.
Original language | English |
---|---|
Title of host publication | Proceedings - Beyond Time and Errors |
Subtitle of host publication | Novel Evaluation Methods for Visualization, BELIV 2014 - 5th Bi-Annual Workshop |
Editors | Petra Isenberg, Tobias Isenberg, Michael Sedlmair, Heidi Lam |
Publisher | Association for Computing Machinery |
Pages | 27-34 |
Number of pages | 8 |
ISBN (Electronic) | 9781450332095 |
DOIs | |
State | Published - Nov 10 2014 |
Event | 5th Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, BELIV 2014 - Paris, France Duration: Nov 10 2014 → … |
Publication series
Name | ACM International Conference Proceeding Series |
---|---|
Volume | 10-November-2015 |
Conference
Conference | 5th Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, BELIV 2014 |
---|---|
Country/Territory | France |
City | Paris |
Period | 11/10/14 → … |
Funding
This manuscript has been authored by UT-Battelle, LLC, under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.
Keywords
- Analytic provenance
- Evaluation
- Process memory
- Visual history