Crowdsourcing quality control for Dark Energy Survey images

P. Melchior; E. Sheldon; A. Drlica-Wagner; E. S. Rykoff; T. M.C. Abbott; F. B. Abdalla; S. Allam; A. Benoit-Lévy; D. Brooks; E. Buckley-Geer; A. Carnero Rosell; M. Carrasco Kind; J. Carretero; M. Crocce; C. B. D'Andrea; L. N. da Costa; S. Desai; P. Doel; A. E. Evrard; D. A. Finley; B. Flaugher; J. Frieman; E. Gaztanaga; D. W. Gerdes; D. Gruen; R. A. Gruendl; K. Honscheid; D. J. James; M. Jarvis; K. Kuehn; T. S. Li; M. A.G. Maia; M. March; J. L. Marshall; B. Nord; R. Ogando; A. A. Plazas; A. K. Romer; E. Sanchez; V. Scarpine; I. Sevilla-Noarbe; R. C. Smith; M. Soares-Santos; E. Suchyta; M. E.C. Swanson; G. Tarle; V. Vikram; A. R. Walker; W. Wester; Y. Zhang

doi:10.1016/j.ascom.2016.04.003

Crowdsourcing quality control for Dark Energy Survey images

P. Melchior, E. Sheldon, A. Drlica-Wagner, E. S. Rykoff, T. M.C. Abbott, F. B. Abdalla, S. Allam, A. Benoit-Lévy, D. Brooks, E. Buckley-Geer, A. Carnero Rosell, M. Carrasco Kind, J. Carretero, M. Crocce, C. B. D'Andrea, L. N. da Costa, S. Desai, P. Doel, A. E. Evrard, D. A. FinleyB. Flaugher, J. Frieman, E. Gaztanaga, D. W. Gerdes, D. Gruen, R. A. Gruendl, K. Honscheid, D. J. James, M. Jarvis, K. Kuehn, T. S. Li, M. A.G. Maia, M. March, J. L. Marshall, B. Nord, R. Ogando, A. A. Plazas, A. K. Romer, E. Sanchez, V. Scarpine, I. Sevilla-Noarbe, R. C. Smith, M. Soares-Santos, E. Suchyta, M. E.C. Swanson, G. Tarle, V. Vikram, A. R. Walker, W. Wester, Y. Zhang

Workflow Systems

Research output: Contribution to journal › Article › peer-review

12 Scopus citations

Abstract

We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the "DES exposure checker", it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net.

Original language	English
Pages (from-to)	99-108
Number of pages	10
Journal	Astronomy and Computing
Volume	16
DOIs	https://doi.org/10.1016/j.ascom.2016.04.003
State	Published - Jul 1 2016

Funding

Funding for the DES Projects has been provided by the U.S. Department of Energy, the U.S. National Science Foundation, the Ministry of Science and Education of Spain, the Science and Technology Facilities Council of the United Kingdom, the Higher Education Funding Council for England, the National Center for Supercomputing Applications at the University of Illinois at Urbana/Champaign, the Kavli Institute of Cosmological Physics at the University of Chicago, the Center for Cosmology and Astro/Particle Physics at the Ohio State University, the Mitchell Institute for Fundamental Physics and Astronomy at Texas A&M University, Financiadora de Estudos e Projetos, Funda\u00E7\u00E3o Carlos Chagas Filho de Amparo \u00E0 Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de Desenvolvimento Cient\u00EDfico e Tecnol\u00F3gico and the Minist\u00E9rio da Ci\u00EAncia, Tecnologia e Inova\u00E7\u00E3o, the Deutsche Forschungsgemeinschaft and the Collaborating Institutions in the Dark Energy Survey. The DES data management system is supported by the National Science Foundation under Grant Number AST/1138766 . The DES participants from Spanish institutions are partially supported by MINECO under grants AYA2012/39559 , ESP2013/48274 , FPA2013/47986 , and Centro de Excelencia Severo Ochoa SEV/2012/0234 . Research leading to these results has received funding from the European Research Council under the European Union\u2019s Seventh Framework Programme (FP7/2007/2013) including ERC grant agreements 240672 , 291329 , and 306478 . PM is supported by the U.S. Department of Energy under Contract No. DE-FG02/91ER40690 . ES is supported by DOE grant DE-AC02/98CH10886 . This work was supported in part by the National Science Foundation under Grant No. PHYS/1066293 and the hospitality of the Aspen Center for Physics.

Keywords

Human-centered computing: Collaborative filtering
Information systems: Crowdsourcing
Surveys

Access to Document

10.1016/j.ascom.2016.04.003

Cite this

Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D'Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., ... Zhang, Y. (2016). Crowdsourcing quality control for Dark Energy Survey images. Astronomy and Computing, 16, 99-108. https://doi.org/10.1016/j.ascom.2016.04.003

@article{38c06526a86845958cda20bf3809f5bc,

title = "Crowdsourcing quality control for Dark Energy Survey images",

abstract = "We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the {"}DES exposure checker{"}, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net.",

keywords = "Human-centered computing: Collaborative filtering, Information systems: Crowdsourcing, Surveys",

author = "P. Melchior and E. Sheldon and A. Drlica-Wagner and Rykoff, \{E. S.\} and Abbott, \{T. M.C.\} and Abdalla, \{F. B.\} and S. Allam and A. Benoit-L{\'e}vy and D. Brooks and E. Buckley-Geer and \{Carnero Rosell\}, A. and \{Carrasco Kind\}, M. and J. Carretero and M. Crocce and D'Andrea, \{C. B.\} and \{da Costa\}, \{L. N.\} and S. Desai and P. Doel and Evrard, \{A. E.\} and Finley, \{D. A.\} and B. Flaugher and J. Frieman and E. Gaztanaga and Gerdes, \{D. W.\} and D. Gruen and Gruendl, \{R. A.\} and K. Honscheid and James, \{D. J.\} and M. Jarvis and K. Kuehn and Li, \{T. S.\} and Maia, \{M. A.G.\} and M. March and Marshall, \{J. L.\} and B. Nord and R. Ogando and Plazas, \{A. A.\} and Romer, \{A. K.\} and E. Sanchez and V. Scarpine and I. Sevilla-Noarbe and Smith, \{R. C.\} and M. Soares-Santos and E. Suchyta and Swanson, \{M. E.C.\} and G. Tarle and V. Vikram and Walker, \{A. R.\} and W. Wester and Y. Zhang",

note = "Publisher Copyright: {\textcopyright} 2016 Elsevier B.V.",

year = "2016",

month = jul,

day = "1",

doi = "10.1016/j.ascom.2016.04.003",

language = "English",

volume = "16",

pages = "99--108",

journal = "Astronomy and Computing",

issn = "2213-1337",

}

Melchior, P, Sheldon, E, Drlica-Wagner, A, Rykoff, ES, Abbott, TMC, Abdalla, FB, Allam, S, Benoit-Lévy, A, Brooks, D, Buckley-Geer, E, Carnero Rosell, A, Carrasco Kind, M, Carretero, J, Crocce, M, D'Andrea, CB, da Costa, LN, Desai, S, Doel, P, Evrard, AE, Finley, DA, Flaugher, B, Frieman, J, Gaztanaga, E, Gerdes, DW, Gruen, D, Gruendl, RA, Honscheid, K, James, DJ, Jarvis, M, Kuehn, K, Li, TS, Maia, MAG, March, M, Marshall, JL, Nord, B, Ogando, R, Plazas, AA, Romer, AK, Sanchez, E, Scarpine, V, Sevilla-Noarbe, I, Smith, RC, Soares-Santos, M, Suchyta, E, Swanson, MEC, Tarle, G, Vikram, V, Walker, AR, Wester, W & Zhang, Y 2016, 'Crowdsourcing quality control for Dark Energy Survey images', Astronomy and Computing, vol. 16, pp. 99-108. https://doi.org/10.1016/j.ascom.2016.04.003

TY - JOUR

T1 - Crowdsourcing quality control for Dark Energy Survey images

AU - Melchior, P.

AU - Sheldon, E.

AU - Drlica-Wagner, A.

AU - Rykoff, E. S.

AU - Abbott, T. M.C.

AU - Abdalla, F. B.

AU - Allam, S.

AU - Benoit-Lévy, A.

AU - Brooks, D.

AU - Buckley-Geer, E.

AU - Carnero Rosell, A.

AU - Carrasco Kind, M.

AU - Carretero, J.

AU - Crocce, M.

AU - D'Andrea, C. B.

AU - da Costa, L. N.

AU - Desai, S.

AU - Doel, P.

AU - Evrard, A. E.

AU - Finley, D. A.

AU - Flaugher, B.

AU - Frieman, J.

AU - Gaztanaga, E.

AU - Gerdes, D. W.

AU - Gruen, D.

AU - Gruendl, R. A.

AU - Honscheid, K.

AU - James, D. J.

AU - Jarvis, M.

AU - Kuehn, K.

AU - Li, T. S.

AU - Maia, M. A.G.

AU - March, M.

AU - Marshall, J. L.

AU - Nord, B.

AU - Ogando, R.

AU - Plazas, A. A.

AU - Romer, A. K.

AU - Sanchez, E.

AU - Scarpine, V.

AU - Sevilla-Noarbe, I.

AU - Smith, R. C.

AU - Soares-Santos, M.

AU - Suchyta, E.

AU - Swanson, M. E.C.

AU - Tarle, G.

AU - Vikram, V.

AU - Walker, A. R.

AU - Wester, W.

AU - Zhang, Y.

PY - 2016/7/1

Y1 - 2016/7/1

N2 - We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the "DES exposure checker", it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net.

AB - We have developed a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the "DES exposure checker", it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. We conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a sufficiently large user base. We also release open-source code of the web application and host an online demo version at http://des-exp-checker.pmelchior.net.

KW - Human-centered computing: Collaborative filtering

KW - Information systems: Crowdsourcing

KW - Surveys

UR - https://www.scopus.com/pages/publications/84971264239

U2 - 10.1016/j.ascom.2016.04.003

DO - 10.1016/j.ascom.2016.04.003

M3 - Article

AN - SCOPUS:84971264239

SN - 2213-1337

VL - 16

SP - 99

EP - 108

JO - Astronomy and Computing

JF - Astronomy and Computing

ER -

Crowdsourcing quality control for Dark Energy Survey images

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this