Challenges in Building an End-to-End System for Acquisition, Management, and Integration of Diverse Data from Sensor Networks in Watersheds: Lessons from a Mountainous Community Observatory in East River, Colorado

The U.S. Department of Energy's Watershed Function Scientific Focus Area (SFA), centered in the East River, Colorado, generates diverse datasets including hydrological, geological, geochemical, geophysical, ecological, microbiological and remote sensing data. The project has deployed extensive field infrastructure involving hundreds of sensors that measure highly diverse phenomena (e.g. stream and groundwater hydrology, water quality, soil moisture, weather) across the watershed. Data from the sensor network are telemetered and automatically ingested into a queryable database. The data are subsequently quality checked, integrated with the United States Geological Survey's stream monitoring network using a custom data integration broker, and published to a portal with interactive visualizations. The resulting data products are used in a variety of scientific modeling and analytical efforts. This paper describes the SFA's end-to-end infrastructure and services that support the generation of integrated datasets from a watershed sensor network. The development and maintenance of this infrastructure, presents a suite of challenges from practical field logistics to complex data processing, which are addressed through various solutions. In particular, the SFA adopts a holistic view for data collection, assessment and integration, which dramatically improves the products generated, and enables a co-design approach wherein data collection is informed by model results and vice-versa.