A scalable pipeline data processing framework using database and visualization techniques

Intelligent pipeline inspection gauges (PIGs) are inspection vehicles that move along within a gas (or oil) pipeline and acquire signals from their surrounding rings of sensors. By analyzing the signals captured by intelligent PIGs, we can detect pipeline defects, such as holes, curvatures and other potential causes of gas explosions. We notice that the size of collected data using a PIG is usually in GB range. Thus, analyzer software must handle such scalable data and provide various kinds of visualization tools so that analysts can easily detect any defects in the pipeline. In this paper, we propose a scalable pipeline data processing framework using database and visualization techniques. Specifically, we analyze requirements for our system, giving its overall architecture of our system. Second, we describe several important subsystems in our system: such as a scalable pipeline data store, integrated multiple visualization, and automatic summary report generator. Third, by performing experiments with GB-range real data, we show that our system is scalable to handle such large pipeline data. Experimental results show that our system outperforms a relational database management system (RDBMS) based repository by up to 31.9 times.