Skip to Content

Integrating Open Source Data Turbine with the GCE Data Toolbox for MATLAB

Printer-friendly versionPrinter-friendly version
Issue: 
Spring 2013

Corinna Gries (NTL), Wade Sheldon (CGE), Tony Fountain, Chad Sebranek (NTL), Matt Miller, and Sameer Tilak

North Temperate Lakes LTER's streaming sensor data are being used as one of three "science experiments" in a NSF Software Infrastructure for Sustained Innovation (SI2) project led by Tony Fountain (CalIT2, UCSD). A major focus of this collaborative project is software integration in complex science environments. This involves strategic software development, systems integration, and testing through demonstration projects (i.e., science experiments). Major requirements for the software developed by this project include performance, usability, interoperability, and cyber-security. In addition to NTL LTER, these software products will be integrated into production research infrastructures at Purdue University, the University of Connecticut, and the Monterey Bay Aquarium Research Institute to answer important science questions, including: (1) What is the impact of uncertainty in the design of civil infrastructure? (2) How sensitive are ocean systems to pH changes? (3) What is the variability of lake metabolic parameters such as gross primary productivity and respiration?

One goal of this collaboration is to make integrating the Open Source Data Turbine (OSDT) streaming data middleware with other environmental community software tools more robust and accessible to information managers. In the first project phase, the existing OSDT - MATLAB interface was improved by developing a toolkit (DTMatlabTK) of easy-to-use MATLAB functions for interacting with Data Turbine servers. Building on these improvements, code was developed to directly access data in OSDT using the GCE Data Toolbox for MATLAB (developed at Georgia Coastal Ecosystems LTER) to provide a robust, automated and configurable QA/QC environment for harvesting real-time sensor data. The GCE Data Toolbox was then used to transform data to be compatible with the CUAHSI Observations Data Model (ODM, see Resources section below for links), and insert processed OSDT data into an ODM database to support an end-to-end workflow from NTL data buoys to a CUAHSI Hydrologic Information Server (figure below).

Figure. Data harvesting workflow from the NTL data buoys to CUAHSI ODM.

The functionality developed during the first phase of the project was introduced in a recent training, titled "Software tools and strategies for managing sensor networks", held at the LTER Network Office in April 2013 (see Henshaw & Gries this issue and Resources section below).

Following this workshop we will implement the new functionality in a production level streaming sensor data management system at NTL. Our experiences will be documented in detailed user guides and tutorials. Additional developments will include creating OSDT software interfaces that are compliant with the Open Geospatial Consortium Sensor Web Enablement standards, and making improvements to the OSDT – Kepler workflow system interface.

Resources:

Home page for the training workshop "Software tools and strategies for managing sensor networks" http://im.lternet.edu/im_training/sensors_2013. Relevant presentations from that workshop: http://im.lternet.edu/node/1169 and http://im.lternet.edu/im_training/sensors_2013/gce_toolbox

NSF award information for this project: http://www.nsf.gov/awardsearch/showAward?AWD_ID=1148458&HistoricalAwards=false

Open Source Data Turbine: http://www.dataturbine.org/

GCE Data Toolbox for MATLAB: https://gce-svn.marsci.uga.edu/trac/GCE_Toolbox

Open Geospatial Consortium, Sensor Web Enablement: http://www.opengeospatial.org/projects/groups/sensorwebdwg

Kepler Project workflow system: https://kepler-project.org/

CUAHSI ODM: http://his.cuahsi.org/documents/ODM1.1DesignSpecifications.pdf