Skip to Content

Notes on Design

Printer-friendly versionPrinter-friendly version
Issue: 
Spring 2011

James Conners (PAL, CCE)

In the last issue of Databits I wrote about the motivations for, and the details of, our current approach to data access within the data system architecture at our site. As described, this approach relies on a web service that accepts the registration of data queries against multiple storage backends. Queries are registered by clients using a standardized query syntax. Clients can then access status responses for the query being processed and data and metadata results are available for access upon successful completion. Currently, we have two applications interacting with this service, DataZoo and an application in use by an education outreach partner. More recently, we’ve had the opportunity to work with a data management group colocated at UCSD/SIO to coordinate the sharing of data through the use of this data access service. From this experience we have obtained valuable feedback for the improvement of our designs.

Realizing from the start that sharing access to our primary web service would risk unwanted increases in server loads and potential negative effects on general performance, we decided to implement an additional service tier that would provide access to a limited set of pre-configured data queries which would in turn be passed through to the underlying service. As developers from the partnering group began testing the API, however, it soon became apparent that additional steps for the mitigation of increased load would be necessary. A mechanism for re-using—within a specified time period—the results for identical queries was implemented. Additional feedback led to other improvements such as improved response headers and updates to general documentation.

In summary, this experience provided us the opportunity to test and improve weaknesses within our design, in addition to assessing its effectiveness. Feedback from collaborators outside our immediate group uncovered issues that wouldn’t have been identified by the limited architectural scope and set of use cases provided by a single data system or group of developers. From my experience, when circumstances allow for it, this type of interaction and feedback frequently provides valuable insight and perspective. Within the LTER network, efforts such as the Tiger Team concept provide a good example of a collaborative arrangement that creates a structure within the development timeline for supporting this type of interaction.