Skip to Content

LTER All-Site Bibliography 2002 - Update

Printer-friendly versionPrinter-friendly version
Issue: 
Fall 2002

- James W. Brunt and Troy Maddux, LTER Network Office (NET)

The first serious discussion of an LTER all-site bibliography (ASBIB) was at the Information Management Committee meeting in Toronto, Canada in 1989. At that time, the task of building a publication of the LTER site publication information was daunting to most site information managers because of the heterogeneity of site solutions to storage and management of bibliography data. Caroline Bledsoe, working then with the LTER Network Office in Seattle, proposed that IM's create the LTER All-Site Bibliography as a group project. After the waves of excitement for this new proposal calmed, Caroline Bledsoe and Harvey Chinn pursued implementation of this vision with funding from the LTER Network Office. The vision has evolved, as in punctuated equilibrium, with periods of relative inactivity after some major jumps. However, to-date, none of the implementations has been maintainable without a lot of work by site and network IMs. At present, we are standardizing on a proprietary end-user software package to take advantage of its web-posting solution. The problem is that neither this nor the previous solution has had the power of an open relational database management system behind them: a big drawback when attempting to provide any value-added components to this very useful data set. We are now at the point where storage and maintenance of an all-site bibliography is much more feasible; we can do it with minimal impact on the sites and in some cases actually provide a valuable service to the sites. But getting there has been a trial.

The Bledsoe/Chinn model:

Harvey Chinn programmed unique scripts in UNIX shell and awk programs to parse each of the (then) 18 LTER site bibliography formats. Each format was text which meant that some sites were already doing conversions from some other format to a unique text format - the first source of divergence. The scripts converted each of these unique text formats to a "standard" format that was a really a laundry list of attribute-value pairs from all the sites. This wasn't a bibliographic standard into which the entries were being placed but a loosely standardized list that included numerous ambiguities - a second source of divergence. Sites would submit their bibliographies to the central site via email or ftp and then the scripts for each site would be run to produce the "LTER" bibliographic format. All of the 18 common files were then indexed together with WAIS index - a pre-www information service product. The indexing produced a bibliography product that was searchable via gopher and later via the www. This system is documented in Bledsoe and Chinn (1997).

The Current Model:

The all-site bibliography is currently housed in an Endnote library file compiled from the all of the bibliography entries sent to the network office from all of the 24 sites. The decision was made to use Endnote as it was already being used by a majority of sites and it allowed for deployment of LTER references to the web in a relatively quick and painless manner. Sites that didn't have EndNote were provided with a copy by the LTER Network Office. Reference Web Poster (RWP) is the software package that handles the web display that can be found on the intranet page http://intranet.lternet.edu under All Site Bibliography. Someone processes Endnote files by hand and concatenates them into a master list which is pulled into RWP. The search capabilities in RWP are limited and you cannot query the library for accounting and statistical purposes. The only way to separate sites is by using one of the EndNote fields to encode the site information.

The Emerging Model:

Taking a strategic look at the needs for an all-site bibliography reveals a variety of potential and actual uses:

  • Accounting of publications (lingua franca) for the LTER Network Program
  • Facilitation of cross-site and synthetic studies
  • Generating new interest in LTER sites and LTER research

In addition, there is a need for some features useful to sites and other organizations, including,

  • Site/organization-specific searching and exporting
  • Turn-key bibliography for site information management systems (See Box 1)

Inherent in these goals is the challenge of maintaining integrity in the information and not forcing sites to do anything special with their data (e.g., encoding their site name in certain fields).

The emerging model for the all-site bibliography takes these needs and challenges into account by providing a variety of input and output channels organized around a flexible relational database infrastructure (Figure 1). A user can employ the web interfaces exclusively to manage and manipulate a site bibliography, can manage their bibliography in their own relational database management system (exposing their bibliography through Web Services), or can use Endnote - or other software producing an Endnote export file - in a variety of configurations to manage their bibliography.

Back End: The new model for the LTER All Site Bibliography is to house the data in a SQL Server database management system that reproduces Endnote format exactly; sites can still use Endnote as their desktop bibliography application.

Front End: The web interfaces to the bibliography database are being built for:

  1. Single entry of a reference
  2. Bulk upload of numerous references
  3. Updating an existing reference in the database
  4. Deleting a reference from the database

ASBIB Interfaces

The Single Entry Page. The script that generates this page will ask you to select the reference type (e.g. book, journal article) you are entering and then will dynamically generate the page with the fields associated with that reference type in EndNote. Next you will be asked to enter your LTER site from a pull down list. The script will check to see if the reference you are entering shares the same name, author, date, and site as an existing record in the database. You can change the record that is in the database (if you have permission); or you can leave the existing reference record and add another site association with that record so it will show up with your site's references; or you can discard the pending reference; or you can put the pending reference in the database as a new record.

The Bulk Upload Page. This script will allow you to select a properly formatted file from your local computer and upload it to the database. A report is returned to the screen with the number of records entered and number of duplicates. Duplicates are displayed on the screen and the questions mentioned above will have to be answered before they are processed. Eventually, this will also handle the configuration for dynamic database connections.

The Update/Delete Page. After proper authentication, this script will display information about records selected for update or deletion. If you are satisfied that you are looking at the right reference you can answer yes to proceed with the update or delete.

The Search/View Page. The script that generates this page is patterned loosely after the Reference Web Poster Interface but will make searching easier and more full-featured. You will enter a search string and you will be able to specify which fields you want to search; or you can choose your site from a pull down to get a quick list of all of your site's publications.

LTER All-Site Bibliography
Figure 1. Diagrammatic representation of the planned LTER All-Site Bibliography

Status: The new database is 95% complete - we're loading the current 16,000 entries and are testing various configurations. This is being done with a perl program that will become the central component of the on-demand bulk upload page system after the initial loading of the database. The perl program, "asbibloader", is complete but we are re-writing it to take advantage of some new perl features. Asbibloader contains the quality assurance component that checks to see if a record already exists in the database. We are tweaking this feature to make it more robust. In addition, it parses out authors into a separate table to allow for some faster search indexing and for links from LTER researchers to publications and vice-versa. We currently get authors in many different forms. One author to a line; all authors on one line separated by commas; all on one line with commas and an "and" for the last author; all on one line delimited by a semicolon; etc. And sometimes they are mixed within one set of references. Parsing these without losing data has been our most recent challenge. We are hoping that providing this feature back to the user will result in a gradual homogenization of method.

Several other features deserve mention. We've tested the cocoon connection to the CAP LTER bibliographic database and are currently mapping EML 2.0 to the database schema. We're mapping the Z39.50 bib.1 attribute set to the database and coding the web interfaces for the asbibloader and the individual entries. We would like to role this out in January if the force be with us.

References Cited

Harvey Chinn and Caroline Bledsoe. "Internet Access to Ecological Information: The U.S. LTER All-Site Bibliography Project". 1997 January . BioScience 47(1).

BOX 1: A Bibtime Story

Ingrid the information manager needs to keep track of the publications produced by her site but she wants to do the least amount of work possible to get the desired results. Her desired results are that the references need to be searchable from the web by site, author, year, or keywords. They need to be locally available in a form that Endnote can read so researchers can easily prepare publications for various publishing sources. They need to be available to the greater Ecological community.

What does Ingrid do? She puts all her references into EndNote export format and bulk uploads them to the LTER All Site Bibliography and she pulls them back out in EndNote export format. She compares this with what she started with. Any problems she encounters, she reports to the Network Office where the amazing people there make things all better. She keeps a copy locally in case those bozos at the Network Office…I mean just-in-case of emergencies…and every time she adds a new reference to the All Site Bibliography through the convenient web interface, she updates her local copy.

So Ingrid just has to keep the references reasonably current and she gets:

  1. The ability to search on any field in the database record
  2. An up to date EndNote export file whenever she or a researcher wants it
  3. The ability to search her site's references or any LTER site's publications or all LTER site publications
  4. Statistics on publications through time, by author, across sites