How the USGS uses the WQX standard

The USGS is expanding its implementation of the Water Quality Exchange (WQX) standard for delivering water quality and sample data.

Overview

• The Water Quality Exchange (WQX) standard is an internationally recognized data standard that makes it easier to share and use water quality data
• The USGS is expanding its current use of the WQX standard from a relatively coarse mapping for data delivered on the Water Quality Portal (WQP) to a comprehensive mapping of USGS sample data on both the WQP and the modernized USGS data portal, Water Data For the Nation (WDFN)
• Overall, USGS metadata will be more visible to users and it will be human-readable

What is WQX?

The WQX schema is a standardized data exchange format specifically for sharing water quality observational data. This data standard was developed by the Environmental Protection Agency (EPA) in conjunction with the US Geological Survey and other partner environmental organizations. The WQX schema provides a set of metadata attributes that are foundational for understanding ambient water quality observations. Users can define information about who collected samples and where, results and metadata about samples, and information that helps to contextualize results, such as habitat assessment scores and metrics.

WQX data uses the Extensible Markup Language (xml) format, which enables easy data sharing in a standardized format. While xml format is popular with developers of integrated software applications, it uses a structure that many users would find difficult to read and use in downstream analyses. We therefore convert the data into a tabular format, such as comma-separated values (csv). This requires us to convert some field names from the native WQX to ensure that the field names are unique and traceable back to the xml format of the schema. The example below shows example activity-level data in the native xml format and the converted csv format. Note that there are certain common field names that may be repeated in a sample record and therefore could cause problems upon conversion to the csv format. For example, the field “Time” appears up to 6 times for a single record: there are start and end times associated with the activity, start and end times for the sample preparation, and again for the sample analysis. Each of these pieces of information are referred to simply as “Time” in the xml format. To associate each value with a unique field name, we attach additional information from the WQX schema so that users can understand exactly where the data for a particular field originated. It can make the field names longer, but is essential for users to understand the data.

The WQX standard helps deliver rich and complex metadata for water quality and sample data following structured guidelines so that data providers anywhere can contribute in a way that makes it easy to combine with data from other providers. By using this standard, we at the USGS are advancing our efforts to make water quality and sample data more findable, accessible, interoperable and reusable (FAIR).

How is the USGS using WQX now?

USGS data have been served on the Water Quality Portal (WQP) since 2012 and users have been successfully accessing our data in this standardized format. Many users have noticed that USGS data weren’t fully mapped to metadata fields, however. Missing or unclear metadata has sometimes made it hard to compare data from USGS with data from other agencies. Our mapping of water quality observations and sample data to the WQX schema has needed a refresh, to make these USGS data more FAIR.

As part of larger data model changes, the USGS is redefining—and improving–how its manually collected water quality and other sample data map to the WQX schema. We anticipate that the new mapping will be available in early 2024. The change in mapping will allow for a more modern delivery of discrete data and supporting metadata. The improved mapping will allow USGS data to be more easily machine readable and interoperable, which among many benefits, will enable USGS sample data to be integrated into large-scale syntheses.

How will USGS sample data look in WQX format?

We’re moving away from codes

USGS historically used codes to convey what was measured as well as to describe various attributes of the measurement such as the methodology and data qualifiers. These codes link to descriptions that are packed full of important information; parameter codes (pcodes for short), for example, link to explicitly defined measured properties. However, these codes must be translated so that users can understand what each code represents. Furthermore, pcodes are very detailed and high-resolution, meaning that sometimes a user must search for many pcodes to fully discover all data that may be of interest. This can make it hard for people to search for the data they want and to interpret a downloaded dataset for downstream use.

We’re improving parameter mappings

One of the major changes users will find is that our many thousands of USGS parameters will be comprehensively mapped to the WQX standard. While we currently map much of our manually collected sample data, it was incomplete and inconsistent. When completed, we will be able to deliver more USGS sample data to the WQP than ever before. This mapping effort requires countless hours by many subject matter experts, and we are working with our EPA partners to ensure that mappings align with existing ‘domain lists’.

The WQX standard has many metadata fields that allow users to gain a deep understanding of the data. One great example is that it breaks up measurement information into its constituent parts. This makes it easier for users to optimize their data to make sure they are getting exactly the data they need. Currently, USGS delivers information about a measured property as part of the parameter description (parm_ds) for a pcode. In WQX, we report this information in multiple WQX fields.

Here’s an example of what one parameter will look like in WQX format:

Users will notice that we are taking advantage of a new field, CharacteristicUserSupplied, which is part of version 3.0 of the WQX schema, to deliver a modernized parameter description, which we call an Observed Property. Observed Properties essentially drop the units and reshuffle the content in the original parameter description, with the added benefit of putting the content into a standardized format and using more controlled text descriptors.

We’re standardizing more metadata

In addition to mapping more USGS parameters, we are also improving the way that we populate USGS metadata into the WQX format. Our descriptions for analytical methods are a great example. We are making it easier for users to know how a sample was analyzed and to determine whether a set of results is directly comparable with results obtained from a different method. To illustrate, let’s take a look at how methodological information will appear for a method measuring ammonia and organic nitrogen in water:

These updates will help the USGS achieve the goal of standardizing the organization and content of USGS measured properties so that they can be more easily understood by users and can be more easily combined with non-USGS datasets. Furthermore, the consistency across the USGS WDFN and WQP data portals will make it easier for users to download and use our data for downstream analyses.

What this means for data downloads

Once these changes are implemented, USGS water quality observations and sample data will be available from WDFN and WQP in the WQX data standard. We will stop delivering data in the current relational database (rdb) format found on NWISWeb.

Users who typically download water quality observational and sample data from NWISWeb will see a big change from the USGS-specific format to the WQX format: most notably, there will be numerous additional metadata fields, more descriptive and plain-language field names, and less reliance on codes that are unique to USGS. For WQP users, there will be a few subtle, but important, changes that we will highlight in a future blog post.

Changes that reflect users’ needs

Over the years, we’ve had users request that we change how we deliver various metadata elements to discrete sample results so they can better use our data. An example of this is providing reporting limits for laboratory data more clearly and comprehensively; this enables users to more easily handle non-detect data and understand measurement thresholds for a laboratory’s analytical capabilities.

Our new, robust WQX mapping will address many needs that users have brought up over years of using sample data. If you are a frequent user of USGS data and have been retrieving it from the Water Quality Portal or API services, you may notice a few changes to our data mapping. We hope these changes improve your workflows and are always interested in hearing your feedback. We will continue to improve this mapping over time and are committed to making sure that our data are able to be understood and consumed.

Let us know what you think!

If you would like to share your thoughts with us, please email wdfn@usgs.gov.

We often seek volunteers to help us test our products. If you are interested in getting involved, please contact us at wdfn_usabilitytesting@usgs.gov.

Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.