Improved Public Delivery of Water Quality and Field Samples Data

USGS is improving the way that sample data are stored and served to the public. These changes will affect our back-end data management systems, web services, application programming interfaces (APIs), computational tools, and user interfaces for accessing data.

What are sample data?

Sample data are derived from manual field collection and laboratory analyses, including for water quality, sediment, biological, air and soil samples (waterdata.usgs.gov/nwis/qwdata). Sample data do not include data collected by continuously-measuring instruments, data from models, or simulated data.

Why is this change happening?

Our current system for storing, processing, and delivering sample data needs a major overhaul in order to deliver reliable data and to keep up with new data types as they come online. A few years ago, we began a modernization project that improves many aspects of our sample data storage and delivery systems, from initial field data entry all the way to public data delivery. These changes will provide a stable foundation for our publicly available web interfaces, APIs, and computational tools for years to come.

What is changing?

Today, we serve sample data through NWISweb as text-delimited .rdb files in a USGS-specific data format at waterdata.usgs.gov/nwis/qwdata. Soon, we will serve sample data as comma separated files in the Water Quality Exchange (WQX) format, a standard data structure that is widely used by the water quality community. The WQX format provides a rich and complex set of metadata with many data fields. All data previously served will continue to be available in the WQX-format. This change will provide access to more robust metadata for discrete samples and remove the need for extensive knowledge of USGS data structures.

In addition, we are moving away from parameter names and their 5-digit parameter codes (pcodes) and towards observed properties (OPs) for defining measured properties or substances. For example, the parameter with pcode ‘53954’ would be represented as the OP:

Total anatoxin-a, water, unfiltered, recoverable, freeze/thaw extraction and then filtered (0.7 micron filter), enzyme-lined immunosorbent assay (ELISA)

We have a team of experts leading this complex process of mapping thousands of pcodes to OPs, which are more standardized and human-readable. Other legacy codes are also being mapped to human-readable text as part of the new mappings. We will provide users with a mapping of pcodes to OPs. We’ll dive into more detail about these mappings in a future blog post. Stay tuned!

USGS data delivery updates make sample data more findable, accessible, interoperable, and reusable (FAIR) to enable combining USGS data with other data sources. Our metadata will be more completely mapped to the WQX standard than before, allowing users to more easily understand sample data. Changes to USGS sample data format also better reflect the needs of users, and will make sample data easier to understand by replacing USGS-specific codes with human-readable text in a standardized format.

How will users get sample data in the future?

We will continue to provide multiple options for obtaining sample data. After rolling out the modernized data flow, USGS sample data will be available on:

• The Water Quality Portal web page and associated web services
• WDFN Monitoring Location Pages site-level summary table
• WDFN Consolidated Download feature (in development)
• USGS-supported dataretrieval code packages (R, Python, Julia)

When will changes happen?

We expect to release changes to sample data in the spring of 2024. Unlike some changes to USGS water data, sample data updates will happen all at once, meaning that every public data delivery system will switch to the new data flow simultaneously.

For a period of time after modernization a static copy of historical sample data in the legacy format will be available at waterdata.usgs.gov/nwis/qwdata.

Where can you learn more and get updates?

The unprecedented changes to how USGS presents sample data will impact how current users interact with and use the data. Our goal is to minimize the impacts as much as possible by preparing users with timely updates, guidance, and training materials. Please stay tuned for updates here on our Water Data For the Nation Blog. We will also notify users of upcoming changes on the Water Data for the Nation and Water Quality Portal websites. For users of one of our dataretrieval code packages, we will provide relevant updates to code packages in the documentation for each respective GitHub repository.

Also, we know that these are big changes for users! We expect that it will take some time to work out the kinks, and it will also take users time to adjust to the new data structure and delivery format. We appreciate your patience as we navigate these changes.

Follow our blog for updates: https://waterdata.usgs.gov/blog/

Reach out to the USGS WDFN team with questions or comments at: wdfn@usgs.gov

If interested in testing sample data please contact us at: wdfn_usabilitytesting@usgs.gov