Improved Public Delivery of Water Quality and Field Samples Data
USGS is improving the way that sample data are stored and served to the public. These changes will affect our back-end data management systems, web services, application programming interfaces (APIs), computational tools, and user interfaces for accessing data.
What are sample data?
Sample data are derived from manual field collection and laboratory analyses, including for water quality, sediment, biological, air and soil samples (waterdata.usgs.gov/nwis/qwdata ). Sample data do not include data collected by continuously-measuring instruments, data from models, or simulated data.
Why is this change happening?
Our current system for storing, processing, and delivering sample data needs a major overhaul in order to deliver reliable data and to keep up with new data types as they come online. A few years ago, we began a modernization project that improves many aspects of our sample data storage and delivery systems, from initial field data entry all the way to public data delivery. These changes will provide a stable foundation for our publicly available web interfaces, APIs, and computational tools for years to come.
What is changing?
Today, we serve sample data through NWISweb as text-delimited .rdb files in a USGS-specific data format at waterdata.usgs.gov/nwis/qwdata . Soon, we will serve sample data as comma separated files in the Water Quality Exchange (WQX) format, a standard data structure that is widely used by the water quality community. The WQX format provides a rich and complex set of metadata with many data fields. All data previously served will continue to be available in the WQX-format. This change will provide access to more robust metadata for discrete samples and remove the need for extensive knowledge of USGS data structures.
In addition, we are moving away from parameter names and their 5-digit parameter codes (pcodes) and towards observed properties (OPs) for defining measured properties or substances. For example, the parameter with pcode ‘53954’ would be represented as the OP:
Total anatoxin-a, water, unfiltered, recoverable, freeze/thaw extraction and then filtered (0.7 micron filter), enzyme-lined immunosorbent assay (ELISA)
We have a team of experts leading this complex process of mapping thousands of pcodes to OPs, which are more standardized and human-readable. Other legacy codes are also being mapped to human-readable text as part of the new mappings. We will provide users with a mapping of pcodes to OPs. We’ll dive into more detail about these mappings in a future blog post. Stay tuned!
USGS data delivery updates make sample data more findable, accessible, interoperable, and reusable (FAIR ) to enable combining USGS data with other data sources. Our metadata will be more completely mapped to the WQX standard than before, allowing users to more easily understand sample data. Changes to USGS sample data format also better reflect the needs of users, and will make sample data easier to understand by replacing USGS-specific codes with human-readable text in a standardized format.
How will users get sample data in the future?
We will continue to provide multiple options for obtaining sample data. After rolling out the modernized data flow, USGS sample data will be available on:
- WDFN Download Discrete Sample Data page
- The Water Quality Portal web page and associated web services (in development)
- WDFN Monitoring Location Pages site-level summary table
- WDFN Consolidated Download feature (in development)
- USGS-supported dataretrieval code packages (R , Python , Julia )(in development)
When will changes happen?
We expect to release changes to sample data in the spring of 2024. Unlike some changes to USGS water data, sample data updates will happen all at once, meaning that every public data delivery system will switch to the new data flow simultaneously.
For a period of time after modernization a static copy of historical sample data in the legacy format will be available at waterdata.usgs.gov/nwis/qwdata .
Where can you learn more and get updates?
The unprecedented changes to how USGS presents sample data will impact how current users interact with and use the data. Our goal is to minimize the impacts as much as possible by preparing users with timely updates, guidance, and training materials. Please stay tuned for updates here on our Water Data For the Nation Blog. We will also notify users of upcoming changes on the Water Data for the Nation and Water Quality Portal websites. For users of one of our dataretrieval code packages, we will provide relevant updates to code packages in the documentation for each respective GitHub repository.
Also, we know that these are big changes for users! We expect that it will take some time to work out the kinks, and it will also take users time to adjust to the new data structure and delivery format. We appreciate your patience as we navigate these changes.
Follow our blog for updates: https://waterdata.usgs.gov/blog/
Reach out to the USGS WDFN team with questions or comments at: wdfn@usgs.gov
If interested in testing sample data please contact us at: wdfn_usabilitytesting@usgs.gov
Related Posts
How the USGS uses the WQX standard
December 27, 2023
Overview The Water Quality Exchange (WQX) standard is an internationally recognized data standard that makes it easier to share and use water quality data The USGS is expanding its current use of the WQX standard from a relatively coarse mapping for data delivered on the Water Quality Portal (WQP) to a comprehensive mapping of USGS sample data on both the WQP and the modernized USGS data portal, Water Data For the Nation (WDFN) Overall, USGS metadata will be more visible to users and it will be human-readable What is WQX?
A tool to aid in the transition from the legacy NWIS discrete data format to the WQX format
September 15, 2024
Overview Recently, the USGS modernized the format of its discrete water quality and sample data from a USGS-specific output format to the internationally recognized Water Quality Exchange (WQX) standard (for more background, read this blog post).
Accessing discrete water quality and sample data on WDFN
March 19, 2024
Introduction We are excited to share with you some updates to how USGS discrete (occurring as separate and distinct events) water quality and field sample (a.
WQX3.0 Data Now Available on the Water Quality Portal
May 20, 2024
The Water Quality Portal (WQP) is the largest publicly available water quality data warehouse in the United States. It integrates publicly available data from many providers and outputs those data in a common format, known as the Water Quality Exchange (WQX) standard.
Advanced Check Status Users
May 19, 2024
Here at WDFN, we employ user-centered design to build better products and reduce risk. This blog is one of a five-part series on users of WDFN: Are You a USGS Water Data User?