Vortex is importing some precip grids but not others

[cen4abc]

Hello,

I have been successfully using vortex (version 0.11.10-rc.1) to import ECWMF ERA5 reanalysis precip data (in both netcdf and grib format) and applying it in HECRAS rain on grid models with some nice results (big thanks to the vortex developers). I have also successfully used MRMS QPE data. However, vortex does not seem to want to import some other precip datasets that I would like to use in HECRAS, namely:

• It will not import this precip data from the UK’s national centre for atmospheric science (NCAS):

<xarray.Dataset>
Dimensions:            (ny: 1200, nx: 1500, pal01_colors: 256, pal_RGB: 3, pal02_colors: 256)
Coordinates:
    lat                (ny, nx) float32 ...
    lon                (ny, nx) float32 ...
  * ny                 (ny) float32 1.527e+06 1.524e+06 ... -2.067e+06 -2.07e+06
  * nx                 (nx) float32 3.961e+05 3.991e+05 ... 4.891e+06 4.894e+06
Dimensions without coordinates: pal01_colors, pal_RGB, pal02_colors
Data variables:
    crr                (ny, nx) float32 ...
    crr_intensity      (ny, nx) float32 ...
    crr_accum          (ny, nx) float32 ...
    crr_status_flag    (ny, nx) float32 ...
    crr_conditions     (ny, nx) float32 ...
    crr_quality        (ny, nx) float32 ...
    crr_pal            (pal01_colors, pal_RGB) uint8 ...
    crr_intensity_pal  (pal02_colors, pal_RGB) uint8 ...
Attributes: (12/46)
    Conventions:                  CF-1.6
    title:                        NWC GEO Convective Rainfall Rate Product
    history:                      2021-08-11T00:03:04Z ncas-sat Product Creat...
    institution:                  National Centre For Atmospheric Science
    source:                       NWC/GEO version v2018
    comment:                      Copyright 2021, EUMETSAT, All Rights Reserved
    ...                           ...
    product_quality:              100.0
    product_completeness:         100.0
    geospatial_lat_max:           15.47179
    geospatial_lat_min:           -21.608143
    geospatial_lon_max:           67.83071
    geospatial_lon_min:           3.561473

• it will not import this precip data from ECWMF’s HRES forecast model:

<xarray.Dataset>
Dimensions:              (isobaricInhPa: 9, latitude: 451, longitude: 900)
Coordinates:
    time                 datetime64[ns] ...
    step                 timedelta64[ns] ...
  * isobaricInhPa        (isobaricInhPa) float64 1e+03 925.0 ... 200.0 50.0
  * latitude             (latitude) float64 90.0 89.6 89.2 ... -89.2 -89.6 -90.0
  * longitude            (longitude) float64 -180.0 -179.6 ... 179.2 179.6
    valid_time           datetime64[ns] ...
    depthBelowLandLayer  float64 ...
    surface              float64 ...
    meanSea              float64 ...
    heightAboveGround    float64 ...
    entireAtmosphere     float64 ...
Data variables: (12/18)
    t                    (isobaricInhPa, latitude, longitude) float32 ...
    st                   (latitude, longitude) float32 ...
    v                    (isobaricInhPa, latitude, longitude) float32 ...
    u                    (isobaricInhPa, latitude, longitude) float32 ...
    tp                   (latitude, longitude) float32 ...
    gh                   (isobaricInhPa, latitude, longitude) float32 ...
    ...                   ...
    tcwv                 (latitude, longitude) float32 ...
    q                    (isobaricInhPa, latitude, longitude) float32 ...
    lsm                  (latitude, longitude) float32 ...
    d                    (isobaricInhPa, latitude, longitude) float32 ...
    vo                   (isobaricInhPa, latitude, longitude) float32 ...
    ro                   (latitude, longitude) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-11-23T17:50 GRIB to CDM+CF via cfgrib-0.9.1...

Vortex is either creating an empty output file or gets stuck during the import process. I am quite new to gridded climate datasets and cannot see anything that might be causing vortex an issue. Do you have any ideas as to why vortex is not handling these two datasets, and how you might process them to make them compatible? Below are screenshots of the parameters that are visible when first loading the datasets into vortex.

Many thanks,
Andy

ps. for info, this is the ERA5 reanalysis data that vortex will work with:

Dimensions:    (longitude: 5, latitude: 5, time: 35064)
Coordinates:
  * longitude  (longitude) float32 38.5 38.75 39.0 39.25 39.5
  * latitude   (latitude) float32 9.5 9.25 9.0 8.75 8.5
  * time       (time) datetime64[ns] 2017-01-01 ... 2020-12-31T23:00:00
Data variables:
    e          (time, latitude, longitude) float32 ...
    tp         (time, latitude, longitude) float32 ...
Attributes:
    Conventions:  CF-1.6
    history:      2022-01-21 20:00:46 GMT by grib_to_netcdf-2.23.0: /opt/ecmw...

[tombrauer]

@cen4abc if you post sample files I can take a look. Despite communal attempts at conventionality (climate and forecasting metadata conventions) no two data distributors ever format their files in the exact same way which means every file from a new source is an adventure!

[cen4abc]

@tombrauer that would be great, thank you. I have placed some sample files here: https://leeds365-my.sharepoint.com/:f:/g/personal/cen4abc_leeds_ac_uk/EhvnQ56Y3vdLtKOTR64rbwQB3ujM44kP799qrQyCnXsFGg?e=1sch7V

It might be better to start with the ECWMF files as these are more widely used / available than the NCAS data.

Thanks again!

[tombrauer]

I investigated the ECMWF file and it is hitting UnsupportedOperationException: Unsupported DRS type = 42. This looks like a known issue in the netcdf-java library that vortex depends on. See discussion here: Unidata/netcdf-java#753. So, this won't be fixed in vortex until the netcdf group implements a solution.

[cen4abc]

Thank you for getting to the botton of this. If you could look at the NCAS dataset at some point as well, that would be much appreciated.

[tombrauer]

@cen4abc I'm looking at the NCAS dataset and it does not appear to follow any convention. Vortex is expecting CF. Passing this dataset to Vortex is kind of like passing a Canadian bill to an ATM programed for U.S. legal tender. Does the originator of the dataset have an explanation for what follow variables mean:

[cen4abc]

@tombrauer let me talk to NCAS and get back to you. I can guess but not with certainty.

[cen4abc]

Hello again,

All variables are defined as follows (crr intensity is the one we are interested in):

crr - Convective Rainfall Rate Class
values from 0 to 11 relating to rain rate bands "00_02_mm_h 02_1_mm_h 1_2_mm_h 2_3_mm_h 3_5_mm_h 5_7_mm_h 7_10_mm_h 10_15_mm_h 15_20_mm_h 20_30_mm_h 30_50_mm_h 50____mm_h"

crr_accum- Convective Hourly Rainfall Accumulation
units = "mm"
valid range 0-50 mm

crr_conditions- Common geophysical and processing conditions
22 catagories that realte to the catagories: "space night day twilight sunglint land sea coast not_used not_used all_satellite_channels_available useful_satellite_channels_missing mandatory_satellite_channels_missing all_NWP_fields_available useful_NWP_fields_missing mandatory_NWP_fields_missing all_product_data_available useful_product_data_missing mandatory_product_data_missing all_auxiliary_data_available useful_auxiliary_data_missing mandatory_auxiliary_data_missing"

crr_intensity- convective precipitation rate
units = "mm/h"
valid range 0-50 mm/hr

crr_intensity_pal- RGB palette for crr_intensity
The standard colour palette for plotting the data.

crr_pal- RGB palette for crr
The standard colour palette for plotting the data.

crr_quality- Common Quality Indicators
7 possible values relating to the following catagories: "nodata internal_consistency temporal_consistency good questionable bad interpolated"

crr_status_flag- Information on specific NWC GEO CRR processing
14 possible values relating to the following catagories: "humidity_correction_applied evolution_correction_applied gradient_correction_applied parallax_correction_applied orographic_correction_applied solar_data_used lightning_data_used data_filtered data_hole_filled all_bands_for_accumulation_available one_band_for_accumulation_missing several_no_consecutive_bands_for_accumulation_missing several_consecutive_bands_for_accumulation_missing accumulation_quality_flag"

[tombrauer]

@cen4abc I just published a build that should read in the NCAS dataset:
https://github.com/HydrologicEngineeringCenter/Vortex/releases/download/v0.11.11-rc.2/vortex-0.11.11-rc.2-win-x64.zip

The originators of this dataset didn't do anyone any favors by following no particular convention with the dataset. The dataset uses an irregular grid (kind of like a spider web) that must be re-indexed into a cartesian grid. It takes some time to re-index the 1.8 million grid cells but the good news is that if you are importing multiple grids at a time, the re-indexing only has to be done once for the multiple grids.

The dataset does not include a time dimension so there is no conventional way to attribute time. I found a global attribute "nominal_product_time" that gives some notion of time. Inferring time from attributes isn't conventional but I added handling logic to read the attribute if it exists and apply it as both start and end time for the grid.

The variable names "crr_accum" and "NWC GEO CRR Convective Hourly Rainfall Accumulation" are non-standard but I added logic to recognize them as precipitation. Standard name table: https://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html

Because there is only a single time attribute to each grid record, you'll have to use the time-shifter to shift either the start date or the end date of the data to create period cumulative precipitation data. Based on the 3 grids provided I'm guessing you'll want to shift the start time by -15 minutes.

Feel free to provide this feedback to the originators of the data. I will quickly run out of time if I add handling logic like this for every non-conventional dataset that's out there. The burden should really be on the originator to clean up their data and make it more transferrable to others.

[cen4abc]

@tombrauer - very appreciative of this, thanks. I will raise this with the dataset provider!

[cen4abc]

@tombrauer I have just sent you a PM about this.