Exploring Python: RTMA Analysis with Bokeh

Exploring Python: RTMA Analysis with Bokeh

This notebook takes a look at accessing some of the data available on UCAR’s THREDDS Server using Siphon.

Here we will be looking at the Real-Time Mesoscale Analysis (RTMA) dataset and how to plot it interactively in a notebook using holoviews and bokeh!

This will allow us to not only visualize the data onto a geographic map, but also interact with it through Bokeh. Let’s take a look at how this is done!

View the gist here! https://gist.github.com/Daviology38/38dd57b5a9957e73b12b567e2ecf0bc4

1
2
3
4
5
6
7
8
9
10
11
12
#Load in packages
from siphon.catalog import TDSCatalog, Dataset
import pandas as pd
import numpy as np
import metpy
import cartopy.crs as ccrs
import holoviews as hv
from holoviews import opts
import xarray as xr
import hvplot.xarray 
from geoviews import tile_sources as gvts
import panel as pn

Now that we have everything loaded, we need to set the backend for our holoviews (hv) plot. Typically we use ‘matplotlib’ as the backend, however in this case we will use ‘bokeh’ since we want an interactive plot. We will also set the output size of the plot to 250.

1
2
hv.extension('bokeh')
hv.output(size=250)
</img> </img>

Now we need to set an instance of the TDSCatalog from Unidata. We are looking at the RTMA data, which can be found under the Forecast Products and Analyses directory.

I will not be going into detail on Siphon here, however you can find some very helpful notebooks on using siphon to search for, parse, and access data thanks to the folks at Unidata here: https://unidata.github.io/pyaos-ams-2021/resources.html

1
2
3
#Get the list of current files on the THREDDS server
#Note that the TDSCatalog takes the url of the folder you want the files from as an argument
cat = TDSCatalog('https://thredds.ucar.edu/thredds/catalog/grib/NCEP/RTMA/CONUS_2p5km/latest.xml')
1
2
#Now let's look at the dataset that are available
cat.datasets
1
['RTMA_CONUS_2p5km_20210601_1500.grib2']

We are grabbing the latest dataset that is available for our plotting purposes. Now let’s access it using OPENDAP. We can access it using the use_xarray option in Siphon’s remote_access function, however it is currently unable to parse RTMA data successfully.

1
2
3
nc2 = cat.datasets[0].access_urls
nc3 = xr.open_dataset(nc2['OPENDAP']).metpy.parse_cf() # Parse with metpy to make it easier to plot
nc3 = nc3.metpy.assign_latitude_longitude() # Latitude and Longitude are not included, so have metpy add them

Let’s take a look at the variables that are available to us in the file:

1
list(nc3.keys())
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
['LambertConformal_Projection',
 'time1_bounds',
 'Dewpoint_temperature_error_height_above_ground',
 'Dewpoint_temperature_Analysis_height_above_ground',
 'Geopotential_height_Analysis_surface',
 'Pressure_error_surface',
 'Pressure_Analysis_surface',
 'Temperature_error_height_above_ground',
 'Temperature_Analysis_height_above_ground',
 'Total_cloud_cover_Analysis_entire_atmosphere_single_layer',
 'Total_cloud_cover_error_entire_atmosphere_single_layer',
 'Total_precipitation_Forecast_altitude_above_msl_1_Hour_Accumulation',
 'Visibility_Analysis_surface',
 'Visibility_error_surface',
 'Wind_direction_from_which_blowing_error_height_above_ground',
 'Wind_direction_from_which_blowing_Analysis_height_above_ground',
 'Wind_speed_error_height_above_ground',
 'Wind_speed_Analysis_height_above_ground',
 'Wind_speed_gust_error_height_above_ground',
 'Wind_speed_gust_Analysis_height_above_ground',
 'u-component_of_wind_Analysis_height_above_ground',
 'v-component_of_wind_Analysis_height_above_ground']

We have a lot of different Analysis available to us to plot on our map! Let’s look at the Temperature analysis

1
nc3['Temperature_Analysis_height_above_ground']
<xarray.DataArray 'Temperature_Analysis_height_above_ground' (time: 1, height_above_ground: 1, y: 1377, x: 2145)>
[2953665 values with dtype=float32]
Coordinates:
    reftime              datetime64[ns] 2021-06-01T15:00:00
  * x                    (x) float32 -2.763e+06 -2.761e+06 ... 2.682e+06
  * y                    (y) float32 -2.638e+05 -2.612e+05 ... 3.231e+06
  * time                 (time) datetime64[ns] 2021-06-01T15:00:00
  * height_above_ground  (height_above_ground) float32 2.0
    metpy_crs            object Projection: lambert_conformal_conic
    latitude             (y, x) float64 20.19 20.2 20.2 ... 50.12 50.11 50.11
    longitude            (y, x) float64 -121.6 -121.5 -121.5 ... -60.92 -60.89
Attributes:
    long_name:                      Temperature Analysis @ Specified height l...
    units:                          K
    abbreviation:                   TMP
    grid_mapping:                   LambertConformal_Projection
    Grib_Variable_Id:               VAR_0-0-0_L103
    Grib2_Parameter:                [0 0 0]
    Grib2_Parameter_Discipline:     Meteorological products
    Grib2_Parameter_Category:       Temperature
    Grib2_Parameter_Name:           Temperature
    Grib2_Level_Type:               103
    Grib2_Level_Desc:               Specified height level above ground
    Grib2_Generating_Process_Type:  Analysis

The temperature analysis is available in 4-dimensions (time, height_above_ground,y,x). Since we added in the latitude and longitude coordinates earlier, we will be able to substitute them in for the y and x dimensions.

Both the time and height_above_ground dimensions have 1 value, which makes our analysis even easier for the plotting program.

Under the attributes, note that the temperature is in kelvin, let’s change that over the celsius.

To do this, we need to find the indices of each variable that has temperature in it. Note that in the above variables temperature is found as both Temperature and temperature, so we will need to search for both using list comprehension.

1
2
3
4
5
6
inds = []
[inds.append(x) for x in nc3.variables if 'Temperature' in x or 'temperature' in x]
for ind in inds:
    nc3[ind] = nc3[ind]-273.15
    nc3[ind].attrs['units'] = 'C'
nc3['Temperature_Analysis_height_above_ground']
<xarray.DataArray 'Temperature_Analysis_height_above_ground' (time: 1, height_above_ground: 1, y: 1377, x: 2145)>
array([[[[19.059998 , 19.059998 , 18.899994 , ..., 27.179993 ,
          27.179993 , 27.170013 ],
         [18.899994 , 18.899994 , 18.899994 , ..., 27.179993 ,
          27.179993 , 27.179993 ],
         [18.899994 , 18.899994 , 18.899994 , ..., 27.179993 ,
          27.179993 , 27.179993 ],
         ...,
         [11.339996 , 11.339996 , 11.350006 , ...,  5.830017 ,
           5.830017 ,  5.830017 ],
         [11.339996 , 11.339996 , 11.350006 , ...,  6.330017 ,
           6.080017 ,  6.080017 ],
         [11.440002 , 11.350006 , 11.350006 , ...,  6.5899963,
           6.580017 ,  6.3399963]]]], dtype=float32)
Coordinates:
    reftime              datetime64[ns] 2021-06-01T15:00:00
  * x                    (x) float32 -2.763e+06 -2.761e+06 ... 2.682e+06
  * y                    (y) float32 -2.638e+05 -2.612e+05 ... 3.231e+06
  * time                 (time) datetime64[ns] 2021-06-01T15:00:00
  * height_above_ground  (height_above_ground) float32 2.0
    metpy_crs            object Projection: lambert_conformal_conic
    latitude             (y, x) float64 20.19 20.2 20.2 ... 50.12 50.11 50.11
    longitude            (y, x) float64 -121.6 -121.5 -121.5 ... -60.92 -60.89
Attributes:
    units:    C

Now that we have this in useful units, let’s plot the dataset in holoviews with bokeh!

To use holoviews with bokeh effectively, we need to declare some functions: one for plotting and one for selecting the variable. Holoviews/bokeh allows plots to be updated through the use of widgets which can correspond to one or more of the dimensions of the dataarray.

Plotting Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
def plot(var=None):
    # Choose the basemap from geoviews tile_source
    base_map = gvts.StamenTerrain
    
    # Create a hvplot quadmesh, call is similar to xarray.plot()
    # For each variable, we want all the values for each dimension, except for 
    #      height_above_ground since it can only take on one value
    # We want to plot the variable on a lat/lon grid, so we set x and y to lon/lat.
    # Rasterize = True helps to speed up the plotting/display
    #
    # For our example, we are only using one time step. If we had more than one time step, 
    #      the groupby call would make a holoviews widget next to the graph that would
    #      allow for the time to be changed and the plot would then update.
    mesh = nc3[var][:,0,:,:].hvplot.quadmesh(x='longitude', y='latitude',
                                rasterize=True, geo=True, title=' '.join(var.split('_')[0:2]),
                                crs=ccrs.PlateCarree(),
                                groupby=list(nc3[var].dims[0:1]), 
                                cmap='jet').opts(frame_width=200, 
                                frame_height=200, colorbar=True, axiswise=True)
    
    # Now we want to overlay the hvplot.quadmesh onto the basemap and include the wheel_zoom tool
    #     from Bokeh to allow the user to zoom in and out on the map. NOTE: There are many other
    #     Bokeh tools that can be used, see the Bokeh documentation for more info:
    #     https://docs.bokeh.org/en/latest/docs/user_guide/tools.html
    overlay = (base_map * mesh.opts(alpha=0.7)).opts(active_tools=['wheel_zoom'])
    
    # Return the holoviews plot
    return overlay

Variable Select Widget and Function

Now that we have the plotting function, let’s see how to call it.

To do this, we need to provide a variable to the plotting function. To get this variable we need to create a widget that displays the variables available in the dataset and a function that updates the plot when a new variable is selected

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Widget to select the variable. We start with a panel Select widget
# We provide the name for it to display above it (name), the options within it,
#     and the starting value when it loads. NOTE: We are taking variables after
#     the first 2 in the list since the first two describe the projection and 
#     time boundaries.
var_select = pn.widgets.Select(name='Variables:', options=list(nc3.keys())[2:], 
                               value='Temperature_Analysis_height_above_ground')

# Define a function to call the plotting function when a new variable is chosen
def on_var_select(event):
    var = event.obj.value
    dashboard[-1] = plot(var=var)

# Function to watch the var_select widget event and call the on_var_select function 
#     when it is updated.
var_select.param.watch(on_var_select, parameter_names=['value']);

Dashboard

Now that we have our plotting and variable select functions/widgets created, we can put them all together into a dashboard.

A holoviews dashboard is a clean way to layout the plots and widgets and connect their interactions to each other.

1
2
3
4
5
6
7
# Create the dashboard. We want them to be organized in a column, with the variable select
#     menu on the top and the plot below it

# Please note that using this in browser, changing variables may not work. Download this notebook and run it on your own
#     to experience the full effects!
dashboard = pn.Column(var_select, plot(var_select.value))
dashboard
1

comments powered by Disqus