Ocean data restoration
Restoring missing coverages of HF Radar datastes.
Your browser should support HTML5. You may visit html5test.com to check if your browser score higher than 470. We recommend Firefox, Chrome, Chrome for Android, Android Browser, Opera, Edge and Safari (only >=10.1). We do not recommend Safari (all <= 10.0) and any version of iOS Safari, Opera Mini and IE.
If your internet connection is behind a firewall, make sure that ports 3000 to 3010 are not blocked. Otherwise non of the features on this webpage will function even though you can visit the page. In such case you will receive a notification message indicating that a connection is not established to our computing server. Some, but not all public wifi connections such as eduroam block these ports, and you need to switch to other network connections.
For a quick start, you do not need to provide an input dataset. Simply from the form above, find the drop-down menu on the right side of the Dataset URL entry. Click on the drop-down menu and select one of the sample datasets to auto fill the URL of data. Scroll down to the buttons and click on Restore, wait a few moments till the computation is finished, then click on either Visualize or Download. You may also watch a demo video below.
The input dataset should be provided as netcdf file *.nc or netcdf markup language file *.ncml. The NetCDF file should contain at least the variables in the table shown below. The file must pass the CF-1.6 compliance test available at online CF-Convension Compliance chekcers. To identify variables in file, the variables should have either the standard_name or name attributes according the table below.
| Variable | Standard name | Acceptable names | Array dimension |
|---|---|---|---|
| Datetime | time |
t,time,datetime |
1-dimensional, [time] |
| Longitude | longitude |
lon,long,longitude |
1-dimensional, [lon] |
| Latitude | latitude |
lat,latitude |
1-dimensional, [lat] |
| East Velocity | surface_eastward_sea_water_velocity eastward_sea_water_velocity |
east_vel,eastward_vel,u, east_velocity,eastward_velocity |
3-dimensional, [time,lat,lon] |
| North Velocity | surface_northward_sea_water_velocity northward_sea_water_velocity |
north_vel,northward_vel,v,north_velocity,northward_velocity |
3-dimensional, [time,lat,lon] |
Time variable: The time variable should have these additional attributes
units, for example days since 1970-01-01 00:00:00 UTC.calendar, for example gregorian.lon/lat variables: These variables should be 1-dimensional to represent axes of a rectangular grid. The 2-dimensional grid point arrays are not supported.
Velocity variables: In locations where the velocity is unknown, the velocity variable should be masked. This can be specified with numpy.ma.mask in python or with fill_value in MATLAB. The fill_value should be specified as an attribute to each of the velocity variables. Also note that the order of writing variable indices (dimension) in python and MATLAB are opposite.
[time,lan,lon].(lon,lat,time).The input netcdf file should be provided with an OpenDap URL protocol. To provide your data with OpenDap protocol, you may host your dataset on any remote THREDDS server or you may request to host your data on our THREDDS server at http://transport.me.berkeley.edu/thredds.
Once you data are hosted on THREDDS server, find its OpenDap URL as follow:
A demo of finding the OpenDap URL is shown in the animated figure (click to play/stop).
Finding OpenDap URL from THREDDS catalog
We have provided two sample dataset below with compatible netcdf format.
http://transport.me.berkeley.edu/thredds/dodsC/root/WHOI-HFR/WHOI_HFR_2014_restored.nc
http://transport.me.berkeley.edu/thredds/dodsC/root/MontereyBay/MontereyBay_2km_restored.nc
The OpenDap URL of these data are also provided in the form as samples. Simply click on the drop-down menu on the right side of the input data text box to autofill the textbox with their URL.
The set of grid points in the input file are divided into two groups of points where the velocity is known and unknown. Options in the form allows you to have control on the locations where the unknown velocity is best to be restored.
Detect domain: By selecting the convex hull the unknown points that are inside a convex hull around the points with known velocity is flagged to be restored. Alternatively, by selecting the concave hull, the unknown points inside a convex hull (or alpha shape) around the known locations are restored. In this case, a parameter (alpha shape radius) can adjust the compactness of the convex hull. For sporadic missing domains, we recommend to use concave hull.
Detect land: If the detect land option is selected, the part of the points inside the convex/concave hull that overlaps with the land domain is excluded. Two methods of detecting land is considered. The high accuracy option might take a long time and we do not recommend to use it. If the detecting land option is selected, you can also opt for extending the restoration domain to the coastline.
Refine grid: The result can be interpolated to refine the output grid by a factor (an integer).
You can start the restoration process on your data by clicking on Restore. The restoration is processed on each of the time frames on the input file independently. Once the process is finished, you may Visualize or Download the results as a netcdf file. To interpret the output netcdf file, you may see the tutorial below.
Note: Your output data is discarded after your browser session ends. Hence, by refreshing your browser you no longer will be able to download and visualize the output results, and should start a new process.
For large output datasets (when number of observations as well as the grid size is large) a poor internet connection might not allow to establish a steady connection with the TerriaJS server. You can switch to 2D Map display if for lighter graphic.
The following are brief description of using the output variables in the NetCDF file. The dataset that will be used in the following examples can be accessed here. Please select your preferred scripting language.
To read NetCDF files in python you should install netCDF4 package. You can install this package with either of the following package managers
Anaconda: If you are using Anaconda framework, install netCDF4 with:
$ conda install -c conda-forge netcdf4
pip: If you are using python's pip package manager with PyPI package indexing, install netCDF4 with:
$ sudo pip install netCDF4
A minimalistic example of loading the data and working with variables are as follow:
# Import the netCDF4 package
>>> import netCDF4
# Change the directory to where the data is downloaded
>>> cd Directory_Of_Data
# Create a netcdf object from the data
>>> nc = netCDF4.Dataset('MarthaVineyard.nc')
The dataset is stored in Lagrangian representation, hence each tracer at a location and an observation time can be specified with two netCDF dimensions time and trajectory. For additional variables the extra dimensions dim and dir are used for indexing arrays and tensors.
| Name | Range | Description |
|---|---|---|
trajectory |
From 0 to N-1 where N is the number of all released tracers, including those outside of the data domain. | The trajectory Id for each tracer. This index counts all tracers including masked tracers, i.e. those initiated outside the domain or leave the domain later. |
time |
From 0 to n-1 where n is the number of time observations along trajectory. | The time index for each variable during their observations along trajectories. |
dim |
Either 0 or 1 | (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 0 corresponds to x or longitude axis and index 1 corresponds to y or latitude axis. |
dir |
Either 0 or 1 | (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 0 corresponds to x or longitude axis and index 1 corresponds to y or latitude axis. This is similar to dim, except it is used for the second indices of tensors to indicate the direction rather than dimension.
|
You can print the list of dimensions in the netCDF object with:
# Print the list of all dimensions in the data >>> nc.dimensions.keys() odict_keys(['time', 'trajectory', 'dim', 'dir'])
The variables in the netCDF file are time, lon and lat. The user may opt for computing additional variables VelocityGradient and/or FTLE.
| Name | Indexing | Unit | Description |
|---|---|---|---|
| Time | Time[time] |
days since 1970/01/01 T00:00:00 | Time of observations along trajectory |
| Longitude | lon[trajectory][time] |
degrees (east is positive) | Longitudes for each trajectory during the observation times |
| Latitude | lat[trajectory][time] |
degrees (north is positive) | Latitudes for each trajectory during the observation times |
| Velocity gradient | VG[trajectory][time][dim][dir] |
meter/second | (Additional variable) Velocity gradient tensor along trajectories. |
| FTLE | FTLE[trajectory][time][dim] |
1/second | (Additional variable) Finite time Lyapunov exponent from start of trajectory till the observation time. Note that this variable is evolving over time, not with a fixed integration time. |
You may print the list of available variables via:
# List of all variables >>> nc.variables.keys() odict_keys(['time', 'lon', 'lat', 'Velocity_Gradient', 'FTLE'])
With above variable names, you can assign them to objects as
# Assigning nc objects to variables >>> t = nc.variables['time'] >>> lon = nc.variables['lon'] >>> lat = nc.variables['lat'] >>> VG = nc.variables['VelocityGradient'] >>> FTLE = nc.variables['FTLE']
As an example, we get the location and velocity gradient of the 910th tracer at the third observation time:
# Specifying index of time and location, reminding Python arrays are zero indexed
>>> TimeIndex = 2
>>> TracerId = 909
# Printing variables
>>> t[TimeIndex]
16252.020066889632
>>> lon[TracerId,TimeIndex]
-70.666096571020375
>>> lat[TracerId,TimeIndex]
41.13184570059471
>>> VG[TracerId,TimeIndex]
array([[ 7.31280004e-07, 1.08090440e-05],
[ 1.12415518e-05, 5.06882105e-06]])
>>> FTLE[TracerId,TimeIndex]
array([ 1.89297108e-05, -2.51548179e-06])])
You can view the content of the NetCDF file with:
>> % Import the netCDF4 package
>> ncdisp('MarthaVineyard.nc')
>> % Get info of the file
>> info = ncinfo('MarthaVineyard.nc');
>> % Get dimensions info
>> info.Dimensions
>> % Get variables info
>> info.Variables
Above will print struct array of netCDF dimensions and variables. In the following we will describe each.
The dataset is stored in Lagrangian representation, hence each tracer at a location and an observation time can be specified with two netCDF dimensions time and trajectory. For additional variables the extra dimensions dim and dir are used for indexing arrays and tensors.
| Name | Range | Description |
|---|---|---|
trajectory |
From 1 to N where N is the number of all released tracers, including those outside of the data domain. | The trajectory Id for each tracer. This index counts all tracers including masked tracers, i.e. those initiated outside the domain or leave the domain later. |
time |
From 1 to n where n is the number of time observations along trajectory. | The time index for each variable during their observations along trajectories. |
dim |
Either 1 or 2 | (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 1 corresponds to x or longitude axis and index 2 corresponds to y or latitude axis. |
dir |
Either 1 or 2 | (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 1 corresponds to x or longitude axis and index 2 corresponds to y or latitude axis. This is similar to dim, except it is used for the second indices of tensors to indicate the direction rather than dimension.
|
The variables in the netCDF file are time, lon and lat. The user may opt for computing additional variables VelocityGradient and/or FTLE.
| Name | Indexing | Unit | Description |
|---|---|---|---|
| Time | Time(time) |
days since 1970/01/01 T00:00:00 | Time of observations along trajectory |
| Longitude | lon(time,trajectory) |
degrees (east is positive) | Longitudes for each trajectory during the observation times |
| Latitude | lat(time,trajectory) |
degrees (north is positive) | Latitudes for each trajectory during the observation times |
| Velocity gradient | VG(dir,dim,time,trajectory) |
meter/second | (Additional variable) Velocity gradient tensor along trajectories. |
| FTLE | FTLE(dim,time,trajectory) |
1/second | (Additional variable) Finite time Lyapunov exponent from start of trajectory till the observation time. Note that this variable is evolving over time, not with a fixed integration time. |
With above variable names, you can assign them to objects as
>> % Assigning nc objects to variables
>> t = ncread('MarthaVineyard.nc','time');
>> lon = ncread('MarthaVineyard.nc','lon');
>> lat = ncread('MarthaVineyard.nc','lat');
>> VG = ncread('MarthaVineyard.nc','VG');
>> FTLE = ncread('MarthaVineyard.nc','FTLE');
As an example, we get the location and velocity gradient of the 910th tracer at the third observation time:
>> % Specifying index of time and location
>> TimeIndex = 3
>> TracerId = 910
>> % Printing variables
>> t(TimeIndex)
ans =
1.6252e+04
>> lon(TimeIndex,TracerId)
ans =
-70.6661
>>> lat(TimeIndex,TracerId)
ans =
41.1318
>> VG(:,:,TimeIndex,TracerId)
ans =
1.0e-04 *
0.0073 0.1124
0.1081 0.0507
>> FTLE(:,TimeIndex,TracerId)
ans =
1.0e-04 *
0.1893
-0.0252
Note: Matlab loads netCDF arrays with reversed order of indices of Python. That is, for instance a variable that is written as vg[trajectory,time,dim,dir] in the netCDF file with Python, should be accessed with vg(dir,dim,time,trajectory) in Matlab.
Comparison of original and restored datasets
Ameli, S., & Shadden, S. C. (2019). A transport method for restoring incomplete ocean current measurements. Journal of Geophysical Research: Oceans, 124, 227– 242. https://doi.org/10.1029/2018JC014254
Bug reports may be addressed to ude.yelekreb@ilemas. If an issue occurred during the computation, you may provide the followings within your email:
FormData.json file. If this was not available, describe what settings you have entered.This material is based upon work supported by the National Science Foundation grant number No. 1520825.
Not connected to the server.