Ocean data restoration

Restoring missing coverages of HF Radar datastes.


Scroll Down
Input data
Restore data
Alpha shape radius (in Km)

Extend data restoration upto the coastline.
Refinement level to increase the resolution:

0%

User Guide


Browser compability

Your browser should support HTML5. You may visit html5test.com to check if your browser score higher than 470. We recommend Firefox, Chrome, Chrome for Android, Android Browser, Opera, Edge and Safari (only >=10.1). We do not recommend Safari (all <= 10.0) and any version of iOS Safari, Opera Mini and IE.


Network Connection

If your internet connection is behind a firewall, make sure that ports 3000 to 3010 are not blocked. Otherwise non of the features on this webpage will function even though you can visit the page. In such case you will receive a notification message indicating that a connection is not established to our computing server. Some, but not all public wifi connections such as eduroam block these ports, and you need to switch to other network connections.


Getting started

For a quick start, you do not need to provide an input dataset. Simply from the form above, find the drop-down menu on the right side of the Dataset URL entry. Click on the drop-down menu and select one of the sample datasets to auto fill the URL of data. Scroll down to the buttons and click on Restore, wait a few moments till the computation is finished, then click on either Visualize or Download. You may also watch a demo video below.


Input data format

The input dataset should be provided as netcdf file *.nc or netcdf markup language file *.ncml. The NetCDF file should contain at least the variables in the table shown below. The file must pass the CF-1.6 compliance test available at online CF-Convension Compliance chekcers. To identify variables in file, the variables should have either the standard_name or name attributes according the table below.

Variable Standard name Acceptable names Array dimension
Datetime time t,time,datetime 1-dimensional, [time]
Longitude longitude lon,long,longitude 1-dimensional, [lon]
Latitude latitude lat,latitude 1-dimensional, [lat]
East Velocity surface_eastward_sea_water_velocity
eastward_sea_water_velocity
east_vel,eastward_vel,u,
east_velocity,eastward_velocity
3-dimensional, [time,lat,lon]
North Velocity surface_northward_sea_water_velocity
northward_sea_water_velocity
north_vel,northward_vel,v,
north_velocity,northward_velocity
3-dimensional, [time,lat,lon]

Time variable: The time variable should have these additional attributes

lon/lat variables: These variables should be 1-dimensional to represent axes of a rectangular grid. The 2-dimensional grid point arrays are not supported.

Velocity variables: In locations where the velocity is unknown, the velocity variable should be masked. This can be specified with numpy.ma.mask in python or with fill_value in MATLAB. The fill_value should be specified as an attribute to each of the velocity variables. Also note that the order of writing variable indices (dimension) in python and MATLAB are opposite.


Input data URL

The input netcdf file should be provided with an OpenDap URL protocol. To provide your data with OpenDap protocol, you may host your dataset on any remote THREDDS server or you may request to host your data on our THREDDS server at http://transport.me.berkeley.edu/thredds.

Once you data are hosted on THREDDS server, find its OpenDap URL as follow:

  • Go to the catalog page of your data (see a sample data catalog).
  • From Access options select OpenDap (see OpenDap page of the sample data).
  • On the new window, use the URL in the DataURL text box.

A demo of finding the OpenDap URL is shown in the animated figure (click to play/stop).

Finding Opendap URL

Finding OpenDap URL from THREDDS catalog

We have provided two sample dataset below with compatible netcdf format.

The OpenDap URL of these data are also provided in the form as samples. Simply click on the drop-down menu on the right side of the input data text box to autofill the textbox with their URL.


Restore data

The set of grid points in the input file are divided into two groups of points where the velocity is known and unknown. Options in the form allows you to have control on the locations where the unknown velocity is best to be restored.

Detect domain: By selecting the convex hull the unknown points that are inside a convex hull around the points with known velocity is flagged to be restored. Alternatively, by selecting the concave hull, the unknown points inside a convex hull (or alpha shape) around the known locations are restored. In this case, a parameter (alpha shape radius) can adjust the compactness of the convex hull. For sporadic missing domains, we recommend to use concave hull.

Detect land: If the detect land option is selected, the part of the points inside the convex/concave hull that overlaps with the land domain is excluded. Two methods of detecting land is considered. The high accuracy option might take a long time and we do not recommend to use it. If the detecting land option is selected, you can also opt for extending the restoration domain to the coastline.

Refine grid: The result can be interpolated to refine the output grid by a factor (an integer).

You can start the restoration process on your data by clicking on Restore. The restoration is processed on each of the time frames on the input file independently. Once the process is finished, you may Visualize or Download the results as a netcdf file. To interpret the output netcdf file, you may see the tutorial below.

Note: Your output data is discarded after your browser session ends. Hence, by refreshing your browser you no longer will be able to download and visualize the output results, and should start a new process.


Visualization

For large output datasets (when number of observations as well as the grid size is large) a poor internet connection might not allow to establish a steady connection with the TerriaJS server. You can switch to 2D Map display if for lighter graphic.

Tutorial


The following are brief description of using the output variables in the NetCDF file. The dataset that will be used in the following examples can be accessed here. Please select your preferred scripting language.



Instructions for Python users


Installing NetCDF package

To read NetCDF files in python you should install netCDF4 package. You can install this package with either of the following package managers

  • Anaconda: If you are using Anaconda framework, install netCDF4 with:

    $ conda install -c conda-forge netcdf4
  • pip: If you are using python's pip package manager with PyPI package indexing, install netCDF4 with:

    $ sudo pip install netCDF4

Loading NetCDF file

A minimalistic example of loading the data and working with variables are as follow:

# Import the netCDF4 package
>>> import netCDF4

# Change the directory to where the data is downloaded
>>> cd Directory_Of_Data

# Create a netcdf object from the data
>>> nc = netCDF4.Dataset('MarthaVineyard.nc')

NetCDF Dimensions

The dataset is stored in Lagrangian representation, hence each tracer at a location and an observation time can be specified with two netCDF dimensions time and trajectory. For additional variables the extra dimensions dim and dir are used for indexing arrays and tensors.

Name Range Description
trajectory From 0 to N-1 where N is the number of all released tracers, including those outside of the data domain. The trajectory Id for each tracer. This index counts all tracers including masked tracers, i.e. those initiated outside the domain or leave the domain later.
time From 0 to n-1 where n is the number of time observations along trajectory. The time index for each variable during their observations along trajectories.
dim Either 0 or 1 (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 0 corresponds to x or longitude axis and index 1 corresponds to y or latitude axis.
dir Either 0 or 1 (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 0 corresponds to x or longitude axis and index 1 corresponds to y or latitude axis. This is similar to dim, except it is used for the second indices of tensors to indicate the direction rather than dimension.

You can print the list of dimensions in the netCDF object with:

# Print the list of all dimensions in the data
>>> nc.dimensions.keys()
odict_keys(['time', 'trajectory', 'dim', 'dir'])

Variables

The variables in the netCDF file are time, lon and lat. The user may opt for computing additional variables VelocityGradient and/or FTLE.

Name Indexing Unit Description
Time Time[time] days since 1970/01/01 T00:00:00 Time of observations along trajectory
Longitude lon[trajectory][time] degrees (east is positive) Longitudes for each trajectory during the observation times
Latitude lat[trajectory][time] degrees (north is positive) Latitudes for each trajectory during the observation times
Velocity gradient VG[trajectory][time][dim][dir] meter/second (Additional variable) Velocity gradient tensor along trajectories.
FTLE FTLE[trajectory][time][dim] 1/second (Additional variable) Finite time Lyapunov exponent from start of trajectory till the observation time. Note that this variable is evolving over time, not with a fixed integration time.

You may print the list of available variables via:

# List of all variables
>>> nc.variables.keys()
odict_keys(['time', 'lon', 'lat', 'Velocity_Gradient', 'FTLE'])

With above variable names, you can assign them to objects as

# Assigning nc objects to variables
>>> t = nc.variables['time']
>>> lon = nc.variables['lon']
>>> lat = nc.variables['lat']
>>> VG = nc.variables['VelocityGradient']
>>> FTLE = nc.variables['FTLE']

As an example, we get the location and velocity gradient of the 910th tracer at the third observation time:

# Specifying index of time and location, reminding Python arrays are zero indexed
>>> TimeIndex = 2
>>> TracerId = 909

# Printing variables
>>> t[TimeIndex]
16252.020066889632

>>> lon[TracerId,TimeIndex]
-70.666096571020375

>>> lat[TracerId,TimeIndex]
41.13184570059471

>>> VG[TracerId,TimeIndex]
array([[  7.31280004e-07,   1.08090440e-05],
      [  1.12415518e-05,   5.06882105e-06]])

>>> FTLE[TracerId,TimeIndex]
array([  1.89297108e-05,  -2.51548179e-06])])

Instructions for Matlab users


Loading NetCDF file

You can view the content of the NetCDF file with:

>> % Import the netCDF4 package
>> ncdisp('MarthaVineyard.nc')

>> % Get info of the file
>> info = ncinfo('MarthaVineyard.nc');

>> % Get dimensions info
>> info.Dimensions

>> % Get variables info
>> info.Variables

Above will print struct array of netCDF dimensions and variables. In the following we will describe each.


NetCDF Dimensions

The dataset is stored in Lagrangian representation, hence each tracer at a location and an observation time can be specified with two netCDF dimensions time and trajectory. For additional variables the extra dimensions dim and dir are used for indexing arrays and tensors.

Name Range Description
trajectory From 1 to N where N is the number of all released tracers, including those outside of the data domain. The trajectory Id for each tracer. This index counts all tracers including masked tracers, i.e. those initiated outside the domain or leave the domain later.
time From 1 to n where n is the number of time observations along trajectory. The time index for each variable during their observations along trajectories.
dim Either 1 or 2 (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 1 corresponds to x or longitude axis and index 2 corresponds to y or latitude axis.
dir Either 1 or 2 (Only for additional variables) Dimension index of a array-like variable in x,y plane. Index 1 corresponds to x or longitude axis and index 2 corresponds to y or latitude axis. This is similar to dim, except it is used for the second indices of tensors to indicate the direction rather than dimension.

Variables

The variables in the netCDF file are time, lon and lat. The user may opt for computing additional variables VelocityGradient and/or FTLE.

Name Indexing Unit Description
Time Time(time) days since 1970/01/01 T00:00:00 Time of observations along trajectory
Longitude lon(time,trajectory) degrees (east is positive) Longitudes for each trajectory during the observation times
Latitude lat(time,trajectory) degrees (north is positive) Latitudes for each trajectory during the observation times
Velocity gradient VG(dir,dim,time,trajectory) meter/second (Additional variable) Velocity gradient tensor along trajectories.
FTLE FTLE(dim,time,trajectory) 1/second (Additional variable) Finite time Lyapunov exponent from start of trajectory till the observation time. Note that this variable is evolving over time, not with a fixed integration time.

With above variable names, you can assign them to objects as

>> % Assigning nc objects to variables
>> t = ncread('MarthaVineyard.nc','time');
>> lon = ncread('MarthaVineyard.nc','lon');
>> lat = ncread('MarthaVineyard.nc','lat');
>> VG = ncread('MarthaVineyard.nc','VG');
>> FTLE = ncread('MarthaVineyard.nc','FTLE');

As an example, we get the location and velocity gradient of the 910th tracer at the third observation time:

>> % Specifying index of time and location
>> TimeIndex = 3
>> TracerId = 910

>> % Printing variables
>> t(TimeIndex)

ans =

1.6252e+04

>> lon(TimeIndex,TracerId)

ans =

-70.6661

>>> lat(TimeIndex,TracerId)

ans =

41.1318

>> VG(:,:,TimeIndex,TracerId)

ans =

   1.0e-04 *

    0.0073    0.1124
    0.1081    0.0507

>> FTLE(:,TimeIndex,TracerId)

ans =

   1.0e-04 *

    0.1893
    -0.0252

Note: Matlab loads netCDF arrays with reversed order of indices of Python. That is, for instance a variable that is written as vg[trajectory,time,dim,dir] in the netCDF file with Python, should be accessed with vg(dir,dim,time,trajectory) in Matlab.

Gallery


Comparison of original and restored datasets


Martha's Vineyard 2014

Martha's Vineyard, MA


Demonstration of ocean surface velocities at Martha's Vineyard, MA on July 2014. Catalog of original dataset can be found here. The restored dataset can also be found here.

Monterey Bay

Monterey Bay, CA


Demonstration of ocean surface velocities at Monterey Bay, CA on January 2017. Catalog of original dataset can be found here. The restored dataset can also be found here.

About


How RESTORE works?

Read about our method here (PDF).

How to cite

Ameli, S., & Shadden, S. C. (2019). A transport method for restoring incomplete ocean current measurements. Journal of Geophysical Research: Oceans, 124, 227– 242. https://doi.org/10.1029/2018JC014254


How to report bugs

Bug reports may be addressed to ude.yelekreb@ilemas. If an issue occurred during the computation, you may provide the followings within your email:


Session ID:

Acknowledgement

This material is based upon work supported by the National Science Foundation grant number No. 1520825.