Using the Client with the OpenET Raster API

Examples

One common need is issuing a raster export command and then waiting to proceed until the image is available and downloaded to the current machine. To do that:

Warning

The current approach will make all exported rasters public in order to be able to automatically download them - do not proceed to use this code if that isn’t acceptable.

import openet_client

# arguments are in the form of a dictionary with keys and
# values conforming to https://open-et.github.io/docs/build/html/ras_export.html
# In the future, geometry may accept OGR or GEOS objects and create the string itself
arguments = {
    'start_date': '2016-01-01',
    'end_date': '2016-03-20',
    'geometry': '-120.72612533471566,37.553211935016215,-120.72612533471566,37.474782294423676,-120.59703597924691,37.474782294423676,-120.59703597924691,37.553211935016215',
    'filename_suffix': 'client_test',
    'variable': 'et',
    'model': 'ensemble',
    'units': 'metric'
}

client = openet_client.OpenETClient("your_open_et_token_value_here")

# note that the path matches OpenET's raster export endpoint
client.raster.export(arguments, synchronous=True)  # synchronous says to wait for it to download before proceeding
print(client.raster.downloaded_raster_paths)  # get the paths to the downloaded rasters (will be a list, even for a single raster)

Batching it

You may also want to queue up multiple rasters, then wait to download them all. To do that, run the raster.export commands with synchronous=False (the default), then issue a call to wait_for_rasters

import openet_client

client = openet_client.OpenETClient("your_open_et_token_value_here")
arguments1 = {}  # some set of arguments, similar to the first example
arguments2 = {}  # same
client.raster.export(arguments1)
client.raster.export(arguments2)
client.raster.wait_for_rasters()  # this will keep running until all rasters are downloaded - it will wait up to a day by default, but that's configurable by providing a `max_time` argument in seconds
print(client.raster.downloaded_raster_paths)  # a list with all downloaded rasters
# or
rasters = client.raster.registry.values()  # get all the Raster objects including remote URLs and local paths

Doing work while you wait + manual control

You might also not want to wait around for the rasters to export, but still have control over the process. Here’s how to manually control the flow

import openet_client

client = openet_client.OpenETClient("your_open_et_token_value_here")
arguments = {}  # some set of arguments, similar to the first example
my_raster = client.raster.export(arguments)

# ... any other code you like here - the OpenET API will do its work and make the raster ready - or not, depending on your place in their queue ...

client.raster.check_statuses()  # check the API's all_files endpoint to see which rasters are ready
if my_raster.status == openet_client.raster.STATUS_AVAILABLE  # check that the raster we want is now ready
    client.raster.download_available_rasters()  # try to download the ones that are ready and not yet downloaded (from this session)

Raster API Class and Methods

class openet_client.raster.Raster(request_result)

Internal object for managing raster exports - tracks current status, the remote URL and the local file path once it exists. Users of this package shouldn’t need to instantiate this object directly in most cases.

download_file(retry_interval=20, max_wait=600)

Attempts to download a raster, assuming it’s ready for download. Will make multiple attempts over a few minutes because sometimes it takes a while for the permissions to propagate, so the first few responses may give a 403 error. We then have a timeout (max_wait) where if we exceed that value, we exit anyway without downloading.

Downloads the file to a tempfile path - the user may move the file after that if they wish.

Parameters:

  • retry_interval – time in seconds between repeated attempts

  • max_wait – How long, in seconds should we wait for the correct permissions before stopping attempts to download.

Returns:

class openet_client.raster.RasterManager(client)

The manager that becomes the .raster attribute on the OpenETClient object. Handles submitting raster export requests and polling for completed exports.

As constructed, could slow down if it handles many thousands of raster exports, or if the all_files OpenET endpoint displays lots of files as options.

Generally speaking, you won’t create this object yourself, but you can set client.raster.wait_interval to the length of time, in seconds, that the manager should wait between polling the all_files endpoint for new exports when waiting for new rasters.

property available_rasters

Which rasters have we marked as ready to download, but haven’t yet been retrieved?

Returns:

check_statuses(rasters=None)

Updates the status information on each raster only - does not attempt to download them.

Parameters:

rasters – the list of rasters to update the status of - if not provided, defaults to all rasters that are queued and not downloaded

Returns:

None

download_available_rasters()

Attempts to download all available rasters individually

Returns:

export(params=None, synchronous=False, public=True, transform=False)

Handles the raster/export endpoint for OpenET. Optionally waits for the raster to be exported and downloaded before proceeding. See documentation examples for usage details.

Parameters:

  • params – A dictionary of arguments with keys matching the raster/export endpoints parameters and values matching the requirements for the values of those keys. If the “geometry” key of this value is a GEOS or an OGR object with a .coords attribute, then the correct geometry string to send to the API will be composed for you. Otherwise, it is your responsibility to match the values with the API’s requirements.

  • synchronous – Whether or not to wait for the raster to export and be downloaded before exiting this function and proceeding

  • public – Whether or not to make the raster public - at this point, keeping this as True is required for all the features of this package to work, but if you just want to use the package to submit a bunch of raster jobs, but not to download those rasters, then you may set this to False.

  • transform – If a GEOS or OGR object with transform and coords attributes is passed in as part of params[“geography”], such as an object from GeoDjango, then you can optionally specify that this function handle a simple transformation to WGS84 (EPSG 4326) for you. For more complicated transformations or datum transformations, it may be best to handle the transformation before running this function. Setting it to False doesn’t control usage of a GEOS/OGR object is used, only if its coordinates are transformed first.

Returns:

Raster object - when synchronous, the local_file attribute will have the path to the downloaded raster on disk - otherwise it will have the status of the raster

property queued_rasters

Which rasters are we still waiting for?

Returns:

wait_for_rasters(uuid=None, max_time=86400)

When we want to just wait until the rasters are ready, we call this method, which polls the all_files endpoint at set intervals and checks which rasters are done.

Then updates the statuses on individual rasters and calls the download functions for the individual rasters and time some are ready to download, but won’t exit until all rasters are available and have had at least one download attempt.

It is recommended to use this after queueing a batch of rasters for running so that they may be exporting all at the same time before waiting - it will wait until all are exported before returning flow control to the calling function. Running this after each raster export will result in much longer runtimes (because exports will not run in parallel).

Parameters:

  • uuid –

  • max_time – Maximum time in seconds to wait for all rasters to complete - defaults to 86400 (a day)

Returns: