DAG Data Access Gateway

DAG principal goal is dispatch ODATA query to different catalogues, performing query translation and results retrieving.

It interrogates two different catalogue type:

• External ESA RSS catalogue via Opensearch OGC protocol query
• Internal catalogues via CSW protocol query

DAG config

DAG has two config

• dev for developing on local machine (set on default)
• prod for deploy on pre-prod machine. To test prod enviroment run mvn install with this parameter -Denv=prod

ODATA query to DAG

At the moment, DAG accepts specific queries in ODATA protocol at this endpoint http://localhost:8080/ctep-dag-client/DAG/Product?

Query filter is passed as $filter=.

ODATA query examples supported by DAG

For collection

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=Collection eq 'SENTINEL'

Once a product is retrieved you can perform a query for product

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=Product eq 'S1A_EW_GRDM'

For time period

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=BeginPosition eq datetime'2017-01-01T00:00:00'

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=EndPosition eq datetime'2017-01-01T00:00:00'

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=BeginPosition eq datetime'2017-01-01T00:00:00' and EndPosition eq datetime'2017-01-03T00:00:00'

For footprint

http://localhost:8080/ctep-dag-client/DAG/Product?$filter=Footprint eq '14.3,39.2,17.9,42.1'

Dag supports also WKT formats for POLYGON and MULTIPOLYGON.

Other kind of supported query filters are:

• Resolution eq 'resolution'
• SensorType eq 'sensor'
• BinningTime eq 'time'
• ParametersList eq 'parameters'

DAG architectures

ctep dag client

This layer offers a basic interface to interact with DAG server. It's just for develop and allow to perform all kind of ODATA queries.

ctep dag data

This layer is the core of downloading products. It implements three differents endpoints:

• /productdownload
• /gpoddownload
• /acridownload

Further details below.

ctep dag server

This layer dispatches ODATA query to respectives endpoints.

ctep dag utils

This layer translates, dispatches and produces/consumes all kind of data to different catalogues. Offer some utility to translate ODATA query to CSW and OPENSEARCH. Manages a connection to Redis for reactive_cache.

Products download

After a product has been retrieved DAG exposes an endpoint to download it

http://localhost:8080/ctep-dag-data/productdownload?

Obligatory filters for this endpoint are:

• token: this is generated from DAG and exposed in the link for the download
• userId: It represents the id of a user authenticated by UAM. This value must be appended from the client, because the DAG has no knowledge of the identity of services user.

Eg. http://localhost:8080/ctep-dag-data/productdownload?token=bd268f43aec28b55e625f250eda2cee9&userId=test

productdownloadis the central endpoint for download. It checks the presence of token and userId, interrogates UAM to receive permission for resource download and then dispatches the request of download to two different endpoints:

• gpoddownload that performs download from ESA GPOD service grid-eo-engine02.esrin.esa.int;
• acridownload that performs download from ftp.hermes.acri.fr that is ACRI internal ftp channel.

If download has success, items are saved in reactive_cache folder and a following request to productdownload check if the file is present in cache and then retrieves it, or perform a new download request.

Notes

updating submodules

When updating Acri gitlab repository, remember to update ctep-dockerized submodules. First, clone ctep-dockerized on your machine, then run:

git submodule update --remote dag

Then you can commit the successive change to remote project.

if you want to check pre-ops machine DAG status

You have to connect to pre-ops machine via ssh. Then run

sudo docker ps to get dag container name and sudo docker exec -it dag_name bash to work inside the container.