Skip to content

File Transfer Azure to S3.md

Latest commit

 

History

History
100 lines (92 loc) · 3.92 KB

File Transfer Azure to S3.md

File metadata and controls

100 lines (92 loc) · 3.92 KB

File Transfer: Azure Blob Storage to Amazon S3

1. Description

This tutorial illustrates the process through which a provider participant can transfer a file stored in its Azure Blob Storage to Amazon S3 belonging to a consumer participant.
For this tutorial, we assume Alice as a provider participant and Bob as consumer participant.

  • We use a self-contained version of Amazon S3 named MinIO, and we deploy two instances, one each for Alice and Bob.
  • We also use a self-contained version of Azure Blob Storage named Azurite, and we deploy single Azurite instance for both connectors, due to a limitation in overriding endpoints during trasfer process.

2. Prerequisites

  • [File Transfer: Azure Blob Storage to Azure Blob Storage](./File Transfer Azure to Azure.md)
  • [File Transfer: Amazon S3 to Amazon S3](./File Transfer S3 to S3.md)

This tutorial is similar to existing tutorial [File Transfer: Azure Blob Storage to Azure Blob Storage](./File Transfer Azure to Azure.md). Only difference is in the initiate transfer process. So please follow existing tutorial to initiate negotiation.

3. Initiate Transfer

While initiating transfer, we need to provide a data destination, so in this case we will be providing an Amazon S3 destination.

curl --location 'http://localhost/bob/management/v2/transferprocesses' \
--header 'Content-Type: application/json' \
--header 'X-Api-Key: password' \
--data-raw '{
  "@context": {
    "odrl": "http://www.w3.org/ns/odrl/2/"
  },
  "assetId": "<Asset Id>",
  "connectorAddress": "http://alice-controlplane:8084/api/v1/dsp",
  "connectorId": "BPNL000000000001",
  "contractId": "<Contract Agreement Id from Get Negotiation Response>",
  "dataDestination": {
    "type": "AmazonS3",
    "keyName": "alice-test-document.txt",
    "bucketName": "bob-bucket",
    "region": "us-east-1",
    "endpointOverride": "http://bob-minio:9000",
    "accessKeyId": "bobawsclient",
    "secretAccessKey": "bobawssecret"
  },
  "protocol": "dataspace-protocol-http"
}'

This API should return below response.

{
  "@type": "IdResponse",
  "@id": "b227dc51-8108-44cb-bc62-dea7bea56482",
  "createdAt": 1700740670220,
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}

Please take note of the transfer process id (@id field) in the response.

4. Get Transfer

Just wait for few seconds and check the transfer state. Please replace transfer process id before running command.

curl --location 'http://localhost/bob/management/v2/transferprocesses/<TRANSFER_PROCESS_ID>' \
--header 'X-Api-Key: password'

It should return below response:

{
  "@id": "b227dc51-8108-44cb-bc62-dea7bea56482",
  "@type": "TransferProcess",
  "correlationId": "b227dc51-8108-44cb-bc62-dea7bea56482",
  "state": "COMPLETED",
  "stateTimestamp": 1704173831187,
  "type": "CONSUMER",
  "assetId": "30",
  "contractId": "e0b386a4-c46f-4d38-bbe0-6db2dfccf0e2",
  "callbackAddresses": [],
  "dataDestination": {
    "@type": "DataAddress",
    "secretAccessKey": "bobawssecret",
    "accessKeyId": "bobawsclient",
    "endpointOverride": "http://bob-minio:9000",
    "region": "us-east-1",
    "type": "AmazonS3",
    "keyName": "alice-test-document.txt",
    "bucketName": "bob-bucket"
  },
  "connectorId": "BPNL000000000001",
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "tx": "https://w3id.org/tractusx/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}

Please note, state should be COMPLETED. If it is TERMINATED, it means, transfer has failed. Please check logs for more details.

Once transfer has completed, the asset file should be available in bob-bucket.