eurofuels-back

A frontend for this project has been created: eurofuels-front

What's this project about

eurofuels is an improved version of eurocombustibles, being this the backend part.

Retrieve the fuel prices data from the EU Weekly Oil Bulletin, store them on an AWS S3 bucket and display it on a webpage.

For retrieving the data, an AWS Lambda function will be used, which will store the data (as a JSON file) on an AWS S3 bucket. The lambda will run periodically triggered by an AWS EventBridge (by CloudWatch) event.

How to set up the project

AWS S3 Bucket

1. Create an S3 Bucket on AWS
2. Change Permissions Policy for the following one (to grant public read permission)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

3. Give public access: disable the Block public access option
4. Change Object Ownership to Bucket Owner Preferred
5. Manage ACL (Access Control List): enable read to everyone
6. Add CORS policy (under bucket's permission tab). Without it, data cannot be accessed. It can be in JSON or in XML

[
  {
    "AllowedHeaders": ["Authorization"],
    "AllowedMethods": ["GET"],
    "AllowedOrigins": ["*"],
    "MaxAgeSeconds": 3000
  }
]

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

7. The AWS S3 Bucket is now set up. Optionally upload the JSON file.

To get the public URL of the file, go to the file and on the Properties tab there it is under Object URL.

AWS Lambda

The code in charge of retrieving the data from its source and saving it as a JSON file on the S3 Bucket.

1. Create a Lambda Function with the following params:

• Runtime: Python 3.13
• Architecture: x86_64
• Memory: 512MB
• Timeout: 60s

2. Add Pandas Layer AWSSDKPandas (name varies with version, serach for it)

3. Update the IAM Role. Go to Configuration -> Permissions -> Click on the role -> Add new inline role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

4. Add the following environment variables:

• BUCKET_NAME: the name given to the bucket when it was created
• FILE_NAME: the name of the JSON file to be saved on the bucket
• EXCEL_URL: the URL of the EU Weekly Oil Bulletin, in Excel format

5. On the code editor insert the lambda code and click Deploy button to deploy changes.

Everything should be ready now, go to test tab and test that it works okay. Go to the S3 Bucket and check that the last time modified was updated.

AWS CloudWatch and EventBridge

Here an event will be created that will trigger the execution of the Lambda Function.

1. Go to CloudWatch → Rules → Events and Create Rule
2. Give name, description and select Schedule as Event Source
3. It will redirect to EventBridge service, where the Schedule Pattern will be Recurring Schedule → Cron-based
4. Enter the desired cron expression, for example 0 30 8 ? * WED * (every Wednesday at 08:30)
5. In the Select targets section choose Lambda function as the target type and in the Function dropdown, select the Lambda function.
6. Select to not retry if fail.

Now the Lambda Function will run on the schedule.

Here you can find a useful CRON expression generator.

Useful links

• AWS S3 Buckets
• AWS Lambda Layers
• AWS Lambda Functions
• AWS EventBridge