Bucket storage retention policy

[ssharma9125]

Hi Everyone,

I am using cortex for the long term storage of prometheus with the gcs (google cloud storage) bucket as the backend storage. I wanted to know how to configure the retention period for the metrics in the bucket. For my use case i only want to retain the metrics from the last 15 days and delete the data older than that from the bucket. Can i configure this in the cortex or do i need to apply the policy at the bucket level?

[yeya24]

@dosu

[dosubot[bot]]

Hey @yeya24, I'm here to help you out with any issues or questions you have. I'm a bot that can assist with bug fixes, answer your queries, and guide you on how to contribute. Let me know how I can assist you!

To configure a retention policy in Cortex to retain metrics for the last 15 days and delete older data, you should use the RetentionPeriod field in the runtime configuration. This allows you to set a per-tenant retention period. You can set the RetentionPeriod to 15 days for the specific tenant or globally if no tenant-specific configuration is needed. This is done through the runtime config, which is reloaded periodically.

Here's a brief outline of how you can configure it:

1. Use the overrides mechanism in the runtime configuration file to set the RetentionPeriod for the desired tenant(s).
2. Set the RetentionPeriod to 15 days (in seconds) for the tenant(s) you want to apply this policy to.

The retention logic will then automatically handle the deletion of blocks older than 15 days by comparing the retention value to the block maxTime.

This policy should be applied within Cortex using the runtime configuration, rather than at the Google Cloud Storage bucket level, as Cortex handles the retention logic internally ^[1].

To continue talking to Dosu, mention @dosu.

[yeya24]

@dosu what's the right config name for RetentionPeriod?

[dosubot[bot]]

The correct configuration name for setting the RetentionPeriod in Cortex is RetentionPeriod in the Limits struct ^[1].

To continue talking to Dosu, mention @dosu.

[yeya24]

Hey @ssharma9125, sorry for the spam. For your question, it is recommended to set up Cortex to apply retention via compactor instead of using GCS bucket level retention policy.

The config in compactor is called compactor_blocks_retention_period.

[rohan-bb]

hi @yeya24 can we limit the put and push-interval operations for s3 bucket, is there any configuration for limiting for size of bucket