Skip to content

refactor sync-to-s3.sh per directory #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RobHooper merged 6 commits into from
May 14, 2026
Merged

refactor sync-to-s3.sh per directory #623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e9bd21d
refactor: sync per directory for added configuration
RobHooper May 12, 2026
4a1da63
feat: dynamic sync options
RobHooper May 13, 2026
d983d08
feat: pass args directly to aws cli
RobHooper May 13, 2026
56e2502
Update salt/aws/files/sync-to-s3.sh
RobHooper May 14, 2026
9effd1a
fix: remove no longer needed var
RobHooper May 14, 2026
b512ae3
style: use default
RobHooper May 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pillar/registry.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ sync:
/home/collect/scrapyd/dbs:
/home/collect/scrapyd/eggs:
/home/collect/scrapyd/jobs:
exclude: "*/requests.queue/*"
/home/collect/scrapyd/logs:

apache:
Expand Down
25 changes: 15 additions & 10 deletions salt/aws/files/sync-to-s3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# Backup multiple directories and upload to AWS S3
# Sync a single directory into S3.

set -euo pipefail

Expand All @@ -24,16 +24,21 @@ if [ ! -x "$AWS_CLI" ]; then
exit 3
fi

if [ -z "$SYNC_DIRECTORIES" ]; then
echo "Error: SYNC_DIRECTORIES isn't set or is empty"
if [ -z "$S3_SYNC_BUCKET" ]; then
echo "Error: S3_SYNC_BUCKET isn't set or is empty"
exit 4
fi

for DIRECTORY in "${SYNC_DIRECTORIES[@]}"; do
SAFENAME="${DIRECTORY/#\//}"
SAFENAME="${SAFENAME/%\//}"
if [ ! -e "$1" ]; then
echo "Error: DIRECTORY isn't set or doesn't exist."
exit 5
fi

DIRECTORY=$1
SAFENAME=${DIRECTORY/#\//}
SAFENAME="${SAFENAME/%\//}"
shift

set +e
$AWS_CLI s3 sync "$DIRECTORY" "s3://$S3_SYNC_BUCKET/$SAFENAME/" --only-show-errors --delete 2>&1 | grep -v "You did not provide the number of bytes specified by the Content-Length HTTP header"

@jpmckinney jpmckinney May 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not need this anymore?

2>&1 | grep -v "You did not provide the number of bytes specified by the Content-Length HTTP header"

@RobHooper RobHooper May 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may still want to ignore these alerts (files changing as the script runs).
I want to see how much the exclude functionality reduces these warnings first.

set -e
done
set +e
$AWS_CLI s3 sync --only-show-errors --delete "${@}" "$DIRECTORY" "s3://$S3_SYNC_BUCKET/$SAFENAME/"
set -e
Comment thread
jpmckinney marked this conversation as resolved.
16 changes: 5 additions & 11 deletions salt/aws/sync.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,10 @@ include:
file.managed:
- contents: |
MAILTO=root
15 03,15 * * * root /home/sysadmin-tools/bin/sync-to-s3.sh
{%- for directory, entry in pillar.sync.directories|items %}
{%- set minute = (loop.index0 * 5) % 60 %}
{{minute}} 03,15 * * * root /home/sysadmin-tools/bin/sync-to-s3.sh {{ directory }}
{%- for option, value in entry | default({}, true) | items %} --{{ option }} "{{ value }}"{% endfor %}
{%- endfor %}
- require:
- file: /home/sysadmin-tools/bin/sync-to-s3.sh

set SYNC_DIRECTORIES setting:
file.keyvalue:
- name: /home/sysadmin-tools/aws-settings.local
- key: SYNC_DIRECTORIES
- value: '( "{{ pillar.sync.directories|join('" "') }}" )'
- append_if_not_found: True
- require:
- file: /home/sysadmin-tools/bin
- sls: aws