You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _sources/configuration/overview.rst.txt
+43-1Lines changed: 43 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,45 @@ Config File Structure
30
30
A zrepl configuration file is divided in to two main sections: ``global`` and ``jobs``.
31
31
``global`` has sensible defaults. It is covered in :ref:`logging <logging>`, :ref:`monitoring <monitoring>` \&:ref:`miscellaneous <miscellaneous>`.
32
32
33
+
34
+
``conf.d``: including config files
35
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
36
+
37
+
It is possible to distribute zrepl configurations over multiple YAML configuration
38
+
files. This is achieved by using the `include` key which can only exist in the main
39
+
configuration file.
40
+
41
+
The list of included paths must point to either individual YAML files or directories.
42
+
If the path points to a directory then all YAML files with a `.yml` extention in the
43
+
directory will be included.
44
+
45
+
.. code-block:: yaml
46
+
47
+
# /etc/zrepl/zrep.yml
48
+
global: ...
49
+
include:
50
+
- ./jobs.d/
51
+
- /opt/some_job.yml
52
+
53
+
# /etc/zrepl/jobs.d/backup.yml
54
+
jobs:
55
+
- name: backup
56
+
type: push
57
+
- ...
58
+
59
+
# /opt/some_job.yml
60
+
jobs:
61
+
- name: another_job
62
+
...
63
+
- ...
64
+
65
+
66
+
67
+
The paths are treated as absolute when starting with `/`, otherwise
68
+
as a relative path to the main config file's parent directory.
69
+
70
+
Job names must be unique across all included configuration files.
71
+
33
72
.. _job-overview:
34
73
35
74
Jobs \& How They Work Together
@@ -130,7 +169,7 @@ The following high-level steps take place during replication and can be monitore
130
169
* Move the **replication cursor** bookmark on the sending side (see below).
131
170
* Move the **last-received-hold** on the receiving side (see below).
132
171
* Release the send-side step-holds.
133
-
172
+
134
173
The idea behind the execution order of replication steps is that if the sender snapshots all filesystems simultaneously at fixed intervals, the receiver will have all filesystems snapshotted at time ``T1`` before the first snapshot at ``T2 = T1 + $interval`` is replicated.
135
174
136
175
ZFS Background Knowledge
@@ -258,6 +297,9 @@ On your setup, ensure that
258
297
259
298
* all ``filesystems`` filter specifications are disjoint
260
299
* no ``root_fs`` is a prefix or equal to another ``root_fs``
300
+
301
+
* For ``sink`` jobs, consider all possible ``root_fs/${client_identity}``.
302
+
261
303
* no ``filesystems`` filter matches any ``root_fs``
Copy file name to clipboardExpand all lines: _sources/configuration/prune.rst.txt
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,6 +50,7 @@ Example Configuration:
50
50
regex: "^manual_.*"
51
51
52
52
.. DANGER::
53
+
53
54
You might have **existing snapshots** of filesystems affected by pruning which you want to keep, i.e. not be destroyed by zrepl.
54
55
Make sure to actually add the necessary ``regex`` keep rules on both sides, like with ``manual`` in the example above.
55
56
@@ -71,6 +72,14 @@ It only makes sense to specify this rule for the ``keep_sender``.
71
72
The reason is that, by definition, all snapshots on the receiver have already been replicated to there from the sender.
72
73
To determine whether a sender-side snapshot has already been replicated, zrepl uses the :ref:`replication cursor bookmark <replication-cursor-and-last-received-hold>` which corresponds to the most recent successfully replicated snapshot.
73
74
75
+
.. ATTENTION::
76
+
77
+
If you use ``not_replicated`` in your pruning rules, make sure to :ref:`monitor <monitoring>` replication.
78
+
If your replication gets stuck then ``not_replicated`` causes snapshots to pile uip on the sender.
79
+
ZFS and especially the ``zfs`` management command are known to degrade in performance with a lot of snapshots.
80
+
Such degradation impacts zrepl, any other scripts, and human ability to manage your zpool.
Copy file name to clipboardExpand all lines: _sources/configuration/snapshotting.rst.txt
+37-17Lines changed: 37 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,8 +25,9 @@ The following snapshotting types are supported:
25
25
26
26
The ``periodic`` and ``cron`` snapshotting types share some common options and behavior:
27
27
28
-
* **Naming:** The snapshot names are composed of a user-defined ``prefix`` followed by a UTC date formatted like ``20060102_150405_000``.
29
-
We use UTC because it will avoid name conflicts when switching time zones or between summer and winter time.
28
+
* **Naming:** The snapshot names are composed of a user-defined ``prefix`` followed by a UTC date formatted like ``20060102_150405_000`` by default.
29
+
We either use UTC or timestamp with timezone information because it will avoid name conflicts when switching time zones or between summer and winter time.
30
+
Note that if timestamp with time zone information is used, the "+" of the timezone (i.e. +02:00) is replaced by "_" (i.e. _02:00) to conform to allowed characters in ZFS snapshots.
30
31
* **Hooks:** You can configure hooks to run before or after zrepl takes the snapshots. See :ref:`below <job-snapshotting-hooks>` for details.
31
32
* **Push replication:** After creating all snapshots, the snapshotter will wake up the replication part of the job, if it's a ``push`` job.
32
33
Note that snapshotting is decoupled from replication, i.e., if it is down or takes too long, snapshots will still be taken.
@@ -65,6 +66,9 @@ The ``periodic`` and ``cron`` snapshotting types share some common options and b
65
66
# Timestamp format that is used as snapshot suffix.
66
67
# Can be any of "dense" (default), "human", "iso-8601", "unix-seconds" or a custom Go time format (see https://go.dev/src/time/format.go)
67
68
timestamp_format: dense
69
+
# Specifies in which time zone the snapshot suffix is generated, optional, defaults to UTC, used only if time zone information is part of timestamp_format.
70
+
# Can be "UTC" (default), "Local" (time zone information from the OS) or any of the IANA Time Zone names (see https://nodatime.org/TimeZones)
71
+
timestamp_location: UTC
68
72
hooks: ...
69
73
pruning: ...
70
74
@@ -97,27 +101,16 @@ The snapshotter uses the ``prefix`` to identify which snapshots it created.
97
101
# Timestamp format that is used as snapshot suffix.
98
102
# Can be any of "dense" (default), "human", "iso-8601", "unix-seconds" or a custom Go time format (see https://go.dev/src/time/format.go)
99
103
timestamp_format: dense
104
+
# Specifies in which time zone the snapshot suffix is generated, optional, defaults to UTC, used only if time zone information is part of timestamp_format.
105
+
# Can be "UTC" (default), "Local" (time zone information from the OS) or any of the IANA Time Zone names (see https://nodatime.org/TimeZones)
106
+
timestamp_location: UTC
100
107
pruning: ...
101
108
102
109
In ``cron`` mode, the snapshotter takes snaphots at fixed points in time.
103
110
See https://en.wikipedia.org/wiki/Cron for details on the syntax.
104
111
zrepl uses the ``the github.com/robfig/cron/v3`` Go package for parsing.
105
112
An optional field for "seconds" is supported to take snapshots at sub-minute frequencies.
106
113
107
-
.. _job-snapshotting-timestamp_format:
108
-
109
-
Timestamp Format
110
-
~~~~~~~~~~~~~~~~
111
-
112
-
The ``cron`` and ``periodic`` snapshotter support configuring a custom timestamp format that is used as suffix for the snapshot name.
113
-
It can be used by setting ``timestamp_format`` to any of the following values:
114
-
115
-
* ``dense`` (default) looks like ``20060102_150405_000``
116
-
* ``human`` looks like ``2006-01-02_15:04:05``
117
-
* ``iso-8601`` looks like ``2006-01-02T15:04:05.000Z``
118
-
* ``unix-seconds`` looks like ``1136214245``
119
-
* Any custom Go time format accepted by `time.Time#Format <https://go.dev/src/time/format.go>`_.
120
-
121
114
122
115
``manual`` Snapshotting
123
116
-----------------------
@@ -137,6 +130,33 @@ See :ref:`this quickstart guide <quickstart-backup-to-external-disk>` for an exa
137
130
138
131
To trigger replication after taking snapshots, use the ``zrepl signal wakeup JOB`` command.
139
132
133
+
.. _job-snapshotting-timestamp_format:
134
+
135
+
Timestamp Format
136
+
----------------
137
+
138
+
The ``cron`` and ``periodic`` snapshotter support configuring a custom timestamp format that is used as suffix for the snapshot name.
139
+
140
+
By default, the current time is converted to UTC and then formatted using the ``dense`` format.
141
+
142
+
Both time zone and format can be customized by setting ``timestamp_format`` and/or ``timestamp_location``.
143
+
144
+
``timestamp_location`` is fed verbatim into `time.LoadLocation <https://pkg.go.dev/time#LoadLocation>`_.
145
+
146
+
``timestamp_format`` supports the following values (case-insensitive).
147
+
148
+
* ``dense`` => Go format string ``20060102_150405_000``
149
+
* ``human`` => Go format string ``2006-01-02_15:04:05``
150
+
* ``iso-8601`` => Go format string ``2006-01-02T15:04:05.000Z07:00``
151
+
* ``unix-seconds`` => Unix seconds since the epoch, formatted as a decimal string (`time.Time.Unix <https://pkg.go.dev/time#Time.Unix>`_)
152
+
* Any custom Go time format accepted by `time.Time#Format <https://go.dev/src/time/format.go>`_.
153
+
154
+
The ``dense``, ``human``, and ``unix-seconds`` formats require the ``timestamp_location`` to be ``UTC`` because they do not contain timezone information.
155
+
156
+
The ``+`` character is `not allowed in ZFS snapshot names <https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/zcommon/zfs_namecheck.c#L334-L351>`_.
157
+
Format strings are disallowed to contain the ``+`` character (it's not a character that is interpreted by ``time.Time.Format``, so there's no reason to use it).
158
+
If a format _produces_ a ``+``, that ``+`` is replaced by an `_`` character.
159
+
140
160
.. _job-snapshotting-hooks:
141
161
142
162
Pre- and Post-Snapshot Hooks
@@ -173,7 +193,7 @@ Most hook types take additional parameters, please refer to the respective subse
Copy file name to clipboardExpand all lines: _sources/configuration/transports.rst.txt
+11-11Lines changed: 11 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -165,7 +165,7 @@ The connection fails if either do not match.
165
165
Mutual-TLS between Two Machines
166
166
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
167
167
168
-
However, for a two-machine setup, self-signed certificates distributed using an out-of-band mechanism will also work just fine:
168
+
For a two-machine setup, self-signed certificates distributed using an out-of-band mechanism will also work just fine:
169
169
170
170
Suppose you have a push-mode setup, with `backups.example.com` running the :ref:`sink job <job-sink>`, and `prod.example.com` running the :ref:`push job <job-push>`.
171
171
Run the following OpenSSL commands on each host, substituting HOSTNAME in both filenames and the interactive input prompt by OpenSSL:
@@ -218,19 +218,19 @@ Tools like `EasyRSA <https://github.com/OpenVPN/easy-rsa>`_ make this very easy:
0 commit comments