ns8-imapsync

Install

Instantiate the module with:

add-module imapsync

The output of the command will return the instance name. Output example:

{"module_id": "imapsync1", "image_name": "imapsync", "image_url": "ghcr.io/nethserver/imapsync:1.0.5"}

Configure

Let's assume that the imapsync instance is named imapsync1.

We need to bind imapsync to a mail server inside the cluster, we use the MODULE_UUID of mail server to configure imapsync, reveal the vmail master secret of local users and start the container waiting for cron tasks or individual user tasks

Launch configure-module, by setting the following parameters:

• mail_server: module uuid of the mail server
• mail_host: local ip of the mail server

Example:

api-cli run module/imapsync1/configure-module --data '{
    "mail_server":"8dd3b3fe-609c-42f7-a2d1-cecba9461bea",
    "mail_host":"10.5.4.1"
}'

The above command will start and configure the imapsync instance.

start to sync a remote imap account to local user account

Example:

api-cli run module/imapsync1/create-task --data '{
    "localuser": "administrator",
    "remotehostname": "imap.foo.com",
    "remoteport": 143,
    "security": "tls",
    "delete_local": false,
    "delete_remote": false,
    "delete_remote_older": 0,
    "exclude": "folder1,folder2",
    "remoteusername": "username",
    "remotepassword": "password",
    "cron": "5m",
    "task_id": "a0241w",
    "foldersynchronization": "all"
}'

Field descriptions:

• security: tls, ssl, or "" (empty for no security)

• delete_local: true or false – if true, deletes mails and folders on the local account that are not in the remote account.

• delete_remote: true or false – if true, deletes mails on the remote account after synchronization to the local account.

• delete_remote_older: Number of days after which remote emails are deleted. If set to 0, messages are removed after the transfer.

• exclude: Comma-separated list of folders to exclude (e.g., folder1,folder2,^folder3$). Use ^ to mark folder names that must start with a pattern and $ for those that must end with one.

• cron: Defines task scheduling (e.g., 5m runs every 5 minutes, "2h" runs every 2 hours). Internally converted to crontab format (0/5 * * * * for minutes, 1 */2 * * * for hours).

• remote credentials: Provide the username and password of the remote IMAP account. Alternatively, use the IMAP master administrator credentials (e.g., user*vmail with the corresponding password).

• foldersynchronization: Controls which folders are synchronized:

  • all – synchronizes all folders.
  • inbox – synchronizes only the inbox.
  • exclusion – synchronizes all folders except excluded ones.

• task_id: Unique identifier for the task, automatically generated by the UI.

• sieve_enabled: Boolean (true or false). If true, applies the user's Sieve filters to synchronized messages. This option is only considered when foldersynchronization is set to inbox.

delete env and stop a running synchronisation

Example:

api-cli run module/imapsync1/delete-task --data '{
    "localuser":"administrator"
}'

stop a running synchronisation

Example:

api-cli run module/imapsync1/stop-task --data '{
    "localuser":"administrator"
}'

start a running synchronisation

Example:

api-cli run module/imapsync1/start-task --data '{
    "localuser":"administrator"
}'

start all configured tasks

Read all imapsync/*.env files and start the synchronization

Example:

api-cli run module/imapsync1/start-all-tasks

stop all configured tasks

Read all imapsync/*.env files and stop the synchronization

Example:

api-cli run module/imapsync1/stop-all-tasks

read configuration

Example:

api-cli run module/imapsync1/get-configuration

Answer:

{
  "mail_server": "e8a6177c-9ae5-4356-826b-0a5f93b2dbaf",
  "mail_host": "10.5.4.1",
  "mail_server_URL": [
    {
      "name": "mail2",
      "label": "mail2 (R3.rocky9-3.org)",
      "value": "e8a6177c-9ae5-4356-826b-0a5f93b2dbaf,10.5.4.1"
    }
  ],
}

list-tasks

Read configuration from environment and from imapsync/*.{env,pwd}

api-cli run module/imapsync1/list-tasks

{
  "enabled_mailboxes": [
    {
      "name": "administrator",
      "label": "administrator",
      "value": "administrator"
    },
    {
      "name": "foo",
      "label": "foo",
      "value": "foo"
    },
    {
      "name": "john",
      "label": "john",
      "value": "john"
    }
  ],
  "user_properties": [
    {
      "task_id": "qu1dcb",
      "localuser": "administrator",
      "remoteusername": "user2@domain.com",
      "remotehostname": "imap.domain.com",
      "remoteport": "143",
      "security": "tls",
      "delete_local": false,
      "deletefolder": "",
      "exclude": "",
      "delete_remote": false,
      "delete_remote_older": 0,
      "sieve_enabled": false,
      "expunge_remote": "",
      "cron": "",
      "folder_inbox": "",
      "foldersynchronization": "all",
      "remotepassword": "password",
      "service_running": false
    },
    {
      "task_id": "vd2am3",
      "localuser": "john",
      "remoteusername": "user@domain.com",
      "remotehostname": "imap.domain.com",
      "remoteport": "143",
      "security": "tls",
      "delete_local": false,
      "deletefolder": "",
      "exclude": "",
      "delete_remote": false,
      "delete_remote_older": 0,
      "sieve_enabled": false,
      "expunge_remote": "",
      "cron": "",
      "folder_inbox": "",
      "foldersynchronization": "all",
      "remotepassword": "password",
      "service_running": false
    }
  ]
}

list-informations

You can retrieve the email and folder numbers and the email size for the local and remote account Example:

api-cli run module/imapsync9/list-informations --data '{
  "localuser": "john",
  "task_id": "vd2am3"
  }'

output:

{"status": true, "host1Folders": 78, "host2Folders": 78, "host1Messages": 164625, "host2Messages": 155, "host1Sizes": 3060488650, "host2Sizes": 72036894}

in case of error you will have

{"status": false}

custom scripts

If necessary, you can add your custom scripts inside the container. To facilitate this, two volumes, included in the backup, are mounted from their respective directories under the ./state module.

• cron (e.g. ~imapsync1/.config/state/cron) This folder is designated for storing cron files, each of which should end its name with .custom, while adhering to the cron syntax.

  1 */1 * * *  root /usr/bin/imapsync --host1 SOURCE_SERVER --user1 SOURCE_USERNAME --password1 SOURCE_PASSWORD --host2 DESTINATION_SERVER --user2 DESTINATION_USERNAME --password2 DESTINATION_PASSWORD
  

  Instead of directly add an imapsync command, you have the option to run a script, which can be stored in the imapsync volume.

  1 */1 * * *  root /etc/imapsync/script.custom
  

• imapsync (e.g. ~imapsync1/.config/state/imapsync) In this location, you can store custom scripts (ending with .custom) intended for execution by cron. These scripts must be set to be executable by all users.

troubleshot issues

The state/imapsync folder contains the following files:

• Environment files (*.env) – Stores task-related environment variables.
• Password files (*.pwd) – Contains credentials. The special vmail.pwd file holds the Dovecot master user's credentials.
• Lock files (*.lock) – Created when a task is running to prevent multiple simultaneous executions.
• Log files (*.log) – Stores the last Imapsync run log. This file is overwritten with each task execution. A summary, including the number of transferred messages and the Imapsync exit code, is always sent to the system log.

You can filter these files by task name, which is a combination of the localuser and the task_id.

Example of files for two tasks, both for local user foo. One of them is running, the other is stopped:

.config/state/
├── cron
│   ├── foo_a0241w.cron
│   └── foo_a1j44r.cron
└── imapsync
    ├── foo_a0241w.env
    ├── foo_a0241w.lock
    ├── foo_a0241w.pwd
    ├── foo_a0241w.log
    ├── foo_a1j44r.env
    ├── foo_a1j44r.pwd
    ├── foo_a1j44r.log
    └── vmail.pwd

to launch manually a task

runagent -m imapsync1
podman exec -ti imapsync /usr/local/bin/syncctl start foo_a1j44r
podman exec -ti imapsync /usr/local/bin/syncctl stop foo_a1j44r
podman exec -ti imapsync /usr/local/bin/syncctl status foo_a1j44r

Bulk import tasks from CSV

Create multiple synchronization tasks at once using a CSV file with the import-csv-tasks utility script.

Overview

This Python script automates the bulk creation of IMAP synchronization tasks by reading user data from a CSV file and calling the create-task API endpoint for each entry. It's ideal for migrating multiple email accounts from a remote IMAP server to your local mail system.

CSV File Format

The CSV file must contain a header row with these 6 required columns:

This users.csv file can be found at .config/examples/users.csv

runagent -m imapsync1
cat ../examples/users.csv

localusername,remoteusername,remotepassword,remotehostname,remoteport,security
pansy.dumbledore5,user1@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,993,ssl
lavender.umbridge7,user2@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,993,ssl
dolores.slughorn3,user3@example.org,"enquotedPasswordIfSeparatorInside",imap.example.org,143,tls

Required columns:

• localusername – Local user account name (must exist on your mail server)
• remoteusername – Remote IMAP account username/email
• remotepassword – Remote IMAP account password (quote if contains special characters)
• remotehostname – Remote IMAP server hostname or IP
• remoteport – Remote IMAP server port (typically 993 for SSL/TLS, 143 for STARTTLS)
• security – Security protocol: ssl, tls, or empty string "" for no encryption

Important notes:

• Column order does not matter – the script maps columns by header name
• Delimiter is comma (,)
• Passwords with special characters should be quoted: "myP@ss,word"
• All fields are mandatory except security – rows with empty required values will be rejected
• The security field can be empty (for no encryption), but the other 5 fields must contain values
• Empty lines in the CSV are automatically skipped

Usage

The script is designed to be used with NS8 runagent to bulk import synchronization tasks.

Basic usage (file redirection):

runagent -m imapsync1 import-csv-tasks < users.csv

Alternative (using pipe):

cat users.csv | runagent -m imapsync1 import-csv-tasks

Check CSV format before creating tasks (check-only mode):

runagent -m imapsync1 import-csv-tasks -c < users.csv

This performs comprehensive validation without making API calls:

• Checks CSV structure (required columns present)
• Validates delimiter (comma-separated)
• Verifies all mandatory fields have values (except security which can be empty)
• Validates port numbers are numeric
• Reports specific errors for each invalid row

Example validation output:

📋 CSV Column Validation:
   Delimiter: ',' (comma-separated)
   Found 6 column(s): localusername, remotehostname, remotepassword, remoteport, remoteusername, security
   Column order: does not matter (mapped by header name)
✓ All 6 required columns present
✓ Found 2 data row(s) (empty lines skipped)

🔍 Validating row data...
✓ All 2 rows validated successfully
✓ Generated 2 unique task IDs

✓ CSV file is valid. No tasks were created (check-only mode).

Display help:

runagent -m imapsync1 import-csv-tasks -h

Options

• -c, --check – Check-only mode: validates CSV format without creating tasks
• -h, --help – Display comprehensive help message

Features

• Flexible input methods:

  • Reads from standard input (stdin)
  • Works with file redirection (<), pipes (|), or heredoc
  • Compatible with NS8 runagent for direct module integration
  • No file path arguments needed

• CSV parsing:

  • Comma-separated delimiter (,) - required
  • Validates all 6 required columns are present before processing
  • Handles quoted fields with embedded delimiters
  • Reports missing columns with clear error messages
  • Skips empty lines automatically

• Data validation:

  • Mandatory field checking: ensures 5 required fields are not empty (localusername, remoteusername, remotepassword, remotehostname, remoteport)
  • Optional field: security can be empty (for no encryption)
  • Port validation: ensures remoteport contains only numeric values
  • Check mode (-c flag): validates all rows without creating tasks
  • Clear error messages with row numbers for easy debugging

• Automatic field population:

  • Generates unique random 6-character task ID (lowercase alphanumeric) for each user
  • Auto-fills optional fields with sensible defaults:
    • cron: "" (empty - no automatic scheduling)
    • delete_local: false (preserve local emails)
    • delete_remote: false (preserve remote emails)
    • delete_remote_older: 0 (no age-based deletion)
    • exclude: "" (no folder exclusions)
    • foldersynchronization: "all" (sync all folders)
    • sieve_enabled: false (don't apply Sieve filters)

• Robust error handling:

  • Validates port numbers are numeric
  • Continues processing remaining users if one fails
  • Displays detailed error messages per user
  • Returns non-zero exit code if any task fails

• Progress tracking:

  • Shows real-time progress for each user creation
  • Displays success/failure status per task
  • Provides final summary with success/failure counts

• API integration:

  • Calls action/create-task for each user
  • Passes data as properly formatted JSON
  • Captures and displays API response output

Example Output

Create 3 synchronization tasks from a CSV file:

runagent -m imapsync1 import-csv-tasks < users.csv

Validating CSV file: users.csv

📋 CSV Column Validation:
   Delimiter detected: ';'
   Found 6 column(s): localusername, remotepassword, remotehostname, remoteport, remoteusername, security
✓ All 6 required columns present
✓ Found 3 data row(s)

📦 Starting task creation with module: imapsync1
   Processing 3 user(s)...

Creating task for pansy.dumbledore5 (ID: a1b2c3)...
✓ Success: pansy.dumbledore5

Creating task for lavender.umbridge7 (ID: d4e5f6)...
✓ Success: lavender.umbridge7

Creating task for dolores.slughorn3 (ID: g7h8i9)...
✓ Success: dolores.slughorn3

======================================================================
📊 Summary: 3 successful, 0 failed
======================================================================

Troubleshooting

"Missing required columns"

• Check CSV header row contains all 6 required column names exactly as specified
• Verify no typos in column names (case-sensitive)

"Missing required value(s)"

• The script rejects rows with empty mandatory fields
• Ensure all 5 required fields have values: localusername, remoteusername, remotepassword, remotehostname, remoteport
• Only security can be empty (for no encryption)
• Use -c flag to validate CSV before creating tasks

"Invalid port number"

• Ensure remoteport column contains only numeric values (e.g., 993, 143)

Remove all tasks after a bad import

If you need to clean up after a failed or incorrect bulk import, you can remove all task configuration files:

# Stop all running tasks first
api-cli run module/imapsync1/stop-all-tasks

# Remove all task configuration files (replace imapsync1 with your module ID)
rm -f /home/imapsync1/.config/state/imapsync/*.env
rm -f /home/imapsync1/.config/state/imapsync/*.pwd
rm -f /home/imapsync1/.config/state/imapsync/*.log
rm -f /home/imapsync1/.config/state/imapsync/*.lock
rm -f /home/imapsync1/.config/state/cron/*.cron

# Verify cleanup
ls -la /home/imapsync1/.config/state/imapsync/

Note: This preserves the vmail.pwd file (master password). To start fresh completely, you can also remove it, but you'll need to reconfigure the module afterward.

Uninstall

To uninstall the instance:

remove-module --no-preserve imapsync1

Running tests locally

This module uses the NS8 standard testing infrastructure. For instructions on how to run the test suite locally, refer to the Running tests locally section of the ns8-github-actions repository.

UI translation

Translated with Weblate.

To setup the translation process:

• add GitHub Weblate app to your repository
• add your repository to [hosted.weblate.org]((https://hosted.weblate.org) or ask a NethServer developer to add it to ns8 Weblate project