fix(activesync): separate PING watermark from SYNC modseq to stop iOS…

[TDannhauer]

Fix ActiveSync PING/SYNC loops, state corruption, and initial-sync reliability

Branch: fix/ActiveSync_loops → FRAMEWORK_6_0
Repository: horde/ActiveSync
HEAD: 22a44863 (8 commits, 22 files, +3181 / −221 lines)

---

Summary

This branch fixes several interacting ActiveSync 3.0 regressions that caused iOS PING loops, Invalid sync_data / KEYMISMATCH cycles, stuck sync_pending (MOREAVAILABLE) batches, and missing mail on initial sync. The changes introduce a clearer separation between SYNC and PING watermarks, harden state persistence against concurrent workers, validate collection state on load/save, and improve MIME export for nested/signed messages.

Production testing on Horde 6 / activesync 3.0.0-beta3 reports significantly improved stability after these fixes.

---

Problems addressed

1. iOS PING loops (~3s wake/sleep cycle)

Email collections used a single modseq for both:

• SYNC (CHANGEDSINCE / $_status)
• PING (IMAP STATUS polling)

Additionally, PING treated sync_pending (in-flight MOREAVAILABLE batches) as a change signal. When SYNC modseq lagged behind the server, or a partial batch remained in sync_pending, PING would wake repeatedly without delivering mail.

2. State corruption under concurrency

Parallel SYNC and PING workers could load and save the same sync_key concurrently, overwriting sync_data and sync_pending. This produced Invalid sync_data corruption, especially during heavy initial sync on large mailboxes.

3. Corrupt sync_data written or silently accepted

Empty arrays (a:0:{}), non-folder blobs, and FOLDERSYNC-shaped data could be treated as valid collection state, breaking subsequent PING/SYNC and blocking new-mail push.

4. Initial sync and MIME export gaps

• Initial sync could be marked complete before exported UIDs were acknowledged.
• Failed exports could advance progress or drop sync_pending entries on retriable backend errors.
• Deeply nested MIME (e.g. inline PGP keys on Dovecot) could fail body fetch and silently omit messages.

---

Solution

PING / SYNC watermark separation

Introduce a three-state model for mail collections:

State	Purpose
$_status	SYNC watermark (CHANGEDSINCE)
$_pingStatus	PING watermark (IMAP STATUS via pingModseq())
sync_pending	SYNC-only; MOREAVAILABLE batch tracking

PING now:

• Polls IMAP STATUS only (does not use sync_pending as a change signal)
• Advances $_pingStatus on detection
• Saves with preservePending => true so in-flight MOREAVAILABLE batches survive PING saves

Files: State/Base.php, Folder/Imap.php, Imap/Adapter.php

Concurrency control

Row locks (device + sync_key level):

• SQL: SELECT … FOR UPDATE held from load through save() / updateSyncStamp()
• Mongo: sync_lock field with findAndModify(), including stale-lock recovery

Collection locks (per folder):

• Migration 24: horde_activesync_collection_lock (SQL) / HAS_collection_lock (Mongo)
• Acquired around loadState(), save(), and updateServerIdInState()
• Row locks remain held through save()

Files: State/Sql.php, State/Mongo.php, migrations/24_horde_activesync_addcollectionlock.php

State validation and recovery

• _normalizeSyncFolderData() — missing/unparseable blobs are rebuilt; clearly corrupt payloads raise StaleState
• _assertValidSyncFolderBeforeSave() / _assertSyncDataBlob() — refuse to persist invalid folder objects or empty/array-shaped blobs
• KEYMISMATCH on corrupt load — corrupt sync_data forces a collection resync via STATUS_KEYMISMATCH instead of silently reinitializing at the same synckey

Files: State/Base.php, Collections.php, Request/Sync.php

Safer persistence (SQL / Mongo)

• Scope UPDATE/INSERT to sync_key, sync_folderid, sync_devid, sync_user
• Persist sync_pending as TEXT; only sync_data uses Horde_Db_Value_Binary
• Wrap persist in transactions; retry concurrent INSERT as UPDATE

Initial sync and export reliability

• Defer marking initial sync complete until exported UIDs are acknowledged
• Fix haveInitialSync=false being treated as complete when the hi key is present
• Stop SYNC export loop from treating failed exports as progress
• Retry IMAP body fetch without BODY[].SIZE when nested parts return no data (Dovecot)
• Treat application/pgp-keys like S/MIME signatures for attachment detection
• Log message build failures instead of silently omitting messages

Files: Connector/Exporter/Sync.php, Folder/Imap.php, Imap/MessageBodyData.php, Mime.php, Mime/Iterator.php, Request/Sync.php

---

Commits (oldest → newest)

Commit	Description
ba01d163	Separate PING watermark from SYNC modseq; preservePending on PING save
d5038717	Reject corrupt sync_data on load and save
b1c288f5	Initial sync acknowledgement, export loop fixes, nested MIME / PGP-key export
e480db3a	SQL and Mongo row locks for parallel state access
c7e372fb	Doc clarifications (row lock behavior, sendNextChange contract)
0dfff8d2	Collection locks, scoped state writes, corrupt-state recovery
161d301e	Migration filename typo fix
22a44863	Force KEYMISMATCH when loading clearly corrupt sync_data

---

Database migration

Run Horde migrations after merge:

php /path/to/horde/bin/horde-upgrade activesync

Adds collection-lock storage required for per-folder serialization.

---

Tests

New/expanded unit tests under test/Horde/ActiveSync/StateTest/:

Test	Coverage
StateSqlPreservePendingTest	PING saves preserve sync_pending; watermark separation
Sql/RowLockTest, Mongo/RowLockTest	Concurrent load/save serialization
Sql/CollectionLockTest, Mongo/CollectionLockTest	Per-collection lock acquire/release
Sql/InitialSyncTest	Initial sync acknowledgement
ImapFolderTest, ImapAdapterTest	PING checkpoint / modseq behavior
MessageBodyDataTest, MimeTest	Nested MIME body fetch fallback, PGP-key structures

cd vendor/horde/activesync   # or package root
vendor/bin/phpunit test/Horde/ActiveSync/StateTest/
vendor/bin/phpunit test/Horde/ActiveSync/ImapFolderTest.php
vendor/bin/phpunit test/Horde/ActiveSync/MessageBodyDataTest.php

---

Production validation

Tested on Horde 6 deployment with:

• iOS Mail (EAS 16.0) — PING loops resolved; INBOX synckeys advance cleanly
• PostgreSQL backend — no recurring Invalid sync_data after fixes + PHP-FPM reload
• Tester feedback: “quite stable” under normal use

---

Related work (not in this branch)

PostgreSQL deployments may also need a complementary horde/db fix for PDO multi-LOB binding: when sync_data and sync_pending are bound as LOBs in a single execute(), PostgreSQL bytea can be corrupted. That amplifies the state bugs fixed here. Track separately in horde/Db.

Window size configuration (maximumwindowsize / maximumrequestwindowsize = 0) is orthogonal but increases batch size and MOREAVAILABLE pressure. Recommend finite values (25–100) in production conf.php regardless of this merge.

---

Test plan

• Run full StateTest suite on SQL (PostgreSQL/MySQL) and Mongo backends
• PING loop: Large INBOX, new mail arrives → single PING wakeup → SYNC delivers mail; no ~3s PING loop
• MOREAVAILABLE: Force large initial sync → verify sync_pending clears across multiple SYNC rounds; PING does not false-wake on pending batch
• Concurrency: Parallel SYNC + PING on same device/collection → no Invalid sync_data in logs
• Corrupt state recovery: Inject corrupt sync_data row → client receives STATUS_KEYMISMATCH and resyncs cleanly
• Signed/nested mail: Message with inline PGP key exports and appears on device
• Migration 24: Upgrade path creates collection-lock table/collection on fresh and existing installs
• Regression: Small-folder sync, calendar/contacts PING unaffected

---

Deployment notes

• Requires horde-db-migration before relying on collection locks
• Reload PHP-FPM/Apache after deploy to clear opcode cache
• Monitor device logs for Invalid sync_data, KEYMISMATCH, and COLLECTIONS: Found changes! without subsequent SYNC

[ralflang]

Please rebase before adding code to this. I will put it back into draft status now.

[TDannhauer]

@ralflang please review

[ralflang]

SYNCSTAMP_UPDATE_THRESHOLD and STATE_ROW_LOCK_STALE_SECONDS defined in Base.php but also Sql.php and Mongo.php - harmless but unnecessary and potential source of future trouble.

_acknowledgeExportedChange() only handles CHANGE/DRAFT, not ADD. For initial sync the messages come through as ADDs, but the code path that primes
_messages works through acknowledgeExportedMessage() invoked from updateState-driven export — verify that ADD changes during initial sync hit -> I don't know if this is an issue protocol-wise.

Did you test

horde-db-migrate 24; horde-db-migrate 23; horde-db-migrate 24

To make sure it's a viable roundtrip without data loss?

Overall I think it's OK to merge.

[TDannhauer]

horde migrate 24 id used to create my table. downgrade to 23 I did not test, but it is trivial : up adds a table, and down drops it. no modification on existing tables.

[TDannhauer]

thanks for the review, wil ladjust the code slightly to address it.