allow users of mdk to choose whether to sort by createdAt or processedAt first (marmot-protocol#171)

Co-authored-by: JeffG <202880+erskingardner@users.noreply.github.com>

Author: dannym-arx

crates/mdk-core/CHANGELOG.md

@@ -41,6 +41,7 @@
 
 ### Added
 
+- **Custom Message Sort Order**: `get_messages()` now supports custom sort orders via the `Pagination::sort_order` field. Added `get_last_message(group_id, sort_order)` method to retrieve the most recent message under a given sort order, enabling clients using `ProcessedAtFirst` ordering to get a consistent "last message" value. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
 - **`create_key_package_for_event_with_options`**: New function that allows specifying whether to include the NIP-70 protected tag. Use this if you need to publish to relays that accept protected events. ([#173](https://github.com/marmot-protocol/mdk/pull/173), related: [#168](https://github.com/marmot-protocol/mdk/issues/168))
 - **MIP-04 Epoch Fallback for Media Decryption**: `decrypt_from_download` now resolves the correct decryption key via an O(1) epoch hint lookup instead of only using the current epoch's exporter secret. Added `NoExporterSecretForEpoch` variant to `EncryptedMediaError` for programmatic error matching. ([#167](https://github.com/marmot-protocol/mdk/pull/167))
 - **`PreviouslyFailed` Result Variant**: Added `MessageProcessingResult::PreviouslyFailed` variant to handle cases where a previously failed message arrives again but the MLS group ID cannot be extracted. This prevents crashes in client applications by returning a result instead of throwing an error. ([#165](https://github.com/marmot-protocol/mdk/pull/165), fixes [#154](https://github.com/marmot-protocol/mdk/issues/154), [#159](https://github.com/marmot-protocol/mdk/issues/159))

crates/mdk-core/src/messages/mod.rs

@@ -20,8 +20,8 @@ mod process;
 mod proposal;
 mod validation;
 
-use mdk_storage_traits::groups::Pagination;
 use mdk_storage_traits::groups::types as group_types;
+use mdk_storage_traits::groups::{MessageSortOrder, Pagination};
 use mdk_storage_traits::messages::types as message_types;
 use mdk_storage_traits::{GroupId, MdkStorageProvider};
 use nostr::{EventId, Timestamp};
@@ -226,6 +226,34 @@ where
             .map_err(|_e| Error::Message("Storage error while getting messages".to_string()))
     }
 
+    /// Returns the most recent message in a group according to the given sort order.
+    ///
+    /// This is useful for clients that use [`MessageSortOrder::ProcessedAtFirst`] and
+    /// need a "last message" value that is consistent with their [`get_messages()`](Self::get_messages)
+    /// ordering. The cached [`Group::last_message_id`](group_types::Group::last_message_id) always
+    /// reflects [`MessageSortOrder::CreatedAtFirst`], so clients using a different sort order
+    /// can call this method instead.
+    ///
+    /// # Arguments
+    ///
+    /// * `mls_group_id` - The MLS group ID
+    /// * `sort_order` - The sort order to use when determining the "last" message
+    ///
+    /// # Returns
+    ///
+    /// * `Ok(Some(Message))` - The most recent message under the given ordering
+    /// * `Ok(None)` - If the group has no messages
+    /// * `Err(Error)` - If the group does not exist or a storage error occurs
+    pub fn get_last_message(
+        &self,
+        mls_group_id: &GroupId,
+        sort_order: MessageSortOrder,
+    ) -> Result<Option<message_types::Message>> {
+        self.storage()
+            .last_message(mls_group_id, sort_order)
+            .map_err(|_e| Error::Message("Storage error while getting last message".to_string()))
+    }
+
     // =========================================================================
     // Storage Save Helpers
     // =========================================================================

crates/mdk-memory-storage/CHANGELOG.md

@@ -27,6 +27,8 @@
 
 ### Added
 
+- **Custom Message Sort Order**: `messages()` now respects the `sort_order` field in `Pagination`, supporting both `CreatedAtFirst` (default) and `ProcessedAtFirst` orderings. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
+- **Last Message by Sort Order**: Implemented `last_message()` to return the most recent message under a given sort order. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
 - **Epoch Lookup by Tag Content**: Implemented `find_message_epoch_by_tag_content` for in-memory storage, scanning cached group messages and matching serialized tags. ([#167](https://github.com/marmot-protocol/mdk/pull/167))
 - **Retryable Message Support**: Updated storage implementation to handle `ProcessedMessageState::Retryable` transitions and persistence. ([#161](https://github.com/marmot-protocol/mdk/pull/161))
 

crates/mdk-memory-storage/src/groups.rs

@@ -5,7 +5,7 @@ use std::collections::BTreeSet;
 use mdk_storage_traits::GroupId;
 use mdk_storage_traits::groups::error::GroupError;
 use mdk_storage_traits::groups::types::*;
-use mdk_storage_traits::groups::{GroupStorage, MAX_MESSAGE_LIMIT, Pagination};
+use mdk_storage_traits::groups::{GroupStorage, MAX_MESSAGE_LIMIT, MessageSortOrder, Pagination};
 use mdk_storage_traits::messages::types::Message;
 use nostr::{PublicKey, RelayUrl};
 
@@ -116,14 +116,23 @@ impl GroupStorage for MdkMemoryStorage {
             return Err(GroupError::InvalidParameters("Group not found".to_string()));
         }
 
+        let sort_order = pagination.sort_order();
+
         match inner.messages_by_group_cache.peek(mls_group_id) {
             Some(messages_map) => {
                 // Collect values from HashMap into a Vec for sorting
                 let mut messages: Vec<Message> = messages_map.values().cloned().collect();
 
-                // Sort by created_at DESC, processed_at DESC, id DESC (newest first).
-                // Uses the centralized display_order_cmp (reversed for DESC).
-                messages.sort_by(|a, b| b.display_order_cmp(a));
+                // Sort newest-first using the requested sort order.
+                // Both comparators are called with (b, a) to get DESC ordering.
+                match sort_order {
+                    MessageSortOrder::CreatedAtFirst => {
+                        messages.sort_by(|a, b| b.display_order_cmp(a));
+                    }
+                    MessageSortOrder::ProcessedAtFirst => {
+                        messages.sort_by(|a, b| b.processed_at_order_cmp(a));
+                    }
+                }
 
                 // Apply pagination
                 let start = offset.min(messages.len());
@@ -136,6 +145,35 @@ impl GroupStorage for MdkMemoryStorage {
         }
     }
 
+    fn last_message(
+        &self,
+        mls_group_id: &GroupId,
+        sort_order: MessageSortOrder,
+    ) -> Result<Option<Message>, GroupError> {
+        let inner = self.inner.read();
+
+        if inner.groups_cache.peek(mls_group_id).is_none() {
+            return Err(GroupError::InvalidParameters("Group not found".to_string()));
+        }
+
+        match inner.messages_by_group_cache.peek(mls_group_id) {
+            Some(messages_map) if !messages_map.is_empty() => {
+                // Find the maximum element under the requested ordering.
+                // Both comparators compare (b, a) to find the DESC-first element via max_by.
+                let winner = match sort_order {
+                    MessageSortOrder::CreatedAtFirst => {
+                        messages_map.values().max_by(|a, b| a.display_order_cmp(b))
+                    }
+                    MessageSortOrder::ProcessedAtFirst => messages_map
+                        .values()
+                        .max_by(|a, b| a.processed_at_order_cmp(b)),
+                };
+                Ok(winner.cloned())
+            }
+            _ => Ok(None),
+        }
+    }
+
     fn admins(&self, mls_group_id: &GroupId) -> Result<BTreeSet<PublicKey>, GroupError> {
         match self.find_group_by_mls_group_id(mls_group_id)? {
             Some(group) => Ok(group.admin_pubkeys.clone()),

crates/mdk-sqlite-storage/CHANGELOG.md

@@ -27,7 +27,10 @@
 
 ### Added
 
-- **Group `last_message_processed_at` Column**: Added `last_message_processed_at` column to the `groups` table via V003 migration to track when the last message was processed/received by this client. This enables consistent ordering between `group.last_message_id` and `get_messages()[0].id`. Existing groups are backfilled with their `last_message_at` value as a reasonable default. ([#166](https://github.com/marmot-protocol/mdk/pull/166))
+- **Custom Message Sort Order**: `messages()` now respects the `sort_order` field in `Pagination`, supporting both `CreatedAtFirst` (default) and `ProcessedAtFirst` orderings via different SQL `ORDER BY` clauses. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
+- **Last Message by Sort Order**: Implemented `last_message()` to return the most recent message under a given sort order via `SELECT ... ORDER BY ... LIMIT 1`. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
+- **Processed-At-First Sort Index**: Added V003 migration creating `idx_messages_sorting_processed_at` composite index on `messages(mls_group_id, processed_at DESC, created_at DESC, id DESC)` for efficient `ProcessedAtFirst` queries. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
+- **Group `last_message_processed_at` Column**: Added `last_message_processed_at` column to the `groups` table via V002 migration to track when the last message was processed/received by this client. This enables consistent ordering between `group.last_message_id` and `get_messages()[0].id`. Existing groups are backfilled with their `last_message_at` value as a reasonable default. ([#166](https://github.com/marmot-protocol/mdk/pull/166))
 
 - **Message `processed_at` Column**: Added `processed_at` column to the `messages` table via V002 migration to store when messages were processed/received by the client. Existing messages are backfilled with their `created_at` value as a reasonable default. ([#166](https://github.com/marmot-protocol/mdk/pull/166))
 

crates/mdk-sqlite-storage/migrations/V003__add_processed_at_first_sort_index.sql

@@ -0,0 +1,6 @@
+-- Add a composite index for the processed_at-first sort order
+-- (processed_at DESC, created_at DESC, id DESC) scoped to mls_group_id.
+-- This allows efficient queries when clients request MessageSortOrder::ProcessedAtFirst,
+-- which prioritises local reception time over the sender's timestamp.
+CREATE INDEX IF NOT EXISTS idx_messages_sorting_processed_at
+  ON messages(mls_group_id, processed_at DESC, created_at DESC, id DESC);

crates/mdk-sqlite-storage/src/groups.rs

@@ -5,7 +5,7 @@ use std::collections::BTreeSet;
 use mdk_storage_traits::GroupId;
 use mdk_storage_traits::groups::error::GroupError;
 use mdk_storage_traits::groups::types::{Group, GroupExporterSecret, GroupRelay};
-use mdk_storage_traits::groups::{GroupStorage, MAX_MESSAGE_LIMIT, Pagination};
+use mdk_storage_traits::groups::{GroupStorage, MAX_MESSAGE_LIMIT, MessageSortOrder, Pagination};
 use mdk_storage_traits::messages::types::Message;
 use nostr::{PublicKey, RelayUrl};
 use rusqlite::{OptionalExtension, params};
@@ -182,12 +182,23 @@ impl GroupStorage for MdkSqliteStorage {
             return Err(GroupError::InvalidParameters("Group not found".to_string()));
         }
 
+        let sort_order = pagination.sort_order();
+
         self.with_connection(|conn| {
-            let mut stmt = conn
-                .prepare(
-                    "SELECT * FROM messages WHERE mls_group_id = ? ORDER BY created_at DESC, processed_at DESC, id DESC LIMIT ? OFFSET ?",
-                )
-                .map_err(into_group_err)?;
+            let query = match sort_order {
+                MessageSortOrder::CreatedAtFirst => {
+                    "SELECT * FROM messages WHERE mls_group_id = ? \
+                     ORDER BY created_at DESC, processed_at DESC, id DESC \
+                     LIMIT ? OFFSET ?"
+                }
+                MessageSortOrder::ProcessedAtFirst => {
+                    "SELECT * FROM messages WHERE mls_group_id = ? \
+                     ORDER BY processed_at DESC, created_at DESC, id DESC \
+                     LIMIT ? OFFSET ?"
+                }
+            };
+
+            let mut stmt = conn.prepare(query).map_err(into_group_err)?;
 
             let messages_iter = stmt
                 .query_map(
@@ -207,6 +218,37 @@ impl GroupStorage for MdkSqliteStorage {
         })
     }
 
+    fn last_message(
+        &self,
+        mls_group_id: &GroupId,
+        sort_order: MessageSortOrder,
+    ) -> Result<Option<Message>, GroupError> {
+        if self.find_group_by_mls_group_id(mls_group_id)?.is_none() {
+            return Err(GroupError::InvalidParameters("Group not found".to_string()));
+        }
+
+        self.with_connection(|conn| {
+            let query = match sort_order {
+                MessageSortOrder::CreatedAtFirst => {
+                    "SELECT * FROM messages WHERE mls_group_id = ? \
+                     ORDER BY created_at DESC, processed_at DESC, id DESC \
+                     LIMIT 1"
+                }
+                MessageSortOrder::ProcessedAtFirst => {
+                    "SELECT * FROM messages WHERE mls_group_id = ? \
+                     ORDER BY processed_at DESC, created_at DESC, id DESC \
+                     LIMIT 1"
+                }
+            };
+
+            conn.prepare(query)
+                .map_err(into_group_err)?
+                .query_row(params![mls_group_id.as_slice()], db::row_to_message)
+                .optional()
+                .map_err(into_group_err)
+        })
+    }
+
     fn admins(&self, mls_group_id: &GroupId) -> Result<BTreeSet<PublicKey>, GroupError> {
         // Get the group which contains the admin_pubkeys
         match self.find_group_by_mls_group_id(mls_group_id)? {

crates/mdk-storage-traits/CHANGELOG.md

@@ -27,6 +27,8 @@
 
 ### Added
 
+- **Custom Message Sort Order**: Added `MessageSortOrder` enum with `CreatedAtFirst` (default) and `ProcessedAtFirst` variants to allow clients to choose how messages are ordered. Added `sort_order` field to `Pagination` struct and `Pagination::with_sort_order()` constructor. Added `Message::processed_at_order_cmp()` and `Message::compare_processed_at_keys()` comparison methods for the processed-at-first ordering. ([#171](https://github.com/marmot-protocol/mdk/pull/171))
+- **Last Message by Sort Order**: Added `GroupStorage::last_message()` method to query the most recent message in a group according to a given sort order. This allows clients using `ProcessedAtFirst` ordering to get a "last message" consistent with their `messages()` call, independent of the cached `Group::last_message_id` (which always reflects `CreatedAtFirst`). ([#171](https://github.com/marmot-protocol/mdk/pull/171))
 - **Epoch Lookup by Tag Content**: Added `find_message_epoch_by_tag_content` method to `MessageStorage` trait for looking up a message's epoch by searching serialized tag content. Used for MIP-04 media decryption epoch hint resolution. ([#167](https://github.com/marmot-protocol/mdk/pull/167))
 
 ### Breaking changes