feat: pre-messages / next version of download on demand #7371

Simon-Laux · 2025-10-29T20:53:40Z

feat: Remove "Download maximum available until" and remove stock string DC_STR_DOWNLOAD_AVAILABILITY #7369
remove: partial downloads (remove creation of the stub messages) #7373
feat: pre messages can now get read receipts #7433
feat: send pre-message on messages with large attachments #7410
feat: receive pre-messages and adapt download on demand #7431
... for now read the issue for more info on todo items and what this is about

This is the branch for #7367
closes #7367

Currently removed tests

test_something.py::test_reaction_to_partially_fetched_msg
python/tests/test_1_online.py::test_webxdc_download_on_demand
src/calls/calls_tests.py::test_no_partial_calls
src/message/message_tests.rs::test_markseen_not_downloaded_msg
src/receive_imf/receive_imf_tests.rs::test_download_later
src/receive_imf/receive_imf_tests.rs::test_create_group_with_big_msg
src/receive_imf/receive_imf_tests.rs::test_prefer_references_to_downloaded_msgs
src/receive_imf/receive_imf_tests.rs::test_partial_download_key_contact_lookup
src/webxdc/webxdc_tests.rs::test_webxdc_update_for_not_downloaded_instance
src/download.rs::test_download_limit
src/download.rs::test_partial_receive_imf
src/download.rs::test_partial_download_and_ephemeral
src/download.rs::test_status_update_expands_to_nothing
src/download.rs::test_mdn_expands_to_nothing
src/download.rs::test_partial_download_trashed
src/reaction.rs::test_partial_download_and_reaction

Progress of the tests

Overview about the recycled(♻️) and dropped(🗑️) tests of the tests that I removed in #7373

🗑️test_reaction_to_partially_fetched_msg
🗑️ test_partial_download_and_ephemeral
🗑️ test_status_update_expands_to_nothing
♻️ test_partial_download_trashed
♻️ test_partial_download_and_reaction
♻️ test_partial_group_consistency
🗑️ test_partial_download_key_contact_lookup
♻️ test_webxdc_update_for_not_downloaded_instance
🗑️test_no_partial_calls
♻️ test_markseen_not_downloaded_msg
♻️ test_msg_seen_on_imap_when_downloaded
♻️ test_create_group_with_big_msg
♻️ test_download_later
🗑️ test_prefer_references_to_downloaded_msgs
♻️test_download_on_demand
♻️test_download_limit_chat_assignment
🗑️test_webxdc_download_on_demand

All the recycled tests were already recycled/re-made except for some which are still to do:
(none)

Furthermore there need to be new tests to test the downloading/scheduling changes.

TODO Tests

postponed to do tests

(none)

discarded/ignored test ideas

maybe we should reconsider / discuss those?

test that full message, replacing pre-message does not work if the message encryption state is not the same for both messages (both full- and pre-message)
- unencrypted pre- and full-message are not sent

src/sql/migrations.rs

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <hocuri@gmx.de>

…ty summaries

…e optimization

Hocuri · 2025-12-18T15:24:35Z

src/sql/migrations.rs

+            "CREATE TABLE download_new (
+                rfc724_mid TEXT NOT NULL DEFAULT '',
+                msg_id INTEGER NOT NULL DEFAULT 0
+            ) STRICT;


@link2xt I'm wondering whether we should add UNIQUE(rfc724_mid) or UNIQUE(msg_id)? Or even both? I guess it doesn't really make sense to have the same message in there twice?

Do we need both msg_id and rfc724_mid in this table? rfc724_mid can be obtained from msg_id.
EDIT: Got it, msg_id is 0 when rows are inserted to download in fetch_new_msg_batch(). But UNIQUE(rfc724_mid) may still make sense.

Previously, the uid_next wasn't advanced, which didn't create any problems, but it also was inefficient, because another loop was done trying to fetch the message again (and then finally skipping it, because it's already known).

iequidoo · 2025-12-22T06:47:30Z

deltachat-ffi/deltachat.h

 * - `download_limit` = Messages up to this number of bytes are downloaded automatically.
- *                    For larger messages, only the header is downloaded and a placeholder is shown.
+ *                    For messages with large attachments, two messages are sent:
+ *                    a Pre-Message containing metadata and a Post-Message containing the attachment.


"Full message" was a better term, "post-message" is literally "smth going after the message". And the full message should contain metadata as well, but the documentation doesn't say that

deltachat-ffi/deltachat.h

deltachat-jsonrpc/src/api/types/message.rs

src/chat.rs

src/download.rs

src/chat.rs

iequidoo · 2025-12-24T09:59:50Z

src/download/pre_msg_metadata.rs

+
+/// Metadata contained in Pre-Message that describes the Post-Message.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct PreMsgMetadata {


Actually this is PreMsgData, it's metadata for the "post-message" (and even not metadata, but a part thereof)

src/download/pre_msg_metadata.rs

iequidoo · 2025-12-24T10:01:19Z

src/scheduler.rs

        .log_err(ctx)
        .ok();

+    download_msgs(ctx, &mut session)


I think it's better to move this right after download_known_post_messages_without_pre_message() so that it goes before scan_folders(). Scanning folders isn't that important, it's even debounced to once per minute.

EDIT: Before, it even preceded the whole fetch_idle().

src/scheduler.rs

iequidoo · 2025-12-24T10:01:25Z

src/sql/migrations.rs

+             JOIN msgs m ON d.msg_id = m.id
+             WHERE m.rfc724_mid IS NOT NULL AND m.rfc724_mid != '';
+            DROP TABLE download;
+            ALTER TABLE download_new RENAME TO download;


Consider renaming to downloads, we usually use plurals for tables

src/stock_str.rs

src/tests.rs

iequidoo · 2025-12-27T07:23:51Z

src/imap.rs

+                    uids_fetch.push(uid);
+                    uid_message_ids.insert(uid, message_id);


if download_limit.is_none_or(|download_limit| size <= download_limit) { uids_fetch.push(uid); uid_message_ids.insert(uid, message_id); } else { download_later.push(message_id.clone()); largest_uid_skipped = Some(uid); }

Maybe do smth like this here? This way pre-messages should still be downloaded immediately and big emails are downloaded later to speed up downloading of pre-messages and other small messages.

I've chacked that JSON-RPC tests work with this suggestion.

The test just passes, it seems there's no need to remove it.

…imap_when_downloaded

src/download.rs

…essage before There is a bug in pre-messages: - A pre-message adds an entry to the `msgs` table with the `rfc724_mid` of the post-message - If the pre-message and post-message are fetched in separate cycles: - `prefetch_should_download()` returns false, because `rfc724_mid_exists()` returns something - so, the message is not added to `download_later`, and never automatically downloaded.

…e-messages modify group members There are no partial messages anymore, so i have no idea what else to check.

iequidoo · 2025-12-30T04:33:09Z

Force-pushed the branch to have #7625 as a single commit with the description from the PR. Also fixed test_download_limit_chat_assignment

iequidoo · 2025-12-30T19:52:38Z

src/message.rs

            trans.execute(
                "UPDATE imap SET target=? WHERE rfc724_mid=?",
-                (target, msg.rfc724_mid),
+                (target, &msg.rfc724_mid),


This deletes the post-message on IMAP, but the pre-message is never deleted. We need to save rfc724_mid of pre-messages to the db to delete them as well. Maybe add msgs.rfc724_mid_pre column?

It currently fails because pre-messages aren't deleted on IMAP.

Simon-Laux changed the title ~~[Branch] Pre-messages / next version of download on demand~~ [Branch] pre-messages / next version of download on demand Oct 29, 2025

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

link2xt force-pushed the pre-messages branch 3 times, most recently from 2f1c383 to 6bcc795 Compare December 11, 2025 12:00

link2xt mentioned this pull request Dec 11, 2025

feat: receive pre-messages and adapt download on demand #7431

Merged

26 tasks

link2xt force-pushed the pre-messages branch 6 times, most recently from 8443b17 to 702771f Compare December 11, 2025 21:24

Hocuri reviewed Dec 11, 2025

View reviewed changes

src/sql/migrations.rs Show resolved Hide resolved

link2xt force-pushed the pre-messages branch from 126c64f to bb0fd7c Compare December 14, 2025 17:21

link2xt force-pushed the pre-messages branch from bb0fd7c to a98fe05 Compare December 14, 2025 17:54

remove "Contact" stock string again, use emoji like in summaries

eb489bc

link2xt changed the title ~~[Branch] pre-messages / next version of download on demand~~ feat: pre-messages / next version of download on demand Dec 14, 2025

link2xt force-pushed the main branch from e0ae864 to 1ea8ed6 Compare December 14, 2025 19:44

Hocuri and others added 8 commits December 14, 2025 21:00

fix: Add additional_text also to the summary, in order to prevent emp…

78f884f

…ty summaries

refactor: Rename download_when_normal_starts->download_later

411126f

refactor: use more idiomatic way of creating Vec, and remove prematur…

96fa64d

…e optimization

improve error message

a8f0a6d

move download_msgs call

4458663

log request to download

60b28cb

rustfmt

53dc1c7

test: add pre-message test

73a2c94

Hocuri reviewed Dec 18, 2025

View reviewed changes

Hocuri added 2 commits December 18, 2025 18:28

Merge branch 'main' into pre-messages

3f5fc45

iequidoo reviewed Dec 22, 2025

View reviewed changes

iequidoo reviewed Dec 24, 2025

View reviewed changes

iequidoo reviewed Dec 27, 2025

View reviewed changes

iequidoo added 2 commits December 27, 2025 05:05

restore test_something.py::test_download_limit_chat_assignment

ee24466

The test just passes, it seems there's no need to remove it.

fix: Update message state on full download, recover test_msg_seen_on_…

fedd0b6

…imap_when_downloaded

iequidoo reviewed Dec 28, 2025

View reviewed changes

src/download.rs Outdated Show resolved Hide resolved

src/download.rs Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

iequidoo force-pushed the pre-messages branch from 13c8b29 to 4a119fc Compare December 29, 2025 08:02

Hocuri and others added 4 commits December 30, 2025 01:27

Add a replacement for test_partial_group_consistency checking that pr…

4c5ce60

…e-messages modify group members There are no partial messages anymore, so i have no idea what else to check.

Make DownloadLimit inclusive like it was before

447982e

review comments

84ce946

iequidoo force-pushed the pre-messages branch from 4a119fc to 84ce946 Compare December 30, 2025 04:30

iequidoo added 3 commits December 30, 2025 01:49

remove TODO

37f96d5

review suggestions

f438488

test_msg_text_on_lost_pre_msg

d8fcb3e

iequidoo reviewed Dec 30, 2025

View reviewed changes

Add test_pre_and_post_msgs_deleted

318c17d

It currently fails because pre-messages aren't deleted on IMAP.

		uids_fetch.push(uid);
		uid_message_ids.insert(uid, message_id);

Uh oh!

feat: pre-messages / next version of download on demand #7371

Are you sure you want to change the base?

feat: pre-messages / next version of download on demand #7371

Uh oh!

Conversation

Simon-Laux commented Oct 29, 2025 • edited by iequidoo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Currently removed tests

Progress of the tests

TODO Tests

postponed to do tests

discarded/ignored test ideas

Uh oh!

Uh oh!

Hocuri Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iequidoo Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iequidoo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iequidoo Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iequidoo Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iequidoo Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

iequidoo Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

iequidoo commented Dec 30, 2025

Uh oh!

iequidoo Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Simon-Laux commented Oct 29, 2025 •

edited by iequidoo

Loading

Hocuri Dec 18, 2025 •

edited

Loading

iequidoo Dec 20, 2025 •

edited

Loading

iequidoo Dec 24, 2025 •

edited

Loading

iequidoo Dec 27, 2025 •

edited

Loading