[bug]: Not sure how to recover from data corruption with RocksDB

[jhd85]

What happened?

I am running two instances of Stalwart on two different servers. For both I use RocksDB. I'm not sure when, but for a few months now it seems that one of my instances has data corruption in the database. Fortunately it has to do with the metrics. I run the enterprise edition and the only thing that does not seem to work are the metric dashboards (opening the metric dashboard also generates the ERROR below).

Don't think I can ever find out what caused it, but I do need to recover from it. Any suggestions?

Data corruption error:

2024-11-26T21:49:09Z ERROR Data corruption detected (store.data-corruption) causedBy = crates/store/src/write/key.rs:670, key = base64:AAAAAABmj89sAn95aC9gEJA=, causedBy = crates/store/src/dispatch/store.rs:125, causedBy = crates/smtp/src/reporting/scheduler.rs:137, details = Failed to read from store

How can we reproduce the problem?

Not sure.

Version

v0.10.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

LDAP

What operating system are you using?

Linux

Relevant log output

2024-11-26T21:49:09Z ERROR Data corruption detected (store.data-corruption) causedBy = crates/store/src/write/key.rs:670, key = base64:AAAAAABmj89sAn95aC9gEJA=, causedBy = crates/store/src/dispatch/store.rs:125, causedBy = crates/smtp/src/reporting/scheduler.rs:137, details = Failed to read from store

Code of Conduct

• I agree to follow this project's Code of Conduct

[mdecimus]

Have you upgraded Stalwart recently and forgot to follow the upgrading instructions?
If not this might be an old bug related to outgoing reports, you can use the console to delete the broken report.

[lic0san]

I am experiencing the same error, which appears periodically in the logs :
causedBy = crates/store/src/write/key.rs:670, key = base64:AAAAAABmmO1gAoPtCFKqQCA=, causedBy = crates/store/src/dispatch/store.rs:125, causedBy = crates/smtp/src/reporting/scheduler.rs:137, details = Failed to read from store

I carefully observed the upgrading instruction for each upgrade.

I am unsure how to isolate any broken report (nothing stands out in DMARC or TLS Aggregates).

[mdecimus]

Steps to fix this error:

1. Download this Python script

import base64
import sys

def prepend_and_encode(char, base64_string):
    # Decode the base64 string
    decoded = base64.b64decode(base64_string)
    
    # Prepend the character to the decoded bytes
    prepended = char.encode('ascii') + decoded
    
    # Encode the result back to base64
    result = base64.b64encode(prepended).decode('ascii')
    
    return result

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: python script.py <ascii_char> <base64_string>")
        sys.exit(1)
    
    char = sys.argv[1]
    base64_string = sys.argv[2]
    
    if not char.isalpha() or len(char) != 1:
        print("Error: First argument must be a single alphabetic character")
        sys.exit(1)
    
    try:
        result = prepend_and_encode(char, base64_string)
        print(result)
    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)

2. Obtain the full key name by running ./python h <key_name>, for example ./python h AAAAAABmmO1gAoPtCFKqQCA=
3. Stop Stalwart.
4. Start the console /opt/stalwart-mail/bin/stalwart-mail --config=/opt/stalwart-mail/etc/config.toml --console
5. Delete the key name returned by the script using the delete <full_key_name> command, for example delete aAAAAAABmmO1gAoPtCFKqQCA=
6. Exit the console.
7. Start Stalwart

[mdecimus]

Sorry, it should be:

> delete base64:KEY_NAME_FROM_PYTHON

You might see errors for other keys, if there are too may you can delete the entire reports table as follows:

> delete h\x00\x00\x00 h\x00\x00\xff 

[lic0san]

Indeed, this updated command appeared to work. I had to repeatedly delete 7 keys (after each deletion, the logs showed another problematic base64 key) until the error finally vanished. It also led to a great number of dmarc activity, which I suspect was held back by the data corruption for weeks...

[mdecimus]

Just to clarify, although the error says data corruption, this error occurred because some versions ago the incoming reports were incorrectly stored in the outgoing reports table.

[lic0san]

Thank you for your support and clarification !

[jhd85]

I ended up deleting the whole reports table. No more errors, thanks!