Error saving dashboard with an specific chart

[ziggekatten]

Scenario:

• Database: Postgres 16, Encoding UTF-8

• Superset 4.0.2

• An bar chart created with no issues, and the query also runs fine in SQL Lab
  

• Adding this chart to an existing or new blank dashboard gives an error in UI: Sorry, there was an error saving this dashboard: Fatal error

The traceback when saving the dashboard shows following error: 2024-11-19 15:03:27,014:ERROR:flask_appbuilder.api:'latin-1' codec can't encode character '\u2013' in position 399: ordinal not in range(256)

Changing dimension on the chart solves the problem. This is where it gets weird, as first of all, my database have encoding UTF-8, and the column used for dimension have no '\u2013'.

Where should I troubleshoot?

[dosubot[bot]]

I found a similar closed issue related to special characters in a CSV export, which might provide some insights: Special characters in a csv export. This issue was last updated on May 02, 2024 ^[1].

For your specific problem, the error message indicates that there is a character encoding issue involving the 'latin-1' codec, which cannot encode certain characters like '\u2013' (an en dash). Here are some steps you can take to troubleshoot:

1. Check Data Encoding: Ensure that all data being used in the chart and dashboard is correctly encoded in UTF-8. This includes any data transformations or external data sources.

2. Inspect Chart Configuration: Since changing the dimension resolves the issue, inspect the specific dimension causing the problem. Look for any special characters or transformations that might introduce encoding issues.

3. Review Superset Configuration: Verify if there are any settings in Superset that might be defaulting to 'latin-1' encoding. Although I couldn't find specific configuration options for character encoding in Superset, it's worth checking your Superset and database connection settings.

4. Database Connection: Ensure that the connection string and settings for your PostgreSQL database explicitly specify UTF-8 encoding.

5. Superset Logs: Review the Superset logs for any additional error messages or warnings that might provide more context about the encoding issue.

These steps should help you identify and resolve the encoding issue you're encountering.

To continue talking to Dosu, mention @dosu.

[ziggekatten]

Nope, that does not resolve anything, as I have gone through these before submitting this question. No latin-1 references in superset config. db-encoding is utf8.

Just to adjust my self, the dimension DO have the en dash in the data: "- Functionality – Incorrect". Now, creating the chart works fine and it visualizes the data correct and can be saved, but the problem is that I can't add it to any dashboard. It's when adding the chart to dashboard I get the error.

The only thing that I can't test is to enforce sqlalchemy to utf8 in db connection dialogue. Whatever I pass to psycopg2 it rejects. Example: {'client_encoding': 'UTF-8'} in engine parameters gives: ERROR: 'str' object does not support item assignment

[ziggekatten]

It seems that the problem is not the source database, but rather the superset backen db, as I checket with our admin, and for some reason the superset db used latin-1. Switching to utf8 solved the issue. Strange that unicode characters in data should impact the metadata database. Well, it works now

[ampil]

Hi @ziggekatten,
Can you please share how did you (your admin) switched the database to utf8?

I changed the connection line in config.py to:

SQLALCHEMY_DATABASE_URI = (
    f"""sqlite:///{os.path.join(DATA_DIR, "superset.db")}?check_same_thread=false&charset=utf8"""
)

But the issue still persists.