Regarding some issues with reading mathematical symbols.

[ishidai]

Description of the bug

test.pdf
cmex10.ttf.zip

`import fitz # PyMuPDF

def normalize_color(color):
r = (color >> 16) & 255
g = (color >> 8) & 255
b = color & 255
return (r / 255.0, g / 255.0, b / 255.0)

input_pdf_path = "test.pdf"
input_pdf = fitz.open(input_pdf_path)

output_pdf = fitz.open()

for page_num in range(input_pdf.page_count):
page = input_pdf.load_page(page_num)
blocks = page.get_text("dict")["blocks"]

new_page = output_pdf.new_page(width=page.rect.width, height=page.rect.height)

for block in blocks:
    if block["type"] == 0:
        for line in block["lines"]:
            for span in line["spans"]:
                text = span["text"]
                font_size = span["size"]
                font = span["font"]
                color = normalize_color(span["color"])
                origin = span["origin"]
                font_path = None
                print(f'text:{text}, font:{font}')
                if font == "CMEX10":
                    # TODO test this font
                    font_path = "cmex10.ttf"
                else:
                    font = "helv"
                new_page.insert_text(
                    origin,
                    text,
                    fontfile=font_path,
                    fontsize=font_size,
                    fontname=font,
                    color=color
                )

output_pdf_path = "new.pdf"
output_pdf.save(output_pdf_path)

print(f"saved: {output_pdf_path}")
`

I provided the original document and the corresponding mathematical symbol font cmex10.ttf, but you can see that when I insert it, it is still garbled.

I hope can help me analyze the reason. Thank you.

How to reproduce the bug

Simply run the code, original document, and font file that I provided.

PyMuPDF version

1.24.7

Operating system

MacOS

Python version

3.11

[JorjMcKie]

This is no issue / bug but a Discussions item -> transferring.

[JorjMcKie]

Where did you get the font from? If you extracted it from the PDF, then the sad news is that there is no guarantee that it is usable as you want it to be.
To confirm, try to output a few mathematical symbols with it in an extra little script and see if the expected output is happening.

[ishidai]

Sorry, I cannot output symbols from this font file. The font file source is: https://www.ctan.org/tex-archive/fonts/cm/ps-type1/bakoma/ttf/. Could you please advise on the best approach to read the mathematical symbols from the test.pdf file and write them separately?

[JorjMcKie]

so the font is not really usable. There is no way but go find a suitable one. On one of the font download sites, maybe also Google's NOTO font library. Try them out by writing your desired symbols with each of them...

[ishidai]

I extracted the font file from the PDF document (see the attached package) and matched the corresponding font file to the traversed font attribute for text insertion. However, some symbols are still missing. I noticed through the log that the text contains content like text="\x10", which is missing when inserted. The font is "CMEX10".
code.zip

So, how should I handle this better?

[JorjMcKie]

I cannot give you more advice here. If text extraction delivers a \x10, then that is all that can ever be known. If the font in question has proper documentation, then maybe there is some mapping information as to which Unicode name \x10 is pointing to, which you could use subsequently.
Otherwise ... bad luck!

I think I mentioned that fonts in general may not be error-free nor complete in terms of these back-references.

[JorjMcKie]

In any case, these are problems related to the specific PDF and / or font and definitely outside PyMuPDF.

[ishidai]

Thank you for the suggestion. Currently, I'm resolving it by matching the existing text information to the corresponding Unicode or glyph name information in the font file. @JorjMcKie