Facing Issues after applying redactions they delete some Image or Icons

[aleem75321]

Description of the bug

I am trying to change the colour of the dot in the PDF. For this, I have applied redaction on the old bindi and added the new bindi with the desired colour.

Problem:-

• they deleted some Images or icons from the pages
• It also changes the colour of some images automatically.
• He is also changing the space between the words like I have not Annotated

ScreenShort:-

1. . Problem Delete object

screenshot before applying redaction

screenshot after applying redaction (I have circled the area where the change in the Images has been deleted)

2. Problem Image distribution
screenshot before applying redaction (I have circled the area where the change the Image)

screenshot after applying redaction (I have circled the area where the change the Image)

3. . Problem text distribution

screenshot before applying redaction (I have circled the area where the change the text )

screenshot after applying redaction (I have circled the area where the change the text )

How to reproduce the bug

import fitz
from pathlib import Path


file_path=Path(r"test_pages/test.pdf")
# file_path=Path(r"out/MT#MTMumbaiBS#18-04-2024#Mumbai#1#MTM9#4.pdf")
doc=fitz.open(file_path)
page=doc[0]


blocks=page.get_text("rawdict",flags=fitz.TEXTFLAGS_TEXT,sort=True)["blocks"]  
#Set Colour for outoput PDF
Red = fitz.pdfcolor["red"]

for b in  blocks:
    for l in b["lines"]:  
        for s in l["spans"]:
            for c in s["chars"]:
                if s["size"]>15 and s['color']==2236191: 
                    if c['c']== "ं":
                        try:
                            font = fitz.Font(fontname=s['font'],fontfile=f"{s['font']}.ttf")  
                        except Exception as e:
                            print(str(e))  
                        redact_box = fitz.Rect(c["bbox"]) 
                        origin_text = fitz.Point(c["origin"]) 
                        redact_box.y1 = redact_box.y1-s['size'] 
                        page.add_redact_annot(redact_box) 
                        # Apply reactions after all text replacements
                        page.apply_redactions(0,0,0)
                       
                        # Create Text writer to Write in Page with choose Color
                        tw = fitz.TextWriter(page.rect,color=Red)  
                        #re-insert same text - different color
                        tw.append((origin_text.x,origin_text.y), text=c['c'],fontsize=s['size'],font=font)
                        tw.write_text(page) 

#Saving Backup File furture use 
out_fpath="OUT/"+file_path.stem+".pdf"
doc.save(out_fpath,garbage=3, deflate=True)
doc.close()

Original PDFS:-

test.pdf
test2.pdf

After applying PDFS:-
test_after_apply_redaction.pdf
test2_after_apply_redaction.pdf

PyMuPDF version

1.24.2

Operating system

Windows

Python version

3.11

[JorjMcKie]

Sorry, what I meant was this:

1. add redaction annotations as desired
2. save as a new PDF without applying redactions!
3. open the just saved PDF and apply redactions.
4. save the result as another PDF.
5. only if PDF from step 4 shows more deletions than the crossed-out red rectangles of PDF from step 2 indicate, we have a potential problem.

[aleem75321]

As you said above, I have done all the steps and found that there is no change in the PDF even though they are not deleting the crossed-out red rectangle

It is interesting when I apply redaction to code some changes in pdf like the dot colour and one image colour also. but if follow the above steps nothing changes

MY question:- According to the documentation if we set the Image, Graphic and Text values to zero it should ignore any overlapping pixels. So why doesn't that parameter work?

test2_Redacted.pdf
test2.pdf

[JorjMcKie]

We are talking past each other!
What I said was: generate a PDF with crossed out redaction rectangles.
Then apply redactions and look at the result.
Your two most recent examples look equal. What is this telling me?

• That you did not mark redactions at all?

Previously you said that more stuff was removed than intended. So what is the relationship here?

• Have you ever checked the character bbox? If it is an empty rectangle, a redaction can never be successful - respectively can have unpredictable effects.
• Replacing (resp. inserting) text in Devanagari (or another complex scripting system) with a character-based output method like fill_textbox() will likely produce unreadable results. Devanagari can reliably only written using a method with "text shaping" support, like insert_htmlbox.

[aleem75321]

Yes, we have discussed this before, let me tell you that I have resolved many issues. Below is the complete code I use in production.

I would like to focus your attention on this first and second image where they removed the icon image (Ganesh photo removed) and filled red color in place of my ad in the second image.

So I think there is some issue with apply_reaction. Even though I am not creating any reduction box at the place where the icon is, still it is getting deleted.

import fitz
from pathlib import Path
file_path=Path(r"test_pages/test.pdf")
# file_path=Path(r"out/MT#MTMumbaiBS#18-04-2024#Mumbai#1#MTM9#4.pdf")
doc=fitz.open(file_path)
page=doc[0]
#Set Colour for outoput PDF
Red = fitz.pdfcolor["red"]
# Black = fitz.pdfcolor["black"]

page.clean_contents(sanitize=True)

# Extracting text Form PDF 
fitz.TOOLS.set_small_glyph_heights(True)          
count=0

current_char = None # Update previous character for next iteration
prev_char = None # Update previous character for next iteration
pre_prev_char=None
pre_pre_prev_char=None

char_box=fitz.Rect.is_empty
prev_bbox= fitz.Rect.is_empty
prev_prev_bbox= fitz.Rect.is_empty
pre_pre_prev_bbox= fitz.Rect.is_empty

char_origin=fitz.Rect.is_empty
prev_char_origin=fitz.Rect.is_empty
prev_prev_char_origin=fitz.Rect.is_empty
pre_pre_prev_origin=fitz.Rect.is_empty
pre_pre_prev_origin=fitz.Rect.is_empty


blocks=page.get_text("rawdict",flags=fitz.TEXTFLAGS_TEXT,sort=True)["blocks"]  

for b in  blocks:
    for l in b["lines"]:  
        for s in l["spans"]:
            if s["size"]>15 and s['color']==2236191: 
                #Defining Font Here and path of font 
                try:
                    font = fitz.Font(fontname=s['font'],fontfile=f"{s['font']}.ttf")  # this must be known somehow - or simply try some font else
                except Exception as e:
                    print(str(e))  

                Black = fitz.sRGB_to_pdf(s["color"])  # re-use original color
                # Create Text writer to Write in Page with choose Color
                tw = fitz.TextWriter(page.rect,color=Red)              
                tw1 = fitz.TextWriter(page.rect,color=Black)              
                
                # Accurate Bounding Box Calculation
                a = s["ascender"]
                d = s["descender"]              
             
                for char in s["chars"]:
                    redact_box = fitz.Rect(char["bbox"]) 
                    origin_text = fitz.Point(char["origin"])  # origin text  location 

                    # Update previous words
                    pre_pre_pre_prev_char=pre_pre_prev_char
                    pre_pre_prev_char=pre_prev_char
                    pre_prev_char = prev_char
                    prev_char = current_char
                    current_char = char["c"]

                    pre_pre_pre_prev_bbox=pre_pre_prev_bbox
                    pre_pre_prev_bbox=prev_prev_bbox
                    prev_prev_bbox=prev_bbox
                    prev_bbox=char_box
                    char_box=redact_box
                    
                    pre_pre_pre_prev_origin=pre_pre_prev_origin
                    pre_pre_prev_origin=prev_prev_char_origin
                    prev_prev_char_origin=prev_char_origin
                    prev_char_origin=char_origin
                    char_origin=origin_text
                   
                    font_size=((origin_text.y - s["size"] * d / (a - d))+a)-(origin_text.y - s["size"] * d / (a - d) - s["size"]+d)
                    
                    for k,c in char.items():
                        if type(c)==str:
                            # Single condition for red dot (anusvara)
                            if c== "ं":
                                buffer = 5  # Adjust as needed
                                redact_box.y0 = redact_box.y1 - redact_box.height + buffer  # Adjust top
                                redact_box.y1 += buffer  # Adjust bottom

                                prev_redact_box = fitz.Rect(prev_bbox) 
                                prev_char_origin = fitz.Point(prev_char_origin) 

                                applied_condition=False                                  
                                
                                if not applied_condition and prev_char=="ै" or prev_char=="ु" or prev_char=="े": 

                                    redact_box.y1 = redact_box.y1-s['size'] 
                                    page.add_redact_annot(redact_box)   
                                    # Apply reactions after all text replacements   
                                    page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE,graphics=fitz.PDF_REDACT_LINE_ART_NONE)   
                                
                                    #re-insert same text - different color
                                    tw.append((origin_text.x+a-d,origin_text.y),text=c,font=font,fontsize=font_size,language="mr")
                                    tw1.append((prev_char_origin.x-0.5,prev_char_origin.y),text=prev_char,font=font,fontsize=font_size,language="mr")
                                    tw.write_text(page)
                                    tw1.write_text(page)
                                    applied_condition=True                                    
                                
                                elif not applied_condition  and (pre_pre_prev_char =='र' and  pre_prev_char=='्') or (pre_pre_pre_prev_char =='र' and  pre_pre_prev_char=='्') :
                                    print(pre_pre_pre_prev_char,pre_pre_pre_prev_bbox,pre_pre_prev_char,pre_pre_prev_bbox,pre_prev_char,prev_prev_bbox,prev_char,prev_bbox,char['c'],char_box)
                                 
                                    applied_condition=True
                                    
                                    
                                elif not applied_condition and prev_char=="ा": 
                      
                                    redact_box.y0=redact_box.y0+7
                                    redact_box.y1 = redact_box.y1-s['size']+7
                                    page.add_redact_annot(redact_box)

                                    # Apply reactions after all text replacements   
                                    page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE,graphics=fitz.PDF_REDACT_LINE_ART_NONE)    
                                    #re-insert same text - different color
                                    tw.append((origin_text.x,origin_text.y),text=c,font=font,fontsize=font_size,language="mr")
                                    tw.write_text(page)
                                    applied_condition=True
                            
                                elif not applied_condition and prev_char=="ी" or prev_char=="ो" or prev_char=="ौ":
                                 
                                    prev_redact_box.y1 = prev_redact_box.y1-s['size']          
                                    page.add_redact_annot(prev_redact_box)
                                    # Apply reactions after all text replacements   
                                    page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE,graphics=fitz.PDF_REDACT_LINE_ART_NONE)  
                                    #re-insert same text - different color
                                    tw.append((origin_text.x+a-d,origin_text.y),text=c,font=font,fontsize=font_size,language="mr")
                                    tw1.append((prev_char_origin.x,prev_char_origin.y-0.01),text=prev_char,font=font,fontsize=s['size'],language="mr")
                                    tw1.write_text(page)
                                    tw.write_text(page)
                                    applied_condition=True

                                else: 
                                    if not applied_condition:
                                        # print(prev_char,prev_bbox,char['c'],char_box)
                                        redact_box.y1 = redact_box.y1-s['size']                                    
                                        page.add_redact_annot(redact_box)
                                        # Apply reactions after all text replacements   
                                        page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE,graphics=fitz.PDF_REDACT_LINE_ART_NONE)  
                                        #re-insert same text - different colour
                                        tw.append((origin_text.x,origin_text.y), text=c, font=font,fontsize=s['size'],language="mr")
                                        tw.write_text(page) 
                                        applied_condition=True   

#Saving Backup File future use 
out_fpath="OUT/"+file_path.stem+".pdf"
doc.save(out_fpath,garbage=3, deflate=True)
doc.close()

[JorjMcKie]

Again: please let me have a file that has visible, crossed-out redaction annotations - with apply_redacions() must not have been executed yet.
Whether there is a bug or not must be based on that PDF. There is no way to demystify 150+ lines of your code for this.

[aleem75321]

For you to understand the issue better. I have the same Advt in English PDF also. I have given that to you also. I know this issue is happening because of this Advt.

But my question is that before applying redactions, I have given image, text and graphics value "page.apply_redactions(0,0,0)" but still it is touching the image while doing any redaction. there is no box

this file is without applying redaction
test_without_redaction.pdf

english_without_redaction.pdf

[JorjMcKie]

Thanks for supplying the examples.
The English document indeed misbehaves in that legal redactions annotations remove unintended page areas. This may be a problem duplication (issue #3376) for which a fix is pending publication.
The other document contains redactions annotations all with empty redaction rectangles. This is currently not properly intercepted / rejected.
I will submit a report to our MuPDF colleagues.

[aleem75321]

@JorjMcKie Still have the same problem when applying _redaction after updating pymupdf to ('1.24.3', '1.24.2', '202405090000001'). I have checked with the same English PDF and the same old problem persists. He still touched that image with no close to any comments.

supplying same Eample PDF to you
PDF without applying redaction.
english.pdf

Applying redaction pDF
english_apply_redaction.pdf