[script] find and display all tagged tables

[mllife]

can anyone help?

[michelcrypt4d4mus]

not sure what is even being asked here?

[mllife]

Can you help me analyze pdfs with tagged tables? https://accessible-pdf.info/en/basics/general/overview-of-the-pdf-tags/#table-elements

[michelcrypt4d4mus]

analyze how? have you run the pdfalyzer and looked at the output yet to see what you get?
there's no special table handling in pdfalyzer but everything in a PDF is made up of tags so you should be able to easily identify the tables if you know the relevant PDF tags.

[mllife]

Hello, I created a sample pdf with tagged Table using Foxit PDF.

And I see this table is appearing in the PDFalyzer
"""
│ │ ├── ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ ┃ 96./StructElem:Table ┃ /Root/StructTreeRoot/K[0]/K[4] ┃ Dictionary ┃
│ │ │ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ AddressInParent │ /K[4] │ │
│ │ │ │ /P │ <91:StructElem:Part(Dictionary)> │ Indirect │
│ │ │ │ /A │ {/BBox: [206.338, 422.665, 404.91, 472.478], /O: /Layout} │ Dictionary │
│ │ │ │ /S │ /Table │ Name │
│ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ /K │ [<97:StructElem:TR(Dictionary)>, <101:StructElem:TR(Dictionary)>, <105:StructElem:TR(Dictionary)>, │ Array │
│ │ │ │ │ <109:StructElem:TR(Dictionary)>] │ │
│ │ │ └──────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┘
│ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ ┃ 97./StructElem:TR ┃ /Root/StructTreeRoot/K[0]/K[4]/K[0] ┃ Dictionary ┃
│ │ │ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ AddressInParent │ /K[0] │ │
│ │ │ │ │ /P │ <96:StructElem:Table(Dictionary)> │ Indirect │
│ │ │ │ │ /S │ /TR │ Name │
│ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ /K │ [<98:StructElem:TH(Dictionary)>, <99:StructElem:TH(Dictionary)>, <100:StructElem:TH(Dictionary)>] │ Array │
│ │ │ │ └───────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┘
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 98./StructElem:TH ┃ /Root/StructTreeRoot/K[0]/K[4]/K[0]/K[0] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[0] │ │
│ │ │ │ │ │ /P │ <97:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /A │ {/Scope: /Column, /O: /Table} │ Dictionary │
│ │ │ │ │ │ /S │ /TH │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 4, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └───────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 99./StructElem:TH ┃ /Root/StructTreeRoot/K[0]/K[4]/K[0]/K[1] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[1] │ │
│ │ │ │ │ │ /P │ <97:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /A │ {/Scope: /Column, /O: /Table} │ Dictionary │
│ │ │ │ │ │ /S │ /TH │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 5, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └───────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 100./StructElem:TH ┃ /Root/StructTreeRoot/K[0]/K[4]/K[0]/K[2] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[2] │ │
│ │ │ │ │ │ /P │ <97:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /A │ {/Scope: /Column, /O: /Table} │ Dictionary │
│ │ │ │ │ │ /S │ /TH │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 6, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └────────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ ┃ 101./StructElem:TR ┃ /Root/StructTreeRoot/K[0]/K[4]/K[1] ┃ Dictionary ┃
│ │ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ AddressInParent │ /K[1] │ │
│ │ │ │ │ /P │ <96:StructElem:Table(Dictionary)> │ Indirect │
│ │ │ │ │ /S │ /TR │ Name │
│ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ /K │ [<102:StructElem:TD(Dictionary)>, <103:StructElem:TD(Dictionary)>, <104:StructElem:TD(Dictionary)>] │ Array │
│ │ │ │ └────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┘
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 102./StructElem:TD ┃ /Root/StructTreeRoot/K[0]/K[4]/K[1]/K[0] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[0] │ │
│ │ │ │ │ │ /P │ <101:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /S │ /TD │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 7, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └────────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 103./StructElem:TD ┃ /Root/StructTreeRoot/K[0]/K[4]/K[1]/K[1] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[1] │ │
│ │ │ │ │ │ /P │ <101:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /S │ /TD │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 8, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └────────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│ │ │ │ ├── ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
│ │ │ │ │ ┃ 104./StructElem:TD ┃ /Root/StructTreeRoot/K[0]/K[4]/K[1]/K[2] ┃ Dictionary ┃
│ │ │ │ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ │ │ │ │ │ AddressInParent │ /K[2] │ │
│ │ │ │ │ │ /P │ <101:StructElem:TR(Dictionary)> │ Indirect │
│ │ │ │ │ │ /S │ /TD │ Name │
│ │ │ │ │ │ /Pg │ <4:Page(Dictionary)> │ Indirect │
│ │ │ │ │ │ /Type │ /StructElem │ Name │
│ │ │ │ │ │ /K │ [{/MCID: 9, /Pg: <4:Page(Dictionary)>, /Type: /MCR}] │ Array │
│ │ │ │ │ └────────────────────┴──────────────────────────────────────────────────────┴────────────┘
│ │ │ │ │ └── ╭───────────────────────────────────────────────────╮
│ │ │ │ │ │ /Pg => <4:Page(Dictionary)> (Non Child Reference) │
│ │ │ │ │ ╰───────────────────────────────────────────────────╯
│
"""

So, my question is that I want to create a code that can help me extract the table.
as a markdown or json.

[mllife]

Do you provide any code interface to parse through this tree structure and then extract the data values inside each node; to build the whole "Table" by parsing the required nodes?

[michelcrypt4d4mus]

there's a section in the README about this but i suspect there are other tools that are probably better suited to extracting tables from PDFs.

[mllife]

@michelcrypt4d4mus , Can you suggest which tools, from what I have researched your library is the closest one, that helped me visualize what I need; I only have one issue how to see/fetch the "data values" in each node?

Also, I have a couple of follow up questions?

1. As I see the StructRootTree as a tree but very similar to some xml/html representation in fact; Can I just dump the whole grabbing the root to HTML (including values in each cell)?
2. Do nodes have some spatial information stored or linked with them? That is when I click the particular node in the Foxit app; it highlights the bounding box around it. Can, I extract the same information using your library to save this as a source link? (like page_number, bounding boxes of table and contained cells)

[michelcrypt4d4mus]

Can you suggest which tools

i don't really know but i feel like pdfminer might do something like that? just a name that came to mind, might be wrong but extracting tables from PDFs is definitely a thing data scientists do so some googling will probably get you somewhere.

be aware that pdfalyzer is mostly designed for analyzing/working with PDF metadata and not the actual data in the PDF. if you want to do anything with the data in the PDF (other than scan for YARA patterns and a couple other kinds of binary extractions) pdfalyzer is probably not the right tool.

As I see the StructRootTree as a tree but very similar to some xml/html representation in fact; Can I just dump the whole grabbing the root to HTML (including values in each cell)?

i'm not sure what you mean but pdfalyzer has an HTML output mode.

Do nodes have some spatial information stored or linked with them? That is when I click the particular node in the Foxit app; it highlights the bounding box around it. Can, I extract the same information using your library to save this as a source link? (like page_number, bounding boxes of table and contained cells)

All information about all nodes is part of the PDF metadata tree so you should see bounding box information. you may want to refer to the official PDF spec to find what the relevant tag is.