data_extraction/notebooks/pdf_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PDF Extraction\n",
    "\n",
    "This notebook shows examples of text extraction from pdf files with different packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- 1. [Non OCR extraction methods](#toc1_)    \n",
    "  - 1.1. [Pypdf2         ](#toc1_1_)    \n",
    "  - 1.2. [Pymupdf         ](#toc1_2_)    \n",
    "  - 1.3. [Fitz         ](#toc1_3_)    \n",
    "  - 1.4. [Unstructured         ](#toc1_4_)    \n",
    "    - 1.4.1. [Unstructured local pdf loader](#toc1_4_1_)    \n",
    "    - 1.4.2. [Unstructured api loader](#toc1_4_2_)    \n",
    "- 2. [OCR and table extraction methods](#toc2_)    \n",
    "  - 2.1. [Unstructured Pytesseract loader](#toc2_1_)    \n",
    "  - 2.2. [Paddle OCR loader](#toc2_2_)    \n",
    "- 3. [Evaluate loaded docs by embedding similarity](#toc3_)    \n",
    "  - 3.1. [Embedding & Storage](#toc3_1_)    \n",
    "  - 3.2. [Similarity search](#toc3_2_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=true\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "current_dir = os.getcwd()\n",
    "kit_dir = os.path.abspath(os.path.join(current_dir, \"..\"))\n",
    "repo_dir = os.path.abspath(os.path.join(kit_dir, \"..\"))\n",
    "\n",
    "sys.path.append(kit_dir)\n",
    "sys.path.append(repo_dir)\n",
    "\n",
    "import glob\n",
    "from dotenv import load_dotenv\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. <a id='toc1_'></a>[Non OCR extraction methods](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide location of pdf files\n",
    "folder_loc = os.path.join(kit_dir,'data/sample_data/sample_pdfs/')\n",
    "pdf_files = [f for f in glob.glob(f'{folder_loc}/*.pdf')]\n",
    "sample_pdf = pdf_files[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load text splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "        # Set a small chunk size, just to make splitting evident.\n",
    "        chunk_size = 500,\n",
    "        chunk_overlap  = 100,\n",
    "        length_function = len,\n",
    "        separators = [\"\\n\\n\\n\",\"\\n\\n\", \"\\n\", \".\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. <a id='toc1_1_'></a>[Pypdf2](https://pypdf2.readthedocs.io/en/3.0.0/)          [&#8593;](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accelerate Data-Driven \n",
      "Decision-Making with \n",
      "State-of-the-Art AI\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These \n",
      "challenges include:\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like \n",
      "Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a\n",
      "---\n",
      "solution to this challenge and it’s time that companies meet this need.\n",
      "  01 Copyright © 2021 SambaNova Systems, Inc. All rights reserved.ROADBLOCKS TO INNOVATION\n",
      "UNLOCKING THE FUTURE\n",
      "SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution \n",
      "computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use,\n",
      "---\n",
      "SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house \n",
      "infrastructure and machine learning expertise. \n",
      "Aﬀordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution \n",
      "or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with \n",
      "pretrained models that can save signiﬁcant time for customers.HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "---\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive \n",
      "today as the advent of the internet a few decades ago. And like the internet—AI promises decisive \n",
      "competitive and operational advantages to organizations that can leverage it for innovation sooner \n",
      "rather than later.\n",
      "Dataﬂow-as-a-ServiceTM \n",
      "Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst \n",
      "subscription-based AI acceleration services \n",
      "platform that enables enterprises to jump-start AI\n",
      "---\n",
      "subscription-based AI acceleration services \n",
      "platform that enables enterprises to jump-start AI \n",
      "initiatives by leveraging SambaNova’s expertise to \n",
      "augment their existing ML capabilities and staﬃng. \n",
      "Dataﬂow-as-a-service provides comprehensive \n",
      "API services and models for businesses to deploy \n",
      "fully managed AI solutions within weeks as \n",
      "opposed to the typical 12-18 months.\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable \n",
      "Dataﬂow Architecture, DataScale® is an\n",
      "---\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable \n",
      "Dataﬂow Architecture, DataScale® is an \n",
      "integrated AI so/f_tware and hardware \n",
      "accelerator that delivers unrivaled performance, \n",
      "accuracy, scale, and ease of use. Optimized \n",
      "from algorithms to silicon, DataScale delivers \n",
      "eﬃciency with a so/f_tware-deﬁned-hardware \n",
      "approach and a highly ﬂexible modular \n",
      "architecture. DataScale can scale seamlessly \n",
      "from one to hundreds of systems to meet the \n",
      "demands of modern AI computing. OUR PRODUCTS\n",
      "---\n",
      "from one to hundreds of systems to meet the \n",
      "demands of modern AI computing. OUR PRODUCTS\n",
      "Computer Vision\n",
      "Natural Language \n",
      "Processing Recommendation\n",
      "SystemsDataScale®\n",
      "ONE PLATFORM\n",
      "YOUR CHOICE OF DEPLOYMENT\n",
      "SambaNova managedDATAFLOW-AS-A-SERVICETMDATASCALE®\n",
      "Customer managed\n",
      "Optimized pretrained models Build your own models\n",
      "Ability to ﬁne-tune with your custom data Your data\n",
      "Opex Opex + Capex• Critical skill gaps due to machine learning talent scarcity\n",
      "---\n",
      "Opex Opex + Capex• Critical skill gaps due to machine learning talent scarcity\n",
      "• Lack of expertise in computing architectures\n",
      "• Diﬃculty in keeping on top of latest models and techniques\n",
      "• Investment justiﬁcation without proof of prior impact\n",
      "---\n",
      "Financial Services \n",
      "Compliance, risk mitigation, \n",
      "algorithmic trading\n",
      "  02 Copyright © 2021 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153”We’re interested in technology that can train models faster than individual GPUs, that has \n",
      "more scalability and has more performance and so the SambaNova DataScale System \n",
      "met that criteria.”\n",
      "---\n",
      "met that criteria.”\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, \n",
      "natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship oﬀering, \n",
      "Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational eﬃciency.\n",
      "---\n",
      "SambaNova’s DataScale® is an integrated so/f_tware and hardware system using Reconﬁgurable Dataﬂow Architecture (RDA), along with \n",
      "open standards and user interfaces. Headquartered in Palo Alto, California, SambaNova Systems was founded in 2017 by industry \n",
      "luminaries, hardware, and so/f_tware design experts from Sun/Oracle and Stanford University. Investors include So/f_tBank Vision Fund 2, funds\n",
      "---\n",
      "and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC, Redline Capital, Atlantic Bridge Ventures, \n",
      "Celesta, and several others.AI Use Cases\n",
      "Across Industries\n",
      "RICK STEVENS\n",
      "ASSOCIATE LABORATORY DIRECTOR\n",
      "ARGONNE NATIONAL LABORATORY\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at  \n",
      "SambaNova.AI.DON’T JUST TAKE OUR WORD FOR ITHealthcare & Life Sciences\n",
      "Genomics, drug discovery,\n",
      "---\n",
      "SambaNova.AI.DON’T JUST TAKE OUR WORD FOR ITHealthcare & Life Sciences\n",
      "Genomics, drug discovery, \n",
      "telehealth, disease diagnosisManufacturing & Automotive \n",
      "Digital transformation, predictive \n",
      "maintenance, visual inspection\n",
      "Oil & Gas, Energy \n",
      "Predictive asset maintenance, \n",
      "seismic processing, explorationRetail & E-Commerce\n",
      "Digital acceleration, \n",
      "personalized services deliveryPublic Sector \n",
      "Intelligent decision-making, \n",
      "ISR, predictive analytics\n",
      "---\n",
      "Intelligent decision-making, \n",
      "ISR, predictive analytics\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen \n",
      "access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. \n",
      "Together, we are enabling ScaleWorX end-users access to disruptive and innovative \n",
      "solutions that can quickly generate new revenue opportunities with AI.”\n",
      "JUN MATSUURA\n",
      "MANAGING DIRECTOR\n",
      "---\n",
      "JUN MATSUURA\n",
      "MANAGING DIRECTOR\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of \n",
      "our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared \n",
      "to a comparable GPU running the same models.”\n",
      "BRONIS DE SUPINSKI,\n",
      "CHIEF TECHNOLOGY OFFICER\n",
      "LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "https://sambanova.ai/\n",
      "mailto:info@sambanova.ai\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "https://twitter.com/SambaNovaAI\n",
      "https://www.linkedin.com/company/sambanova/\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(sample_pdf)\n",
    "docs_pypdf2 = loader.load_and_split(text_splitter = text_splitter)\n",
    "\n",
    "for doc in docs_pypdf2:\n",
    "    print(f'{doc.page_content}\\n---')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. <a id='toc1_2_'></a>[Pymupdf](https://pymupdf.readthedocs.io/en/latest/)          [&#8593;](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accelerate Data-Driven \n",
      "Decision-Making with \n",
      "State-of-the-Art AI\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These \n",
      "challenges include:\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like \n",
      "Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a\n",
      "---\n",
      "solution to this challenge and it’s time that companies meet this need.\n",
      "  01\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved.\n",
      "ROADBLOCKS TO INNOVATION\n",
      "UNLOCKING THE FUTURE\n",
      "SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution \n",
      "computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use,\n",
      "---\n",
      "SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house \n",
      "infrastructure and machine learning expertise. \n",
      "Afordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution \n",
      "or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with \n",
      "pretrained models that can save signiﬁcant time for customers.\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "---\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive \n",
      "today as the advent of the internet a few decades ago. And like the internet—AI promises decisive \n",
      "competitive and operational advantages to organizations that can leverage it for innovation sooner \n",
      "rather than later.\n",
      "Dataﬂow-as-a-ServiceTM \n",
      "Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst \n",
      "subscription-based AI acceleration services\n",
      "---\n",
      "Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst \n",
      "subscription-based AI acceleration services \n",
      "platform that enables enterprises to jump-start AI \n",
      "initiatives by leveraging SambaNova’s expertise to \n",
      "augment their existing ML capabilities and stafng. \n",
      "Dataﬂow-as-a-service provides comprehensive \n",
      "API services and models for businesses to deploy \n",
      "fully managed AI solutions within weeks as \n",
      "opposed to the typical 12-18 months.\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable\n",
      "---\n",
      "opposed to the typical 12-18 months.\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable \n",
      "Dataﬂow Architecture, DataScale® is an \n",
      "integrated AI sofware and hardware \n",
      "accelerator that delivers unrivaled performance, \n",
      "accuracy, scale, and ease of use. Optimized \n",
      "from algorithms to silicon, DataScale delivers \n",
      "efciency with a sofware-deﬁned-hardware \n",
      "approach and a highly ﬂexible modular \n",
      "architecture. DataScale can scale seamlessly \n",
      "from one to hundreds of systems to meet the\n",
      "---\n",
      "architecture. DataScale can scale seamlessly \n",
      "from one to hundreds of systems to meet the \n",
      "demands of modern AI computing. \n",
      "OUR PRODUCTS\n",
      "Computer Vision\n",
      "Natural Language \n",
      "Processing \n",
      "Recommendation\n",
      "Systems\n",
      "DataScale®\n",
      "ONE PLATFORM\n",
      "YOUR CHOICE OF DEPLOYMENT\n",
      "SambaNova managed\n",
      "DATAFLOW-AS-A-SERVICETM\n",
      "DATASCALE®\n",
      "Customer managed\n",
      "Optimized pretrained models\n",
      "Build your own models\n",
      "Ability to ﬁne-tune with your custom data\n",
      "Your data\n",
      "Opex\n",
      "Opex + Capex\n",
      "---\n",
      "Build your own models\n",
      "Ability to ﬁne-tune with your custom data\n",
      "Your data\n",
      "Opex\n",
      "Opex + Capex\n",
      "• Critical skill gaps due to machine learning talent scarcity\n",
      "• Lack of expertise in computing architectures\n",
      "• Difculty in keeping on top of latest models and techniques\n",
      "• Investment justiﬁcation without proof of prior impact\n",
      "---\n",
      "Financial Services \n",
      "Compliance, risk mitigation, \n",
      "algorithmic trading\n",
      "  02\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153\n",
      "”We’re interested in technology that can train models faster than individual GPUs, that has \n",
      "more scalability and has more performance and so the SambaNova DataScale System \n",
      "met that criteria.”\n",
      "---\n",
      "met that criteria.”\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, \n",
      "natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship ofering, \n",
      "Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efciency.\n",
      "---\n",
      "SambaNova’s DataScale® is an integrated sofware and hardware system using Reconﬁgurable Dataﬂow Architecture (RDA), along with \n",
      "open standards and user interfaces. Headquartered in Palo Alto, California, SambaNova Systems was founded in 2017 by industry \n",
      "luminaries, hardware, and sofware design experts from Sun/Oracle and Stanford University. Investors include SofBank Vision Fund 2, funds\n",
      "---\n",
      "and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC, Redline Capital, Atlantic Bridge Ventures, \n",
      "Celesta, and several others.\n",
      "AI Use Cases\n",
      "Across Industries\n",
      "RICK STEVENS\n",
      "ASSOCIATE LABORATORY DIRECTOR\n",
      "ARGONNE NATIONAL LABORATORY\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at  \n",
      "SambaNova.AI.\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "Healthcare & Life Sciences\n",
      "Genomics, drug discovery, \n",
      "telehealth, disease diagnosis\n",
      "---\n",
      "Healthcare & Life Sciences\n",
      "Genomics, drug discovery, \n",
      "telehealth, disease diagnosis\n",
      "Manufacturing & Automotive \n",
      "Digital transformation, predictive \n",
      "maintenance, visual inspection\n",
      "Oil & Gas, Energy \n",
      "Predictive asset maintenance, \n",
      "seismic processing, exploration\n",
      "Retail & E-Commerce\n",
      "Digital acceleration, \n",
      "personalized services delivery\n",
      "Public Sector \n",
      "Intelligent decision-making, \n",
      "ISR, predictive analytics\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen\n",
      "---\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen \n",
      "access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. \n",
      "Together, we are enabling ScaleWorX end-users access to disruptive and innovative \n",
      "solutions that can quickly generate new revenue opportunities with AI.”\n",
      "JUN MATSUURA\n",
      "MANAGING DIRECTOR\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of\n",
      "---\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of \n",
      "our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared \n",
      "to a comparable GPU running the same models.”\n",
      "BRONIS DE SUPINSKI,\n",
      "CHIEF TECHNOLOGY OFFICER\n",
      "LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "https://sambanova.ai/\n",
      "mailto:info@sambanova.ai\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "https://twitter.com/SambaNovaAI\n",
      "https://www.linkedin.com/company/sambanova/\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import PyMuPDFLoader\n",
    "\n",
    "loader = PyMuPDFLoader(sample_pdf)\n",
    "docs_pymupdf = loader.load_and_split(text_splitter = text_splitter)\n",
    "\n",
    "for doc in docs_pymupdf:\n",
    "    print(f'{doc.page_content}\\n---')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. <a id='toc1_3_'></a>[Fitz](https://pymupdf.readthedocs.io/en/latest/module.html)          [&#8593;](#toc0_)\n",
    "\n",
    "Pymupdf uses the fitz package underneath but using Fitz provides more flexibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accelerate Data-Driven \n",
      "Decision-Making with \n",
      "State-of-the-Art AI\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive \n",
      "today as the advent of the internet a few decades ago. And like the internet—AI promises decisive \n",
      "competitive and operational advantages to organizations that can leverage it for innovation sooner \n",
      "rather than later.\n",
      "ROADBLOCKS TO INNOVATION\n",
      "---\n",
      "rather than later.\n",
      "ROADBLOCKS TO INNOVATION\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These \n",
      "challenges include:\n",
      "• Critical skill gaps due to machine learning talent scarcity\n",
      "• Lack of expertise in computing architectures\n",
      "• Difculty in keeping on top of latest models and techniques\n",
      "• Investment justiﬁcation without proof of prior impact\n",
      "UNLOCKING THE FUTURE\n",
      "---\n",
      "• Investment justiﬁcation without proof of prior impact\n",
      "UNLOCKING THE FUTURE\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like \n",
      "Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a \n",
      "solution to this challenge and it’s time that companies meet this need.\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "---\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution \n",
      "computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, \n",
      "SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house \n",
      "infrastructure and machine learning expertise.\n",
      "---\n",
      "infrastructure and machine learning expertise. \n",
      "Afordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution \n",
      "or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with \n",
      "pretrained models that can save signiﬁcant time for customers.\n",
      "OUR PRODUCTS\n",
      "Dataﬂow-as-a-ServiceTM \n",
      "Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst \n",
      "subscription-based AI acceleration services\n",
      "---\n",
      "Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst \n",
      "subscription-based AI acceleration services \n",
      "platform that enables enterprises to jump-start AI \n",
      "initiatives by leveraging SambaNova’s expertise to \n",
      "augment their existing ML capabilities and stafng. \n",
      "Dataﬂow-as-a-service provides comprehensive \n",
      "API services and models for businesses to deploy \n",
      "fully managed AI solutions within weeks as \n",
      "opposed to the typical 12-18 months.\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable\n",
      "---\n",
      "opposed to the typical 12-18 months.\n",
      "DataScale® \n",
      "Built on SambaNova Systems’ Reconﬁgurable \n",
      "Dataﬂow Architecture, DataScale® is an \n",
      "integrated AI sofware and hardware \n",
      "accelerator that delivers unrivaled performance, \n",
      "accuracy, scale, and ease of use. Optimized \n",
      "from algorithms to silicon, DataScale delivers \n",
      "efciency with a sofware-deﬁned-hardware \n",
      "Computer Vision\n",
      "Natural Language \n",
      "Processing \n",
      "Recommendation\n",
      "Systems\n",
      "DataScale®\n",
      "DATAFLOW-AS-A-SERVICETM\n",
      "DATASCALE®\n",
      "Optimized pretrained models\n",
      "---\n",
      "Recommendation\n",
      "Systems\n",
      "DataScale®\n",
      "DATAFLOW-AS-A-SERVICETM\n",
      "DATASCALE®\n",
      "Optimized pretrained models\n",
      "Build your own models\n",
      "ONE PLATFORM\n",
      "YOUR CHOICE OF DEPLOYMENT\n",
      "SambaNova managed\n",
      "Customer managed\n",
      "---\n",
      "AI Use Cases\n",
      "Across Industries\n",
      "Financial Services \n",
      "Compliance, risk mitigation, \n",
      "algorithmic trading\n",
      "Oil & Gas, Energy \n",
      "Predictive asset maintenance, \n",
      "seismic processing, exploration\n",
      "Healthcare & Life Sciences\n",
      "Genomics, drug discovery, \n",
      "telehealth, disease diagnosis\n",
      "Retail & E-Commerce\n",
      "Digital acceleration, \n",
      "personalized services delivery\n",
      "Manufacturing & Automotive \n",
      "Digital transformation, predictive \n",
      "maintenance, visual inspection\n",
      "Public Sector \n",
      "Intelligent decision-making,\n",
      "---\n",
      "maintenance, visual inspection\n",
      "Public Sector \n",
      "Intelligent decision-making, \n",
      "ISR, predictive analytics\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "”We’re interested in technology that can train models faster than individual GPUs, that has \n",
      "more scalability and has more performance and so the SambaNova DataScale System \n",
      "met that criteria.”\n",
      "RICK STEVENS\n",
      "ASSOCIATE LABORATORY DIRECTOR\n",
      "ARGONNE NATIONAL LABORATORY\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen\n",
      "---\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen \n",
      "access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. \n",
      "Together, we are enabling ScaleWorX end-users access to disruptive and innovative \n",
      "solutions that can quickly generate new revenue opportunities with AI.”\n",
      "JUN MATSUURA\n",
      "MANAGING DIRECTOR\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of\n",
      "---\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of \n",
      "our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared \n",
      "to a comparable GPU running the same models.”\n",
      "BRONIS DE SUPINSKI,\n",
      "CHIEF TECHNOLOGY OFFICER\n",
      "LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at  \n",
      "SambaNova.AI.\n",
      "https://sambanova.ai/\n",
      "mailto:info@sambanova.ai\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "---\n",
      "SambaNova.AI.\n",
      "https://sambanova.ai/\n",
      "mailto:info@sambanova.ai\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "https://twitter.com/SambaNovaAI\n",
      "https://www.linkedin.com/company/sambanova/\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, \n",
      "natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship ofering,\n",
      "---\n",
      "Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efciency.\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "import fitz\n",
    "from src.multi_column import column_boxes\n",
    "from langchain.schema import Document\n",
    "\n",
    "docs = fitz.open(sample_pdf)\n",
    "\n",
    "docs_fitz=[]\n",
    "for page, page in enumerate(docs):\n",
    "    full_text = ''\n",
    "    bboxes = column_boxes(page, footer_margin=100, no_image_text=True)\n",
    "    for rect in bboxes:\n",
    "        full_text += page.get_text(clip=rect, sort=True)\n",
    "    metadata={\"source\":sample_pdf}\n",
    "    doc = Document(page_content=full_text, metadata=metadata)\n",
    "    docs_fitz.append(doc)\n",
    "docs_fitz = text_splitter.split_documents(docs_fitz)\n",
    "for doc in docs_fitz:\n",
    "    print(f'{doc.page_content}\\n---')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4. <a id='toc1_4_'></a>[Unstructured](https://unstructured.io/)          [&#8593;](#toc0_)\n",
    "\n",
    "The Unstructured package offers text extraction capabilities from various document types and seamlessly integrates with [Langchain]((https://python.langchain.com/docs/integrations/document_loaders/unstructured_file)).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.1. <a id='toc1_4_1_'></a>[Unstructured local pdf loader](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accelerate Data-Driven Decision-Making with State-of-the-Art AI\n",
      "\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive today as the advent of the internet a few decades ago. And like the internet—AI promises decisive competitive and operational advantages to organizations that can leverage it for innovation sooner rather than later.\n",
      "\n",
      "ROADBLOCKS TO INNOVATION\n",
      "---\n",
      "ROADBLOCKS TO INNOVATION\n",
      "\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These challenges include:\n",
      "\n",
      "Critical skill gaps due to machine learning talent scarcity • Lack of expertise in computing architectures • Difficulty in keeping on top of latest models and techniques • Investment justiﬁcation without proof of prior impact\n",
      "\n",
      "UNLOCKING THE FUTURE\n",
      "---\n",
      "UNLOCKING THE FUTURE\n",
      "\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a solution to this challenge and it’s time that companies meet this need.\n",
      "---\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house infrastructure and machine learning expertise.\n",
      "---\n",
      "Affordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with pretrained models that can save signiﬁcant time for customers.\n",
      "---\n",
      "OUR PRODUCTS Dataﬂow-as-a-ServiceTM Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst subscription-based AI acceleration services platform that enables enterprises to jump-start AI initiatives by leveraging SambaNova’s expertise to augment their existing ML capabilities and staffing. Dataﬂow-as-a-service provides comprehensive API services and models for businesses to deploy fully managed AI solutions within weeks as opposed to the typical 12-18 months.\n",
      "\n",
      "Natural Language Processing\n",
      "---\n",
      "Natural Language Processing\n",
      "\n",
      "Computer Vision\n",
      "\n",
      "DataScale®\n",
      "\n",
      "Recommendation Systems\n",
      "---\n",
      "Recommendation Systems\n",
      "\n",
      "DataScale® Built on SambaNova Systems’ Reconﬁgurable Dataﬂow Architecture, DataScale® is an integrated AI software and hardware accelerator that delivers unrivaled performance, accuracy, scale, and ease of use. Optimized from algorithms to silicon, DataScale delivers efficiency with a software-deﬁned-hardware approach and a highly ﬂexible modular architecture. DataScale can scale seamlessly from one to hundreds of systems to meet the demands of modern AI computing.\n",
      "---\n",
      "ONE PLATFORM\n",
      "\n",
      "YOUR CHOICE OF DEPLOYMENT\n",
      "\n",
      "DATAFLOW-AS-A-SERVICETM\n",
      "\n",
      "DATASCALE®\n",
      "\n",
      "SambaNova managed\n",
      "\n",
      "Customer managed\n",
      "\n",
      "Optimized pretrained models\n",
      "\n",
      "Build your own models\n",
      "\n",
      "Ability to ﬁne-tune with your custom data\n",
      "\n",
      "Your data\n",
      "\n",
      "Opex\n",
      "\n",
      "Opex + Capex\n",
      "\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved.\n",
      "\n",
      "01\n",
      "\n",
      "AI Use Cases Across Industries\n",
      "\n",
      "Financial Services Compliance, risk mitigation, algorithmic trading\n",
      "\n",
      "Healthcare & Life Sciences Genomics, drug discovery, telehealth, disease diagnosis\n",
      "---\n",
      "Healthcare & Life Sciences Genomics, drug discovery, telehealth, disease diagnosis\n",
      "\n",
      "Manufacturing & Automotive Digital transformation, predictive maintenance, visual inspection\n",
      "\n",
      "Oil & Gas, Energy Predictive asset maintenance, seismic processing, exploration\n",
      "\n",
      "Retail & E-Commerce Digital acceleration, personalized services delivery\n",
      "\n",
      "Public Sector Intelligent decision-making, ISR, predictive analytics\n",
      "\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "---\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "\n",
      "”We’re interested in technology that can train models faster than individual GPUs, that has more scalability and has more performance and so the SambaNova DataScale System met that criteria.”\n",
      "\n",
      "RICK STEVENS ASSOCIATE LABORATORY DIRECTOR ARGONNE NATIONAL LABORATORY\n",
      "\n",
      "https://sambanova.ai/\n",
      "---\n",
      "RICK STEVENS ASSOCIATE LABORATORY DIRECTOR ARGONNE NATIONAL LABORATORY\n",
      "\n",
      "https://sambanova.ai/\n",
      "\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. Together, we are enabling ScaleWorX end-users access to disruptive and innovative solutions that can quickly generate new revenue opportunities with AI.”\n",
      "\n",
      "mailto:info@sambanova.ai\n",
      "\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "---\n",
      "mailto:info@sambanova.ai\n",
      "\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "\n",
      "https://twitter.com/SambaNovaAI\n",
      "\n",
      "https://www.linkedin.com/company/sambanova/\n",
      "\n",
      "JUN MATSUURA MANAGING DIRECTOR\n",
      "\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared to a comparable GPU running the same models.”\n",
      "\n",
      "BRONIS DE SUPINSKI, CHIEF TECHNOLOGY OFFICER LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "---\n",
      "BRONIS DE SUPINSKI, CHIEF TECHNOLOGY OFFICER LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at SambaNova.AI.\n",
      "---\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship offering, Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efficiency\n",
      "---\n",
      ". SambaNova’s DataScale® is an integrated software and hardware system using Reconﬁgurable Dataﬂow Architecture (RDA), along with open standards and user interfaces. Headquartered in Palo Alto, California, SambaNova Systems was founded in 2017 by industry luminaries, hardware, and software design experts from Sun/Oracle and Stanford University\n",
      "---\n",
      ". Investors include SoftBank Vision Fund 2, funds and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC, Redline Capital, Atlantic Bridge Ventures, Celesta, and several others.\n",
      "---\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153\n",
      "\n",
      "02\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import UnstructuredPDFLoader\n",
    "\n",
    "loader = UnstructuredPDFLoader(sample_pdf)\n",
    "docs_unstructured = loader.load_and_split(text_splitter = text_splitter)\n",
    "\n",
    "for doc in docs_unstructured:\n",
    "    print(f\"{doc.page_content}\\n---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.2. <a id='toc1_4_2_'></a>[Unstructured api loader](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accelerate Data-Driven Decision-Making with State-of-the-Art AI\n",
      "\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive today as the advent of the internet a few decades ago. And like the internet—AI promises decisive competitive and operational advantages to organizations that can leverage it for innovation sooner rather than later.\n",
      "\n",
      "ROADBLOCKS TO INNOVATION\n",
      "---\n",
      "ROADBLOCKS TO INNOVATION\n",
      "\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These challenges include:\n",
      "\n",
      "Critical skill gaps due to machine learning talent scarcity • Lack of expertise in computing architectures • Difficulty in keeping on top of latest models and techniques • Investment justiﬁcation without proof of prior impact\n",
      "\n",
      "UNLOCKING THE FUTURE\n",
      "---\n",
      "UNLOCKING THE FUTURE\n",
      "\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a solution to this challenge and it’s time that companies meet this need.\n",
      "---\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house infrastructure and machine learning expertise.\n",
      "---\n",
      "Affordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with pretrained models that can save signiﬁcant time for customers.\n",
      "---\n",
      "OUR PRODUCTS Dataﬂow-as-a-ServiceTM Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst subscription-based AI acceleration services platform that enables enterprises to jump-start AI initiatives by leveraging SambaNova’s expertise to augment their existing ML capabilities and staffing. Dataﬂow-as-a-service provides comprehensive API services and models for businesses to deploy fully managed AI solutions within weeks as opposed to the typical 12-18 months.\n",
      "\n",
      "Natural Language Processing\n",
      "---\n",
      "Natural Language Processing\n",
      "\n",
      "Computer Vision\n",
      "\n",
      "DataScale®\n",
      "\n",
      "Recommendation Systems\n",
      "---\n",
      "Recommendation Systems\n",
      "\n",
      "DataScale® Built on SambaNova Systems’ Reconﬁgurable Dataﬂow Architecture, DataScale® is an integrated AI software and hardware accelerator that delivers unrivaled performance, accuracy, scale, and ease of use. Optimized from algorithms to silicon, DataScale delivers efficiency with a software-deﬁned-hardware approach and a highly ﬂexible modular architecture. DataScale can scale seamlessly from one to hundreds of systems to meet the demands of modern AI computing.\n",
      "---\n",
      "ONE PLATFORM\n",
      "\n",
      "YOUR CHOICE OF DEPLOYMENT\n",
      "\n",
      "DATAFLOW-AS-A-SERVICETM\n",
      "\n",
      "DATASCALE®\n",
      "\n",
      "SambaNova managed\n",
      "\n",
      "Customer managed\n",
      "\n",
      "Optimized pretrained models\n",
      "\n",
      "Build your own models\n",
      "\n",
      "Ability to ﬁne-tune with your custom data\n",
      "\n",
      "Your data\n",
      "\n",
      "Opex\n",
      "\n",
      "Opex + Capex\n",
      "\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved.\n",
      "\n",
      "01\n",
      "\n",
      "AI Use Cases Across Industries\n",
      "\n",
      "Financial Services Compliance, risk mitigation, algorithmic trading\n",
      "\n",
      "Healthcare & Life Sciences Genomics, drug discovery, telehealth, disease diagnosis\n",
      "---\n",
      "Healthcare & Life Sciences Genomics, drug discovery, telehealth, disease diagnosis\n",
      "\n",
      "Manufacturing & Automotive Digital transformation, predictive maintenance, visual inspection\n",
      "\n",
      "Oil & Gas, Energy Predictive asset maintenance, seismic processing, exploration\n",
      "\n",
      "Retail & E-Commerce Digital acceleration, personalized services delivery\n",
      "\n",
      "Public Sector Intelligent decision-making, ISR, predictive analytics\n",
      "\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "---\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "\n",
      "”We’re interested in technology that can train models faster than individual GPUs, that has more scalability and has more performance and so the SambaNova DataScale System met that criteria.”\n",
      "\n",
      "RICK STEVENS ASSOCIATE LABORATORY DIRECTOR ARGONNE NATIONAL LABORATORY\n",
      "\n",
      "https://sambanova.ai/\n",
      "---\n",
      "RICK STEVENS ASSOCIATE LABORATORY DIRECTOR ARGONNE NATIONAL LABORATORY\n",
      "\n",
      "https://sambanova.ai/\n",
      "\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. Together, we are enabling ScaleWorX end-users access to disruptive and innovative solutions that can quickly generate new revenue opportunities with AI.”\n",
      "\n",
      "mailto:info@sambanova.ai\n",
      "\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "---\n",
      "mailto:info@sambanova.ai\n",
      "\n",
      "https://www.facebook.com/SambaNovaAI/\n",
      "\n",
      "https://twitter.com/SambaNovaAI\n",
      "\n",
      "https://www.linkedin.com/company/sambanova/\n",
      "\n",
      "JUN MATSUURA MANAGING DIRECTOR\n",
      "\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared to a comparable GPU running the same models.”\n",
      "\n",
      "BRONIS DE SUPINSKI, CHIEF TECHNOLOGY OFFICER LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "---\n",
      "BRONIS DE SUPINSKI, CHIEF TECHNOLOGY OFFICER LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at SambaNova.AI.\n",
      "---\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship offering, Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efficiency\n",
      "---\n",
      ". SambaNova’s DataScale® is an integrated software and hardware system using Reconﬁgurable Dataﬂow Architecture (RDA), along with open standards and user interfaces. Headquartered in Palo Alto, California, SambaNova Systems was founded in 2017 by industry luminaries, hardware, and software design experts from Sun/Oracle and Stanford University\n",
      "---\n",
      ". Investors include SoftBank Vision Fund 2, funds and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC, Redline Capital, Atlantic Bridge Ventures, Celesta, and several others.\n",
      "---\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153\n",
      "\n",
      "02\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import UnstructuredAPIFileLoader\n",
    "# register at Unstructured.io to get a free API Key\n",
    "load_dotenv(os.path.join(repo_dir,'.env'))\n",
    "loader = UnstructuredAPIFileLoader(sample_pdf, \n",
    "                                   api_key=os.environ.get('UNSTRUCTURED_API_KEY'),\n",
    "                                   url =os.environ.get('UNSTRUCTURED_URL'))\n",
    "docs_unstructured_api = loader.load_and_split(text_splitter = text_splitter)\n",
    "for doc in docs_unstructured_api:\n",
    "    print(f\"{doc.page_content}\\n---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. <a id='toc2_'></a>[OCR and table extraction methods](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. <a id='toc2_1_'></a>[Unstructured Pytesseract loader](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For runing this loader you should install the pyteseract and poppler-utils packages in your machine, or run this notebook over the data_extarction docker container\n",
    "\n",
    "This loader uses behind the scenes Unstructured and pytesseract module to perform a layout detection, then transcribe text, and tables as Html tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jorgep/Documents/ask_public_own/test_env/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Some weights of the model checkpoint at microsoft/table-transformer-structure-recognition were not used when initializing TableTransformerForObjectDetection: ['model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked']\n",
      "- This IS expected if you are initializing TableTransformerForObjectDetection from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TableTransformerForObjectDetection from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "commercial and public sector industries and is as disruptive decades ago. And like the internet—Al promises decisive to organizations that can leverage it for innovation sooner\n",
      "\n",
      "Accelerate Data-Driven Decision-Making with State-of-the-Art AI\n",
      "\n",
      "AI is increasingly being adopted across commercial and public sector industries and is as disruptive today as the advent of the internet a few decades ago. And like the internet—AI promises decisive competitive and operational advantages to organizations that can leverage it for innovation sooner rather than later.\n",
      "\n",
      "ROADBLOCKS TO INNOVATION\n",
      "\n",
      "Today, all but the big tech giants face seemingly insurmountable obstacles to bring AI applications into production. These challenges include:\n",
      "\n",
      "Critical skill gaps due to machine learning talent scarcity • Lack of expertise in computing architectures • Difficulty in keeping on top of latest models and techniques • Investment justiﬁcation without proof of prior impact\n",
      "\n",
      "UNLOCKING THE FUTURE\n",
      "\n",
      "The AI talent shortage continues to burden companies across all industries. And the reality is, unless you’re a company like Google or Facebook, you’ll be hard-pressed to attract top talent, resulting in a subpar AI solution. It’s time you demand a solution to this challenge and it’s time that companies meet this need.\n",
      "\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house infrastructure and machine learning expertise.\n",
      "\n",
      "Affordable and attainable, SambaNova’s AI platform can be delivered as a CapEx-modeled rack Infrastructure solution or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely conﬁgured with pretrained models that can save signiﬁcant time for customers.\n",
      "\n",
      "OUR PRODUCTS Dataﬂow-as-a-ServiceTM Dataﬂow-as-a-ServiceTM is the industry’s ﬁrst subscription-based AI acceleration services platform that enables enterprises to jump-start AI initiatives by leveraging SambaNova’s expertise to augment their existing ML capabilities and staffing. Dataﬂow-as-a-service provides comprehensive API services and models for businesses to deploy fully managed AI solutions within weeks as opposed to the typical 12-18 months.\n",
      "\n",
      "Computer Vision\n",
      "\n",
      "Natural Language Processing\n",
      "\n",
      "Recommendation Systems\n",
      "\n",
      "DataScale®\n",
      "\n",
      "ONE PLATFORM\n",
      "\n",
      "DataScale® Built on SambaNova Systems’ Reconﬁgurable Dataﬂow Architecture, DataScale® is an integrated AI software and hardware accelerator that delivers unrivaled performance, accuracy, scale, and ease of use. Optimized from algorithms to silicon, DataScale delivers efficiency with a software-deﬁned-hardware approach and a highly ﬂexible modular architecture. DataScale can scale seamlessly from one to hundreds of systems to meet the demands of modern AI computing.\n",
      "\n",
      "<table><tr><td>DATAFLOW-AS-A-SERVICE™</td><td>DATASCALE®</td></tr><tr><td>SambaNova managed</td><td>Customer managed</td></tr><tr><td>Optimized pretrained models</td><td>Build your own models</td></tr><tr><td>Ability to fine-tune with your custom data</td><td>Your data</td></tr><tr><td>Opex</td><td>Opex + Capex</td></tr></table>\n",
      "\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved.\n",
      "\n",
      "01\n",
      "\n",
      "eau 10 (Oro)\n",
      "\n",
      "AX\n",
      "\n",
      "DataScales\n",
      "\n",
      "AI Use Cases Across Industries\n",
      "\n",
      "Manufacturing & Automotive Digital transformation, predictive maintenance, visual inspection\n",
      "\n",
      "Healthcare & Life Sciences Genomics, drug discovery, telehealth, disease diagnosis\n",
      "\n",
      "Financial Services Compliance, risk mitigation, algorithmic trading\n",
      "\n",
      "Oil & Gas, Energy Predictive asset maintenance, seismic processing, exploration\n",
      "\n",
      "Public Sector Intelligent decision-making, ISR, predictive analytics\n",
      "\n",
      "Retail & E-Commerce Digital acceleration, personalized services delivery\n",
      "\n",
      "DON’T JUST TAKE OUR WORD FOR IT\n",
      "\n",
      "”We’re interested in technology that can train models faster than individual GPUs, that has more scalability and has more performance and so the SambaNova DataScale System met that criteria.”\n",
      "\n",
      "RICK STEVENS ASSOCIATE LABORATORY DIRECTOR ARGONNE NATIONAL LABORATORY\n",
      "\n",
      "”In this partnership with SambaNova, we’re providing businesses with never-before-seen access to artiﬁcial intelligence that will certainly lead to greater adoption of AI in Asia. Together, we are enabling ScaleWorX end-users access to disruptive and innovative solutions that can quickly generate new revenue opportunities with AI.”\n",
      "\n",
      "JUN MATSUURA MANAGING DIRECTOR\n",
      "\n",
      "”The support for running multiple models built into the hardware was a signiﬁcant aspect of our selection of SambaNova Systems. We’re seeing roughly a 5x improvement compared to a comparable GPU running the same models.”\n",
      "\n",
      "BRONIS DE SUPINSKI, CHIEF TECHNOLOGY OFFICER LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "\n",
      "To learn how your organization can innovate with next-generation AI, please visit us at SambaNova.AI.\n",
      "\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship offering, Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efficiency. SambaNova’s DataScale® is an integrated software and hardware system using Reconﬁgurable Dataﬂow Architecture (RDA), along with open standards and user interfaces. Headquartered in Palo Alto, California, SambaNova Systems was founded in 2017 by industry luminaries, hardware, and software design experts from Sun/Oracle and Stanford University. Investors include SoftBank Vision Fund 2, funds and accounts managed by BlackRock, Intel Capital, GV, Walden International, Temasek, GIC, Redline Capital, Atlantic Bridge Ventures, Celesta, and several others.\n",
      "\n",
      "Copyright © 2021 SambaNova Systems, Inc. All rights reserved. | Palo Alto, CA and Austin, TX | (650) 263-1153\n",
      "\n",
      "02\n",
      "---\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from data_extraction.src.pdf_table_text_extraction import UnstructuredPdfPytesseractLoader\n",
    "\n",
    "loader = UnstructuredPdfPytesseractLoader(sample_pdf)\n",
    "docs=loader.load()\n",
    "\n",
    "for doc in docs:\n",
    "    print(f'{doc.page_content}\\n---\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. <a id='toc2_2_'></a>[Paddle OCR loader](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For runing this loader you should run this notebook over the paddle-ocr environment the data_extarction_paddel docker container\n",
    "\n",
    "This loader uses behind the scenes Paddle OCR and Paddle Structure modules to perform a layout detection, then mask images and equations, transcribe text, and tables as Html tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2024/02/06 15:47:02] ppocr DEBUG: Namespace(help='==SUPPRESS==', use_gpu=False, use_xpu=False, ir_optim=True, use_tensorrt=False, min_subgraph_size=15, shape_info_filename=None, precision='fp32', gpu_mem=500, image_dir=None, det_algorithm='DB', det_model_dir='/Users/jorgep/.paddleocr/whl/det/en/en_PP-OCRv3_det_infer', det_limit_side_len=960, det_limit_type='max', det_db_thresh=0.3, det_db_box_thresh=0.6, det_db_unclip_ratio=1.5, max_batch_size=10, use_dilation=False, det_db_score_mode='fast', det_east_score_thresh=0.8, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_sast_score_thresh=0.5, det_sast_nms_thresh=0.2, det_sast_polygon=False, det_pse_thresh=0, det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_box_type='quad', det_pse_scale=1, scales=[8, 16, 32], alpha=1.0, beta=1.0, fourier_degree=5, det_fce_box_type='poly', rec_algorithm='SVTR_LCNet', rec_model_dir='/Users/jorgep/.paddleocr/whl/rec/en/en_PP-OCRv3_rec_infer', rec_image_shape='3, 48, 320', rec_batch_num=6, max_text_length=25, rec_char_dict_path='/Users/jorgep/Documents/ask_public_own/paddle_env/lib/python3.10/site-packages/paddleocr/ppocr/utils/en_dict.txt', use_space_char=True, vis_font_path='./doc/fonts/simfang.ttf', drop_score=0.5, e2e_algorithm='PGNet', e2e_model_dir=None, e2e_limit_side_len=768, e2e_limit_type='max', e2e_pgnet_score_thresh=0.5, e2e_char_dict_path='./ppocr/utils/ic15_dict.txt', e2e_pgnet_valid_set='totaltext', e2e_pgnet_mode='fast', use_angle_cls=False, cls_model_dir='/Users/jorgep/.paddleocr/whl/cls/ch_ppocr_mobile_v2.0_cls_infer', cls_image_shape='3, 48, 192', label_list=['0', '180'], cls_batch_num=6, cls_thresh=0.9, enable_mkldnn=False, cpu_threads=10, use_pdserving=False, warmup=False, sr_model_dir=None, sr_image_shape='3, 32, 128', sr_batch_num=1, draw_img_save_dir='./inference_results', save_crop_res=False, crop_res_save_dir='./output', use_mp=False, total_process_num=1, process_id=0, benchmark=False, save_log_path='./log_output/', show_log=True, use_onnx=False, output='./output', table_max_len=488, table_algorithm='TableAttn', table_model_dir=None, merge_no_span_structure=True, table_char_dict_path=None, layout_model_dir=None, layout_dict_path=None, layout_score_threshold=0.5, layout_nms_threshold=0.5, kie_algorithm='LayoutXLM', ser_model_dir=None, ser_dict_path='../train_data/XFUND/class_list_xfun.txt', ocr_order_method=None, mode='structure', image_orientation=False, layout=True, table=True, ocr=True, recovery=False, save_pdf=False, lang='en', det=True, rec=True, type='ocr', ocr_version='PP-OCRv3', structure_version='PP-Structurev2')\n",
      "True\n",
      "True\n",
      "True\n",
      "**figure l**\n",
      "Al is increasingly being adopted across commercial and public sector industries and is as disruptive\n",
      "today as the advent of the internet a few decades ago.And like the internet-Al promises decisive\n",
      "competitive and operational advantages to organizations that can leverage it for innovation sooner\n",
      "rather than later.\n",
      "Today,all but the big tech giants face seemingly insurmountable obstacles to bring Al applications into production.These\n",
      "challenges include:\n",
      "Critical skill gaps due to machine learning talent scarcity\n",
      "Lack of expertise in computing architectures\n",
      "Difficulty in keeping on top of latest models and technigues\n",
      "Investment justification without proof of prior impact\n",
      "UNLOCKING THE FUTURE\n",
      "The Al talent shortage continues to burden companies across all industries. And the reality is, unless you're a company lik\n",
      "Google or Facebook,you'll be hard-pressed to attract top talent,resulting in a subpar Al solution.It's time you demand\n",
      "solution to this challenge and it's time that companies meet this need.\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "SambaNova enables organizations to build and deploy Al solutions for natural language processing,high-resolution\n",
      "computer vision,and recommendation.With state-of-the-art accuracy,unmatched scalability,and ease of use\n",
      "SambaNova delivers Al capabilities at a fraction of the time and expense it takes to develop complex in-house\n",
      "infrastructure and machine learning expertise\n",
      "Affordable and attainable, SambaNova's Al platform can be delivered as a CapEx-modeled rack Infrastructure solution\n",
      "or an OpEx modeled subscription service that is fully managed by SambaNova and comes completely configured with\n",
      "pretrained models that can save significant time for customers.\n",
      "OUR PRODUCTS\n",
      "Dataflow-as-a-ServiceTM\n",
      "Dataflow-as-a-ServiceTM is the industry's first\n",
      "subscripfion-basedAlacceleration services\n",
      "platform that enables enterprises to jump-start Al\n",
      "initiatives by leveraging SambaNova's expertise to\n",
      "augment their existing ML capabilities and staffing\n",
      "Dataflow-as-a-service provides comprehensive\n",
      "API services and models for businesses to deploy\n",
      "fully managed Al solutions within weeks as\n",
      "opposed to the typical 12-18 months.\n",
      "DataScale\n",
      "../data/extraction/SambaNova_Accelerate-Data-Driven-Decision-Making/SambaNova_Accelerate-Data-Driven-Decision-Making_page_0_figure_0.jpg\n",
      "ght 2021 SambaNova Systems, Inc.All rights reserved.\n",
      "ONE PLATFORM\n",
      "01\n",
      "---\n",
      "\n",
      "Al Use Cases\n",
      "Across Industries\n",
      "../data/extraction/SambaNova_Accelerate-Data-Driven-Decision-Making/SambaNova_Accelerate-Data-Driven-Decision-Making_page_1_figure_0.jpg\n",
      "../data/extraction/SambaNova_Accelerate-Data-Driven-Decision-Making/SambaNova_Accelerate-Data-Driven-Decision-Making_page_1_figure_2.jpg\n",
      "DON'T JUSTTAKEOURWORD FORIT\n",
      "'re interested in technology that can train models faster than individual Gp\n",
      "re scalability and has more performance and so the SambaNova DataSco\n",
      "t that criteria.\"\n",
      "In this partnership with sambaNova,we're providing businesses with never-before-seen\n",
      "access to artificial intelligence thatwill certainlylead to greaferadoption ofAl in Asia\n",
      "Together, we are enabling Scaleworx end-users access to disruptive and innovative\n",
      "solutions that can quickly generate new revenue opportunities with Al.'\n",
      "JUN MATSUURA\n",
      "MANAGING DIRECTOR\n",
      "The support for running multiple models built into the hardware was a significant aspect of\n",
      "our selection of SambaNova Systems.We're seeing roughly a 5x improvement compared\n",
      "to a comparable GPUrunning the same models.\n",
      "BRONIS DE SUPINSKI,\n",
      "CHIEF TECHNOLOGY OFFICER\n",
      "LAWRENCE LIVERMORE NATIONAL LABORATORY\n",
      "earn how your organization can innovate with next-generation Al, please vi\n",
      "nbaNova.Al\n",
      "iny\n",
      "SSambaNova\n",
      "../data/extraction/SambaNova_Accelerate-Data-Driven-Decision-Making/SambaNova_Accelerate-Data-Driven-Decision-Making_page_1_figure_1.jpg\n",
      "RICK STEVENS\n",
      "ASSOCIATE LABORATORY DIRECTOR\n",
      "ARGONNE NATIONALLABORATORY\n",
      "---\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from data_extraction.src.multi_column_ocr import PaddleOCRLoader\n",
    "\n",
    "loader = PaddleOCRLoader(sample_pdf, output_folder=os.path.join(kit_dir,'data/extraction'))\n",
    "docs=loader.load()\n",
    "\n",
    "for doc in docs:\n",
    "    print(f'{doc.page_content}\\n---\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. <a id='toc3_'></a>[Evaluate loaded docs by embedding similarity](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. <a id='toc3_1_'></a>[Embedding & Storage](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load INSTRUCTOR_Transformer\n",
      "max_seq_length  512\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings import HuggingFaceInstructEmbeddings\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "encode_kwargs = {'normalize_embeddings': True}\n",
    "embd_model = HuggingFaceInstructEmbeddings( model_name='intfloat/e5-large-v2',\n",
    "                                            embed_instruction=\"\", # no instructions needed for candidate passages\n",
    "                                            query_instruction=\"Represent this sentence for searching relevant passages: \",\n",
    "                                            encode_kwargs=encode_kwargs)\n",
    "vectorstore_pypdf2 = FAISS.from_documents(documents=docs_pypdf2, embedding=embd_model)\n",
    "vectorstore_pymupdf = FAISS.from_documents(documents=docs_pymupdf, embedding=embd_model)\n",
    "vectorstore_fitz = FAISS.from_documents(documents=docs_fitz, embedding=embd_model)\n",
    "vectorstore_unstructured_local = FAISS.from_documents(documents=docs_unstructured, embedding=embd_model)\n",
    "vectorstore_unstructured_api = FAISS.from_documents(documents=docs_unstructured_api, embedding=embd_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. <a id='toc3_2_'></a>[Similarity search](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------PyPDF2 Loader----------\n",
      "\n",
      "met that criteria.”\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, \n",
      "natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship oﬀering, \n",
      "Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational eﬃciency.\n",
      "--------PyMuPDF Loader------------\n",
      "\n",
      "met that criteria.”\n",
      "SambaNova Systems is an AI innovation company that empowers organizations to deploy best-in-class solutions for computer vision, \n",
      "natural language processing, recommendation systems, and AI for science with conﬁdence. SambaNova’s ﬂagship ofering, \n",
      "Dataﬂow-as-a-Service, helps organizations rapidly deploy AI in days, unlocking new revenue and boosting operational efciency.\n",
      "--------Fitz loader------------\n",
      "\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI\n",
      "SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution \n",
      "computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, \n",
      "SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house \n",
      "infrastructure and machine learning expertise.\n",
      "-------Unstructured local Loader----------\n",
      "\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house infrastructure and machine learning expertise.\n",
      "--------Unstructured api loader------------\n",
      "\n",
      "HARNESS THE POWER OF NEXT-GENERATION AI SambaNova enables organizations to build and deploy AI solutions for natural language processing, high-resolution computer vision, and recommendation. With state-of-the-art accuracy, unmatched scalability, and ease of use, SambaNova delivers AI capabilities at a fraction of the time and expense it takes to develop complex in-house infrastructure and machine learning expertise.\n"
     ]
    }
   ],
   "source": [
    "query = \"what is the valuie offer of sambanova?\"\n",
    "\n",
    "ans = vectorstore_pypdf2.similarity_search(query)\n",
    "print(\"-------PyPDF2 Loader----------\\n\")\n",
    "print(ans[0].page_content)\n",
    "\n",
    "\n",
    "ans_2 = vectorstore_pymupdf.similarity_search(query)\n",
    "print(\"--------PyMuPDF Loader------------\\n\")\n",
    "print(ans_2[0].page_content)\n",
    "\n",
    "\n",
    "ans_3 = vectorstore_fitz.similarity_search(query)\n",
    "print(\"--------Fitz loader------------\\n\")\n",
    "print(ans_3[0].page_content)\n",
    "\n",
    "ans_4 = vectorstore_unstructured_local.similarity_search(query)\n",
    "print(\"-------Unstructured local Loader----------\\n\")\n",
    "print(ans_4[0].page_content)\n",
    "\n",
    "\n",
    "ans_5 = vectorstore_unstructured_api.similarity_search(query)\n",
    "print(\"--------Unstructured api loader------------\\n\")\n",
    "print(ans_5[0].page_content)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "8ef1dd803998dd3041a093f0eac89949e22b02a495b5f7c82a2465356c114caf"
  },
  "kernelspec": {
   "display_name": "Python 3.10.0 ('pocs')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}