# Overview

This example project illustrates how to perform various tasks on documents with pre-trained machine learning models. It covers the following tasks:

- **Optical Character Recognition (OCR)**: The conversion of images of text (e.g. scanned documents) into raw text;
- **Layout Analysis**: The process of analyzing the structure of a document, including the detection and categorization of elements like paragraphs, headings, tables, and images;
- **Document Classification**: The categorization of documents into predefined classes based on their content, such as invoices, forms or emails;
- **Visual Question Answering (VQA)**: The process of answering questions based on the content of a document;
- **Key Information Extraction (KIE)**: The process of extracting specific structured data (like names, dates, or price) from unstructured documents.

This project is designed to be **easily reusable**. You can [download](https://downloads.dataiku.com/public/dss-samples/EX_DOCUMENT_AI/) it and find instructions to reuse it in the [final section](#next-process-your-own-documents-1) of this wiki.

Please note that this project does not cover the case of question answering over a multi-page document or a collection of documents. This case is already the topic of the [Multimodal RAG example project](https://gallery.dataiku.com/projects/EX_MULTIMODAL_RAG/) and its accompanying [blog post](https://medium.com/data-from-the-trenches/beyond-text-taking-advantage-of-rich-information-sources-with-multimodal-rag-0f98ff077308).

# Walkthrough

## Optical Character Recognition and Layout Analysis

The [1. OCR and Layout Analysis](flow_zone:J9WtdNc) Flow zone addresses both optical character recognition (OCR) and layout analysis. We jointly cover these tasks, as they are often interrelated. A document can be viewed as a patchwork of components such as titles, paragraphs, sections, images, and so on. Detecting these components can serve as a valuable prerequisite for OCR.

### Dataset

For these tasks, we selected 4 [images](managed_folder:cFWXLXKE) from the dataset [DocLayNet](https://github.com/DS4SD/DocLayNet) and 3 [images](managed_folder:FAUwCSvY) from the dataset [IAM](https://www.kaggle.com/datasets/naderabdalghani/iam-handwritten-forms-dataset). The dataset DocLayNet offers detailed page-by-page layout segmentation, providing ground truth with bounding boxes for 11 distinct class labels. It is fully hand-annotated by experts, ensuring a really good standard in layout segmentation based on human recognition and interpretation.

![DocLayNet.png](4gARYv1jdElB)
<div align=center style="margin-bottom: 20px; margin-top: -5px"> Images from the DocLayNet dataset </div>

### Approaches

First, we [convert](recipe:compute_FAUwCSvY) PDFs to JPEGs with a Python recipe, as the documents need to be in JPEG format. Then, we implement several OCR approaches in this Flow zone.

The first one is to use Dataiku's [Tesseract plugin](recipe:compute_OCR_output_DSS) on the entire document without performing layout analysis.

However, since we lose the structure of the document (as OCR struggles to differentiate between various layouts), a more step-by-step and structured approach is necessary. Therefore, we perform layout analysis first and then proceed with OCR for the other approaches in this Flow zone.

We use the Python library [Unstructured](https://github.com/Unstructured-IO/unstructured-inference?tab=readme-ov-file) for the layout analysis task. This library leverages the model [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) in order to detect and label the differents parts of the document. The output of this [recipe](recipe:compute_output_region_unstructured) consists of the pixels corresponding to the bounding boxes of the different elements in the layout.

![Layout_analysis.jpg](B8mcWLw4aS3A)
<div align=center style="margin-bottom: 20px; margin-top: -5px"> Result of Layout Analysis using Unstructured - YOLOX</div>
We then [split](recipe:compute_Tz1CoYYW) each document into individual elements and correct their potential skew with the Hough transform. The output [managed folder](managed_folder:Tz1CoYYW) includes a sub-folder for each initial document. Each sub-folder contains one file per layout element detected in the original document.

We choose three approaches (proprietary API with GPT-4o, open source multimodal LLM from Huggingface with InternVL2 and a well-know open source software, Tesseract) to perform the OCR:
- [GPT-4o]. This [recipe](recipe:compute_results_GPT_OCR) does not require local computational resources and demonstrates the best OCR capabilities, particularly with handwritten documents.
- [InternVL2-8B](https://huggingface.co/OpenGVLab/InternVL2-8B). This open-weight multimodal large language model with a permissive license is [efficient](recipe:compute_results_OCR_InternVL) for both printed and handwritten documents.
- [Tesseract](https://github.com/tesseract-ocr/tesseract). This well-known, efficient and very fast OCR solution is available and [easy to use](recipe:compute_results_tesseract) with a [plugin](https://www.dataiku.com/product/plugins/tesseract-ocr/) in Dataiku. However, it struggles with handwritten documents.f

## Document Classification

### Dataset

In the [2. Document Classification](flow_zone:default) Flow zone, we select 10 [images](managed_folder:Rvdjm86o) from the test split of the [RVL-CDIP](https://www.kaggle.com/datasets/pdavpoojan/the-rvlcdip-dataset-test) (Ryerson Vision Lab Complex Document Information Processing) dataset. This dataset consists of 40,000 grayscale images in 16 classes such as letter, form, email, handwritten, news article and presentation.

![RVLCDIP.jpg](TZL2my0DFfBX)
<div align=center style="margin-bottom: 20px; margin-top: -5px"> One example per class in the RVL-CDIP dataset</div>


### Approaches

For the document classification task, we use three differents approaches:
- [GPT-4o](https://platform.openai.com/docs/models/gpt-4o), a multimodal LLM available through the [LLM Mesh](https://doc.dataiku.com/dss/latest/generative-ai/index.html) and the OpenAI API. We test two approaches, in the first one we only provide instructions ([zero-shot](recipe:compute_results_classification)) while we add examples in the second one ([ few-shot](recipe:compute_results_classification_gpt_fs) learning).  Few-shot learning, using only one image per class, shows significantly better performance.
- [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct). An open source and license free multimodal large language model. This [recipe](recipe:compute_results_classification_qwen_7b) shows good performance in zero shot for document classification. The model has been [quantized](https://huggingface.co/docs/transformers/en/main_classes/quantization) in 8 bits in order to use less GPU.
- [UDOP](https://huggingface.co/microsoft/udop-large-512-300k). It adopts an encoder-decoder Transformer architecture and it's specialised on document AI tasks like document image classification, layout analysis and document visual question answering. UDOP performs in [zero-shot](recipe:compute_results_classification_UDOP) but also after a [fine-tuning](recipe:compute_Results_classification_UDOP_FT_PEFT) phase using [Low-Rank Adaptation](https://arxiv.org/pdf/2106.09685) (LoRA) with [160 images](managed_folder:cKQJveH0) (10 images per class). This fine-tuned version has significantly better performance than the zero-shot version of UDOP. We can suppose that a larger fine tuning will lead to even better results.

## Visual Question Answering

### Dataset

For the visual question answering task, we choose 5 [images](managed_folder:JRpzJC6D) from the test split of the [DocVQA](https://www.docvqa.org/datasets) dataset. This dataset is composed of documents and between 2 and 4 questions per document.  It includes a wide range of document types (e.g. letters, memos, notes, reports, etc.) and a mix of printed, typewritten and handwritten content.

![docvqa_dataset.png](y1yfr2LOl0n0)
<div align=center style="margin-bottom: 20px; margin-top: -5px"> Images from the DocVQA dataset with an example question</div>

### Approach

We favor here a multimodal LLM, given the open-ended nature of the questions and the fact that the expected answer is some free-form text. We can for example use the same multimodal LLMs mentioned above, [GPT-4o](recipe:compute_results_VQA_GPT) and [Qwen2-VL](recipe:compute_results_VQA_QWEN_VL2).

We use a simple prompt taken from a [research paper](https://arxiv.org/html/2405.18433v1#A2) – "Answer the question. Do not write a full sentence, just provide a value. Question: {question}" – which helps generate concise answers close to those in the original dataset.

We then [evaluate](recipe:evaluate_results_VQA_GPT) the generated answers with an [LLM-as-a-judge](https://knowledge.dataiku.com/latest/ml-analytics/gen-ai/tutorial-llm-evaluation.html) and ground-truth answers. Both models show strong results.

Visual Question Answering is implemented in a [webapp](web_app:mjWfou5) which allows to:
- Select one of the [5 images](managed_folder:Tmf77vDr);
- Ask a question in full text
- Get the result of the question

## Key Information Extraction

### Dataset

For this task, we selected 10 [images](managed_folder:iJFvJqJA) from the [SROIE](https://github.com/zzzDavid/ICDAR-2019-SROIE) dataset. This dataset is composed of 1000 images of scanned receipts, and has labels for differents tasks such as key information extraction, layout analysis and character recognition. The task here in this project is to find the values corresponding to the four following keys: "company", "address", "date" and "total".

![SROIE.png](EyzihztLrA2Z)
<div align=center style="margin-bottom: 20px; margin-top: -5px"> Receipt from the SROIE dataset</div>

### Approach

We again use the same multimodal LLMs as before – GPT4o and Qwen2-VL. Since the task requires to systematically determine the values for four different keys ("company", "address", "date" and "total"), we leverage techniques to constrain the format of the answers:
- We [use](recipe:compute_results_KIE_GPT) *function calling* with [GPT-4o](https://platform.openai.com/docs/models/gpt-4o) to generate a JSON object with the four required properties and the correct types;
- With [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct), we take advantage of the Python library [Outlines](https://dottxt-ai.github.io/outlines/) to coerce the model to generate an answer with the same format as above.


## Webapp visualization

We propose different visualizations using the [webapp](web_app:mjWfou5) for the three following tasks: Optical character recognition, Visual question answering and Key information extraction.

### OCR
![OCR_webapp.png](MdlEQrGv9tXF)

### Visual question answering
![vqa_webapp.png](9iEXYYDf68UR)

### Key information extraction
![KIE_webapp.png](eY2WWaslrCMv)

# Next: process your own documents

## Technical requirements
This project:
- leverages features available starting from **Dataiku 13.2.2**;
- requires the [Text extraction and OCR plugin](https://www.dataiku.com/product/plugins/tesseract-ocr/);
- takes advantage of the [LLM evaluation recipe](https://doc.dataiku.com/dss/latest/release_notes/13.html#new-feature-llm-evaluation-recipe) which is in Private Preview as part of the Advanced LLM Mesh Early Adopter Program, as of October 2024;
- requires three Python 3.10 code environments, the specifics of which are on this [page](article:2):


## How to reuse this project
The project can be downloaded [here](https://downloads.dataiku.com/public/dss-samples/EX_DOCUMENT_AI/).

Once you have imported the project, you can directly navigate the Flow.

If you want to use your own data, put your images:
- in this [folder](managed_folder:cFWXLXKE) for OCR and layout analysis;
- in this [folder](managed_folder:Rvdjm86o) for document classification;
  - put your images in this [folder](managed_folder:cKQJveH0) for fine-tuning UDOP. Put the labels in a  _label.json_  file where keys are filenames of the pictures and values the classes.
  - to use few-shot learning for document classification, place your example images in this [folder](managed_folder:55DQRQlh). Put the labels in a _label_fs.json_ file where keys are filenames of the pictures and values the classes.
- in this [folder](managed_folder:JRpzJC6D) for visual question answering;
- in this [folder](managed_folder:iJFvJqJA) for key information extraction.

All the datasets are stored in filesystem so no remapping will be needed. However you have the option to [change the connection type](https://knowledge.dataiku.com/latest/courses/flow-views-and-actions/connection-changes-concept-summary.html#connection-changes) if you want to rely on a specific data storage type. 

If you want to leverage certain parts of the Flow directly, keep in mind that Dataiku allows you to reuse elements at different levels. In particular it is possible to:
- [duplicate a whole project](https://knowledge.dataiku.com/latest/kb/governance/How-to-duplicate-a-DSS-project.html)
- [copy and paste entire subflows](https://knowledge.dataiku.com/latest/courses/flow-views-and-actions/connection-changes-concept-summary.html#copy-subflow)
- [copy and paste recipes](https://knowledge.dataiku.com/latest/kb/collaboration/How-to-copy-a-recipe-in-your-Flow.html) and remap input/output datasets
- [copy and paste preparation steps](https://doc.dataiku.com/dss/latest/preparation/copy-steps.html) within a Prepare recipe  

# Related Resources
- [Plugin OCR Tesseract](https://www.dataiku.com/product/plugins/tesseract-ocr/) 
- [Documentation on Dataiku's LLM Mesh](https://doc.dataiku.com/dss/latest/generative-ai/index.html)
- [Document AI](https://huggingface.co/blog/document-ai) 
- "Multimodal LLM" Dataiku project ([demo](https://gallery.dataiku.com/projects/EX_VISION_LLM/), [download site](https://downloads.dataiku.com/public/dss-samples/EX_VISION_LLM/), [blog post](https://medium.com/data-from-the-trenches/demystifying-multimodal-llm-053143c07d6f))
- "Multimodal RAG" Dataiku project ([demo](https://gallery.dataiku.com/projects/EX_MULTIMODAL_RAG/), [download site](https://downloads.dataiku.com/public/dss-samples/EX_MULTIMODAL_RAG/), [blog post](https://medium.com/data-from-the-trenches/beyond-text-taking-advantage-of-rich-information-sources-with-multimodal-rag-0f98ff077308))