[CVPR 2026] Beyond Text: How VinQA Incorporates Visual Elements into Responses for Multimodal Document QA

Real-world documents frequently combine text with visual elements like diagrams, charts, tables, and images. Also, answers to questions can be more helpful when they go beyond a series of sentences and include grounding figures or tables. This is because diagrams are the most effective means for explaining complex procedures and concepts, just as tables and charts are for presenting quantitative data.

Image 1 Example of VinQA data ^[1]

As illustrated in Image 1, when a user asks about a product's installation procedure, placing a diagram of that specific step right before the explanation, instead of relying solely on text, provides a far more intuitive experience. 

Recognizing that visual elements can significantly enhance response clarity, LG AI Research initiated a study on multimodal document question answering (QA). Specifically, the study focused on generating long-form answers that strategically position grounding visuals within text, rather than simply appending images. As a result, the VinQA paper was successfully presented at CVPR 2026. 

This study focuses on two main aspects: generation and evaluation. 

“How can we accurately select and strategically position grounding visual elements from real-world documents to generate long-form answers? And how can we reliably measure their quality?” 

Real-world documents are inherently multimodal. A single page can contain a mixture of various visual elements, including diagrams, charts, tables, and images, presented in different layouts. Nevertheless, MLLM-based document QA studies have mainly advanced in two directions—inputting the entire document into the model or retrieving relevant pages via multimodal RAG. However, both approaches share a common limitation: they only produce answers in text.

VinQA, on the other hand, is not just a model. It proposes a real-world document-based multimodal QA dataset, along with a corresponding evaluation framework. The core contributions of this study can be divided into four key pillars: 

Image 2. VinQA dataset construction process^[1]

Each VinQA instance consists of: (i) a question requiring both textual and visual understanding, (ii) a context comprising the top-K relevant pages retrieved via a multimodal RAG pipeline, and (iii) a multimodal grounded answer drawing upon that context. Within the answer, visual citations are placed immediately before the sentences that describe them.

The dataset gathers only original documents from various public document QA datasets, including MMLongBench-Doc^[2], TAT-DQA^[3], and SlideVQA^[4], disregarding their existing QA labels. These documents were then reorganized into seven domains: academic papers, websites, textbooks, guidebooks, research reports, financial reports, and slides. In scale, the training split comprises roughly 130,000 pages and 42,700 QAs, and the test split includes 9,373 pages and 1,605 QAs. In addition to single-page questions, the dataset encompasses reasoning questions that span multiple pages, multiple documents, and multiple modalities, as well as "unanswerable" questions to identify cases where no answer exists. While existing datasets largely focus on short-form, text-only answers, VinQA simultaneously covers long-form answers and text-visual integration, using the broadest domain scope.

Image 3. Page Encoding & Modality Encoding ^[1]

The study explored two different approaches that convert the page images from original documents into MLLM inputs while enabling the citation of visual elements. Page Encoding encodes the entire page image as it is, while also providing the bounding boxes of detected visual elements to define the unit of citation. This fully preserves visual information like layout but requires the model to read text directly from pixels and interpret box regions. Modality Encoding extracts text through OCR, crops out visual elements for separate encoding, then uses the cropped images as the unit of citation. Despite losing some layout information, it allows for more precise handling of text and visual elements by modality.

Existing metrics have limitations in evaluating the quality of long-form answers that combine text and visual elements. To address this, this study introduces the Multimodal Grounded QA Scoring Evaluator (M-GroSE), a multimodal extension of GroUSE, a text-only grounded response evaluation tool.^[5] M-GroSE evaluates answerable questions based on three criteria: relevance, completeness, and faithfulness. For unanswerable questions, it uses Unanswerability F1 to assess whether the model correctly identifies them as unanswerable. Additionally, it features Visual Source F1, which directly measures the citation accuracy of visual elements, and Visual G-Eval,which uses an MLLM judge to evaluate the effectiveness, position, and faithfulness of the cited visual elements.

In the VinQA test set experiment, frontier commercial models like GPT-4.1 and Claude 3.5 Sonnet still recorded the highest scores. However, once the open-source Qwen2.5-VL-7B was fine-tuned on the VinQA training set, its performance improved significantly, coming far closer to the commercial models. Prior to fine-tuning, Qwen2.5-VL-7B’s M-GroSE scores averaged only about 2, but after fine-tuning, the score surged to around 3.3, nearing the peak performance scores (approximately 3.5 to 3.6).

Image 4. M-GroSE scores by context length (document complexity) ^[1]

An in-depth analysis was conducted using two main factors to understand what enabled this score improvement. 

The first was context length (document complexity). The context token length was divided into segments ranging from 0-2.5k to over 10k, which revealed that VinQA fine-tuning consistently improved response quality across all segments. Additionally, before fine-tuning, Modality Encoding outperformed Page Encoding as context grew longer (See the positive difference in (c) of Image 4). However, this gap virtually disappeared after fine-tuning, demonstrating Page Encoding’s ability to handle complex documents without explicit parsing.

Image 5. Citation accuracy by visual element type (Visual Source F1) ^[1]

Second, citation accuracy by visual element type (Visual Source F1) was analyzed. Before fine-tuning, VinQA training significantly improved citation performance across all visual element types. Mixed cases involving multiple types also displayed improvements, confirming that VinQA enhances the ability to handle heterogeneous visual elements as well. Furthermore, as shown in the context length analysis, the performance gap between Page Encoding and Modality Encoding virtually disappeared after training.

Lastly, to directly evaluate the integration of visual elements into responses, effectiveness, position, and faithfulness were assessed on a scale of 1 to 5 using Visual G-Eval. In both encoding methods, consistent improvements were observed across all three metrics after VinQA fine-tuning. In the case of Page Encoding, the scores for effectiveness, position, and faithfulness increased from 3.44 to 3.94, 4.24 to 4.77, and 3.44 to 3.85, respectively. However, Modality Encoding outperformed Page Encoding by a slight margin in terms of the effective utilization and explanation of visual elements (effectiveness and faithfulness). In summary, the study demonstrated that VinQA provides effective learning signals for enhancing both the groundedness of multimodal answers and the accuracy of visual element citations. 

VinQA is a starting point. The key next step at LG AI Research is now the expansion into the AI agent approach. Currently, VinQA generates responses based on the pages retrieved at once by a multimodal retriever, finding answers within them. However, real-world professional tasks often require more than a single retrieval. Consequently, LG AI Research plans to shift its research toward models that autonomously identify their information gaps, perform proactive (agentic) retrieval to gather necessary materials, and strengthen their grounding by drawing upon multiple documents. VinQA’s ability to read documents and generate text and visual responses serves as a key capability for agents that execute tasks on behalf of users. Furthermore, its grounding process reduces hallucinations to ensure more reliable results.

Through continuous research, LG AI Research aims to evolve this technology into trustworthy AI agents for practical business automation, envisioning a future where AI reads, presents, and responds to documents just as a human would.