[AAAI-25] ImagePiece: A Novel Content-Aware Re-Tokenization Technique for Semanticizing Tokens in Efficient ViT

Lively discussions took place across various AI research fields at the AAAI-25 conference, where many fascinating studies were presented. This year, Large Language Models (LLMs), Multimodal Learning, and reasoning for complex problem solving emerged as key trends. AI alignment and reliability were also discussed, and various approaches were presented, such as exploring how to make large-scale models safer and fairer for use in the real world. The spread of integrated processing of multimodal data and the introduction of large-scale models in various fields reminded us that AI is already being applied to complex problems in the real world, including science, healthcare, urban analysis, and finance, beyond the existing areas of image and natural language processing. 

As the use of such large-scale models has become increasingly common, so has the interest in using computational resources efficiently and improving inference speed. In various models, including LLMs, efficiency is no longer an option but a necessity, and various approaches have been announced to this end. In this overall trend, various attempts to increase the structural efficiency of the internal expression of the model have also gained attention, including work on restructuring the token expression of Vision Transformer to make it more semantic.

Thus, at AAAI-25, we presented the study "ImagePiece: Content-Aware Re-Tokenization for Efficient Image Recognition^[6]". This post will present the main ideas and development of that study, as well as the directions it explored for designing efficient vision models and related research.

LG AI Research’s “ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition^[6]”

Real-World Challenges and Research Inspiration

LG AI Research has been conducting various research projects to improve efficiency and productivity at factories within LG's affiliates. In the process of conducting research, we have mainly utilized transformer structural models, especially Vision Transformer (ViTs) models for image data processing. However, ViTs rely on Multi-Head Self-Attention (MHSA) in their structure, which incurs significant computational cost, which has become a limitation in actual industrial applications. 

In particular, images collected in factory environments often contain repetitive backgrounds or irregular structures rather than specific objects. As a result, ViT's patch-based tokenization process generates a large number of non-semantic patches, leading to inefficiencies. This structural limitation was not sufficiently addressed in previous studies on streamlining (DynamicViT^[1], EViT^[2], ToMe^[3]), such as simply removing tokens or merging similar ones. This is because removing non-semantic tokens too quickly can lead to the loss of important information, while merging them haphazardly can dilute semantic expressions. Therefore, it is essential to accurately determine the importance of tokens before reducing them for efficiency, and existing research has been limited in this regard.

Image 1. Overview of the ImagePiece pipeline compared to WordPiece for NLP^[6].
While text is tokenized into semantic units, tokens generated from image patches are often non-semantic tokenized (even though they are ultimately important areas).

Inspired by the Subword Tokenization approach in NLP, we came up with a way to re-tokenize non-semantic visual tokens into “semantic units.” In NLP, most text tokens have meaning. For example, in Image 1, tokenizing the sentence “This is a Trolleybus on a City Street in Front of a Building.” would break it down as ['This', 'is', 'a', 'Trolley', '##bus', 'on', 'a', 'City', 'Street', 'in', 'Front', 'of', 'a', 'Building', '.'], with each token acting as the smallest semantic unit. In the case of images, however, simply tokenizing them by dividing them into 16 × 16 patches can result in some patches having little meaning at the initial stage, while still containing important information in the broader context.

To prevent these potentially important patches from being initially deemed unimportant tokens and removed, the researchers proposed a new re-tokenization technique called ImagePiece, which preserves important tokens and improves overall performance by inducing tokens that were deemed non-semantic to become semantic when combined with surrounding information. In other words, ImagePiece's re-tokenization process allows these non-semantic tokens to be merged into a single semantic token if they have the potential to be contextually meaningful. This is not a simple approach to reducing the number of tokens, but rather a process of redefining the “semantic of a token” in ViT and reconfiguring the tokenizer itself, which the researchers believe can effectively reduce the amount of computation while maintaining the expressiveness of the model.

Key Research Details

The core idea behind ImagePiece is to re-tokenize visual tokens into “semantic units” like text, which is why we propose a re-tokenization strategy that borrows from NLP's WordPiece method, merging similar non-semantic tokens into semantic chunks and then evaluating them again based on their importance.

Image 2. ImagePiece overview^[6]

1) Re-Tokenizing Non-Semantic Tokens

ImagePiece proposes to solve the problem of “many non-semantic tokens,” a limitation of traditional ViT, by re-tokenizing them into semantic units like NLP. The core idea consists of three steps:

1. Step 1. Assess token importance:
   Calculate the importance of each token based on its Attention Score with the class token ([CLS]).

2. Step 2. Merge non-semantic tokens:
   Divide the bottom k least important tokens into two groups (A, B), and merge the tokens in each group A with the most similar tokens in group B. This uses the Bipartite Soft Matching technique.

3. Step 3. Re-evaluate semantics:
   Re-evaluate the Attention Score, including the merged tokens, the existing important tokens, and the remaining tokens that were not merged. This keeps tokens that have new meaning and removes tokens that are still non-semantic.

 

Note that while ViT goes through the entire layer and meaning is generated after context is formed between tokens, Efficient ViT requires tokens to be removed (or merged) early on, which can result in important tokens being lost before meaning is generated. Therefore, ImagePiece is effective at reconstructing non-semantic tokens early on, minimizing information loss.

2) Local Coherence Bias

In ImagePiece, to make the merging of non-semantic tokens more effective, we take into account that spatially close patches are more likely to be visually similar. Of course, not all adjacent patches are similar, but in most cases, spatially adjacent non-semantic tokens are candidates that can be semantic together, and it is important to induce these pairs of tokens to merge better. To this end, the Local Coherence Bias module based on overlapping convolution is introduced, which strengthens the feature similarity between spatially close tokens. As a result, it helps non-semantic tokens merge more naturally and meaningfully within a real visual scene.

3) Compatibility with Token Reduction Methods

Image 2 shows the low-value tokens being removed after re-tokenization and token importance re-assessment. A variety of existing token removal techniques can be applied in this process, and ImagePiece is highly compatible when combined with them, significantly improving overall performance.

4) Results

Image 3. Performance comparison table with various token pruning methods^[6]

Image 4. Performance comparison table with various token merging methods^[6]

Image 5. Compatibility comparison table with various token removal techniques^[6]

Image 6. Experimental table showing changes in token importance after re-tokenization^[6]

Images 3 and 4 show that ImagePiece achieves higher accuracy than the existing token removal techniques on the image classification task (ImageNet-1K dataset). In particular, it not only outperforms the existing lightweighting techniques, but also outperforms the original non-lightweighted DeiT-Ti and DeiT-S models, demonstrating that ImagePiece's re-tokenization strategy contributes to a substantial performance improvement.

In Image 5, we experimented with compatibility with existing token removal techniques and found an average accuracy improvement of 0.79% when applying ImagePiece, which shows that ImagePiece can also show high synergy with other token pruning/merging-based methods.

In Image 6, we analyze the effectiveness of the re-tokenization strategy in a more quantitative way. We measure the ratio of tokens that were initially rated as inattentive that were reassessed as attentive after re-tokenization because they were reconstructed into semantic chunks, layer by layer. In the sixth layer, we see that 58.77% of tokens are converted to attentive, which supports the idea that ImagePiece is effective at uncovering meaningful information in inattentive tokens that are initially overlooked.

In conclusion, we experimentally demonstrated that ImagePiece, with its strategy of reconstructing non-semantic tokens into semantic units, is an effective way to improve the efficiency of Vision Transformer and can improve performance with less information loss compared to the existing simple token removal approach.

Notable Research on Vision-Language Models (VLM) at AAAI-25

In the field of multimodal models, there is an active research trend toward efficiency, especially in Vision-Language Models (VLMs), which focuses not only on improving inference speed, but also on the connectivity between visual and text information. In other words, multimodal models attempt to leverage the relationship between visual and textual tokens to simultaneously improve the model's inference capabilities and achieve efficient computation. Recently, methods have been proposed to select or restore visual information based on text information, which is essential for performing real-world tasks, and research is also expanding to understand complex object relationships and enable accurate visual-based reasoning. In this context, this post will introduce two studies that simultaneously pursue modeling efficiency and consistency through precise connections between visual and text information.

1) Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information^[4]

Image 7. Key areas that vary depending on the question in the same image^[4]

This study proposes a visual token restoration mechanism utilizing text information to address the problem of important information being lost due to excessive visual token removal in multimodal vision-language models (MM-LLMs).

Existing vision token compression methods mostly evaluate importance and remove tokens based on visual information alone. However, in real-world multimodal tasks, text (such as questions) can be a cue to determine which visual information is important. This is because, for example, the same image can have different areas of attention depending on the question. Image 7 shows that even for the same image, the question text should dictate where to focus attention, depending on the question text. In addition, as only the question needs to be resolved at the task level, the remaining image areas (remaining visual tokens) other than the main area are not needed, which can help with model efficiency.

To solve the aforementioned problem, this paper computes the semantic similarity between text and visual tokens, restores the visual tokens that are relevant to the text (Text Information Recycling), and merges the rest of the less important tokens (Secondary Recovery). This process requires no pre-training and has been shown in experiments based on LLaVA-1.5 to significantly improve inference speed by reducing the number of tokens to the 10% level without sacrificing performance.

Image 8. Multimodal vision token recovery mechanism guided by text information framework^[4]

2) Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models^[5]

Image 9. Comparison with existing LVLM^[5]:
(a) Only corresponds to the entire image/region and text, lacks detailed objects;
 (b) focuses on simple phrases and boxes, lacks relational context; 
(c) uses a hierarchical multi-precision association method that understands complex concepts using simple concepts as clues

PromViL is a framework designed to overcome the limitations of existing multimodal models when dealing with complex concepts or relationships between multiple objects, by decomposing concepts in sentences in stages, progressively grounding visual information at each stage, and making inferences.

The core idea is to parse complex concepts step-by-step. Existing Large Vision-Language Models (LVLMs) tend to process one long sentence or the entire image at once, leaving it unclear where “a Blue Hat” fits in a sentence like “The Woman with a Blue Hat” and how “The Woman” is connected to it. PromViL decomposes these sentences in order of complexity.

1. Level 1: “a Blue Hat”

2. Level 2: “The Woman with a Blue Hat”

3. Level 3: “The Shirt of the Woman with a Blue Hat”

 

This hierarchical division from simple to complex expressions is achieved by having the model perform simple grounding at each level, using the visual information from the previous level as cues for the next level of grounding. This process allows the model to reliably understand increasingly complex expressions.

The model structure and learning method are as follows.

1. Data construction: Based on Visual Genome, we created a new CompoVL dataset with a variety of compound expressions. Each sentence is decomposed into nested expressions, allowing the model to learn visual-verbal alignments of varying difficulty.

2. Learning objective: For each complexity level (Level-i) of expression Ei, the model predicts the corresponding visual location Yi. It is fed with the information [Ei-1, Yi-1] obtained in the previous level to enable more accurate grounding.

3. Learning method: Learning is done by combining the standard next-token prediction approach of language models with grounding supervision. The overall structure follows a progressive decoding algorithm, starting with simple expressions and progressing to increasingly complex ones, creating an inference path.

 

Compared with existing LVLMs such as MiniGPTv2 and Kosmos-2, PromViL has shown strong performance, especially when dealing with complex expressions, demonstrating that designing inference to reflect a “flow of thought” rather than a simple alignment is indeed effective.

A New Path Forward for LG AI Research

LG AI Research aims to overcome the structural limitations of Vision Transformer and open up new possibilities for visual tokenizers based on the recently announced ImagePiece. Inspired by NLP's WordPiece, ImagePiece is a content-aware re-tokenization technique designed to reorganize non-semantic visual patches into semantic units. It is not just an efficiency technique to reduce the amount of computation, but also works as a strategy to improve the expressiveness of visual information, and has the scalability to be universally applicable to various models based on ViT.

In the future, we plan to build on this re-tokenization approach and expand its potential in multimodal learning and complex visual-verbal reasoning. In fact, in multimodal environments, overcoming the structural constraints of visual information has a direct impact on a model's ability to comprehend, and ImagePiece’s approach can also play a positive role in token grounding, the semantic units underlying visual-text associations.

Furthermore, improvements at the tokenizer level can be an important starting point for the development of future general-purpose learning frameworks and multimodal foundation models. By defining a tokenization that is appropriate for the unique structure of the image modality, it enables more robust and flexible learning across different domains and tasks. Furthermore, it is expected to be a key component for balancing efficiency and expressiveness in various scenarios such as few-shot learning, domain adaptation, and reasoning.

LG AI Research will continue to develop the core components of visual AI technology that can create real value in the industrial field, and will lead the way in next-generation modeling techniques that can combine efficiency, expressiveness, and versatility.

AAAI-25 Series

EP.1 [AAAI-25] Study on Denoising Distillation Model Using 3D Coordinates
EP.2 [AAAI-25] A Novel Augmentation Technique in Truncated SVD-Based Representation Space for Improving Tabular Data Prediction Performance
EP.3 [AAAI-25] Study on Efficient Time Series Forecasting Model Learning Strategies That Alleviate Variable Redundancy