GPU Job Scheduling Using an Idle Inference GPU Pool

With the advancement of AI technology, demand for GPU resources has been increasing rapidly. In particular, since training and inference for large language models (LLMs) require massive computational resources, securing stable GPU infrastructure has become a key competitive advantage for AI research and service operations. However, GPUs remain a costly resource, and there are practical limitations to scaling infrastructure within a limited budget. As a result, situations frequently arise where the demand for GPUs in research and service environments cannot be fully met.

Meanwhile, companies that provide AI services tend to allocate resources based on peak times in order to prepare for fluctuations in traffic. In such cases, during periods of low traffic, the allocated resources remain in an “idle state,” consuming GPU memory while remaining underutilized.

The project “GPU Job Scheduling using an idle inference GPU pool” began with this idea. It is an attempt to maximize infrastructure utilization by allocating idle GPU resources to model training or research workloads, while ensuring that service stability is not compromised. In this article, we share how we defined and addressed the problem in order to maximize the efficiency of GPU resource operations.

1. Auto-scaling of LLM services

In typical services, it is relatively easy to predict resource usage based on metrics such as the number of requests, CPU utilization, and memory usage, and to configure auto-scaling policies accordingly. Since these metrics generally increase in proportion to service load, threshold-based scaling strategies tend to operate reliably.

However, it is difficult to apply the same approach to LLM-based services. This is because GPU resources consumed by each request vary depending on the number of input tokens, the number of generated tokens, and the size and architecture of the model. In other words, LLM services have a technical characteristic that makes it difficult to accurately predict resource demand using simple metrics alone.

Metrics such as GPU utilization and memory usage also have limitations in an LLM environment. Due to the nature of LLM inference, a single request can consume most of the available GPU resources, causing utilization to spike momentarily. However, it is difficult to interpret this as an indication of overall system load. Similarly, because a significant portion of memory is already allocated during the model loading phase, memory usage shows only minimal fluctuations as traffic changes, making it difficult to accurately reflect actual load variations.

Due to these characteristics, traditional CPU/memory-based auto-scaling or simple request-count-based approaches do not work effectively in LLM services, and new metrics are needed to better reflect the actual computational load.

To address this issue, we utilized the internal metrics provided by vLLM as auto-scaling signals. Since vLLM is designed for LLM serving, its metrics, such as real-time throughput and queue status, better capture the runtime characteristics of LLM workloads. By using these metrics, we were able to implement a more sophisticated auto-scaling strategy optimized for LLM services.

2. Auto-scaling analysis

Image 1. Replica Auto-scaling graph (24h)

A time-series analysis of changes in service replicas count after auto-scaling was applied revealed distinct GPU resource usage patterns across time periods. During daytime hours (8:00 AM to 8:00 PM), the replica count increased rapidly as traffic rose, reached a peak and then gradually decreased as traffic declined. In contrast, during the off-peak nighttime hours (8:00 PM to 8:00 AM the following day), the system maintained the minimum replica count with minimal fluctuations, leaving a large portion of GPU resources unused for extended periods.

For example, in an environment where a single replica uses four GPUs, an average of 52 GPUs remain idle for approximately 12 hours per day. While this is necessary to ensure service stability by providing a buffer of spare resources, there is room for improvement in terms of GPU utilization efficiency. By leveraging these idle resources for research and experimental tasks, we can significantly increase overall GPU utilization without expanding the infrastructure.

Image 2. GPU Replica Pool Status Example over Time

Based on this analysis, we built a pipeline to run GPU workloads during nighttime hours when service traffic is low. This pipeline is designed to avoid compromising the availability of existing service resources. It operates on a best-effort basis, meaning that running workloads can be interrupted at any time if GPU resources need to be reclaimed due to increased service traffic.

3. Designing a GPU Job Pipeline

Image 3. GPU Job Pipeline Based on Argo Workflows

To utilize idle GPU resources, a pipeline capable of executing a wide variety of tasks is required. In particular, research and experimental workloads often require diverse execution methods, and even the same task is often run repeatedly depending on the configuration; therefore, the pipeline must support versatility, flexibility, and reproducibility at the same time.

1. Versatility: Supports both training and inference tasks for various models

2. Scalability and flexibility: Must be able to execute without additional development burden even when new task types are added

3. Reproducibility: Guarantees consistent behavior and results under identical inputs and settings

Based on these requirements, the GPU job pipeline was designed with scalability and versatility as its primary goals. First, to avoid dependency on specific frameworks or execution environments, each job is packaged as Docker image. The pipeline is structured so that when a user prepares an image containing the optimal execution environment and code required for the job, the pipeline executes that image as-is. This implementation allows the pipeline to accommodate a wide variety of jobs without restrictions on language, libraries, or execution methods.

Image 4. Example of GPU Job Pipeline

The pipeline is composed of execution units called steps. Each step represents a single independent task, and users can combine them sequentially to form a single workflow. For example, in the case of an LLM training process, the data preprocessing, pre-training, supervised fine-tuning, reinforcement learning, and evaluation stages can be divided into steps and linked in sequence. It is also possible to execute multiple tasks in parallel by varying only the parameters of a single step. This structure allows for the clear representation of complex experimental flows and makes it easy to modify or re-execute specific steps.

Each step receives the required input values at runtime. These inputs include data paths, model paths, hyperparameters, and the number of GPUs to be used. A key point here is that all settings required for execution are encapsulated as parameters rather than hard-coded. Because different experiments can be run by changing only the input values while keeping the same Docker image, experiment management and reproducibility are significantly improved.

All execution processes and outputs are managed via Cloud Storage, ensuring the reproducibility and traceability of the experiments. Each step reads the necessary data from storage and saves the results back to it. This structure eliminates the need to share state across execution environments, allowing the pipeline to remain stateless.

Ultimately, this pipeline aims to create a structure where “any task can be defined as a Docker image and executed simply by specifying the required resources and input values.” This allows us to flexibly accommodate new experiments or models without having to modify the pipeline itself, and to make optimal use of available idle GPU resources for a wide range of research tasks.

4. Results of idle resource utilization

Image 5. Operational Results of the Idle GPU Project

After the system was established, we collected approximately three months of operational data, from November 2025 to January 2026, to evaluate the effectiveness of idle GPU utilization. During this period, a total of 85 jobs were executed through the pipeline, including various types of tasks such as model training, performance evaluation, and data generation.

In November 2025, during the early stages of the project, a relatively small number of tasks were executed. However, as the project stabilized and spread internally, we can see that the number of tasks executed and the amount of usage increased significantly by January 2026. The cumulative GPU usage over the three-month period totaled 95,000 GPU hours. In particular, monthly GPU usage in January 2026 increased by approximately 70% compared to November 2025, indicating that the utilization of idle resources is gradually expanding. Converted to the number of GPUs based on a 12-hour day, the figure was approximately 67 in November 2025 but increased to about 110 in January 2026. Converting this figure to a 24-hour basis, it is equivalent to securing approximately 55 additional GPUs for the entire month of January 2026. In other words, this is equivalent to securing a significant amount of computational resources without the need for additional hardware expansion.

Significant cost savings were also observed. When calculating the cost based on a three-year public cloud contract for the same computational workload, we achieved cost savings of approximately KRW 75 million in January 2026 alone, and generated cumulative savings of approximately KRW 185 million over a three-month period.

We also confirmed that by converting previously unused GPU resources into tangible productivity, we can scale the organization’s overall GPU resource capacity without additional infrastructure investments. Furthermore, the steady upward trend in usage metrics also shows that the system, which was initially in an early adoption stage, has gradually become established and expanded naturally within the organization. This can be seen as a result that goes beyond technical implementation, demonstrating sufficient value and convenience from the perspective of actual users.

5. Future plan: Advancing GPU Resource Management

LG AI Research will continue to enhance its GPU resource management to enable more fine-grained control and more efficient utilization.

1. Enhancing auto-scaling metrics: We plan to analyze actual usage patterns for each service in greater detail to implement an optimal GPU resource scheduling logic that can adapt to changing demand.

2. Establishing intelligent GPU scheduling technology: We plan to advance our GPU scheduling technology by leveraging Kubernetes, EXAONE(LLM), and other tools. Our goal is to build an always-on system that can execute tasks as soon as resources become available, rather than being limited to nighttime hours.

3. User Experience (UX) optimization: By improving the interface and designing the process from job requests to real-time monitoring intuitively, we aim to create an environment where researchers can use the infrastructure without technical barriers.

Through these continuous improvements, the Platform&Infra Team plans to maximize the utilization of idle GPU resources and make a tangible contribution to improving research productivity.