SciNO: A Functional Diffusion Model with Neural Operators for Causal Ordering

Introduction

NeurIPS (Neural Information Processing Systems) 2025, the world's leading artificial intelligence conference held in San Diego, USA, served as a platform for researchers from around the world to share the latest research trends and technical directions. Over 20,000 papers were submitted to this year's conference, with research centered on generative models and large language models (LLMs) continuing to dominate. Throughout the conference, discussions prominently focused on how to ensure models' reasoning capabilities and reliability, moving beyond simple quantitative growth in performance.

Within this trend, research on causal discovery to identify causal relationships between variables has also steadily gained attention. This year, the key topics in the causal discovery field were “scalability” and “the integration of LLMs and causal inference”. This suggests that causality research is evolving not only as theoretical exploration but also as a practical technology capable of solving complex real-world problems. LG AI Research has also aligned with this research trend, announcing SciNO (Score-informed Neural Operator)^[1], a framework enabling stable and scalable causal reasoning even in high-dimensional data.

Causal Discovery

Causal discovery, which uncovers causal relationships within complex data, is a core technology that provides a reliable basis for decision-making. However, as the number of variables increases, explicitly searching the entire causal graph structure becomes computationally prohibitive. Consequently, ordering-based causal discovery approaches, which first estimate the causal order of variables to reduce the search space, have recently gained attention. Instead of learning the causal graph in a single stage, this approach sequentially identifies leaf nodes one by one, estimates the causal order, and derives the final causal graph based on this. In such ordering-based approaches, the key challenge lies in defining the criterion for identifying which variables are leaf nodes unaffected by other variables. Under the Additive Noise Model (ANM) assumption, it has been theoretically established that the Hessian diagonal of each variable's log-density function can serve as a key statistic for determining whether that variable is a leaf node. 

Existing ordering-based causal discovery methods such as SCORE^[2] and CaPS^[3] rely on Stein gradient estimators for Hessian diagonal estimation, leading to a limitation where memory usage increases sharply in high-dimensional settings. To address this, DiffAN^[4] was proposed, utilizing a diffusion model to directly learn the score function of the probability density and offering a scalable approach for high-dimensional data compared to existing methods. However, DiffAN still faces limitations in the stability of derivative approximation in high-dimensional settings where the number of variables becomes extremely large, due to its use of an MLP-based model architecture. Based on these observations, LG AI Research recognized the need for a new methodology capable of stably approximating the score function and its higher-order derivatives. To address this, we introduced a functional diffusion model, enabling more stable learning of the score function and its derivatives within a functional space.

Score-informed Neural Operator

SciNO (Score-informed Neural Operator) is a functional diffusion model framework with neural operator proposed to address these limitations. SciNO aims to stably approximate the score function and its higher-order derivatives required for accurate causal ordering, adopting the theoretical perspective of the Hilbert Diffusion Model (HDM)^[5] to achieve this. Unlike existing diffusion models that define data distributions in finite-dimensional Euclidean spaces, HDM defines probability distributions on Hilbert spaces and models the corresponding diffusion processes. This approach plays a crucial role in preserving the functional information and differential structure inherent in the data by treating it as a continuous function rather than a simple vector.

This functional diffusion modeling naturally integrates with neural operators. SciNO considers Sobolev spaces to ensure differentiability and theoretically proves that neural operators can stably approximate not only the score function but also its higher-order derivatives, based on the Sobolev Embedding Theorem. Unlike existing MLP-based diffusion models that exhibited instability in estimating the Hessian diagonal in high-dimensional data, the neural operator performs more accurate approximation of derivatives by learning the mapping between functions.

Image 1. Score-informed Neural Operator (SciNO) model architecture^[1]

SciNO model architecture

Based on this theoretical background, SciNO's architecture is built upon the time-conditioned Fourier Neural Operator (FNO). The overall model architecture consists of three main components: the input embedding layer, the Fourier layers, and the output head.

First, the input data is embedded into a high-dimensional space through the initial MLP layer, then passes through multiple Fourier layers. In each Fourier layer, the input signal is transformed into the frequency domain using the Fast Fourier Transform (FFT), then processed by separating it into real and imaginary parts. This signal passes through an MLP in the frequency domain and is mapped back to the spatial domain via inverse transformation.

The time t in the diffusion model is processed through the Learnable Time Encoding (LTE) module. Unlike standard positional embedding, which assigns fixed vectors to discrete positions, LTE provides learnable embeddings for continuous temporal variables. The resulting time encoding is injected into the frequency-domain signal via element-wise multiplication, enabling the model to simultaneously capture derivative information in both spatial and temporal dimensions.

The final output head predicts the score function for each variable. SciNO utilizes this to approximate the Hessian diagonal and sequentially estimates the causal order by repeatedly identifying and removing variables whose variance is minimized as leaf nodes.

Image 2. Comparison of causal ordering and causal discovery performance of DiffAN and SciNO on synthetic and real-world datasets ^[1]

SciNO achieves a significant improvement in causal ordering accuracy compared to the baseline DiffAN^[4], reducing order divergence by an average of 42.7% on synthetic graphs and 31.5% on real-world datasets. Notably, it maintains high scalability by maintaining stable performance even in high-dimensional settings with 100 nodes.

Image 3. A probabilistic control algorithm for LLM-based causal reasoning ^[1]

Combining LLMs and SciNO

Furthermore, this study proposes a probabilistic control algorithm that combines SciNO with autoregressive models such as LLMs, presenting a method to integrate semantic information into the causal ordering process. The proposed algorithm reinterprets the sequence generation process of LLMs as a causal ordering process.

Specifically, given the current causal order and textual descriptions for each variable, an LLM prompt is constructed to select the most likely leaf node from the unselected variables. In this process, the token probability generated by the LLM is defined as a prior probability based on semantic information. Meanwhile, SciNO estimates a data-driven likelihood of a variable being a leaf node based on statistics from observational data. Finally, these two pieces of information are combined to update the posterior probability, and the variable that maximizes this is selected as the next leaf node. This Bayesian update method allows probabilistic control over the ordering process by calibrating the LLM-derived semantic prior with SciNO's data-driven evidence. 

Unlike existing representative LLM-based causal discovery methods, which query the relationship between all variable pairs and exhibit query complexity $O$($D^2$) for a given number of nodes $D$, the proposed algorithm achieves complexity $O\left(D\right)$ proportional to the number of variables. It also offers practical advantages by being applicable without requiring additional fine-tuning or prompt engineering.
 

Image 4. Comparison of causal ordering performance of GPT-4o, uncontrolled LlaMA, and LlaMA with SciNO-based control on the BNLearn dataset ^[1]

 

Experimental results show that the SciNO-based control method demonstrated significant performance improvements across high-dimensional datasets with 40 or more nodes, achieving up to 75% performance gains on the ARTH150 graph with 100 or more nodes. Furthermore, the fact that it can achieve higher performance than SciNO alone suggests that the semantic information from LLMs can be effectively utilized as auxiliary information in the causal ordering process.

Conclusion

Going forward, LG AI Research plans to continue advancing scalable causal discovery algorithm while expanding research into causal reasoning using large language models and agent-based causal analysis frameworks. By systematically combining domain expertise with data-driven learning, we aim to build automated analysis pipelines capable of effectively identifying causal relationships even in complex real-world settings. Based on this foundation, we will develop causal inference technology into a practical and reliable solution suitable for solving real business problems and supporting strategic decision-making processes.

 

→ Read the NeurIPS - SciNO Paper