Post-doctoral researcher; Hybrid Implicit Neural Representations for Next-Generation Learned Video Compression

Background and Motivation 

The continuous growth of video traffic generated by streaming platforms, immersive media, cloud gaming, autonomous systems, and machine vision is placing unprecedented pressure on communication and storage infrastructures. Although conventional video codecs such as H.266/VVC, AV1 and HEVC continue to improve coding efficiency through decades of algorithmic refinements, their block-based architecture is approaching diminishing returns. Their design relies on handcrafted signal processing modules — including prediction, transforms, quantization, in-loop filtering and entropy coding — that have been carefully optimized but remain fundamentally constrained by assumptions about natural image statistics. 

Learned Video Compression (LVC) has emerged during the past decade as a promising alternative. Instead of optimizing individual codec modules independently, learned codecs formulate compression as a global end-to-end optimization problem. Neural networks jointly learn the analysis transform, synthesis transform, entropy model and motion representation by minimizing a rate-distortion objective where (D) measures reconstruction distortion while (R) estimates the coding rate through differentiable entropy models. 

Recent learned image and video codecs have demonstrated compression performances comparable to — or surpassing — the latest standard codecs for several operating points. More importantly, they provide a flexible framework in which every component can be jointly optimized for a target application. 

Despite these impressive advances, several important challenges remain before learned video codecs can become a practical replacement for conventional standards. 

First, current architectures remain computationally demanding. Deep convolutional encoders, decoders and optical-flow estimation networks require millions of parameters and substantial computational resources during both training and inference. 

Second, entropy modeling remains one of the principal bottlenecks. State-of-the-art autoregressive models improve coding efficiency but introduce strong sequential dependencies that significantly reduce decoding parallelism. 

Third, current codecs still represent images and videos through discrete latent feature maps. While highly effective, these representations inherit many limitations of grid-based signals, including resolution dependence, interpolation artifacts and limited geometric adaptability. 

Finally, temporal redundancy is often modeled explicitly through optical flow or recurrent networks, both of which become increasingly expensive for long sequences or high-resolution content. 

These limitations motivate the exploration of alternative signal representations capable of providing richer spatial and temporal modeling while maintaining efficient entropy coding. 

Hybrid Implicit Neural Representations 

Implicit Neural Representations (INRs) have recently emerged as a fundamentally different paradigm for representing visual signals. Rather than storing an image or a video as discrete samples on a grid, an INR represents the signal as a continuous function where the neural network maps spatial (or spatio-temporal) coordinates 

XΧ  directly to signal values ΥΥ.

Continuous representations naturally provide: 

• arbitrary-resolution reconstruction, 

• smooth interpolation, 

• compact parameterization, 

• geometric continuity. 

However, pure coordinate-based INRs exhibit several limitations for compression. They often require relatively large neural networks, suffer from slow convergence, and struggle to reproduce highly textured regions without substantially increasing model complexity. 

Hybrid Implicit Neural Representations (Hybrid INRs) have recently addressed these limitations by combining learned latent grids with coordinate-based neural decoders. 

Instead of relying solely on coordinates, the decoder receives both 

• local latent features extracted from hierarchical latent grids, 

• continuous spatial coordinates. 

The latent grids capture high-frequency local structures while the implicit decoder reconstructs the continuous signal. Hierarchical latent resolutions allow coarse image structures to be represented at low spatial resolutions and fine details at higher resolutions. 

Among recent examples, Cool-Chic [Ladune et al., 2003] and C3 [Kim et al., 2024] have demonstrated that hybrid INRs constitute an extremely competitive framework for learned image compression. Its hierarchical latent representation, lightweight implicit decoder and learned entropy model achieve state-of-the-art rate-distortion performance while requiring significantly fewer decoder parameters than conventional CNN-based architectures. 

The advantages of hybrid INRs include: 

• continuous image representation, 

• compact decoder architectures, 

• scalable latent hierarchies, 

• compatibility with autoregressive entropy models, 

• excellent adaptation to image geometry. 

Nevertheless, current hybrid INR methods remain largely restricted to still images. Extending this framework to video introduces new scientific questions regarding temporal representation, temporal prediction and entropy modeling [Leguay et al., 2024]. 

Extending Hybrid INRs to Learned Video Compression 

The central objective of this project is to develop a new generation of learned video codecs based on spatio-temporal hybrid implicit neural representations.  Instead of treating each frame independently, the proposed framework will model an entire video sequence as a continuous function where the decoder jointly exploits spatial coordinates, temporal coordinates and hierarchical latent representations.  The key challenge lies in introducing temporal redundancy into the latent hierarchy without sacrificing the compactness that makes hybrid INRs attractive.  Several complementary research directions will be investigated. 

Hierarchical Spatio-Temporal Latent Representation 

• Current hybrid INR codecs organize latent grids across multiple spatial resolutions. This project proposes extending the representation to the temporal dimension. This could include temporal latent pyramids, key-frame and inter-frame latent decomposition, multi-scale temporal feature grids, adaptive temporal resolutions depending on motion complexity… 

• Continuous Temporal Prediction 

• Conventional learned video codecs typically estimate optical flow between successive frames. Hybrid INRs provide an opportunity to replace explicit motion estimation by continuous temporal modeling, such as latent-space temporal prediction, learned temporal embeddings… 

• Hierarchical Context Modeling 

• Entropy coding remains one of the largest contributors to coding performance. Inspired by recent advances in hierarchical latent modeling for image compression, previously decoded latent levels and previous temporal instants can provide rich contextual information for entropy prediction. 

• Lightweight Neural Architectures 

One of the strengths of hybrid INRs is the possibility of using extremely compact decoders. The project aims to continue investigating lightweight architectures aiming to reduce both computational complexity and memory footprint while preserving coding performance. 

Expected Scientific Contributions 

The proposed research will contribute to the development of a unified framework for learned video compression based on implicit neural representation. 

Expected contributions include: 

• novel spatio-temporal hybrid INR architectures for video compression; 

• lightweight decoder architectures suitable for practical deployment; 

• comprehensive rate-distortion-complexity evaluation against state-of-the-art learned video codecs and conventional standards such as VVC.