CLIPort
1 / 5Instruction “put the gray block in the green bowl”
SlotDiT: Object-Centric Representations for Diffusion Transformers
British Machine Vision Conference (BMVC) 2026
Abstract
Text-conditioned latent diffusion models are powerful video generators and promising backbones for robotic applications, but common pixel- or VAE-based latents prioritize reconstruction fidelity and lack explicit semantic structure. Object-centric slots instead decompose a scene into compact representations of its entities.
We introduce SlotDiT, a text-guided Diffusion Transformer operating in a slot-based latent space. Given one reference image, it extracts object-centric slots and, conditioned on the text instruction and scene context, autoregressively denoises future slot trajectories, enabling the prediction of future scene dynamics.
We compare slot, VAE-based, and semantics-aligned latents within one unified DiT setup across four robotic datasets. SlotDiT delivers competitive visual quality, consistently stronger task completion, leading robot-control performance, and substantially faster inference.
SlotDiT
Given a reference image and language instruction, SlotDiT first encodes the scene into a compact set of object-centric slots. Conditioned on the instruction and observed slots, the diffusion transformer iteratively denoises future slot trajectories from Gaussian noise.
The latest predicted slots are reused as context for autoregressive generation, and a frozen object-centric decoder can render the resulting trajectories as future frames.
Evaluation
We isolate the effect of the latent space by comparing object-centric slots with VAE-based and semantics-aligned representations under the same DiT architecture, training procedure, and inference scheme.
Evaluation comprises two distinct tasks: text-guided video generation on CLIPort, LanguageTable-Synthetic, LanguageTable-Real, and BridgeData V2; and robot control on CLIPort and LanguageTable-Synthetic.
Text-Guided Video Generation
SlotDiT outperforms all the VAE-based baselines by achieving the highest task-success rate on CLIPort (87.8%), LanguageTable-Synthetic (79.7%), and LanguageTable-Real (64.6%), while remaining competitive on BridgeData V2 (57.6%). VAE-based baselines can obtain stronger perceptual scores, but visual fidelity alone does not reliably indicate instruction following or task completion.
Qualitative comparison
These rollouts illustrate the metric–task gap: visually plausible baselines may preserve appearance yet fail to execute the instruction, while SlotDiT models the task-relevant object dynamics and its predictions correctly illustrate the completed task.
LanguageTable-Synthetic
1 / 5Instruction “slide the moon towards the cube”
LanguageTable-Real
1 / 5Instruction “push the yellow hexagon to the top center of the board”
BridgeData V2
1 / 5Instruction “move the corn to the stove”
Robot Control
For the robot control setting, we use inverse dynamics models (IDMs) to map predicted latents to actions. SlotDiT reaches 73.0% success in open-loop robot control on CLIPort and 74.5% in the in-distribution closed-loop LanguageTable-Synthetic setting, leading all learned predictors and nearly matching the GT slots oracle.
It remains strongest in the LanguageTable-Synthetic's out-of-distribution scenarios—harder eight-block scenes and unseen instruction templates—showing that object-centric dynamics retain task-relevant information beyond training.
Efficiency
SlotDiT represents each frame with only 10 slot tokens, compared with 256 tokens for the alternative latents. It reaches 9.33 FPS, a 5.68× speedup over DiT + SD-VAE, with most of the gain coming from diffusion sampling, highlighting the efficiency gains from the compact and semantic object-centric latent space.
Controllability
Given the same initial observation, SlotDiT generates distinct future trajectories for two different instructions, while preserving the overall scene layout and object identities. In both cases, SlotDiT is able to identify the new objects and correctly applies the described motion, seamlessly adapting to the new text instruction.
CLIPort
Instruction “put the yellow block in the blue bowl”
Changed “put the red block in the blue bowl”
LanguageTable-Synthetic
Instruction “move the star to the blue cube”
Changed “move the pentagon to the blue cube”
LanguageTable-Real
Instruction “slide the green star on top of yellow hexagon”
Changed “slide the red circle on top of yellow hexagon”
BridgeData V2 - Example 1
Instruction “move the pot to the cooker”
Changed “move the yellow object to the cooker”
BridgeData V2 - Example 2
Instruction “move the corn to the stove”
Changed “move the lit to the stove”
Object-Centric Behaviour
SlotDiT models scene dynamics through persistent object-level representations. The masks show the decomposition of each predicted frame, while the individual slot reconstructions reveal how individual scene entities are represented and how they evolve over time.
BridgeData V2 · Example 1
Instruction-conditioned object-centric rollout
BridgeData V2 · Example 2
Instruction-conditioned object-centric rollout
Key Findings
- Object-centric DiT latents improve instruction following and task completion while retaining competitive visual quality.
- Perceptual video metrics alone do not measure whether a robotic instruction was actually completed successfully.
- Compact object-centric slots make diffusion inference substantially more efficient than dense alternatives.
- Object-centric structure proves to be a powerful inductive bias for diffusion-based generative modeling in robotic environments.
Citation
@inproceedings{plepi2026slotdit,
title = {SlotDiT: Object-Centric Representations for Diffusion Transformers},
author = {Plepi, Gjergj and Behnke, Sven},
booktitle = {British Machine Vision Conference (BMVC)},
year = {2026}
}