What Should a Robot Dream About? Notes on world models for robotics Somewhere in the last eighteen months, “world model” stopped being a research term and became a land grab. Yann LeCun walked out of Meta and raised a billion-dollar seed round in Paris to build them. DeepMind shipped Genie 3 and then opened it to the public as Project Genie: type a sentence, walk around inside the result at 24 frames per second. Waymo bolted a driving-specialized world model on top of Genie 3 so its cars could rehearse tornadoes and elephants. NVIDIA went from Cosmos to Cosmos 3 in eighteen months and now calls world models the operating layer of “physical AI.” Tesla folds its cars and its humanoid into one...

🤖 Training Morphology-Agnostic Locomotion Policies with Heterogeneous Robotic Environments in Isaac Lab 🌍 Introduction: The Dream of Universal Robot Control Imagine a world where a single neural network policy can control any legged robot, from Boston Dynamics’ Spot to ANYmal to Unitree’s quadruped family, without requiring separate training for each platform. This isn’t just theoretical elegance; it’s a practical necessity. As robotics deployments scale, maintaining separate policies for each robot variant becomes a maintenance nightmare. When you need to update locomotion behavior, you’d have to retrain dozens of specialized policies instead of updating one universal controller.

Bridging the Reality Gap: A Survey of Domain Randomization and Future Horizons Simulation is the bedrock of modern robotic learning. It allows us to train agents safely and parallelize data collection at scales impossible in the physical world. However, the “Reality Gap”, the discrepancy between simulated physics and the real world, remains a formidable barrier. Policies trained in perfectly deterministic simulations often fail catastrophically when deployed on physical hardware. The dominant solution to this problem is Domain Randomization (DR). By randomizing the physical parameters of the simulation (friction, mass, damping, etc.) during training, we force the agent to learn robust...

Although transformers have been dominating the generative AI domain, but I still find this ablation study of Bayesian GAN quite insightful, so I am going to publish it on my website. It is an old project. Abstract This work is about exploring the capacity and limitation of the Bayesian GAN algorithm. The existing framework of the Bayesian GAN is still an unexplored method of learning generative adversarial networks in a bayesian context where the key idea to marginalize the posterior over the weights of the generator and discriminator using a variant of the stochastic gradient descent algorithm namely Hamiltonian Monte-Carlo. It has been demonstrated that this method gives state-of-the-art...

Introduction The rand() function generates uniformly-distributed numbers between 0~RAND_MAX, where RAND_MAX depends on the implementation and language. For example, in Matlab, RAND_MAX is 1, while in C/C++ RAND_MAX is the maximum integer number of the int representation. The problem is then how to generate numbers distributed with the Gaussian PDF based on rand(), and how to check that what you generate is in fact Gaussian or Pareto distributed. There are many approaches to generate normally-distributed random numbers starting from a uniform distribution. This report describes three methods: Inverse Transform Sampling, Box-Muller Algorithm, and Ziggurat Algorithm. Moreover, finally, we show...

Distributional RL In common RL approaches, we have a value function which returns a single value for each action. This single value is the expectation of a true distribution which in the distributional RL, we seek to return that for each action. In common RL, value function is defined using the Bellman’s equation: $$Q(x,a) = \mathbb{E} R(x,a) + \gamma \mathbb{E} Q(X', A')$$In distributional RL, we need to drop those expectations and so the distributional Bellman’s equation would look like (onte that this equality sign means that the sides are random variables drawn from the same probability distribution law):

ACER It is an off-policy actor-critic model with experience replay, greatly increasing the sample efficiency and decreasing the data correlation. The reason for doing that is because ACER is off-policy and to control the stability of the off-policy estimator: It has multiple workers (as A2C); It uses replay buffer (as in DQN); It uses Retrace Q-value estimation; It truncates the importance weights with bias correction; It applies TRPO. Deep Q-learning methods are most sample efficient techniques. However, they have two important limitations. First, the deterministic nature of the optimal policy limits its use in adversarial domains. Second, finding the greedy action with respect to the Q...

Conservative uncertainty estimation by fitting prior networks: By using random prior networks and comparing its output with the learnable network, one can easily determine the level of uncertainty regarding the given input data. The more uncertain, the more unseen that data is, thus is a good option for exploration. Once the data is given to the learnable network, it tries to match its output to the one from the random prior network.

1. Introduction Definition of intelligence: perceive, understand, predict, and manipulate a world far larger and more complicated than itself. AI definition lies in these aspects: “thought processes and reasoning” vs “behaviour” and “humanly” vs “rationally”. Acting humanly: Turing test. Thinking humanly: Cognitive modeling approach, in line with cognitive science. Thinking rationally: The “laws of thought” approach, in line with logic. Acting rationally: The rational agent approach, which is an agent acting so as to achieve the best expected outcome. The quest for “artificial flight” succeeded when the Wright brothers and others stopped imitating birds and started using wind tunnels and...

{% raw %} 1. Linear Algebra Linear algebra is the study of vectors and certain rules to manipulate vectors. For a real-valued system of linear equations we obtain either no, exactly one, or infinitely many solutions. In a system of linear equations with two variables $x_1; x_2$, each linear equation defines a line on the $x_1x_2$-plane. Since a solution to a system of linear equations must satisfy all equations simultaneously, the solution set is the intersection of these lines.

A Simple Neural Attentive Meta-Learner, algorithm: SNAIL - Uses a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information. - Rather than training the learner on a single task (with the goal of generalizing to unseen samples from a similar data distribution) a meta-learner is trained on a distribution of similar tasks, with the goal of learning a strategy that generalizes to related but unseen tasks from a similar task distribution. - Combines temporal convolutions, which enable the meta-learner to aggregate contextual information from past experience, with causal...

Methods: Saliency Maps: Intuitively, the absolute value of the gradient indicates those input features (pixels, for image classification) that can be perturbed the least in order for the target output to change the most, with no regards for the direction of this change. Gradient Input: The attribution is computed taking the (signed) partial derivatives of the output with respect to the input and multiplying them feature-wise with the input itself. Integrated Gradient:

In RL, an agent tries to learn the dynamics of an environment by trial and error by interacting with the environment. It is difficult for the agents to learn a general policy that applies across similar environments. Furthermore, they do not get reasonable performances on the same environments of varying difficulty level. To address it, here I try to introduce a new pipeline for generating environments with varying difficulty levels to improve the agents’ performances. Inspired by how humans learn difficult tasks, which basically starts with easy settings and slowly increases the difficulty of the settings, we designed a curriculum learning framework in which the agent first is trained on a...

Lecture 1: Introduction to Reinforcement Learning Planning: rules of the game are given, perfect model inside agent’s head, plan ahead to find optimal policy(look ahead search or tree search). In RL environment is unknown, in planning environment is known. Types of RL agents: Policy based Value function based Actor critic(combines policy and value function) Agent’s model is a representation of the environment in the agent’s head. Agent is our brain, is the algorithm we come up with.