In my last article , we taught a robot to balance a pendulum using reinforcement learning. We never told it what to do. We only told it how well it did, through a reward signal, and let it discover a strategy on its own across millions of simulation steps.

This time, we’re doing the exact opposite. We’re going to show the robot what to do.

Welcome to imitation learning.

From rewards to demonstrations

Let’s put them next to each other before going further.

With reinforcement learning, you don’t need any data. But there’s a catch (actually a few):

• You have to hand-craft a reward function. (Remember how much time we spent tweaking those reward weights for the pendulum?)
• You usually need a simulator, because the agent learns by trial and error, and you can’t crash a real robot a million times.
• And then you let it explore for millions of steps until it stumbles onto something good.

With imitation learning, you flip the whole thing around. The robot learns by copying you. The price you pay is that now you need real-life demonstrations.

That sounds much simpler, and in many ways it is. It’s a big part of why robot manipulation has exploded recently. But imitation learning has its own dirty little secret, one that bit me hard at a hackathon last December. Hold that thought, we’ll get to it.

What is imitation learning, really?

At its core, the simplest form of imitation learning is just supervised learning wearing a robotics costume.

Think back to how we framed things last time. Supervised learning is “here’s the input, here’s the correct label, now learn to produce the label.” Imitation learning, in its most basic form (called behavior cloning), is exactly that:

• Input: what the robot sees and feels right now. The camera image(s) and the position of each joint, sometimes more, like motor currents or force sensors.
• Label: the action I took at that exact moment. Where I moved each joint.

You record a bunch of (observation, action) pairs while doing the task, and you train a neural network to map one to the other. That’s it. The robot is literally cloning your behavior, one moment at a time.

There are fancier flavors of imitation learning out there (things like DAgger, or inverse reinforcement learning where you try to recover the reward function from demonstrations). But the models everyone is excited about right now, including the two we’ll dig into, are built on good old behavior cloning. So that’s what we’ll focus on.

The interesting part isn’t the idea, which is almost embarrassingly simple. It’s everything that goes wrong when you actually try it, and the clever tricks people invented to fix it.

Collecting demonstrations: teleoperation

Before any of that, you need data. And in robotics, the data is the model. There’s no giant dataset of “opening electrical cabinets” sitting on the internet. You have to make it yourself.

The way you do this is teleoperation. You physically move the robot through the task while recording everything.

The classic setup is a leader and a follower. You have two arms: a leader arm that you move by hand, and a follower arm that mirrors its movements. As you guide the leader through the task, the follower copies it, and the whole time you’re recording the joint positions and the camera feed. Each successful run is one “episode.”

I used two different arms across my experiments:

• The SO-101, which I’d worked with before, has 5 degrees of freedom.
• The Seeed Studio arm has 6 degrees of freedom.

That extra degree of freedom matters more than it sounds. With 5 DoF, there are orientations the gripper simply can’t reach. The 6th joint gives you a free wrist, which for a task like reaching in at an angle to grab a cabinet door is the difference between “possible” and “constantly fighting the robot.” The flip side: every degree of freedom you add is one more thing the model has to learn to coordinate.

This is me teleoperating a Seeed Studio follower arm with a leader arm:

Data collection is repetitive and a little tedious. You do the same task over and over, a few dozen times, trying to be consistent but also varied enough that the model doesn’t memorize one exact trajectory. You quickly develop a deep respect for anyone who collects thousands of episodes.

And here’s where I learned my lesson the hard way.

We laughed about it at the time. But that story is actually the perfect doorway into the single most important concept in imitation learning, and the thing every model in this space is fighting against. Let me first show you the model I used, and then come back to why my teammate broke it.

Under the hood: the ACT model

The model I trained from scratch for these tasks is called ACT, which stands for Action Chunking with Transformers. It came out of the ALOHA project and is one of the most popular starting points for robot imitation learning today.

Let’s build it up piece by piece, because every part of it is a fix for a specific problem.

The problem: predicting one action at a time is fragile

The naive behavior cloning we described earlier predicts a single action from a single observation. Then the robot moves, observes again, predicts the next action, and so on.

The problem is that tiny errors compound. The robot makes a small mistake, which puts it in a slightly unfamiliar state, where it makes a slightly bigger mistake, which puts it somewhere even more unfamiliar… and within a second or two it has drifted somewhere it has never seen and has no idea what to do. (We’ll formalize this in the next section, it has a name.)

The fix: action chunking

Instead of predicting one action, ACT predicts a whole chunk of future actions at once, say the next ~100 steps.

Why does this help? Because it shortens the number of decisions the model has to make. If you commit to 100 actions at a time, you’ve effectively divided the length of your task by 100, and there are far fewer opportunities for errors to accumulate and snowball. The robot follows a short, coherent plan instead of nervously re-deciding everything at every single step.

Smoothing it out: temporal ensembling

There’s a subtlety. If you predict a chunk, execute all of it, then predict the next chunk, you get a little “jerk” at the boundary between chunks where the new plan disagrees with the old one.

ACT’s trick is temporal ensembling. It actually queries the model at every timestep, producing overlapping chunks, and then averages the predictions for the current moment across all the chunks that have an opinion about it. The result is smooth, continuous motion instead of jerky stop-and-go. (Notice the philosophy is the opposite of my pendulum, where I penalized jerky motion in the reward. Here there’s no reward to do that, so the smoothing is baked into the architecture.)

The clever bit: the CVAE and human messiness

Here’s the part I find most elegant.

When a human demonstrates a task multiple times, the demonstrations are multimodal. You don’t do the exact same thing every time. Sometimes you approach the cabinet handle from slightly above, sometimes from the side. Both are correct.

A naive network trained to minimize error on this data does something terrible: it averages the two valid approaches and aims for the middle, which might be straight into an obstacle. Averaging two correct answers can give you a very wrong one.

ACT solves this by being a CVAE (Conditional Variational Autoencoder). The intuition, without drowning in the math:

• During training, an encoder looks at the demonstration and compresses its “style” into a small latent variable $z$. Think of $z$ as capturing which way you happened to do it this time.
• The main network (a transformer encoder-decoder) then generates the action chunk conditioned on the images, the joint positions, and this style variable $z$.
• The model is trained with two objectives: an $L_1$ loss that says “reproduce the demonstrated actions,” and a KL-divergence term that keeps the space of styles $z$ well-behaved (close to a simple Gaussian).

At test time, you simply set $z = 0$ (the “average style”) and let the network produce a coherent action chunk. Because the multimodality was absorbed into $z$ during training, the network no longer has to average incompatible behaviors. It picks one clean way to do the task instead of mushing them together.

Put it all together: ResNet backbones turn the camera images into features, a transformer fuses those with the joint positions and the latent $z$, and out comes a chunk of smooth, human-like actions. Trained on a few dozen of my teleoperated episodes, this is what learned to open the cabinet.

The Achilles heel: distribution shift

Now let’s come back to my hackathon teammate.

The fundamental weakness of behavior cloning has a name: distribution shift (or covariate shift). It’s the formal version of the compounding-errors problem we hand-waved at earlier.

Here’s the issue. The model only ever saw the states that appear in your demonstrations. As long as it stays on that familiar path, it knows what to do. But the moment it drifts even slightly off-distribution, into a state it never saw in training, it’s flying blind. And because each mistake pushes it further from familiar territory, errors don’t just add up, they compound.

Action chunking helps a lot here, because committing to a short plan gives errors fewer chances to snowball. But it doesn’t cure the problem. Nothing fully does, short of collecting more and better data.

And this is exactly what happened at the hackathon. The model didn’t actually learn “open the cabinet.” It learned to reproduce the pixels it saw during training, and my teammate sitting in the background was one of those pixels. When he moved, the input no longer matched anything in the training distribution, and the robot was instantly off-distribution, flying blind. It hadn’t learned the task. It had learned the data.

So how do we get past this? How do we get robots that actually understand the task, that generalize to a cabinet they’ve never seen, in a kitchen they’ve never been in, with a teammate sitting wherever he likes? That’s the frontier, and it’s where things get really exciting.

Scaling up: from one task to generalist robots

The ACT model I trained does exactly one thing: open one cabinet. Train it on a new task and you start more or less from scratch. That doesn’t scale. You can’t collect demonstrations for every task a robot might ever face.

The big idea on the frontier is the same one that transformed language and vision: scale. Instead of one small model per task, train one enormous model on a huge, diverse pile of robot data, across many tasks, many robots, and many environments. With enough variety, the model stops memorizing specific trajectories and starts learning something closer to general manipulation skills.

These models are called VLAs: Vision-Language-Action models.

The name says it all. A VLA fuses three things into a single model:

• Vision: what the robot sees through its cameras.
• Language: an instruction in plain English, like “open the electrical cabinet.”
• Action: the joint commands sent to the robot.

The “language” part is the game-changer. These models are typically built on top of a pre-trained vision-language model (the same kind of model that can look at a photo and describe it). That backbone already knows a staggering amount about the world from internet-scale data: what a cabinet is, what a handle is, what “open” means. The robot doesn’t have to learn all of that from your forty demonstrations. It just has to connect that existing knowledge to motor actions.

Fine-tuning Pi0.5

The VLA I got my hands on is Pi0.5 (π0.5), from Physical Intelligence . It’s one of the most capable open generalist robot policies available, explicitly designed for open-world generalization, the ability to operate in homes and environments it has never seen.

The contrast with ACT is striking:

• Training ACT from scratch, the model knows nothing. Your few dozen episodes have to teach it the entire task from zero.
• Fine-tuning Pi0.5, the model already understands cabinets, doors, handles, and the general physics of manipulation. Your episodes just nudge a knowledgeable generalist toward your specific setup. You need far less data to get something that generalizes far better.

One more technical nugget worth knowing, because it’s a genuinely different approach from ACT’s CVAE. To generate its continuous actions, Pi0.5 uses an action expert trained with flow matching.

You can think of flow matching as a cousin of the diffusion models behind AI image generators. Instead of producing an action in one shot, the model starts from pure random noise and learns a “velocity field” that smoothly flows that noise, over a few steps, into a real action chunk. The beautiful thing is that this handles the multimodality problem we discussed earlier very naturally: just like an image generator can produce many different valid images, flow matching can produce many different valid ways to do the task, without averaging them into mush. It’s solving the same problem ACT’s latent $z$ solves, but with a much more powerful and scalable tool.

Imitation learning vs. reinforcement learning

Now that we’ve been through both this article and the RL deep dive , we can put them side by side honestly.

Neither one is “better.” They’re tools for different jobs. RL shines when the behavior is hard to demonstrate but easy to score (how would you teleoperate a robot into balancing my chaotic double pendulum?). Imitation shines when the behavior is easy to demonstrate but hard to score (what’s the reward function for “open a cabinet nicely”?).

What’s next: toward robots that imagine

So we’ve now seen two of the big paradigms in modern robot learning. We tell a robot what good looks like with rewards, or we show it what to do with demonstrations. Both have a frustrating limitation in common, though: they need an enormous amount of experience, either millions of simulation steps or piles of demonstrations, because the robot has no real model of how the world works. It can’t think ahead.

What if a robot could build its own internal simulator of reality, watch a situation, and imagine what would happen if it tried something, all in its head, before moving a single motor?

That’s the idea behind world models, and it might be the thing that ties this whole series together: from rewards, to demonstrations, to imagination. We’ll write an article about these.

In this article, I want to open the black box.

We’ll walk through:

• What reinforcement learning (RL) really is (in practical terms)
• How to think about the agent, environment, states, actions, and rewards
• What policy gradients are and why they’re a good fit for continuous control
• How PPO (Proximal Policy Optimization) works under the hood
• How all of this applies to our double pendulum example

What is reinforcement learning, really?

Reinforcement learning is a type of machine learning. If supervised learning is “here’s the input, here’s the correct label, now learn to produce the correct label”, then reinforcement learning is:

“Here’s the world. Try things. I’ll tell you if you’re doing well or badly.”

Reinforcement learning has two advantages:

• We do not need recorded training data.
• Our model can discover behaviors by itself. As we have seen with our pendulum, it learned a way to swing it up that I could not have demonstrated.

But we do need to explicitly state a reward.

Instead of labeled examples, the agent gets:

• State: what the world looks like right now
• Action: what the agent decides to do
• Reward: a scalar number measuring how good that action (and resulting state) was
• Next state: what the world looks like after the action

This loop repeats over and over.

For the double pendulum:

• State might include:
  • Cart position and velocity
  • First arm angle and angular velocity
  • Second arm angle and angular velocity
• Action:
  • A continuous force to apply to the cart
• Reward:
  • Positive when the arms are upright and the motion is smooth
  • Negative when the arms fall or move too violently (see Part 1 to see our detailed Reward)

The key idea:
The agent is not told what to do. It’s only told how well it did, using the reward signal.

The tuple (state, action, reward, next state) is stored for learning.

After a batch of steps, the agent uses all the accumulated data to update the neural network parameters.

This “collect experience → update network” cycle repeats until the behavior converges to something useful. Over time (and millions of time steps), the agent discovers a strategy (called a policy) that tends to produce high total reward.

Immediate rewards vs. future returns: a crucial distinction

Rewards are computed immediately

At each time step $t$, the environment computes a reward $r_t$ based on:

• The current state $s_t$
• The action $a_t$ that was just taken
• The resulting next state $s_{t+1}$

For our double pendulum, at every control step (every 16.67 ms), the reward function:

• Measures how upright the arms are (position reward)
• Measures how smooth the motion is (velocity penalties)
• Computes a single scalar value: $R_t$

This reward is immediate – it’s computed right after the action is applied, based on what happened in that single time step.

But the agent optimizes for future returns

However, the agent doesn’t just care about the immediate reward. It cares about the total future reward it can expect from this point forward.

This is captured by the return (also called the discounted return):

Where:

• $R_t$ is the immediate reward at time $t$
• $\gamma$ is the discount factor (typically 0.99 or 0.999)
• Future rewards are discounted (weighted less) the further they are in the future

Why discount future rewards?

The discount factor $\gamma$ serves several purposes:

1. Mathematical convenience: Without discounting, infinite-horizon problems can have infinite returns, making optimization impossible.

2. Uncertainty: The further in the future, the less certain we are about what will happen. A reward now is more valuable than the same reward later.

3. Practical behavior: It encourages the agent to achieve rewards sooner rather than later.

For example, with $\gamma = 0.99$:

• A reward of +1.0 at time $t$ contributes +1.0 to the return
• A reward of +1.0 at time $t+1$ contributes +0.99 to the return
• A reward of +1.0 at time $t+10$ contributes only +0.90 to the return

4. Core concepts in RL

Let’s define some core concepts in RL:

4.1 Policy $\pi(a \mid s)$

The policy is the agent’s behavior. Its role is to output an action given a state.

• Input: current state
• Output: probability distribution over actions

For continuous control like our double pendulum, the policy output is usually a Gaussian distribution:

The policy is usually implemented as a neural network.

• The neural network outputs:
  • A mean vector $\mu(s)$
  • A standard deviation vector $\sigma(s)$ (or sometimes a global $\sigma$)
• The action is sampled as: \(a \sim \mathcal{N}(\mu(s), \sigma(s))\)

This stochasticity is important:
The agent must explore different actions to learn what works, so we allow this by having a statistical distribution of actions rather than a single deterministic action.

4.2 Value function $V(s)$

The value function tells us:

“Starting from this state, how much total future reward can I expect if I follow my current policy?”

Formally: \(V^\pi(s) = \mathbb{E}\left[\sum_{t=0}^T \gamma^t r_t \mid s_0 = s, \pi\right]\)

“What is my expected total reward given my current state and my current policy”

In practice, we approximate this with another neural network, the critic.

4.3 Advantage function $A(s,a)$

The advantage $A(s,a)$ measures how much better an action turned out to be, compared to following our policy.

One common definition:

Where:

• $Q(s,a)$ is called the action-value function. It is the expected return if we take action $a$ in state $s$ and follow the policy afterwards.

• $V(s)$ is called the state-value function. It is the average expected return if we just follow the policy from state $s$.

• If an action performed better than expected, advantage > 0 → push policy towards that action.
• If it performed worse than expected, advantage < 0 → push policy away from that action.

Intuition with a Simple Example

You’re playing a game:

• In state $s$, following your policy typically gives you 10 points. (This is $V(s) = 10$)
• But taking action $a_1$ gives you 15 points. (This is $Q(s, a_1) = 15$)
• Taking action $a_2$ gives you 9 points. (This is $Q(s, a_2) = 9$)

Then:

• $A(s, a_1) = 15 - 10 = +5$ → better than average
• $A(s, a_2) = 9 - 10 = -1$ → worse than average

Interpretation:

• Your policy should shift probability toward $a_1$ (because it’s better than average)
• And shift probability away from $a_2$ (because it’s worse than average).

5. Value-based methods vs policy-based methods

There are two broad ways to solve RL problems:

• Value-based methods: learn a value function that maximizes the reward and derive a policy from it (e.g., Q-learning, DQN). Indeed, if I can efficiently predict my future rewards, I can derive a policy that maximizes it.
• Policy-based methods: directly learn a policy that maps states to actions and maximizes the reward.

In the policy-based method, we call $\theta$ the policy parameters of our neural network.
The gradient of the objective with respect to $\theta$ looks like:

Where:

• $\log \pi_\theta(a \mid s)$ is how likely the policy thinks that action is. (we take the log to simplify the gradient maths and provide stability by turning products into sums)
• $G_t$ is our discounted return function seen earlier.

(we took a shortcut to arrive at this formula, if you want the full details, check out here )

Then we can use this gradient to update the policy (neural network) parameters by doing the usual back propagation.
If the reward is high → increase the probability of that action in that state.
If the reward is low → decrease the probability.

Value-based methods are great for small sampled data where the action space is discrete, but they perform weakly in exploration because they tend to directly maximize the reward (greedy).
Policy-based methods are great for continuous action spaces (robotics) and perform better at exploration since stochasticity is built in. They do need more data.

6. The hybrid architecture: actor-critic methods

In policy methods, we need to use $G_t$ in our gradient to update our model parameters. However, $G_t$ is a very noisy signal with high variance: in some episodes an action can lead to a high final reward and in others to a low final reward due to the uncertainty and variability of the future.
Due to this high variance, in order to properly train a policy-based method, we need a lot of episodes and data.
Actor-critic methods solve this high-variance issue by combining both approaches.

In actor-critic methods, we will train both models at the same time:

• A policy that controls our agent and outputs actions
• A value function predicting how good a state is (i.e., the expected future return starting from that state) as we’ve seen earlier.

Instead of using $G_t$ directly, our Actor-Critic method will use the Advantage function we have seen earlier: \(A(s, a) = Q(s, a) - V(s)\)

So the gradient becomes:

Keep in mind that $V(s)$ is a neural network itself now.

Down the rabbit hole: how to compute Q(s, a)?

If the action-value function $Q(s,a)$ is “what is my reward if I perform the action a and follow the policy afterward” and the state-value function $V(s)$ is “what is my reward if I follow the policy”, then we can express everything based on the Critic when we take the difference.
Mathematically:

As we have

Using the value function \(V(s_{t+1})\) of the next state, we can rewrite \(Q(s_t, a_t)\) as:

and so

(it’s an approximation because we use our critic estimate instead of the true value)
($\gamma$ is our discount factor seen before)
It is called the TD-error ($\delta_t$) and we can easily compute it from the $(S_t, A_t, S_{t+1}, R_t)$ data points we collect in each episode and our Critic model $V(s_t)$.

Even deeper: GAE is better than TD-error

TD-error approximates the Advantage function by looking at the difference between the action-value function $Q$ and the state-value function $V$ over one step.
But we can have an even better approximation by looking at the difference over multiple steps.
Instead of just looking at the advantage at time t, we look at the advantages over the rest of the trajectory. We use a parameter $\lambda$ to control how far we look in time.

Recursively we define
\(A_t = \delta_t + (\gamma\lambda)A_{t+1}\) So
\(A_t = \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2\delta_{t+2} + ...\)

In practice we can compute $A_t$ at the end of an episode starting from the end and computing it recursively.
$\lambda$ allows us to control how far we want to look:

• $\lambda$ = 0 is equivalent to the TD-error approach, low variance but high bias
• $\lambda$ = 1 is equivalent to the full sample approach similar to the policy method, low bias but high variance.

Full summary of the training loop

To wrap everything up, here is the complete training cycle used in an actor-critic method with GAE. This gives a clear, high-level view of how all the pieces fit together.

1. Initialize the Models

• Initialize both networks (with random weights):
  • Actor: outputs actions given a state.
  • Critic: predicts the value of each state.

2. Collect a Batch of Experience

• Run the current policy in the environment for a fixed number of steps.
• At each timestep, record:
  • State: $s_t$
  • Action taken: $a_t$
  • Reward received: $r_t$
  • Next state: $s_{t+1}$
• This set of transitions is called a rollout.

3. Estimate the Values

• Use the critic network to compute value estimates for each state:
  • $V(s_t)$
  • $V(s_{t+1})$

4. Compute the TD-Errors

• For each timestep, compute the temporal-difference error (TD-error): \(\delta_t = R_t + \gamma V(s_{t+1}) - V(s_t)\)
• This measures how “surprising” or “better or worse than expected” each outcome was.

5. Compute the Advantages with GAE

• Compute the advantages using the formula: \(A_t = \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)²\delta_{t+2} + ...\)
• This produces a smoothed, low-variance estimate of the advantage for each action.

6. Train and Update the Critic

• To train the critic, we compute the return estimate: \(\hat{G}_t = A_t + V(s_t)\)
• This serves as the supervised learning target for the value network.

• Update the critic by minimizing the mean-squared error between predicted values and return targets: \(L_{\text{critic}} = (V(s_t) - \hat{G}_t)^2\)

7. Update the Actor

• Using the policy, compute the log probability of each action.

• Update the policy so that actions with positive advantage are more likely, and those with negative advantage are discouraged.
• The basic policy loss is: \(L_{\text{actor}} = -\log \pi(a_t | s_t) \, A_t\)

8. Repeat

• Repeat the loop:
  • Collect new data with the updated policy.
  • Recompute value estimates, TD-errors, advantages, and returns.
  • Update the actor and critic again.
• Over time, both networks improve:
  • The critic predicts values more accurately,
  • The actor learns an increasingly effective policy for the environment.

7. Details behind PPO: Proximal Policy Optimization

Now we get to the algorithm I used to train the double pendulum: PPO (Proximal Policy Optimization).

PPO is popular because it strikes a good balance between:

• Performance: It works very well on complex continuous control tasks.
• Stability: It avoids taking too-large, destabilizing policy updates.
• Simplicity: It’s relatively easy to implement and tune.

7.1 The core idea

Naively, if we use policy gradients, we might make huge updates to the policy after each batch of experience. That can completely break a working policy.

PPO prevents this by constraining how much the policy is allowed to change at each update.

It does this through a clipped objective.

7.2 The probability ratio

PPO compares:

• The old policy $\pi_{\theta_{\text{old}}}$ used to generate data.
• The new policy $\pi_{\theta}$ we’re trying to optimize.

For each state–action pair, we compute the probability ratio (not to be confused with the reward $r_t$):

• If $r_t(\theta) > 1$: the new policy makes that action more likely.
• If $r_t(\theta) < 1$: the new policy makes that action less likely.

We want to increase the likelihood of actions with positive advantage, and decrease for negative advantage. But we don’t want $r_t(\theta)$ to change too much in one update.

7.3 The clipped objective

PPO uses the following (simplified) clipped objective:

Where:

• $\epsilon$ is a small number like 0.1 or 0.2.
• The clip function keeps $r_t(\theta)$ within $[1 - \epsilon, 1 + \epsilon]$.

Intuition:

• For good actions (advantage > 0), we want to increase their probability, but not too much.
• For bad actions (advantage < 0), we want to decrease their probability, but not too much either.

PPO takes the minimum of the unclipped and clipped objective. This effectively penalizes updates that would change the policy too drastically.

The result: conservative, stable learning steps that improve the policy without breaking it.

8. Where did we define all of these in our model?

The policy, value function, and PPO hyperparameters above live in the project config at DoublePendulumIsaacLab/DoublePendulumTraining/source/DoublePendulumTraining/DoublePendulumTraining/tasks/manager_based/doublependulumtraining/agents/skrl_ppo_cfg.yaml.

Specific mappings:

• policy and value: Define the two neural nets (both 128×128 ELU MLPs).
• agent: Sets PPO knobs such as discount_factor ($\gamma$), lambda ($\lambda$), and ratio_clip ($\epsilon$).
• trainer: Controls rollouts (batch collection length) and timesteps (total training duration).

Our config for training the model in Isaac Lab is defined in ‘skrl_ppo_cfg.yaml’

Here is the full config:

seed: 42


# Models are instantiated using skrl's model instantiator utility
# https://skrl.readthedocs.io/en/latest/api/utils/model_instantiators.html
models:
  separate: False
  policy:  # see gaussian_model parameters
    class: GaussianMixin
    clip_actions: False
    clip_log_std: True
    min_log_std: -20.0
    max_log_std: 2.0
    initial_log_std: 0.0
    network:
      - name: net
        input: OBSERVATIONS
        # Increased network size from [32, 32] to [128, 128] for better learning capacity
        # Balancing a double pendulum requires more complex control, so a larger network
        # can learn more sophisticated balancing strategies
        # 128 units per layer provides sufficient capacity for this challenging task
        layers: [128, 128]
        activations: elu
    output: ACTIONS
  value:  # see deterministic_model parameters
    class: DeterministicMixin
    clip_actions: False
    network:
      - name: net
        input: OBSERVATIONS
        # Increased network size from [32, 32] to [128, 128] for better value estimation
        # A larger value network can better predict expected returns for complex balancing
        # Matching policy network size for consistency
        layers: [128, 128]
        activations: elu
    output: ONE


# Rollout memory
# https://skrl.readthedocs.io/en/latest/api/memories/random.html
memory:
  class: RandomMemory
  memory_size: -1  # automatically determined (same as agent:rollouts)


# PPO agent configuration (field names are from PPO_DEFAULT_CONFIG)
# https://skrl.readthedocs.io/en/latest/api/agents/ppo.html
agent:
  class: PPO
  rollouts: 32
  learning_epochs: 8
  mini_batches: 8
  discount_factor: 0.99
  lambda: 0.95
  # Learning rate: Slightly reduced for more stable learning with larger network
  # The larger network (128x128) benefits from a slightly lower learning rate
  learning_rate: 3.0e-04
  learning_rate_scheduler: KLAdaptiveLR
  learning_rate_scheduler_kwargs:
    kl_threshold: 0.008
  state_preprocessor: RunningStandardScaler
  state_preprocessor_kwargs: null
  value_preprocessor: RunningStandardScaler
  value_preprocessor_kwargs: null
  random_timesteps: 0
  learning_starts: 0
  grad_norm_clip: 1.0
  ratio_clip: 0.2
  value_clip: 0.2
  clip_predicted_values: True
  entropy_loss_scale: 0.0
  value_loss_scale: 2.0
  kl_threshold: 0.0
  rewards_shaper_scale: 0.1
  time_limit_bootstrap: False
  # logging and checkpoint
  experiment:
    directory: "doublependulumtraining"
    experiment_name: ""
    # write_interval: Number of timesteps between TensorBoard log writes
    # Set to 1 to log every step, or a higher number to log less frequently
    # For training analysis, logging every 10-100 steps is usually sufficient
    write_interval: 10  # Log to TensorBoard every 10 timesteps
    # checkpoint_interval: Number of timesteps between model checkpoints
    # Set to a higher number to save less frequently (saves disk space)
    checkpoint_interval: 100  # Save checkpoint every 100 timesteps


# Sequential trainer
# https://skrl.readthedocs.io/en/latest/api/trainers/sequential.html
trainer:
  class: SequentialTrainer
  # timesteps: Total number of timesteps to train
  # 30000 timesteps = 30000 / (rollouts=32) ≈ 937 iterations
  # Each iteration processes 32 rollouts (32 episodes worth of data)
  # Chosen to give enough updates while keeping wall-clock time reasonable
  timesteps: 30000
  environment_info: log

We define two neural networks, policy and value, corresponding to the policy and value functions discussed earlier. Both are 128×128 ELU MLPs.
The PPO setup uses rollouts: 32, meaning each iteration gathers 32 steps before running updates. learning_epochs: 8 means we reuse that batch eight times per iteration.
Key PPO parameters (discount_factor, lambda, ratio_clip, value_clip) map directly to the concepts above. learning_rate and the KL-based scheduler control step size, and entropy_loss_scale would encourage exploration (kept at 0 here).
—

Other topics in RL

Domain randomization for sim-to-real

• Randomize physics and visuals during simulation (masses, friction, motor gains, sensor noise, textures, lights) so the policy trains on a distribution, not a single “perfect” simulator.
• The goal is a policy that’s robust to reality’s quirks because it has already seen many variations.
• Works well with Isaac Lab: easy to sample parameters each episode, log what was used, and replay troublesome seeds when you see real-world failures.

Two areas I’ll cover in future articles because they pair nicely with continuous control:

Imitation learning (behavior cloning / DAGGER)

• Train a policy directly from demonstrations instead of (or before) reward-based RL.
• Useful when you can teleoperate the robot or script a decent controller to create expert trajectories.
• Often combined with PPO: pretrain with imitation to get a stable starting policy, then fine-tune with RL for robustness.

VLA (Vision–Language–Action) policies

• Combine perception (images), language instructions, and action generation in one model.
• Great for tasks where you want to say “place the upper link over the marker” rather than craft a reward for every nuance.
• Still data-hungry; simulators like Isaac Lab help generate diverse visual data before real-world transfer.

After working with “traditional” navigation robotics (SLAM, navigation, etc.) on my Just1 project , I wanted to try something new with control. Reinforcement learning is one of those cool techniques that can teach robots how to move.

Before learning how to walk, we need to learn how to crawl. So, how about learning how to balance an inverted double pendulum?

First of all, what is Isaac Sim? And Isaac Lab?

Okay, so Nvidia Isaac Sim and Isaac Lab are both part of this ecosystem called “Omniverse”, which honestly can be pretty confusing at first.

Source: Isaac Lab Documentation

Think of Omniverse as the “operating system” for 3D simulation. Isaac Sim sits on top of that and focuses on robotics simulation. It works with ROS, URDF, and all those other robotics standards. Then Isaac Lab sits on top of Isaac Sim and lets you do reinforcement learning with your robots.

Beyond Omniverse, Nvidia also has:

• Nvidia Cosmos: For generating realistic synthetic 3D worlds and videos
• Nvidia Gr00t: A general purpose foundation model to control humanoid robots
• Isaac ROS: A collection of ROS packages built to be accelerated on Nvidia GPUs

In our case, we’ll build our robot within Isaac Sim and train it using reinforcement learning with Isaac Lab.

Let’s build our pendulum

I designed the double pendulum in Onshape. I have more experience with SolidWorks, but Onshape was easy to pick up. Everything is in the cloud, it has a built-in versioning system, and it’s straightforward to export to URDF format.

What is URDF? URDF (Unified Robot Description Format) is basically an XML file that describes your robot. Things like its parts, joints, what it looks like, and how it collides with stuff. It’s the standard way to describe robots in ROS and most simulation tools, including Isaac Sim. We need it because Isaac Sim reads these files to figure out how to load and simulate our robot.

As if balancing an inverted double pendulum wasn’t hard enough, I decided to add a small sign to the second arm to throw it off balance even more.

The double pendulum consists of:

• A rail (fixed base)
• A cart that slides along the rail (controlled by the agent)
• A first arm attached to the cart
• A second arm attached to the first arm
• A small sign attached to the second arm

The tricky part? The agent can only control the horizontal force on the cart. It has to figure out how to keep both pendulum arms standing upright.

Manually balancing our pendulum

After importing the URDF into Isaac Sim, I made sure everything worked. I spent a while tweaking the joint physics, things like stiffness and damping for the cart sliding on the rail, to make it behave as realistically as possible. This matters a lot because if your simulation physics are realistic, your robot will work better when you try it on real hardware. That’s a whole topic on its own, so I’ll save the details for another time.

I tried to balance the pendulum manually using the cart controls. Let me tell you, it’s incredibly hard! The double pendulum is a chaotic system, and even small errors compound quickly. But we live in 2025 now, and we have AI to do the tedious work for us, even balance our pendulums.

It’s training time

I created a new project in Isaac Lab using their template system. Everything, loading the URDF, setting up joints, configuring physics, it’s all done through code.

You can find the code on Github here .

Now here’s the key part of reinforcement learning: we need to define rewards. The model trains a bunch of instances at the same time (4,096 in my case!) and tries to get the highest total reward.

For our inverted double pendulum, we want both arms to stay straight up. Here’s the basic reward setup:

1. Upright position reward: We penalize the agent when the arms tilt away from vertical. The more they tilt, the bigger the penalty. This is the main goal.

2. Smooth motion rewards: We also penalize high speeds, both how fast the arms swing and how fast the cart moves. This stops the robot from being all jerky and wobbly, and makes it move smoothly instead.

Each reward has a weight that says how important it is. The agent gets these rewards every step of the simulation. The neural network figures out how to balance all these competing goals by trying different things across thousands of parallel simulations.

With this reward setup, we’re ready to train! I used the PPO (Proximal Policy Optimization) algorithm with the skrl library, which works really well for this kind of continuous control task.

Finding the right weights for these rewards was one of the biggest headaches. If the upright position reward is too weak, the agent just never learns to balance. If the smooth motion penalties are too strong, the agent gets too cautious and can’t make quick corrections when things go wrong. I spent a lot of time tweaking these weights, running training, watching what happened, and adjusting.

There are a few ways to make this easier. One common trick is curriculum learning. Start with simpler goals (like just balancing from an upright position) and slowly add more complex rewards. Another option is to use reward normalization or scaling, which automatically balances the different reward parts. Some people also use automated tools to search for the best hyperparameters. For this project, I did it the manual way, but those techniques can save you a ton of time.

Here is a gif showing our 4,096 instances learning how to balance themselves in parallel.

Under the hood

If you want to understand the magic behind the scenes, check out Part 2: Behind the scenes . I explain how reinforcement learning actually works, what PPO does, and how neural networks learn from rewards.

For now, just know that the agent tries different things, figures out what works, and gets better over millions of simulation steps.

Our trained policy

After just 20 minutes of training (on my 4080 Super), we have a policy that “learned” how to swing up the arms and keep them balanced. Check out this gif where I push the robot around to mess it up, and it recovers back to standing upright.

What’s next?

So now we have a trained neural network! It takes in the positions and speeds of both arms, plus the position and speed of the cart, and spits out a force to apply to the cart to keep everything balanced.

The neural network itself is pretty simple. It’s just a feedforward network. You give it the current state, it goes through a few hidden layers, and out comes the force. The “magic” is all in the weights, which were learned through millions of trials in simulation.

Note: All neurons in each layer are interconnected to neurons in adjacent layers.

The next big step would be to take this neural network and put it on a real robot with real motors. This is called “sim to real” transfer, and that’s where things get really interesting. Having realistic physics in your simulation becomes super important here. You need to make sure your joint stiffness, controller force, and frequency are actually possible in the real world.

For example, in our simulation, we set up a force of up to 100N and a 60Hz control frequency. This means our real-time controller needs to:

• Read sensors
• Run the neural network (do inference)
• Send commands to the motor

All in just 16.67ms (one control cycle).

How to walk from there?

We’ve trained a system with 3 degrees of freedom (the cart position and two arm angles). A humanoid robot has something like 20 to 30 degrees of freedom just for walking. The idea is the same. Define rewards, train in simulation, and transfer to reality. But the complexity and computing power needed goes way up.

The Problem: Visualizing Robot Data

When building autonomous robots, you’re dealing with a constant stream of data from multiple sensors: lidar scans, camera feeds, IMU readings, motor commands, and navigation states. The challenge isn’t just collecting this data. It’s making sense of it all in real-time.

You can easily spend more time debugging visualization tools than actually working on your robot’s algorithms.

Traditional approaches to visualize your data in robotics usually include:

• Command-line tools: Functional but not intuitive for complex data
• Custom web interfaces: Time-consuming to build and maintain
• Python/Jupyter: Flexible and fast to iterate, but mostly offline and manual
• ROS2 built-in tools (Rviz): Powerful but with a steep learning curve and ugly UI

Enter Foxglove

Foxglove is a powerful robotics visualization tool that works seamlessly with ROS2. It’s like having a professional-grade cockpit for your robot, but without the months of development time.

What Makes Foxglove Special

Real-time Data Visualization: Foxglove connects directly to your robot’s ROS2 topics and displays sensor data in real-time. Whether it’s lidar point clouds, camera feeds, or navigation paths, everything updates live as your robot operates.

Multi-Panel Interface: Unlike single-purpose tools, Foxglove lets you create custom dashboards with multiple panels. You can have a 3D map, camera feed, sensor graphs, and control panels all in one view.

No Coding Required: The interface is drag-and-drop. You add panels, configure them, and start visualizing immediately. No need to write custom visualization code.

Recording and Playback: You can record all your robot’s data and replay it later for analysis. This is invaluable for debugging issues that happened hours ago.

How I Used Foxglove with Just1

For my Just1 robot project, Foxglove became the central hub for development and testing. Here’s how I used it:

1. Real-Time Monitoring

• 3D Map Panel: Visualizes the robot’s SLAM map and current position
• Camera Panel: Shows the camera feed for visual feedback
• Plot Panel: Displays sensor data like IMU readings and motor speeds

2. Navigation Control

• 2D Pose Panel: Click on the map to send navigation goals
• Path Visualization: See the planned path and current trajectory
• Obstacle Detection: Monitor lidar data for obstacle avoidance

3. Debugging and Analysis

• Topic Monitor: Track all ROS2 topics and their message rates
• Log Analysis: Review recorded data to understand what went wrong

How I installed it

There are many ways to connect Foxglove to your robot depending on your stack. In my case, since I was using ROS2, it was as simple as adding another node. I followed their documentation here .
Check out my code here where I add the foxglove-bridge node.
You can whitelist a list of topics and data will start flowing into your Foxglove app.
Want to add a Camera Panel? Just send over your /camera/video topic.

Installing the UI on your computer is even easier. Just download Foxglove and follow the steps here again .

See Foxglove in Action with My Just1 Demo

The Takeaway

Foxglove Studio transformed my robotics development from a frustrating, guesswork-heavy process into an intuitive, visual experience. It’s not just a tool. It’s a paradigm shift in how we interact with robot data.

The features I explored through the Just1 project only seem like the tip of the iceberg of what Foxglove can do, and I’m looking forward to experimenting more with it in future projects.

The Genesis of an Idea

The Hardware Challenge

Building a robot on a budget means making some tough choices. Every component had to justify its cost, and every feature had to serve a clear purpose.

The Brain: Raspberry Pi 4 (8GB) at $80 was my biggest expense, but it needed to handle real-time sensor processing, SLAM algorithms, and navigation planning. The Pi 5 would have been nice, but finding a reliable 5A power supply proved challenging. Looking back, the RAM isn’t much of a bottleneck since we’re using LIDAR instead of camera-only navigation. We’re much more constrained by CPU usage.

The Eyes: This is where things got interesting. I originally planned to use only a camera for Visual SLAM (VSLAM), but after weeks of debugging and optimization, I ended up buying the cheapest 2D Lidar I could find. The LD19 Lidar at $70 became the cornerstone of the robot’s perception system.

The Movement: Mecanum wheels were fun to work with. I initially wanted to experiment with them because they provide interesting control capabilities. These omnidirectional wheels allow the robot to move forward, backward, sideways, and rotate in place, making them perfect for tight spaces and precise positioning. Four TT motors with L298N drivers were the cheapest solution to control them.

The Senses: An MPU6050 IMU ($3!) provides orientation data, while the optional Camera Module 3 adds visual feedback for future enhancements.

The Software Stack: Where the Magic Happens

Hardware is just the beginning. The real challenge was making all these components work together seamlessly. I chose ROS2 as the middleware because it’s the industry standard for robotics, and learning it would be valuable for future projects. My main goal for this project was actually to get hands-on experience with ROS2 to better understand how it works.

Navigation Stack: RTAB-Map handles SLAM (Simultaneous Localization and Mapping), creating a map of the environment while tracking the robot’s position. Nav2 provides the path planning and obstacle avoidance capabilities.

Sensor Fusion: A Madgwick filter combines IMU data with other sensors for accurate orientation estimation.

Visualization: Coming from a company where dozens of engineers work on a single tool to visualize sensor data, I couldn’t afford the time and resources to reinvent the wheel. Foxglove Studio became my best friend during development. This powerful tool lets me visualize sensor data, monitor the robot’s state, and even send navigation goals through an intuitive interface.

The Development Journey

Building Just1 wasn’t just about assembling hardware and writing code. It was a crash course in robotics engineering. Here are some of the key lessons I learned:

Lesson 1: Start Simple, Then Optimize

My first version used JPEG encoding for the camera feed, which consumed massive CPU resources. Switching to H.264 encoding dramatically improved performance, but I didn’t need to spend time on that until late in the development. Sometimes the simplest solution isn’t the most efficient, but don’t spend time on issues you don’t have yet.

Lesson 2: Documentation is Everything

Every wiring connection, every software configuration, every troubleshooting step. I documented it all. This wasn’t just for others who might want to build their own Just1, but for future me who would inevitably forget how I solved that one tricky bug at 2 AM.

Lesson 3: Real-World Testing Reveals Everything

Simulation is great, but there’s no substitute for real-world testing. Coming from the software world into hardware, you have to expect the unexpected. Your robot can fail or behave strangely simply because it’s out of battery. Light and shadows can play tricks on your sensor inputs.

The Foxglove Revelation

One of the most surprising discoveries was how much Foxglove Studio transformed the development experience. This visualization tool became essential for:

• Real-time monitoring: Watching sensor data streams, camera feeds, and navigation status
• Debugging: Visualizing the robot’s understanding of its environment
• Control: Sending navigation goals through an intuitive interface
• Analysis: Reviewing recorded data to understand what went wrong (or right)

Foxglove turned what could have been a command-line nightmare into an intuitive, visual experience. It’s like having a cockpit for your robot. I’ve written a dedicated blog post on Foxglove .

The Bigger Picture

Building Just1 taught me that autonomous robotics doesn’t have to be prohibitively expensive or impossibly complex. Having previously worked with the famous SO-101 , I’m proud to have built something similar for a small autonomous robot, complete with ROS2 integration.

The project is fully open-source, with complete documentation, CAD files, and software available for anyone who wants to build their own. I built Just1 mainly for myself, but I was proud to share it online and received great feedback. Just1 has been shared on Reddit and appeared in the Weekly Robotics newsletter ! I ended up meeting new people and having drinks with others who wanted to build their own Just1 and learn robotics. That’s the beauty of hardware!

The Takeaway

The best way to learn about something is just to dive into it and build a project out of it. In the software world, it’s very easy to do. You just need a computer and you can start coding. But with Just1, I realized that the same is also possible in hardware. You can build a very simple robot and learn principles that are applied in robotaxis roaming the streets of San Francisco.

The journey from idea to working robot was filled with challenges, late-night debugging sessions, and moments of pure joy when everything finally clicked. But most importantly, it showed me that the barrier to entry in robotics isn’t as high as it might seem.

So, what’s your “Just1” project? What have you been wanting to build but thought was too complex or expensive? Sometimes, the best way to find out is to just start building.

This problem is called the Coupon Collector’s Problem!

Simple approach to the problem

To simplify the approach to the problem, let’s say that we are collecting monsters, and we are getting these monsters from “surprise boxes”. For now we’ll assume that there are 100 different monsters with the same probability of getting all of them.

It is your first opening of a box. The probability of getting a new monster is … well 1, as you have no other monsters. For your second opening of a box, the probability of getting a new different one is \(\frac{99}{100}\). Indeed, there is a slight “chance” (or bad luck) of \(\frac{1}{100}\) that you’ll get the same monster as your first opening.

To generalize our finding, if we already have \(k\) different monsters, the probability of getting a new monster out of a pool of \(N\) monsters (in our case \(N = 100\)) is:

• If we already have 99 monsters, the probability of finding a new one (the ultimate one!) is \(\frac{1}{100}\)

This follows a geometric distribution with parameter \(\frac{N-k}{N}\). The expectation of this geometric distribution is \(\frac{N}{N-k}\). We can interpret this expectation as:

After already having \(k\) different monsters, we’d need to open on average \(\frac{N}{N-k}\) boxes to get a new one.

• So if we already have 99 monsters, we’d need on average to open 100 boxes to have a new one.

Let’s use this expectation formula. If we have 0 monsters, the number of box openings to get a new one is:

Once we have 1 monster, the number of box openings on average to get a new one is:

Summing both, the average number of box openings to get 2 monsters is 2.01. As we’ve seen before, there is a slight chance that we’ll get the same monster after the 2 box openings.

Generalizing, in order to get \(k\) different monsters, we’ll need to open this amount of boxes:

Which we can also write:

Let’s compute this sum in Python.

import numpy as np
import matplotlib.pyplot as plt
#We can build the sum with a for loop

N = 100

def openbox_depending_on_monster(k):
    total = 0
    for i in range(1, k + 1):
        total += N / (N - (i - 1))
    return total

#or with a vectorized approach

def openbox_depending_on_monster_np(k):
    # np.arange creates an array starting from 1 up to k
    indices = np.arange(1, k + 1)
    # Directly compute the terms using vectorized operations and compute the sum
    total = np.sum(N / (N - (indices - 1)))
    return total

print("So to get 95% of the monsters (95) we'd need to open {} boxes".format(int(openbox_depending_on_monster_np(95))))
print("And to be sure to have all 100 distinct monsters we'd need to open on average {} boxes".format(int(openbox_depending_on_monster_np(100))))

We can also plot the evolution of the number of distinct monsters, depending on the number of box openings:

# Range of k values from 1 to 100
N = 100
k_values = np.arange(1, N)
# Compute the sum for each k
sum_results = [openbox_depending_on_monster_np(k) for k in k_values]

# Create the plot
plt.figure(figsize=(10, 6))
plt.plot(sum_results, k_values, label=' ')
plt.xlabel('Number of box openings')
plt.ylabel('Number of distinct monsters')
plt.title('Number of distinct monsters depending on the number of box openings')
plt.grid(True)
plt.legend()
plt.show()

We can see that the more monsters we get, the harder it is to find new ones!

The opposite approach to the problem — a bit more complex

What if we approach the problem from the other side? How many distinct monsters do we get after opening \(k\) boxes?

Well, we could always look at our charts above and deduce that, but from a statistical point of view, it would look like this:

The probability that monster \(i\) has not been picked after \(k\) attempts is:

So the probability that monster \(i\) has been picked at least once is:

Given that each of the 100 monsters has the same probability of appearing, we can use the linearity of expectation to find the expected number of different monsters seen after \(k\) attempts:

N = 100

def monster_depending_on_openbox(k):
    total = N*(1-((N-1)/N)**k)
    return total

print("So after opening 100 boxes, we'll have on average {} different monsters".format(int(monster_depending_on_openbox(100))))

Non-uniform distribution — even more complex

Now, what happens if the probabilities of obtaining each monster are not uniform when opening a box? Let’s say we have 90 monsters with a probability of being found of \(\frac{1}{91}\) and 10 rare monsters with a probability of being found of \(\frac{1}{910}\).

How many box openings will it take to collect all 100 different monsters?

Well, the mathematical approach gets a bit more complicated, so we’ll venture into the world of simulation with Monte Carlo methods.

Monte Carlo methods

Monte Carlo methods are a type of computational algorithm used to solve problems through random sampling. We are going to simulate multiple series of box openings until we reach our target of unique monsters, and then we will see on average how many box openings we needed.

import numpy as np

# Define probabilities for each monster
probabilities = [1/91] * 90 + [1/910] * 10  
#probabilities = [1/100] * 100 #we can do that if we want to simulate with uniform distribution
probabilities = np.array(probabilities) / sum(probabilities)  # Normalize probabilities

# Number of total monsters
total_monsters = len(probabilities)

# Target number of unique monsters (we want to reach 100 of unique monsters)
target_unique_monsters = 100

# Function to simulate opening boxes until we reach the target
def simulate_boxes():
    collected_monsters = set()
    box_count = 0
    while len(collected_monsters) < target_unique_monsters:
        monster = np.random.choice(total_monsters, p=probabilities)
        collected_monsters.add(monster)
        box_count += 1
    return box_count

# Number of simulations
num_simulations = 10000
results = [simulate_boxes() for _ in range(num_simulations)]

# Calculate the average number of boxes needed
average_boxes = np.mean(results)
print("To have all 100 distinct monsters we'd need to open on average {} boxes".format(int(average_boxes)))

We can see how this addition of rarer monsters drastically increases the number of boxes we have to open to collect them all!

This number is an average, but each round of the simulation got a different number of box openings to get all the monsters. We can even visualize the distribution of the number of openings needed in each round of the simulation.

import matplotlib.pyplot as plt

# Plot the distribution of results from the simulations
plt.hist(results, bins=30, color='skyblue', edgecolor='black')
plt.title('Distribution of Boxes Needed to Collect all monsters')
plt.xlabel('Number of Boxes')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

Conclusion

We’ve explored the Coupon Collector’s Problem from simple uniform distributions to complex scenarios with rare items. The math shows that collecting everything takes patience, especially when some items are much rarer than others.

Next time you open a pack or plan to collect something, remember: with math and persistence, even the most challenging collections can be completed!