CTC Explained: Train on Robot Sensor Data Without Labeling Every Frame

The problem CTC solves

Say you're building for a robot arm. It has a gripper with a force sensor and an IMU, and together they produce a 6-channel reading 50 times a second. You want a model that watches this stream and reports the events that occurred: the gripper picked something up, then it placed it down.

If you've touched any on-device ML as an app developer, it was probably a classifier, even if nobody used the word. A classifier is a model that takes one input and answers one question about it with one label from a fixed list. Vision framework telling you a photo contains a dog, ML Kit deciding a review is positive or negative, a keyboard model flagging a message as spam: all classifiers. One thing in, one answer out. It's the hello world of machine learning, and it's the natural first idea here too: feed the model the sensor recording, get back an answer.

The problem is shape. A classifier's answer is a single label, but the answer you want here is an ordered list: pick, then place. And the input isn't one thing the way a photo is one thing. Four seconds of sensor data at 50 Hz is 200 consecutive readings, where the two moments you care about occupy maybe a dozen frames each, at positions you don't know in advance. Different clips put the events in different places and a clip could contain one event, or three. A model that emits exactly one label per input has no way to say any of that. You need something that reads 200 timesteps and produces a variable-length sequence, and nothing in the one-input, one-label mold does it.

The obvious workaround is to classify every frame: frame 1 is "nothing", frame 2 is "nothing", frame 88 is "pick", and so on. Now the shapes match, but you've created a labeling problem. To train that model, someone has to go through every recording and mark the exact frame where each event starts and ends. Two issues with that:

1. It's expensive. Frame-accurate annotation of sensor data is slow, and you need thousands of clips.
2. It isn't even well defined. Scrub through a force trace (the gripper's force reading plotted over time, the touch equivalent of an audio waveform) and try to pick the exact frame a "pick" begins. Contact? First force rise? Grip closure? Two annotators will disagree, and your training labels become noise.

CTC removes the requirement entirely. You label each clip with only the ordered list of events, like [pick, place], and the loss function handles every possible way those events could line up with the 200 frames. No timestamps, no boundaries, no frame-level annotation.

This is exactly the trick that made modern speech recognition practical. Audio is just a time series, and transcripts don't say when each word was spoken. Robot sensor streams have the same shape, so the same math applies.

One prerequisite: what a loss function is

CTC is a loss function, so it's worth being precise about that term before using it. A loss function is the number a model is graded by during training. Each training step, the framework runs the model on a batch of examples, computes the loss (a single number where lower means the outputs were closer to correct), and nudges every weight in the model slightly in the direction that reduces it. Repeated a few hundred thousand times, that's the whole of training.

The part that matters for this post: choosing the loss function is choosing what "correct" means. Train with a frame-wise loss and you've declared that correct means matching a label at every frame, which drags in the annotation problem above. Train with CTC and you've declared that correct means producing the right sequence of events somewhere in the window. Same network, same data, different definition of success. And you never implement the grading yourself; it ships with the framework, as you'll see in the PyTorch section.

How it works: the blank token and the collapse rule

A CTC model itself is nothing exotic. It's any network that reads your time series and outputs, for every frame, a probability distribution over your labels. The only structural change CTC asks for is one extra label in the vocabulary: the blank, written here as ∅. Blank means "no event is being emitted at this frame". It isn't the same as a "nothing is happening" class you invented yourself; it's machinery the loss function needs.

So for our gripper, the per-frame output is a softmax over three classes: ∅, pick, and place.

A full sequence of per-frame outputs is called an alignment (or a path). For a 10-frame window, one alignment might be ∅ ∅ pick pick ∅ ∅ place ∅ ∅ ∅. CTC defines a collapse function, usually written $\mathcal{B}$, that turns any alignment into a label sequence with two steps applied in this exact order:

1. Merge consecutive repeats of the same label into one.
2. Remove all blanks.

Watch it happen below: repeats merge, blanks drop out, and whatever survives is the label sequence. The first three alignments have different timings but collapse to the same label sequence. The fourth shows why the blank has to exist at all: it's the only way to express the same event happening twice in a row.

The direction of this function is the whole idea. Many alignments map to one label sequence. So when your training label is [pick, place], the model isn't being told "the pick was at frame 88". It's being told "some alignment that collapses to pick, place is correct, and I don't care which one".

The math

You need surprisingly little notation to state CTC precisely. The input is a time series $X$ with $T$ frames. At each frame $t$ the network outputs $p_t(k \mid X)$, the probability of class $k$ (including blank). CTC treats frames as conditionally independent given the network's encoding, so the probability of one complete alignment $\pi$ is just the product of its per-frame probabilities:

The probability of a label sequence $\mathbf{y}$ is the sum over every alignment that collapses to it:

And the training loss is the negative log likelihood of the correct label sequence $\mathbf{y}^*$:

Minimizing this loss pushes probability mass toward the set of correct alignments as a group. The model is free to concentrate that mass on whichever alignment is easiest to learn.

There's one practical catch in that middle equation: the number of valid alignments is astronomically large. For 200 frames and 3 classes there are $3^{200}$ possible paths. Summing them one by one is impossible. CTC solves this with dynamic programming, the same idea you may know from optimizing recursive functions with memoization. Define $\alpha_t(s)$ as the total probability of all alignment prefixes that account for the first $t$ frames and the first $s$ symbols of the target (with blanks inserted around each label). Each step reuses the previous column:

The three terms are the three legal moves: stay on the same symbol, advance by one, or skip over a blank between two different labels. The full sum collapses to a $T \times S$ table, so the loss is computed in $O(T \cdot S)$ instead of exponential time. You'll never implement this yourself. Every framework ships it, and it's why calling nn.CTCLoss is cheap. But knowing the recurrence explains the one hard constraint CTC has: the input must have at least as many frames as the expanded target has symbols. You can't ask 5 frames to explain 8 events.

What a trained CTC model actually outputs

Because the loss only cares about the collapsed sequence, trained CTC models converge to a characteristic and slightly surprising behavior: they output blank almost everywhere, with a sharp one-or-two-frame spike for each event. This is called the peaky behavior, and it's normal.

Two things to take from this chart:

• Decoding is trivial. At each frame, take the argmax: the label with the highest probability. (The name is short for "argument of the maximum", meaning the index of the largest value in an array. [0.96, 0.03, 0.01] has an argmax of 0, which is blank.) Apply the collapse rule to that per-frame sequence and you have your answer. This is called greedy decoding, and for short event vocabularies it's usually all you need.
• Don't treat spike positions as measured event times. The spike lands somewhere inside the true event window, often near the end, because nothing in the loss rewards precise placement. If your robot control logic needs the exact millisecond of contact, get it from the raw sensor threshold, and use CTC for what it's good at: recognizing that the event happened and in what order.

Training one in PyTorch

Here's a complete, minimal training step. The encoder is a GRU (gated recurrent unit), a network layer built for sequences: it reads the frames in order and carries a running memory of what it has seen, so each frame's output can depend on the history that led up to it. Bidirectional means it runs twice, once forward and once backward, so every frame's output also knows what comes after it. Other sequence encoders (a temporal convolution, a small transformer) work too; a two-layer bidirectional GRU is a reasonable default for sensor streams at these rates. The parts that are actually CTC-specific are the extra blank class, the log_softmax, the transpose to time-first layout, and the four arguments to the loss.

import torch
import torch.nn as nn
 
# Class 0 must be the blank. Real labels start at 1.
BLANK = 0
NUM_CLASSES = 3  # blank, pick, place
 
class EventTagger(nn.Module):
    def __init__(self, in_features=6, hidden=128):
        super().__init__()
        self.encoder = nn.GRU(
            in_features, hidden, num_layers=2,
            batch_first=True, bidirectional=True
        )
        self.head = nn.Linear(2 * hidden, NUM_CLASSES)
 
    def forward(self, x):  # x: (batch, time, features)
        h, _ = self.encoder(x)
        return self.head(h)  # (batch, time, classes)
 
model = EventTagger()
ctc_loss = nn.CTCLoss(blank=BLANK)
 
# 8 clips, 200 frames each, 6 sensor channels.
x = torch.randn(8, 200, 6)
 
# Every clip is labeled "pick, place". No timestamps anywhere.
targets = torch.tensor([1, 2] * 8)          # flattened label ids
input_lengths = torch.full((8,), 200)       # frames per clip
target_lengths = torch.full((8,), 2)        # labels per clip
 
# CTCLoss wants (time, batch, classes) log-probabilities.
log_probs = model(x).log_softmax(-1).transpose(0, 1)
 
loss = ctc_loss(log_probs, targets, input_lengths, target_lengths)
loss.backward()

Note what the labels look like: targets is just the event ids in order, and target_lengths says how many belong to each clip. That's the entire annotation format. If a teammate records 500 demos tomorrow, labeling them is one line of text per demo.

Inference is the greedy decode described above:

def decode(log_probs_one_clip):  # (time, classes)
    ids = log_probs_one_clip.argmax(-1).tolist()
    out, prev = [], None
    for k in ids:
        if k != prev and k != BLANK:
            out.append(k)
        prev = k
    return out  # e.g. [1, 2] meaning pick, place

If you later need higher accuracy on longer vocabularies, the upgrade path is beam search decoding, optionally with a language-model-style prior over event sequences. The trained model doesn't change.

Where this fits in a robotics workflow

CTC is a good match for robot data because robot data is almost always a time series with cheap sequence-level labels and expensive frame-level labels:

• Event detection from proprioception. Proprioception is the robot's sense of its own body: joint angles, motor currents, gripper force, IMU readings. Everything measured from the inside, no cameras. Streams like these carry events worth recognizing (contact, slip, grasp success, a door latching, footsteps on a quadruped), and an operator can say what happened in a 10-second clip far faster than they can mark where.
• Gesture and command recognition. IMU or joint-angle streams mapped to an ordered set of recognized gestures.
• Audio on the robot. Wake words and short spoken commands are the classic CTC use case, and they run fine on edge hardware.
• Skill segmentation of teleop demos. If you're collecting demonstrations, like the pose data workflows in our Telecollect post, CTC gives you a way to tag each demo with its skill sequence and train a recognizer without hand-segmenting anything.

The deployment side is the part we build for. A trained EventTagger exports to ONNX or TorchScript and runs comfortably on a Jetson Orin Nano next to your control loop. On WendyOS it ships as a container with wendy run, the same way you'd ship any other app, so the model that recognized events on your training workstation is byte-for-byte the model running on the robot.

When not to use CTC

Being matter of fact about the limits saves you a rewrite later:

• You need precise event timing. CTC spikes aren't timestamps. Use frame-wise classification with a small hand-labeled set, or read timing from the raw signal.
• Your outputs aren't monotonic with the input. CTC assumes events appear in the stream in the same order as the labels. That holds for sensor events and speech. It doesn't hold for tasks like translation.
• Outputs depend on each other. The conditional independence assumption means the model can't learn "place is likely after pick" inside the loss itself. For short event vocabularies this rarely matters, and a decoding prior covers most of the gap. When it does matter, the successors are RNN-T and attention-based encoder-decoders, both of which cost more to train and deploy.
• The input can be shorter than the target. The recurrence needs $T \geq$ the expanded target length. Downsample less aggressively in your encoder if you hit this.

Summary

CTC turns "label every frame" into "list what happened", which is usually the difference between a labeling job your team will actually do and one it won't. The mechanics are one extra blank class, a collapse rule, and a dynamic-programming loss your framework already implements. For robot sensor streams, that makes it the cheapest way to get from raw recordings to a trained event recognizer, and the result deploys to the edge like any other model. Label the what, let the loss handle the when.