If you've been watching the logistics automation news, you'll have seen a bunch of videos of robots performing package induction (introducing packages into a shipping network or warehousing system). Figure, XSquare, and even Plus One Robotics have shown completely autonomous systems running for an hour or more, inducting a wide variety of package and parcel types. While form factors and specific setups differ, all these examples share a common approach—they use machine learning, specifically learning from examples and demonstrations, to train all or part of the decision-making and behavior. This technology isn't new; it dates back at least 20 years, but recent advances in model complexity, training systems, and computational ability have finally enabled it to move from the lab into the workplace.
When done in advance, imitation learning (IL) allows a system to adapt to the specific variability present in a target domain, while avoiding the need to explicitly model that variation—unless we're talking about sim2real training, in which case an explicit model of that variation is still required. But to handle changes that show up after the system goes live, an increasingly popular approach is human-in-the-loop (HITL) learning, where humans intervene in the running system, provide additional demonstrations, and teach the system how to adapt to the unforeseen changes that abound in warehouse logistics: new package types, new designs, and everything else nobody thought to demonstrate in advance.
The result is a whole autonomy stack that enables a deployed system to adapt itself to changing situations on the fly without requiring lengthy downtime or retraining pauses. A change occurs, and the humans in the loop deal with it until the autonomous elements learn enough, and the whole system stays up and running.
Bringing reinforcement learning into the loop
But there's a limitation here. Systems trained this way are limited to being roughly as good as the source of their data: the humans. They might be a little faster, or a little more accurate, but the basic behavior the automation learns is the same as what the humans demonstrated. In fact, it's often necessary to ensure that if there are multiple human demonstrators, they all demonstrate how to perform the task in the same fashion, lest the autonomous elements become confused. And simply speeding up the autonomous process often doesn't work because the dynamics of the system change at speed. Thus, current work is looking at bringing reinforcement learning (RL) into the mix to enable the automation to discover, for itself, new and better ways of performing the task. This isn't a replacement for IL; it sits on top of it. Exploring from scratch will often not result in successful task performance, so a system starts with what was demonstrated, then explores ways to make it better. Often, this exploration takes place in simulation (see the sim2real point above) so it can run faster than real time, safely, and repeatably.
With IL, HITL, and RL, a single autonomous system can adjust its performance to change and improve over time. Connected systems can even share their learnings and improvements with other autonomies of the same type, leading to improvements across an entire fleet.
Learning across automations
All this works great for a single automation—a fixed set of hardware doing a specific task. And we're starting to see the same approach being applied to the same hardware performing many different tasks (1X, etc.). But in a warehouse, there are often several different autonomy solutions. One might induct parcels, another might depalletize boxes, a third might run the conveyance, and so on. Each system can learn and optimize by itself, but what's often of concern is the coordination between them, how to get the entire collection of autonomous systems to perform, holistically, better.
This is the orchestration problem
Specifically, optimizing each machine individually does not necessarily optimize the operation as a whole. A robot that becomes faster in isolation can create a bottleneck downstream just as easily as it can relieve one upstream. Or an induction system placing packages 'just so' to maximize throughput may cause issues for the scanner tunnel later on. There's potential for high-level improvements if you can get the various systems to work together better, without any need for new hardware or other infrastructure improvements.
What's interesting is that the intersystem optimization problem can be seen as the same as the intra-system optimization problem, just at a different level of abstraction. Within a robot induction cell, for example, optimization involves coordinating the movement of all of the degrees of freedom (DOFs) of the arm(s), as well as the conveyors, cameras, and so on. Between systems, the problem is the same: how to coordinate the DOFs across all the systems, but now at the level of parcel locations and picking targets, rather than camera pixels and joint angles.
So, we already have the necessary techniques to address this problem; it's simply a question of scale.
Bringing new hardware online
However, most warehouses today have a mix of learned autonomous systems, algorithmic (hard-coded) systems, and manual steps. Reworking all of these to be orchestrated by a learned system is a lot of work, and out of scope/budget for most updates. Even a brand-new warehouse may run into problems because by the time the warehouse is complete, newer, better hardware may be available.
So, we need a system that doesn't just adapt to task variation, where the objects or goal changes, but can also adapt to hardware variation, where the actual embodiment of the learned autonomy changes over time. New cameras, conveyors, arms, and such are installed, and old ones are removed.
We believe that hardware changes can be approached under a framework that is similar to orchestration and automation. New hardware is just novel input and output dimensions, for which the mapping between them must be learned. It's the same problem as learning autonomy, just applied to hardware instead of tasks. The difference is that orchestration learns an additional layer on top of existing autonomy, which means learning an additional system alongside it. There are many questions and problems of scale and coordination, but under this model, a system's capability would grow in step with the physical equipment added to it, rather than remaining fixed to whatever configuration existed at initial deployment.
What's exciting about this view is that it doesn't require (re)designing an entire operation at once. The practical path is incremental: automate one bottleneck, evaluate the results, then add the next process while the system carries forward what it has already learned rather than starting from zero. This is a lower-risk way to adopt automation than committing to a single, comprehensive deployment, and it reflects the direction most operations are actually taking.
Supply chains continue to grow more dynamic, and the range of products, packaging, and edge cases that robots are asked to handle keeps expanding. The systems best positioned for that environment won't be the ones that anticipated every possible case in advance—that isn't achievable. They'll be the ones built to keep learning, both individually and across an operation, as new cases continue to appear.
