LegoGPT is here

Some 27 years ago, I was so impressed by Pablo Funes' work (with his PhD advisor the always ideating Jordan Pollack) (P. Funes and J.B. Pollack, “Evolutionary Body Building: Adaptive Physical Designs for Robots,” Artificial Life, vol. 4, no. 4, 1998, pp. 337–357; preprint: link) on evolving physical structures that I wanted to hire him right away -I didn't know him but reached out and he, to my delight, said yes! He used LEGO bricks (a convenient and powerful “toy” model, literally for physical 3D assembly) to evolve, among other things, long bridges, by simply stating the objective of reaching a point on the other side of a canyon. The bigger the gap between origin and destination, the harder it is to design a stable bridge that does not crash. A simple physics model was crafted to evaluate the architectures generated by a genetic algorithm (GA) and a human (Pablo) would then proceed to build the successful structures to check whether they were indeed stable. The physical embodiment was necessary to toss out edge cases that might appear stable in the simple physics model but were, in fact, unstable -a form of active learning from real world physics feedback. In the process, the GA “invented” or discovered the cantilever.

So it was with some trepidation that I came across the clever work of a team from @carnegie mellon university (Ava Pun* Kangle Deng* Ruixuan Liu* Deva Ramanan Changliu Liu Jun-Yan Zhu): “Generating Physically Stable and Buildable LEGO® Designs from Text” (https://avalovelace1.github.io/LegoGPT/). I was disappointed to see that Pablo’s work was not cited -in fact, history seems to begin in 2013, the earliest references in the authors’ list. Guys, you need to do better!

What is cool, of course, in this new development, is that there is no explicit fitness function, just descriptions of objects you want to build. They created a dataset of “over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions”. I wonder what happens when you describe an out-of-distribution object, such as a long bridge... The other cool, and somewhat unrelated, aspect of the work is that the designs can be built automatically by robotic arms. What would be even cooler would be the ability to inject physical feedback directly into the (un)stability dataset.