Mixing generative AI with physics to create personalized objects that work in the real world | MIT News

Have you ever had an idea for something that looks cool, but doesn’t work well in practice? When it comes to designing things like decor and personal accessories, artificial intelligence (genAI) models can be relevant. They can produce creative and complex 3D designs, but when you try to turn such designs into real-world objects, they often don’t support everyday use.

The fundamental problem is that genAI models often lack an understanding of physics. Although tools such as Microsoft’s TRELLIS system can create a 3D model from text or image information, its chair design, for example, may be unstable, or have disconnected parts. A model doesn’t fully understand what your intended object is designed to do, so even if your chair can be 3D printed, it might collapse under the force of someone sitting on the floor.

In an effort to make these designs work in the real world, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) are giving generative AI models a reality check. Their “PhysiOpt” program enhances these tools through physics simulation, creating blueprints for personal items such as mugs, mugs, key holders, and notebooks that work as intended when 3D printed. It quickly checks if the layout of your 3D model is working, fine-tuning small shapes while ensuring the appearance and functionality of the design is preserved.

You can simply type what you want to create and what it will be used for in PhysiOpt, or upload an image to a virtual application, and in about half a minute, you will have a realistic 3D object to build. For example, CSAIL researchers have created a “flamingo-shaped drinking glass,” which they 3D printed on a drinking glass with a handle and base that resemble a tropical bird’s leg. As the design was developed, PhysiOpt made small improvements to ensure the design felt right.

“PhysiOpt combines GenAI and physics-based shape optimization, helping almost anyone create desired designs for unique accessories and decorations,” said MIT Electrical engineering and computer science (EECS) PhD student and CSAIL researcher Xiao Sean Zhan SM ’25, co-author of the paper presenting the work. “It’s an automated program that allows you to make shapes physically reproducible, given certain constraints. PhysiOpt can iterate on its creation as many times as you like, without additional training.”

This approach allows you to create “intelligent design,” where an AI generator creatively creates your objects based on user specifications, while taking performance into account. You can connect your 3D model to the AI ​​generator, and after typing what you want to produce, you specify how much force or weight the object should handle. It’s a neat way to simulate real-world usage, like predicting whether a hook will be strong enough to hold your coat. Users also specify what materials they will use to make the object (such as plastics or wood), and how it is supported — for example, a mug stands on the floor, and a bookshelf rests on a stack of books.

Given a specification, PhysiOpt begins to magnify the object iteratively. Under the hood, it uses a physics simulation called “finite element analysis” to evaluate the design. This comprehensive scanner provides a heat map over your 3D model, showing where your plan is not well supported. If you were building, say, a birdhouse, you might find that the support beams under the house were bright red, meaning that the house would collapse if not reinforced.

PhysiOpt can create even bolder pieces. Researchers saw this evolution firsthand when they created a steampunk (a style that combines Victorian and futuristic aesthetics) with intricate, robotic-looking hooks, and a flat-backed “giraffe table” to place things on. But how do they know what “steampunk” is, or what unique furniture should look like?

Surprisingly, the answer is not extensive training – at least, not for researchers. Instead, PhysiOpt uses a pre-trained model that has already seen thousands of shapes and objects. “Existing systems often require a lot more training to have a semantic understanding of what you want to see,” added co-author Clément Jambon, who is also an MIT EECS PhD student and CSAIL researcher. “But we use a model that has that sense of what you want to create already baked in, so PhysiOpt doesn’t practice.”

By working with a pre-trained model, PhysiOpt can use “shape priors,” or knowledge of what shapes should look like based on prior training, to produce what users want to see. It’s like an artist recreating the style of a famous painter. Their expertise is based on extensive study of various art forms, so they will be able to showcase that beauty. Similarly, the shape familiarity of the pre-trained model helps it generate 3D models.

CSAIL researchers noted that PhysiOpt’s visual awareness helped it create 3D models more effectively than “DiffIPC,” a comparable method that simulates and optimizes shape. When both methods were tasked with generating 3D designs of objects such as chairs, the CSAIL system was about 10 times faster with each iteration, while creating realistic objects.

PhysiOpt presents a possible bridge between ideas and real world human objects. What you might think is a great idea for a cup of coffee, for example, can quickly make you move from your computer screen to your desk. And while PhysiOpt already performs stress testing for designers, it may be able to quickly predict constraints such as loads and constraints, instead of users needing to provide that information. This independent, logical approach is possible by combining linguistic models of vision, which combine human language understanding with computer vision.

In addition, Zhan and Jambon aim to remove artifacts, or random pieces that appear from time to time in PhysiOpt’s 3D models, by making the system more aware of physics. MIT scientists are also considering how to model the complex limitations of various manufacturing techniques, such as shrinking dangling parts for 3D printing.

Zhan and Jambon wrote their paper with MIT-IBM Watson AI Lab Principal Research Scientist Kenney Ng ’89, SM ’90, PhD ’00 and two CSAIL colleagues: undergraduate researcher Evan Thompson and Assistant Professor Mina Konaković Luković, who is the lab’s principal investigator.

The researchers’ work was supported, in part, by the MIT-IBM Watson AI Laboratory and Wistron Corp. They are presenting in December at the Association for Computing Machinery’s SIGGRAPH Conference and the Computer Graphics and Interactive Techniques Exhibition in Asia.