Over the years I’ve picked up a small amount of skill manipulating images in Photopia, MS Paint, Powerpoint (not kidding) and other programs, mostly to make edits or create graphics for different applications.
But one skill I’ve always wished I could have but never had enough patience to learn is 3D modelling. Specifically I mean sculpting, which has more to deal with meshes and their exterior appearance and less to do with functional parts for manufacturing and engineering. And while I’ve designed and 3D printed a few things using 3D Builder, I never learned Blender or Fusion360 to the extent I’d need to do more complex stuff.
What I would love is to be able to push a button, and take an idea in my head all the way to a full 3D model, even if just for reference or further editing.
Low and behold, some literal demigods at Microsoft published a paper on a new generative model they created called TRELLIS, which does exactly that.
The researchers describe TRELLIS as:
“A native 3D generative model built on a unified Structured Latent representation and Rectified Flow Transformers, enabling versatile and high-quality 3D asset creation.”
If you understand any part of that sentence, you can read more about it on the official GitHub page for the paper.
The basic explanation is that TRELLIS works by taking a single image, guessing on what that image would look like from multiple angles, and then building a mesh map of it and outputting as a .glb (3d model) file.
What makes TRELLIS especially cool, is that it also applies a texture to the mesh based on the original image prompt, which means you don’t even have to do painting or texturing.
It can do complex materials, shiny metal objects, even transparent/glass structures.
In this quick post, we’ll take an idea in my head to a 2D image, then to a 3D model, then to a real physical object (via 3D printing) with the use of AI tools.
Honestly it’s a wild time to be alive.
Getting an Image
So first, you need an image to start this process with. I originally generated my avi using Bing’s DALLE 3, which I’ve personally been having the most success with one-shotting something specific and unique. This was the idea I had in my head and it translated to my pfp pretty well:
First step was to clean up the image, so we spliced it into different layers and gave it a consistent background, and then last we removed the “glow/chalk” effect of red around it so as not to screw with the 3D model:
Once I was satisfied with the work, I then moved onto plugging this into TRELLIS.
Using TRELLIS
While you could go to the paper’s GitHub repository and download the code and run it yourself, luckily HuggingFace has a demo you can use for free!
https://huggingface.co/spaces/JeffreyXiang/TRELLIS
We simply went to the website, and followed the instructions to upload our image as input. I left all the settings at default, and just hit generate.
About 30 seconds after uploading, the .mp4 video preview of the model was already done:
From there, I checked to make the video results seemed correct. Honestly I was pretty impressed, this was astoundingly accurate to the photo while still maintaining the artistic look of the original image. All based on a single photo, of something it would have no training data on.
Next, I hit “extract GLB” which took the process of outputting it as a 3D model, and after about another 60 seconds I had a working model of my avi to play with:
Obviously as a bust it’s hollow, but if I wanted to spend the time I could fix that up pretty quickly.
3D Printing a Bust
Now that I had a 3D model of my avi, I just opened the .glb file in 3D Builder, then saved it as a .stl file.
I imported that into my slicer software with no issues:
In order to get it to print right I added autogenerated tree supports, and sliced off the bottom part so it would sit flat on my desk.
Print settings were as follows:
- PETG profile
- 0.16mm layer height
- auto tree supports
- no brim
- 10% gyroid infill
- 2 walls
Final Result
After about 4 hours, I was holding in my hand the 3D print of what was once just an idea in my head.
This is honestly nuts. I’m a terrible sculptor, I don’t have experience in doing it, yet tools like TRELLIS made this possible in a matter of minutes, when it would have probably taken me days to get to this point in Z-brush or Blender.
Shortcomings + Next Steps
TRELLIS seems to work better with simple images, that don’t extend too far in any one axis. I tried doing some full body renders as an Avatar model for Twitch content, but the accuracy got worse. We’ll have to play around and probably learn Blender better before I can pull that off.
What I’ll probably do next is paint this little bust and mount it on my desk, or try reprinting it in multiple colors.