Follow-Your-Creation: Empowering 4D Creation

through Video Inpainting

1HKUST   2HKUST(GZ)   3NUS   4Tsinghua University  5CUHK
* Equal Contribution Corresponding Author

arxiv

[Paper]  |  [Code]

DEMO Video

Pipeline

Overview of our method. We reformulate the 4D video creation as video inpainting task. Left: given a video, we first generate the composite masks from the dynamic point cloud and feed them into video inpainting model to unlock its 4D video creation capability. Right: To unlock the capability of generating 4D video with larger motion, we first generate videos with small motion, then feed them into the model to improve the temporal consistency progressively.

Results

Normal Camera Trajectory

... + fin-like structure ...
... + blue wings ...
... + red can "CAT MEAT" ...
... + a vibrant orange kayak ...
... + bowknot and French fries ...
... + wearing a hat and a green monk's clothe ...
... + wears a gold hoop ...

Bullet Time

... + floating on a red donut ...
... + over-ear headphones ...
... + wearing sunglasses ...
... + eating a meat ...
... + red car ...
... + holding a delicious hamburger ...
... + green scarf ...

Application

... + zoom out ...
... + zoom in ...
... + steady shooting ...
... + camera turns right ...

Frozen Camera

... + panda ...
... + same type of figurine and a robot ...
... + exhibits blue-and-white porcelain ...
... + painting of sunflowers ...
... + robot dog ...
... + applauding audience ...
... + multiple freestanding cameras ...

💖Enjoyed this project? Drop a star ⭐follow-your-creation⭐—it means a lot!

🚀Our project page is borrowed from ISA4D.💡

Citation

                        @article{
                        title={Follow-Your-Creation: Empowering 4D Creation through Video Inpainting},
                        author=
                        journal={},
                        year={},
                        publisher={}
                        }