What’s the Deal with that Content-Aware Fill for Video Thing Adobe Showed Us?
Last month at Adobe MAX in Las Vegas, we saw Project Cloak for the first time. The research project and experiment is still in development, and it may or may not ever be in a Creative Cloud product, but it sure was captivating: draw a box, do some Adobe Sensei magic, and poof: that ugly lamp post is just gone. Logo on a t-shirt? GONE! Weirdo in the backgrond of your shot? Yep: EXTREMELY GONE.
If you’ve ever spent hours of your day rotoscoping something to remove it from a shot, you might share the sentiment: I need this. Are we playing God, or are we taking back precious hours of our time? Why not both!
To find out more about the research, technology, and thinking behind it, I talked to Research Engineer Geoffrey Oxholm.
Creative COW: What drove the development of Project Cloak? Was it knowing that people rotoscoped out stuff in a tedious way all the time? Or was it more like “we’ve done this in Photoshop, I wonder if we can do it on a moving image?” Was the genesis more about solving a problem, or more about experimentation?
Geoffrey: The Adobe Research team has strong partnerships (and friendships) with our colleagues on the product teams. We’ve known for ages that “video inpainting” (removing stuff from video) would be hugely impactful to our customers. One of our researchers published the seminal work on this topic as part of his PhD thesis back in 2004. Since then, it has been a problem that the academic community has been working on, but mostly coming up with solutions that have significant barriers to being ready for consumers (limited use-cases, running times, memory constraints). Recently, there have been some breakthroughs in related areas that inspired us to take a closer look.
How long are things like Project Cloak worked on before they get shown at MAX? I’m sure it must vary, but are people working on these things for months, years, or longer?
This problem has been actively researched for over a decade by the wider academic community. While I can’t speak to other Adobe MAX sneaks, the code that’s running behind Cloak was engineered over about 4 months.
Bye, haters.
What more can you tell me about how Project Cloak works under the hood that wasn’t addressed in the MAX presentation?
I can’t dive into a lot of detail, but we call the area you wish to remove the “hole”. The algorithm works by estimating the motion within the hole. We use dense tracking to keep track of what part of the video has seen the hole region either before, or after, the current frame. Using the motion estimation, we essentially shove the pixels through the hole, and record what the frames look like. If there are portions that are not seen in the video, then we fall back on Content-Aware-Fill to guess what the appearance looks like. There are some details around ensuring the appearance looks good across the whole video, but that’s the basic idea. In summary, with Adobe Sensei, Project Cloak uses an advanced algorithm to dramatically accelerate a time-consuming process, delivering results that previously would have required many hours of editing.
Last year there was a bit of backlash and increased interest in Sneaks thanks to VoCo, which was essentially Photoshop for Audio. In this political climate where there’s been a big focus on what’s fake and what’s real. How do you address people that may be concerned about that?
Our Vice President of Creativity, Mark Randall discussed this topic in detail in a blog post following our demo of VoCo last year.
These questions, this dilemma, is part of our legacy as tool builders. We’re deeply inspired by helping real people create amazing new things, works of art, the future, with our tools. Adobe shows projects like Cloak at MAX Sneaks before they’re ready to be out in the world because we welcome the conversations and feedback from the community. As responsible global citizens, we’re committed to thoughtful development of our tools.
As researchers, our life’s work is on magnifying creativity and opening up new avenues for expression. These are real problems that amazing creative people have, and helping them is what motivates us.
And of course: can we please have Cloak in After Effects? Please?
Glad to hear that you’re excited and eager to see Cloak in one of our shipping products.