Netflix researchers have created a new type of AI-powered green-screen technology that can produce realistic visual effects for film and television in real time.
Green-screen technology is routinely used to capture footage of actors that can then be inserted in the foreground of virtual or prerecorded scenes. To do this, actors are filmed against a bright green background, which is easily isolated and removed digitally.
This process can be done automatically with reasonable accuracy, such as in television weather forecasts, but it can be thrown by items of green clothing or by transparent or fine objects, like wisps of hair. When greater accuracy is needed in films or television series, specialist operators tweak settings manually, sometimes requiring hours to perfect a shot.
In a bid to create a technique that is both fast and accurate, Netflix has come up with a method it calls Magenta Green Screen (MGS). Actors are filmed against a background of bright green LEDs while being lit from the front with red and blue ones, which together create a magenta glow (see video, above).
Because digital cameras work by taking an individual red, green and blue value for each pixel, this technique has the effect of creating a green channel that records only the background, with the foreground appearing black, and red and blue channels that record only the foreground, leaving the background looking black. Together these create the magenta and green look.
Film editors can replace the green channel in real time, realistically and instantly placing the actors in the foreground of another scene, with even potentially tricky areas, such as transparent bottles or the area around strands of hair, working without problems.
“Computers already have provided such powerful tools to make a lot of stuff easier,” says Paul Debevec, one of the Netflix researchers involved. “[This is] another thing that we can make easier, so that the talented artists that we have can focus on the artistry, actually making things look better.”
But there is a problem with the method. Because the foreground is only recorded in blue and red, it leaves the actors looking magenta-tinted. To solve this, Netflix uses artificial intelligence to put the full range of colour back into the foreground, using a photograph of the actors lit normally as a reference to create a realistic-looking green channel. This AI works quickly, but not yet in real time, although fast techniques such as averaging the red and blue channels to create an approximation of a green channel work effectively enough for the director to monitor while filming.
Drew Lahat at video production company Geiger Post in California says the industry always welcomes new methods, but that the technique involves a lot of complex steps, which may not yet be conducive to the fast-paced nature of shooting film and television.
“There are a variety of aspects of this technology that, from a practical perspective, make me wonder if it’s worth it,” he says. “From what I understand, unless you have an excellent green screen in a fully controlled space, this technique falls apart fast.”
Topics:
