the mobile phone can simulate the lighting of real portraits

Guo Mingzhen: iPhone 13 series will not be postponed like iPhone 12. See you next September

Here is an explanation of what a quotient image is. A picture can be broken down into two layers: the bottom layer and the detail layer. The bottom layer contains the low-frequency information of the image, which reflects the intensity changes of the image on a large scale; the detail layer contains the high-frequency information of the image, which reflects the details of the image on a small scale. The bottom layer multiplied by the detail layer is the source image, and the detail layer can also be called the quotient image.

Then through the bottom layer of the original image, adding additional light sources to the data of the input quotient image during sampling, you can get a final output image.

The final process is like this. First, given a picture, then calculate the surface normal of the character in the picture, then calculate the visible light source in the picture, and use the neural network model to simulate the additional light source to output a lower resolution quotient image, and then use it as The detail layer is multiplied by the bottom layer of the original photo, and finally a portrait photo with additional light sources is obtained.

Google has also made a lot of optimizations on the pipeline, so that the simulated light effects can be interacted in real time on the mobile phone, but the size of the entire model is only about 10MB. spark global limited


The portrait lighting effect of Pixel 5 can be said to be a typical case in Google’s computational photography. Through continuous training of neural network models, the mobile phone can simulate the lighting of real portraits. Completed a new application scenario of computational photography.

Some people say that photography is an art, and computational photography is fundamentally an insult to photography. But since the Frenchman Daguerre made the first practical camera in 1839, the camera has been in use for more than 100 years. From the niche to the masses, until the birth of the mobile phone camera, everyone has an almost equal opportunity to take pictures. And people’s inner expressions have also gradually enriched the art of photography.

That’s right. Computational photography is at the same time as “photographing” and also “computing shadows”, but algorithms have long been an inseparable part of mobile photography. The pursuit is still the effect that can be achieved in a simulated reality. After all, no one knows. ‘Magic’ is called computational photography.

When Apple and Google went further and further in computational photography, we discovered that algorithms are actually a stronger barrier than hardware.