会员体验
专利管家(专利管理)
工作空间(专利管理)
风险监控(情报监控)
数据分析(专利分析)
侵权分析(诉讼无效)
联系我们
交流群
官方交流:
QQ群: 891211   
微信请扫码    >>>
现在联系顾问~
热词
    • 1. 发明申请
    • GENERATION OF A VIRTUAL VIEWPOINT IMAGE OF A PERSON FROM A SINGLE CAPTURED IMAGE
    • WO2023057781A1
    • 2023-04-13
    • PCT/GR2021/000061
    • 2021-10-08
    • FACEBOOK TECHNOLOGIES, LLCSARAFIANOS, NikolaosTUNG, Tony
    • SARAFIANOS, NikolaosTUNG, Tony
    • H04N13/268H04N13/261G06T7/11G06T7/194G06T15/04G06T15/20G06T15/205G06T2207/30196
    • In particular embodiments, to generate two videos corresponding to the two views of a human from the single video, multiple neural networks may be used to process a 2D video of a human in motion. Initially, a video of a person may be captured by a standard video camera. The video may comprise a plurality of frames including RGB images of the person. A computing system may process these images to generate the two videos corresponding to the two views of a human. As an example and not by way of limitation, an artificial reality headset may process these images to generate a mapping between the RGB pixels of each of the images to a 3D surface-based model of body part of the person. In order to generate the mapping, the artificial reality headset may use a mapping machine-learning model to process one of the RGB images to generate the mapping between pixels of the single RGB image to the 3D surface-based model of body part of the person.A machine-learning model may be used to refine the 3D surface-based model into a refined 3D surface- based model. The refined 3D surface-based model is then used to warp the single RGB image that was used to generate the 3D surface- based model to generate a texture of the person. The texture may then be used as an input to a full texture machine-learning model that generates a full-body UV texture from the partial texture generated by the warped RGB images. The full texture machine-learning model may be used to inpaint regions that are not seen or are self-occluded areas. As an example and not by way of limitation, if the partial texture generated by the warped RGB images have a partial texture of a person's hands, then the full texture machine-learning model may inpaint the region corresponding to the person's hands to generate a complete texture of the person's hands. After generating the full texture using the full texture machine-learning model, the artificial reality headset may then use a view generation machine- learning model to generate two different views of the person using the refined 3D surface-based model. That is, the view generation machine-learning model may generate a left eye refined 3D surface- based model and a right eye refined 3D surface-based model. The complete texture generated from the full texture machine-learning model may then be warped onto the left eye refined 3D surface-based model and the right eye refined 3D surface-based model. A neural Tenderer machine-learning model may then be applied to the warped left and right images to fill in missing pixel information to generate the stereo pair of images of the person. This method of generating a stereo pair of images and/or an image at a virtual viewpoint may be done without an explicit 3D reconstruction of the person.