{"id":20988,"date":"2023-06-23T06:56:31","date_gmt":"2023-06-23T13:56:31","guid":{"rendered":"http:\/\/jnack.com\/blog\/?p=20988"},"modified":"2025-03-12T18:59:04","modified_gmt":"2025-03-13T01:59:04","slug":"google-uses-generative-imaging-for-virtual-try-on","status":"publish","type":"post","link":"http:\/\/jnack.com\/blog\/2023\/06\/23\/google-uses-generative-imaging-for-virtual-try-on\/","title":{"rendered":"Google uses generative imaging for virtual try-on"},"content":{"rendered":"\n<p>In my time at Google, we tried and failed&nbsp;<em>a lot<\/em>&nbsp;to make virtual try-on happen using AR. It\u2019s extremely hard to\u2026<\/p>\n\n\n\n<ul>\n<li><strong>measure<\/strong>&nbsp;bodies (to make buying decisions based on fit)<\/li>\n\n\n\n<li><strong>render<\/strong>&nbsp;virtual clothing accurately (placing virtual clothing over real clothing, or getting them to disrobe, which is even harder!; simulating materials in realtime)<\/li>\n\n\n\n<li>get a sizable&nbsp;<strong>corpus<\/strong>&nbsp;of 3D assets (in a high-volume, low-margin industry)<\/li>\n<\/ul>\n\n\n<blockquote class=\"wp-block-quote\"><\/blockquote>\n\n\n<p>Outside of a few limited pockets (trying on makeup, glasses, and shoes\u2014all for style, not for fit), I haven\u2019t seen anyone (Amazon, Snap, etc.) crack the code here. Researcher Ira Kemelmacher-Shlizerman (who last I heard was working on virtual mirrors, possibly leveraging Google\u2019s Stargate tech) acknowledges this:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Current techniques like geometric warping can cut-and-paste and then deform a clothing image to fit a silhouette. Even so, the final images never quite hit the mark: Clothes don\u2019t realistically adapt to the body, and they have visual defects like misplaced folds that make garments look misshapen and unnatural.<\/p>\n<\/blockquote>\n\n\n\n<p>So, it\u2019s interesting to see&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/blog.google\/products\/shopping\/ai-virtual-try-on-google-shopping\/?utm_source=kw-newsletter&amp;utm_medium=email&amp;utm_campaign=06152023A&amp;utm_content=cta\" target=\"_blank\">Google trying again<\/a>&nbsp;(\u201cTry on clothes with generative AI\u201d):<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>This week we introduced an AI-powered virtual try-on feature that uses the Google Shopping Graph to show you how clothing will look on a diverse set of real models.<\/p>\n\n\n\n<p>Our new guided refinements can help U.S. shoppers fine-tune products until you find the perfect piece. Thanks to machine learning and new visual matching algorithms, you can refine using inputs like color, style and pattern.<\/p>\n<\/blockquote>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/storage.googleapis.com\/gweb-uniblog-publish-prod\/original_images\/try-on-blog-post_1.gif\" alt=\"\"\/><\/figure><\/div>\n\n\n<p>They\u2019ve posted a\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/blog.google\/products\/shopping\/virtual-try-on-google-generative-ai\/\" target=\"_blank\">technical overview<\/a>\u00a0and a link to their\u00a0<a rel=\"noreferrer noopener\" href=\"https:\/\/tryondiffusion.github.io\/\" target=\"_blank\">project site<\/a>:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Inspired by Imagen, we decided to tackle VTO using diffusion \u2014 but with a twist. Instead of using text as input during diffusion, we use a pair of images: one of a garment and another of a person. Each image is sent to its own&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/blog.google\/technology\/ai\/ask-a-techspert-whats-a-neural-network\/\" target=\"_blank\">neural network<\/a>&nbsp;(a U-net) and shares information with each other in a process called \u201ccross-attention\u201d to generate the output: a photorealistic image of the person wearing the garment. This combination of image-based diffusion and cross-attention make up our new AI model.<\/p>\n<\/blockquote>\n\n\n\n<p>They note that \u201cWe don\u2019t promise fit and for now focus only on visualization of the try on. Finally, this work focused on upper body clothing.\u201d<\/p>\n\n\n\n<p>It\u2019s a bit hard to find exactly where one can try out the experience. They write:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Starting today, U.S. shoppers can virtually try on women\u2019s tops from brands across Google, including Anthropologie, Everlane, H&amp;M and LOFT. Just tap products with the \u201cTry On\u201d badge on Search and select the model that resonates most with you.<\/p>\n<\/blockquote>\n\n\n\n<p><!-- \/wp:quote --><\/p>","protected":false},"excerpt":{"rendered":"<p>In my time at Google, we tried and failed&nbsp;a lot&nbsp;to make virtual try-on happen using AR. It\u2019s extremely hard to\u2026 Outside of a few limited pockets (trying on makeup, glasses, and shoes\u2014all for style, not for fit), I haven\u2019t seen anyone (Amazon, Snap, etc.) crack the code here. Researcher Ira Kemelmacher-Shlizerman (who last I heard [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[66,77],"tags":[],"_links":{"self":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/20988"}],"collection":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/comments?post=20988"}],"version-history":[{"count":4,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/20988\/revisions"}],"predecessor-version":[{"id":20992,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/20988\/revisions\/20992"}],"wp:attachment":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/media?parent=20988"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/categories?post=20988"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/tags?post=20988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}