Legality of Cannabis by U.S. Jurisdiction

Stable Diffusion
Developer(s)CompVis group LMU Munich; Runway; Stability AI[1]
Initial releaseAugust 22, 2022
Stable release
2.1 (model)[2] / December 7, 2022
Repositorygithub.com/Stability-AI/stablediffusion
Written inPython
Operating systemAny that support CUDA kernels
TypeText-to-image model
LicenseCreative ML OpenRAIL-M
Websitestability.ai

Stable Diffusion is a deep learning, text-to-image model released in 2022. It is primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt.[3]

Stable Diffusion is a latent diffusion model, a variety of deep generative neural network developed by the CompVis group at LMU Munich.[4] The model has been released by a collaboration of Stability AI, CompVis LMU, and Runway with support from EleutherAI and LAION.[5][1][6] In October 2022, Stability AI raised US$101 million in a round led by Lightspeed Venture Partners and Coatue Management.[7]

Stable Diffusion's code and model weights have been released publicly,[8] and it can run on most consumer hardware equipped with a modest GPU with at least 8 GB VRAM. This marked a departure from previous proprietary text-to-image models such as DALL-E and Midjourney which were accessible only via cloud services.[9][10]

Technology

Diagram of the latent diffusion architecture used by Stable Diffusion.
The denoising process used by Stable Diffusion. The model generates images by iteratively denoising random noise until a configured number of steps have been reached, guided by the CLIP text encoder pretrained on concepts along with the attention mechanism, resulting in the desired image depicting a representation of the trained concept.

Architecture

Stable Diffusion uses a variant of diffusion model (DM), called latent diffusion model (LDM).[1] Introduced in 2015, diffusion models are trained with the objective of removing successive applications of Gaussian noise on training images which can be thought of as a sequence of denoising autoencoders. Stable Diffusion consists of 3 parts: the variational autoencoder (VAE), U-Net, and an optional text encoder.[11] The VAE encoder compresses the image from pixel space to a smaller dimensional latent space, capturing a more fundamental semantic meaning of the image.[12] Gaussian noise is iteratively applied to the compressed latent representation during forward diffusion.[11] The U-Net block, composed of a ResNet backbone, denoises the output from forward diffusion backwards to obtain latent representation. Finally, the VAE decoder generates the final image by converting the representation back into pixel space.[11] The denoising step can be flexibly conditioned on a string of text, an image, and other modalities. The encoded conditioning data is exposed to denoising U-Nets via a cross-attention mechanism.[11] For conditioning on text, the fixed, pretrained CLIP ViT-L/14 text encoder is used to transform text prompts to an embedding space.[1] Researchers point to increased computational efficiency for training and generation as an advantage of LDMs.[13][14]

Training data

Stable Diffusion was trained on pairs of images and captions taken from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web, where 5 billion image-text pairs were classified based on language, filtered into separate datasets by resolution, a predicted likelihood of containing a watermark, and predicted "aesthetic" score (e.g. subjective visual quality).[15] The dataset was created by LAION, a German non-profit which receives funding from Stability AI.[15][16] The Stable Diffusion model was trained on three subsets of LAION-5B: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+.[15] A third-party analysis of the model's training data identified that out of a smaller subset of 12 million images taken from the original wider dataset used, approximately 47% of the sample size of images came from 100 different domains, with Pinterest taking up 8.5% of the subset, followed by websites such as WordPress, Blogspot, Flickr, DeviantArt and Wikimedia Commons.[17][15]

Training procedures

The model was initially trained on the laion2B-en and laion-high-resolution subsets, with the last few rounds of training done on LAION-Aesthetics v2 5+, a subset of 600 million captioned images which the LAION-Aesthetics Predictor V2 predicted that humans would, on average, give a score of at least 5 out of 10 when asked to rate how much they liked them.[18][15][19] The LAION-Aesthetics v2 5+ subset also excluded low-resolution images and images which LAION-5B-WatermarkDetection identified as carrying a watermark with greater than 80% probability.[15] Final rounds of training additionally dropped 10% of text conditioning to improve Classifier-Free Diffusion Guidance.[20]

The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost of $600,000.[21][22][23]

Limitations

Stable Diffusion has issues with degradation and inaccuracies in certain scenarios. Initial releases of the model were trained on a dataset that consists of 512×512 resolution images, meaning that the quality of generated images noticeably degrades when user specifications deviate from its "expected" 512×512 resolution;[24] the version 2.0 update of the Stable Diffusion model later introduced the ability to natively generate images at 768×768 resolution.[25] Another challenge is in generating human limbs due to poor data quality of limbs in the LAION database.[26] The model is insufficiently trained to understand human limbs and faces due to the lack of representative features in the database, and prompting the model to generate images of such type can confound the model.[27] In addition to human limbs, generating animal limbs have been observed to be challenging as well, with the observed failure rate of 25% when trying to generate an image of a horse.[28]

Accessibility for individual developers can also be a problem. In order to customize the model for new use cases that are not included in the dataset such as generating anime characters ("waifu diffusion"),[29] new data and further training are required. Fine-tuned adaptations of Stable Diffusion created through additional retraining have been used for a variety of different use-cases, from medical imaging[30] to algorithmically-generated music.[31] However, this fine-tuning process is sensitive to the quality of new data; low resolution images or different resolutions from the original data can not only fail to learn the new task but degrade the overall performance of the model. Even when the model is additionally trained on high quality images, it is difficult for individuals to run models in consumer electronics. For example, the training process for waifu-diffusion requires a minimum 30 GB of VRAM,[32] which exceeds the usual resource provided in consumer GPUs, such as Nvidia’s GeForce 30 series having around 12 GB.[33]

The creators of Stable Diffusion acknowledge the potential for algorithmic bias, as the model was primarily trained on images with English descriptions.[22] As a result, generated images reinforce social biases and are from a western perspective as the creators note that the model lacks data from other communities and cultures. The model gives more accurate results for prompts that are written in English in comparison to those written in other languages with western or white cultures often being the default representation.[22]

End-user fine tuning

To address the limitations of the model's initial training, end-users may opt to implement additional training for the purpose of fine-tuning generation outputs to match more specific use-cases. There are three methods in which user-accessible fine tuning can be applied to a Stable Diffusion model checkpoint:

  • An "embedding" can be trained from a collection of user-provided images, and allows the model to generate visually similar images whenever the name of the embedding is used within a generation prompt.[34] Embeddings are based on the "textual inversion" concept developed by researchers from Tel Aviv University in 2022 with support from Nvidia, where vector representations for specific tokens used by the model's text encoder are linked to new pseudo-words. Embeddings can be used to reduce biases within the original model, or mimic visual styles.[35]
  • A "Hypernetwork" is a small pre-trained neural network that is applied to various points within a larger neural network, and refers to the technique created by NovelAI developer Kurumuz in 2021, originally intended for text-generation transformer models. Hypernetworks steer results towards a particular direction, allowing Stable Diffusion-based models to imitate the art style of specific artists, even if the artist is not recognised by the original model; they process the image by finding key areas of importance such as hair and eyes, and then patch these areas in secondary latent space.[36]
  • DreamBooth is a deep learning generation model developed by researchers from Google Research and Boston University in 2022 which can fine-tune the model to generate precise, personalised outputs that depict a specific subject, following training via a set of images which depict the subject.[37]

Capabilities

The Stable Diffusion model supports the ability to generate new images from scratch through the use of a text prompt describing elements to be included or omitted from the output.[1] Existing images can be re-drawn by the model to incorporate new elements described by a text prompt (a process known as "guided image synthesis"[38]) through its diffusion-denoising mechanism.[1] In addition, the model also allows the use of prompts to partially alter existing images via inpainting and outpainting, when used with an appropriate user interface that supports such features, of which numerous different open source implementations exist.[39]

Stable Diffusion is recommended to be run with 10 GB or more VRAM, however users with less VRAM may opt to load the weights in float16 precision instead of the default float32 to tradeoff model performance with lower VRAM usage.[24]

Text to image generation

Demonstration of the effect of negative prompts on image generation.
  • Top: No negative prompt
  • Centre: "green trees"
  • Bottom: "round stones, round rocks"

The text to image sampling script within Stable Diffusion, known as "txt2img", consumes a text prompt in addition to assorted option parameters covering sampling types, output image dimensions, and seed values. The script outputs an image file based on the model's interpretation of the prompt.[1] Generated images are tagged with an invisible digital watermark to allow users to identify an image as generated by Stable Diffusion,[1] although this watermark loses its efficacy if the image is resized or rotated.[40]

Each txt2img generation will involve a specific seed value which affects the output image. Users may opt to randomize the seed in order to explore different generated outputs, or use the same seed to obtain the same image output as a previously generated image.[24] Users are also able to adjust the number of inference steps for the sampler; a higher value takes a longer duration of time, however a smaller value may result in visual defects.[24] Another configurable option, the classifier-free guidance scale value, allows the user to adjust how closely the output image adheres to the prompt.[20] More experimentative use cases may opt for a lower scale value, while use cases aiming for more specific outputs may use a higher value.[24]

Additional text2img features are provided by front-end implementations of Stable Diffusion, which allow users to modify the weight given to specific parts of the text prompt. Emphasis markers allow users to add or reduce emphasis to keywords by enclosing them with brackets.[41] An alternative method of adjusting weight to parts of the prompt are "negative prompts". Negative prompts are a feature included in some front-end implementations, including Stability AI's own DreamStudio cloud service, and allow the user to specify prompts which the model should avoid during image generation. The specified prompts may be undesirable image features that would otherwise be present within image outputs due to the positive prompts provided by the user, or due to how the model was originally trained, with mangled human hands being a common example.[39][2]

Image modification

Stable Diffusion also includes another sampling script, "img2img", which consumes a text prompt, path to an existing image, and strength value between 0.0 and 1.0. The script outputs a new image based on the original image that also features elements provided within the text prompt. The strength value denotes the amount of noise added to the output image. A higher strength value produces more variation within the image but may produce an image that is not semantically consistent with the prompt provided.[1]

The ability of img2img to add noise to the original image makes it potentially useful for data anonymization and data augmentation, in which the visual features of image data are changed and anonymized.[42] The same process may also be useful for image upscaling, in which the resolution of an image is increased, with more detail potentially being added to the image.[42] Additionally, Stable Diffusion has been experimented with as a tool for image compression. Compared to JPEG and WebP, the recent methods used for image compression in Stable Diffusion face limitations in preserving small text and faces.[43]

Demonstration of inpainting and outpainting techniques using img2img within Stable Diffusion
File:Demonstration of inpainting and outpainting using Stable Diffusion, halal edition (step 1 of 4).png
Step 1: An image is generated, coincidentally with the subject having one arm missing.
File:Demonstration of inpainting and outpainting using Stable Diffusion, halal edition (step 2 of 4).png
Step 2: Via outpainting, the bottom of the image is extended by 768 pixels and filled with AI-generated content.
File:Demonstration of inpainting and outpainting using Stable Diffusion, halal edition (step 3 of 4).png
Step 3: In preparation for inpainting, a makeshift arm is drawn using the paintbrush in GIMP.
File:Demonstration of inpainting and outpainting using Stable Diffusion, halal edition (step 4 of 4).png
Step 4: An inpainting mask is applied over the makeshift arm, and img2img generates a new arm while leaving the remainder of the image untouched.

Additional use-cases for image modification via img2img are offered by numerous front-end implementations of the Stable Diffusion model. Inpainting involves selectively modifying a portion of an existing image delineated by a user-provided layer mask, which fills the masked space with newly generated content based on the provided prompt.[39] A dedicated model specifically fine-tuned for inpainting use-cases was created by Stability AI alongside the release of Stable Diffusion 2.0.[25] Conversely, outpainting extends an image beyond its original dimensions, filling the previously empty space with content generated based on the provided prompt.[39]

A depth-guided model, named "depth2img", was introduced with the release of Stable Diffusion 2.0 on November 24, 2022; this model infers the depth of the provided input image, and generates a new output image based on both the text prompt and the depth information, which allows the coherence and depth of the original input image to be maintained in the generated output.[25]

Usage

Stable Diffusion claims no rights on generated images and freely gives users the rights of usage to any generated images from the model provided that the image content is not illegal or harmful to individuals. The freedom provided to users over image usage has caused controversy over the ethics of ownership, as Stable Diffusion and other generative models are trained from copyrighted images without the owner’s consent.[44]

As visual styles and compositions are not subject to copyright, it is often interpreted that users of Stable Diffusion who generate images of artworks should not be considered to be infringing upon the copyright of visually similar works.[45] However, individuals depicted in generated images may be protected by personality rights if their likeness is used,[45] and intellectual property such as recognizable brand logos still remain protected by copyright. Nonetheless, visual artists have expressed concern that widespread usage of image synthesis software such as Stable Diffusion may eventually lead to human artists, along with photographers, models, cinematographers, and actors, gradually losing commercial viability against AI-based competitors.[46]

Stable Diffusion is notably more permissive in the types of content users may generate, such as violent or sexually explicit imagery, in comparison to other commercial products based on generative AI.[47] Addressing the concerns that the model may be used for abusive purposes, CEO of Stability AI, Emad Mostaque, explains that "[it is] peoples' responsibility as to whether they are ethical, moral, and legal in how they operate this technology",[10] and that putting the capabilities of Stable Diffusion into the hands of the public would result in the technology providing a net benefit, in spite of the potential negative consequences.[10] In addition, Mostaque argues that the intention behind the open availability of Stable Diffusion is to end corporate control and dominance over such technologies, who have previously only developed closed AI systems for image synthesis.[10][47] This is reflected by the fact that any restrictions Stability AI places on the content that users may generate can easily be bypassed due to the open-source nature of the license which Stable Diffusion was released under.[44]

License

Unlike models like DALL-E, Stable Diffusion makes its source code available,[48][1] along with pretrained weights. Its license prohibits certain use cases, including crime, libel, harassment, doxing, "exploiting ... minors", giving medical advice, automatically creating legal obligations, producing legal evidence, and "discriminating against or harming individuals or groups based on ... social behavior or ... personal or personality characteristics ... [or] legally protected characteristics or categories".[49][50] The user owns the rights to their generated output images, and is free to use them commercially.[51]

See also

References

  1. ^ a b c d e f g h i j "Stable Diffusion Repository on GitHub". CompVis - Machine Vision and Learning Research Group, LMU Munich. 17 September 2022. Retrieved 17 September 2022.
  2. ^ a b "Stable Diffusion v2.1 and DreamStudio Updates 7-Dec 22". stability.ai. Archived from the original on December 10, 2022.
  3. ^ "Diffuse The Rest - a Hugging Face Space by huggingface". huggingface.co. Archived from the original on 2022-09-05. Retrieved 2022-09-05.
  4. ^ Rombach; Blattmann; Lorenz; Esser; Ommer (June 2022). High-Resolution Image Synthesis with Latent Diffusion Models (PDF). International Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA. pp. 10684–10695. arXiv:2112.10752.
  5. ^ "Stable Diffusion Launch Announcement". Stability.Ai. Archived from the original on 2022-09-05. Retrieved 2022-09-06.
  6. ^ "Revolutionizing image generation by AI: Turning text into images". LMU Munich. Retrieved 17 September 2022.
  7. ^ Wiggers, Kyle (17 October 2022). "Stability AI, the startup behind Stable Diffusion, raises $101M". Techcrunch. Retrieved 2022-10-17.
  8. ^ Stable Diffusion, CompVis - Machine Vision and Learning LMU Munich, 2022-11-04, retrieved 2022-11-04
  9. ^ "The new killer app: Creating AI art will absolutely crush your PC". PCWorld. Archived from the original on 2022-08-31. Retrieved 2022-08-31.
  10. ^ a b c d Vincent, James (15 September 2022). "Anyone can use this AI art generator — that's the risk". The Verge.
  11. ^ a b c d Alammar, Jay. "The Illustrated Stable Diffusion". jalammar.github.io. Retrieved 2022-10-31.
  12. ^ "High-Resolution Image Synthesis with Latent Diffusion Models". Machine Vision & Learning Group. Retrieved 2022-11-04.
  13. ^ "Stable Diffusion launch announcement". Stability.Ai. Retrieved 2022-11-02.
  14. ^ Rombach; Blattmann; Lorenz; Esser; Ommer (June 2022). High-Resolution Image Synthesis with Latent Diffusion Models (PDF). International Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA. pp. 10684–10695. arXiv:2112.10752.
  15. ^ a b c d e f Baio, Andy (2022-08-30). "Exploring 12 Million of the 2.3 Billion Images Used to Train Stable Diffusion's Image Generator". Waxy.org. Retrieved 2022-11-02.
  16. ^ "This artist is dominating AI-generated art. And he's not happy about it". MIT Technology Review. Retrieved 2022-11-02.
  17. ^ Ivanovs, Alex (2022-09-08). "Stable Diffusion: Tutorials, Resources, and Tools". Stack Diary. Retrieved 2022-11-02.
  18. ^ Schuhmann, Christoph (2022-11-02), CLIP+MLP Aesthetic Score Predictor, retrieved 2022-11-02
  19. ^ "LAION-Aesthetics | LAION". laion.ai. Archived from the original on 2022-08-26. Retrieved 2022-09-02.
  20. ^ a b Ho, Jonathan; Salimans, Tim (2022-07-25). "Classifier-Free Diffusion Guidance". arXiv:2207.12598 [cs.LG].
  21. ^ Mostaque, Emad (August 28, 2022). "Cost of construction". Twitter. Archived from the original on 2022-09-06. Retrieved 2022-09-06.
  22. ^ a b c "CompVis/stable-diffusion-v1-4 · Hugging Face". huggingface.co. Retrieved 2022-11-02.
  23. ^ Wiggers, Kyle (2022-08-12). "A startup wants to democratize the tech behind DALL-E 2, consequences be damned". TechCrunch. Retrieved 2022-11-02.
  24. ^ a b c d e "Stable Diffusion with 🧨 Diffusers". huggingface.co. Retrieved 2022-10-31.
  25. ^ a b c "Stable Diffusion 2.0 Release". stability.ai. Archived from the original on December 10, 2022.
  26. ^ "LAION". laion.ai. Retrieved 2022-10-31.
  27. ^ "Generating images with Stable Diffusion". Paperspace Blog. 2022-08-24. Retrieved 2022-10-31.
  28. ^ François Chollet. "(If you were wondering how often Stable Diffusion will give you a horse with more than 4 legs (or sometimes less) when you ask it for a photo of a horse: in my experience it's about 20-25% of the time.)". Twitter. Retrieved 2022-10-31.
  29. ^ "hakurei/waifu-diffusion · Hugging Face". huggingface.co. Retrieved 2022-10-31.
  30. ^ Chambon, Pierre; Bluethgen, Christian; Langlotz, Curtis P.; Chaudhari, Akshay (2022-10-09). "Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains". arXiv:2210.04133 [cs.CV].
  31. ^ Seth Forsgren; Hayk Martiros. "Riffusion - Stable diffusion for real-time music generation". Riffusion. Archived from the original on December 16, 2022.
  32. ^ Mercurio, Anthony (2022-10-31), Waifu Diffusion, retrieved 2022-10-31
  33. ^ Smith, Ryan. "NVIDIA Quietly Launches GeForce RTX 3080 12GB: More VRAM, More Power, More Money". www.anandtech.com. Retrieved 2022-10-31.
  34. ^ Dave James (October 28, 2022). "I thrashed the RTX 4090 for 8 hours straight training Stable Diffusion to paint like my uncle Hermann". PC Gamer. Archived from the original on November 9, 2022.
  35. ^ Gal, Rinon; Alaluf, Yuval; Atzmon, Yuval; Patashnik, Or; Bermano, Amit H.; Chechik, Gal; Cohen-Or, Daniel (2022-08-02). "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion". arXiv:2208.01618 [cs.CV].
  36. ^ "NovelAI Improvements on Stable Diffusion". NovelAI. October 11, 2022. Archived from the original on October 27, 2022.
  37. ^ Yuki Yamashita (September 1, 2022). "愛犬の合成画像を生成できるAI 文章で指示するだけでコスプレ 米Googleが開発". ITmedia Inc. (in Japanese). Archived from the original on August 31, 2022.
  38. ^ Meng, Chenlin; He, Yutong; Song, Yang; Song, Jiaming; Wu, Jiajun; Zhu, Jun-Yan; Ermon, Stefano (August 2, 2021). "SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations". arXiv:2108.01073 [cs.CV].
  39. ^ a b c d "Stable Diffusion web UI". GitHub. 10 November 2022.
  40. ^ invisible-watermark, Shield Mountain, 2022-11-02, retrieved 2022-11-02
  41. ^ "stable-diffusion-tools/emphasis at master · JohannesGaessler/stable-diffusion-tools". GitHub. Retrieved 2022-11-02.
  42. ^ a b Luzi, Lorenzo; Siahkoohi, Ali; Mayer, Paul M.; Casco-Rodriguez, Josue; Baraniuk, Richard (2022-10-21). "Boomerang: Local sampling on image manifolds using diffusion models". arXiv:2210.12100 [cs.CV].
  43. ^ Bühlmann, Matthias (2022-09-28). "Stable Diffusion Based Image Compression". Medium. Retrieved 2022-11-02.
  44. ^ a b Cai, Kenrick. "Startup Behind AI Image Generator Stable Diffusion Is In Talks To Raise At A Valuation Up To $1 Billion". Forbes. Retrieved 2022-10-31.
  45. ^ a b "高性能画像生成AI「Stable Diffusion」無料リリース。「kawaii」までも理解し創造する画像生成AI". Automaton Media (in Japanese). August 24, 2022.
  46. ^ Heikkilä, Melissa (16 September 2022). "This artist is dominating AI-generated art. And he's not happy about it". MIT Technology Review.
  47. ^ a b Ryo Shimizu (August 26, 2022). "Midjourneyを超えた？ 無料の作画AI｢ #StableDiffusion ｣が｢AIを民主化した｣と断言できる理由". Business Insider Japan (in Japanese).
  48. ^ "Stable Diffusion Public Release". Stability.Ai. Archived from the original on 2022-08-30. Retrieved 2022-08-31.
  49. ^ "Ready or not, mass video deepfakes are coming". The Washington Post. 2022-08-30. Archived from the original on 2022-08-31. Retrieved 2022-08-31.
  50. ^ "License - a Hugging Face Space by CompVis". huggingface.co. Archived from the original on 2022-09-04. Retrieved 2022-09-05.
  51. ^ Katsuo Ishida (August 26, 2022). "言葉で指示した画像を凄いAIが描き出す「Stable Diffusion」 ～画像は商用利用も可能". Impress Corporation (in Japanese).

External links

source: https://en.wikipedia.org/w/index.php?title=Stable_Diffusion&oldid=1128886942