Stable Diffusion

An image generated by Stable Diffusion based on the text prompt "a photograph of an astronaut riding a horse"
Developer(s)	StabilityAI
Initial release	August 22, 2022
Stable release	1.5 (model)[1] / August 31, 2022
Repository	github.com/CompVis/stable-diffusion
Written in	Python
Operating system	Any that support CUDA kernels
Type	Text-to-image model
License	Creative ML OpenRAIL-M
Website	stability.ai

Stable Diffusion is a deep learning, text-to-image model released by startup StabilityAI in 2022. It is primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt.^[2]

Stable Diffusion is a latent diffusion model, a variety of generative neural network developed by researchers at LMU Munich. It was developed by Stability AI in collaboration with LMU and Runway, with support from EleutherAI and LAION.^[3][4][5] Stability AI is in talks to raise capital at a valuation of up to one billion dollars as of September 2022.^[6]

Stable Diffusion's code and model weights have been released publicly, and it can run on most consumer hardware equipped with a modest GPU. This marked a departure from previous proprietary text-to-image models such as DALL-E and Midjourney which were accessible only via cloud services.^[7][8]

Architecture

Stable Diffusion is a form of diffusion model (DM). Introduced in 2015, diffusion models are trained with the objective of removing successive applications of Gaussian noise to training images, and can be thought of as a sequence of denoising autoencoders. Stable Diffusion uses a variant known as a "latent diffusion model" (LDM). Rather than learning to denoise image data (in "pixel space"), an autoencoder is trained to transform images into a lower-dimensional latent space. The process of adding and removing noise is applied to this latent representation, with the final denoised output then decoded into pixel space. Each denoising step is accomplished by a U-Net architecture. The researchers point to reduced computational requirements for training and generation as an advantage of LDMs.^[3][9]

The denoising step may be conditioned on a string of text, an image, or some other data. An encoding of the conditioning data is exposed to the denoising U-Nets via a cross-attention mechanism. For conditioning on text, a transformer language model was trained to encode text prompts.^[9]

Usage

The Stable Diffusion model supports the ability to generate new images from scratch through the use of a text prompt describing elements to be included or omitted from the output,^[4] and the redrawing of existing images which incorporate new elements described within a text prompt (a process commonly known as guided image synthesis^[10]) through the use of the model's diffusion-denoising mechanism.^[4] In addition, the model also allows the use of prompts to partially alter existing images via inpainting and outpainting, when used with an appropriate user interface that supports such features, of which numerous different open source implementations exist.^[11]

Stable Diffusion is recommended to be run with 10GB or more VRAM, however users with less VRAM may opt to load the weights in float16 precision instead of the default float32 to lower VRAM usage.^[12]

Text to image generation

Demonstration of the effect of negative prompts on image generation.

• Top: No negative prompt
• Centre: "green trees"
• Bottom: "round stones, round rocks"

The text to image sampling script within Stable Diffusion, known as "txt2img", consumes a text prompt, in addition to assorted option parameters covering sampling types, output image dimensions, and seed values, and outputs an image file based on the model's interpretation of the prompt.^[4] Generated images are tagged with an invisible digital watermark to allow users to identify an image as generated by Stable Diffusion,^[4] although this watermark loses its effectiveness if the image is resized or rotated.^[13] The Stable Diffusion model is trained on a dataset consisting of 512×512 resolution images,^[4][14] meaning that txt2img output images are optimally configured to be generated at 512×512 resolution as well, and deviating from this size can result in poor quality generation outputs.^[12]

Each txt2img generation will involve a specific seed value which affects the output image; users may opt to randomise the seed in order to explore different generated outputs, or use the same seed to obtain the same image output as a previously generated image.^[12] Users are also able to adjust the number of inference steps for the sampler; a higher value takes a longer duration of time, however a smaller value may result in visual defects.^[12] Another configurable option, the classifier-free guidance scale value, allows the user to adjust how closely the output image adheres to the prompt;^[15] more experimentative or creative use cases may opt for a lower value, while use cases aiming for more specific outputs may use a higher value.^[12]

Negative prompts are a feature included in some user interface implementations of Stable Diffusion which allow the user to specify prompts which the model should avoid during image generation, for use cases where undesirable image features would otherwise be present within image outputs due to the positive prompts provided by the user, or due to how the model was originally trained.^[11] The use of negative prompts has a highly statistically significant effect on decreasing the frequency of generating unwanted outputs compared to the use of emphasis markers, which are another alternative method of adding weight to parts of prompts utilised by some open-source implementations of Stable Diffusion, where brackets are added to keywords to add or reduce emphasis.^[16]

Image modification

Stable Diffusion includes another sampling script, "img2img", which consumes a text prompt, path to an existing image, and strength value between 0.0 and 1.0, and outputs a new image based on the original image that also features elements provided within the textual prompt; the strength value denotes the amount of noise added to the output image, with a higher value producing images with more variation, however may not be semantically consistent with the prompt provided.^[4] Image upscaling is one potential use case of img2img, among others.^[4]

Inpainting and outpainting

Additional use-cases for image modification via img2img are offered by numerous different front-end implementations of the Stable Diffusion model. Inpainting involves selectively modifying a portion of an existing image delineated by a user-provided mask, which fills the masked space with newly generated content based on the provided prompt.^[11] Conversely, outpainting extends an image beyond its original dimensions, filling the previously empty space with content generated based on the provided prompt.^[11]

License

Unlike models like DALL-E, Stable Diffusion makes its source code available,^[17][4] along with pre-trained weights. Its license prohibits certain use cases, including crime, libel, harassment, doxxing, "exploiting ... minors", giving medical advice, automatically creating legal obligations, producing legal evidence, and "discriminating against or harming individuals or groups based on ... social behavior or ... personal or personality characteristics ... [or] legally protected characteristics or categories".^[18][19] The user owns the rights to their generated output images, and is free to use them commercially.^[20]

Training

Stable Diffusion was trained on pairs of images and captions taken from LAION-5B, a publically available dataset derived from Common Crawl data scraped from the web. The dataset was created by LAION, a German non-profit which receives funding from Stability AI.^[14][21] The model was initially trained on a large subset of LAION-5B, with the final rounds of training done on "LAION-Aesthetics v2 5+", a subset of 600 million captioned images which an AI predicted that humans would give a score of at least 5 out of 10 when asked to rate how much they liked them.^[14][22] This final subset also excluded low-resolution images and images which an AI identified as carrying a watermark.^[14] A third-party analysis of the model's training data identified that out of a smaller subset of 12 million images taken from the original wider dataset used, approximately 47% of the sample size of images came from 100 different domains, with Pinterest taking up 8.5% of the subset, followed by websites such as WordPress, Blogspot, Flickr, DeviantArt and Wikimedia Commons.^[23][14]

The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost of $600,000.^[24][25][26]

Societal impacts

As visual styles and compositions are not subject to copyright, it is often interpreted that users of Stable Diffusion who generate images of artworks should not be considered to be infringing upon the copyright of visually similar works,^[27] however individuals depicted in generated images may still be protected by personality rights if their likeness is used,^[27] and intellectual property such as recognisable brand logos still remain protected by copyright. Nonetheless, visual artists have expressed concern that widespread usage of image synthesis software such as Stable Diffusion may eventually lead to human artists, along with photographers, models, cinematographers and actors, to gradually lose commercial viability against AI-based competitors.^[21]

Stable Diffusion is notably more permissive in the types of content users may generate, such as violent or sexually explicit imagery, in comparison to similar machine learning image synthesis products from other companies.^[28] Addressing concerns that the model may be used for abusive purposes, CEO of StabilityAI Emad Mostaque explains that "(it is) peoples' responsibility as to whether they are ethical, moral, and legal in how they operate this technology",^[8] and that putting the capabilities of Stable Diffusion into the hands of the public would result in the technology providing a net benefit overall, even in spite of the potential negative consequences.^[8] In addition, Mostaque argues that the intention behind the open availability of Stable Diffusion is to end corporate control and dominance over such technologies, who have previously only developed closed AI systems for image synthesis.^[8][28]

See also

• 15.ai
• Artificial intelligence art
• Craiyon
• Imagen (Google Brain)

References

External links

• Stable Diffusion Demo