description: <p>(It would be greatly appreciated if someone can point to me a clean source of Tokyo 7th Sisters assets. I don't really want to scrape Twitter or reverse the game API.)</p><h1>Mask, Don't Negative Prompt: Dealing with Undesirable Parts of Training Images</h1><h2>Introduction</h2><p>Training images aren't always clean. Sometimes, when training for a given target, unrelated parts in images such as text, frames, or watermarks will also be learned by the model. There are several strategies that can be applied to this problem, each with shortcomings:</p><ol><li><p><strong>Cropping</strong>: Leave out undesired parts. Modifies source composition, not applicable in some cases.</p></li><li><p><strong>Inpainting</strong>: Preprocess the data and replace undesirable parts with generated pixels. Requires a good inpainting prompt / model.</p></li><li><p><strong>Negative Prompts</strong>: Train as is and add negative prompts when generating new images. Requires the model to know how the undesirable parts map to the prompt.</p></li></ol><p>Another simple strategy is effective:</p><ol><li><p><strong>Masking</strong>: Multiply the loss with a predefined mask.</p></li></ol><p>This method <a target="_blank" rel="ugc" href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/6700"><strong><u>is</u></strong></a> <a target="_blank" rel="ugc" href="https://github.com/csyxwei/ELITE"><strong><u>not</u></strong></a> new, but the most popular LoRA training script has yet to have built-in support for it.</p><h2>Experiment</h2><p><a target="_blank" rel="ugc" href="https://huggingface.co/datasets/gustproof/sd-data/blob/main/serizawa-momoka.zip"><strong><u>60 images</u></strong></a> with card text and decorations of <a target="_blank" rel="ugc" href="https://t7s.game-info.wiki/d/%b6%dc%c2%f4%a5%e2%a5%e2%a5%ab"><strong><u>Serizwa Momoka from Tokyo 7th Sisters</u></strong></a> were used.</p><p><a target="_blank" rel="ugc" href="https://huggingface.co/gustproof/sd-models/blob/main/serizawa-momoka/checkpoints/srzwmmk-masked-v1.0-000050.safetensors"><strong><u>A masked LoRA</u></strong></a> and <a target="_blank" rel="ugc" href="https://huggingface.co/gustproof/sd-models/blob/main/serizawa-momoka/checkpoints/srzwmmk-v1.0-000050.safetensors"><strong><u>an plain unmasked LoRA</u></strong></a> were trained.</p><p>For the masked version, <a target="_blank" rel="ugc" href="https://huggingface.co/gustproof/sd-models/resolve/main/posts/images/mask-original.webp"><strong><u>a mask</u></strong></a> was drawn using image editing software over source images. Note that since the VAE has a 8x scaling factor, what seen by the model is the <a target="_blank" rel="ugc" href="https://huggingface.co/gustproof/sd-models/resolve/main/posts/images/mask.webp"><strong><u>8x8 pixelated version</u></strong></a>. Tags that describe the parts masked away were removed.</p><h2>Results</h2><p>(see preview images)</p><h2><strong>Future work</strong></h2><ul><li><p>Auto generation of masks with segmentation models</p></li></ul>
modelVersions.availability: Public
modelVersions.baseModel: SD 1.5
modelVersions.baseModelType: Standard
modelVersions.description: <p>Card art only. Trained with masks.</p>
modelVersions.downloadUrl: https://civitai.com/api/download/models/73034
modelVersions.downloaded: true
modelVersions.downloadUrl: https://civitai.com/api/download/models/73034
modelVersions.AutoV2: 16C6E0EFF5
modelVersions.AutoV3: E724866B44F4
modelVersions.BLAKE3: BB742945C9BD366DA7E8DD679CBBE220EA3417D193AAF37F852503262081E66F
modelVersions.CRC32: 92A211B3
modelVersions.SHA256: 16C6E0EFF5D516DAA92E124F0979AE903BA1A9167DA7E84C894917FF0008A92B
modelVersions.format: SafeTensor
modelVersions.name: srzwmmk-masked-v1.0-000050.safetensors
modelVersions.pickleScanMessage: No Pickle imports
modelVersions.pickleScanResult: Success
modelVersions.scannedAt: 2023-05-17T15:09:48.856Z
modelVersions.type: Model
modelVersions.virusScanResult: Success
modelVersions.hash: UBKLBs0002IU00-,_49F00xXRNnj.m?JE0#8
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/b335fa9e-c592-4c57-a16c-a391357d2d1b/width=450/815112.jpeg
modelVersions.hash: UwMk2~tR?^s.x[t7spV@x]jFoJR+WAWCWEoJ
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/d9151fcc-4048-429d-b299-c5ef9e4bd042/width=450/815113.jpeg
modelVersions.hash: USN0xcD*}^?H=yR%wdozsE%MIVNd-;WDw^s.
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/cbdc10a1-3088-4806-b587-f61e6ea0742e/width=450/815114.jpeg
modelVersions.hash: UVOMjV+}~XOTUEyDtSoMNbbH%2Vs^lVtV?R%
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/dc1f14dc-12c0-4b5a-85e7-34499d48870e/width=450/815115.jpeg
modelVersions.hash: UhN-D$~p^+-ptSjYkBWBa_ofWEf7tSoIaefR
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/b10f0e58-077c-4c3e-ab21-1e1988da787a/width=450/815117.jpeg
modelVersions.name: masked-v1.0
modelVersions.publishedAt: 2023-05-17T08:11:10.498Z
modelVersions.trainedWords: Serizawa Momoka
name: Serizawa Momoka (Tokyo 7th Sisters) | Guide: Mask, Don't Negative Prompt: Dealing with Undesirable Parts of Training Images
tags: anime character game character t7s nanasis