description: <p>Imported into Civitai from <a target="_blank" rel="ugc" href="https://rentry.org/LazyDAdaptationGuide">https://rentry.org/LazyDAdaptationGuide</a></p><p></p><p>This guide is a repository for testing and tweaking DAdaptation V3 LoRAs, introduced by Kohya on 05/25/2023 .</p><p>For reference to my guide on collating a dataset, and the old method of utilizing the</p><p>AdamW8Bit optimizer, see - <a target="_blank" rel="ugc" href="https://rentry.org/lazytrainingguide">https://rentry.org/lazytrainingguide</a></p><p>Useful links:</p><p>Kohya - <a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts">https://github.com/kohya-ss/sd-scripts</a></p><p><a target="_blank" rel="ugc" href="https://github.com/kohya-ss/sd-scripts￼bmaltais"><br />bmaltais</a> training GUI (Dreambooth <a target="_blank" rel="ugc" href="http://localhost">localhost</a>, <strong>DADAPT V3 HAS NOW BEEN IMPLEMENTED</strong>) - <a target="_blank" rel="ugc" href="https://github.com/bmaltais/kohya_ss">https://github.com/bmaltais/kohya_ss</a></p><p></p><p>With the inclusion of DADAPT V3 into bmailtais, I have provided a .json via <a target="_blank" rel="ugc" href="https://files.catbox.moe/8yh7ko.json"><span style="color:rgb(232, 230, 227)">https://files.catbox.moe/8yh7ko.json</span></a> - edit your img folders, max epochs, dim/alpha, weight decay etc.</p><p></p><p></p><p>DAdaptation is an adaptive optimizer that automatically selects and tweaks the TE/UNET/LR values as it trains.</p><p>I have spent the past few months tweaking the older DAdaptation implementation (Assumingly 1.5) via the derriandestro repo. With the release of V3 and other versions of the DAdaptation optimizer, I firmly believe that this is the best optimizer to use and oust AdamW8Bit as the default optimizer.</p><p>Here is an example of the shift in learning rates between DAdapt V2 (green) and DAdapt V3 (blue) - in general, it's less heavy in blasting out a higher learning rate and cooking the image faster, meaning we can keep the weight decay lower than previously practiced:</p><p></p><p><img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/a1bf0f0c-bffe-45c7-8499-61712210de47/width=525/a1bf0f0c-bffe-45c7-8499-61712210de47.jpeg" /></p><p></p><p><strong>DAdaptation removes the need for setting restarts, setting learning rates, text encoders, unet learning rates etc. Simply put, you can flick a switch and it will generate your LoRA with minimal effort.</strong></p><p>DAdapt can be used for all LoRAs. Characters, concepts, artist, style etc.</p><p>DAdapt needs a higher 'weight decay' value, as expressed in the research data and the LoRA results. This is the primary method for tweaking your LoRA. Is your LoRA not learning enough before it overfits? Increase the weight decay in .1 increments. With the release of V3, it seems to blast the learning rate a lot less heavily than V1.5 or 2, and so we can usually get away with reducing the weight decay more than we used to for the previous interations of the optimizer.</p><p>DAdapt should have the learning rate, text encoder and unet learning rate set anywhere from 0.5 to 1. This is personal preference, but I've found keeping it at '1' and tweaking the weight decay is the simplest method of fine-tuning training.</p><p>DAdapt needs the argument <code>--optimizer_args "decouple=True"</code> setting along with the weight decay settings (for example): <code>"weight_decay=0.4" "betas=0.9,0.99"</code><br />This can be done in the Bmaltais Kohya GUI through the parameters boxes.</p><p>Here are a few of the settings I have confirmed work well:</p><p></p><p>Learning Rate (LR) 1</p><p>Text Encoder (TE) 1</p><p>UNET 1</p><p>Weight Decay (WD) - From 0.2-1 - will require longer training to learn more, but FAR less prone to overfitting</p><p>Repeats 1 for each data subfolder - Allows for tweaking of steps via epochs and removes the need to do math</p><p>Resolution 512 - DAdapt is a heavier train and needs better hardware. 768 is not worth it</p><p>Clip Skip 2 - For NAI based models</p><p>Training Model NAI</p><p>Scheduler Cosine - Constant is not good, and I'm not sure why it was originally recommended for DAdaptation</p><p>Batch Size - As high as you can go for your hardware. For example, 6 for a 3060 - From my experience it's better to set this and forget it, and focus on more impactful settings for training</p><p>Epochs - Set it as high as you want. As an example, I have tested a 36 image, 3 character LoRA. This started to burn out at around 230 epochs at weight decay 0.4 totalling around 2400 steps at batch 6 - Once again, DAdapt needs more steps to achieve results as we're actively hampering the learning through decay</p><p><br />DIM32 - (or set it lower to '8' for example, increase training time/reduce weight decay)</p><p>Alpha16 - (match the DIM if set to 8 or lower)</p><p>Warmups - Relatively untested, but I just set it to 0.1 to give it a bit of time to calibrate (?) - set warmups to 0 if you're using the above .json and the "growth_rate=1.02\" argument</p><p>How many training steps for my LoRA? - Final steps are divisible by your batch - for example, at batch 6, for a single character LoRA on DAdaptation I would probably expect anywhere from 600-1200 steps. Generally, training til overfitting and going back down the epochs will save needing to re-train the LoRA all over again with new settings</p><p></p><p>Examples:</p><p>XYZ plot starting at 100 epochs, up to 234 - 1/1/1 TE/UNET/LR, 0.4 weight decay, 0.1 warmup steps - training at batch 6 totalling around 2,400 steps</p><p></p><p><strong>Note that these are trained on a tiny dataset of 12 images per character with one outfit, and should not be a representation of the plability of character clothing, positions etc.</strong></p><p><img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6d70d4ae-0375-493f-a253-58311241f2bd/width=525/6d70d4ae-0375-493f-a253-58311241f2bd.jpeg" /></p><p>LoRA trained on 3 characters, example tags:</p><p>Aru - Aru, long hair, breasts, bangs, shirt, gloves, ribbon, yellow eyes, white shirt, pink hair, horns, collard shirt, red ribbon, coat, blunt bangs, fur-trimmed coat, halo</p><p>Shiroko - Shiroko, long hair, breasts, bangs, blue eyes, animal ears, animal ear fluff, mismatched pupils, cross hair ornament, blue scarf, grey hair, halo</p><p>Karin (Bunny) - Karin, large breasts, black hair, halo, animal ears, cleavage, hair between eyes, fake animal hears, playboy bunny, rabbit tail, yellow eyes, very long hair, dark skin</p><p>(The final epoch version has been uploaded to this Civitai post)</p><p>Example .safetensors -</p><p>Final Epoch - <a target="_blank" rel="ugc" href="https://files.catbox.moe/muah1o.safetensors￼200">https://files.catbox.moe/muah1o.safetensors<br />200</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/luvylf.safetensors￼180">https://files.catbox.moe/luvylf.safetensors<br />180</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/xkemzm.safetensors￼160">https://files.catbox.moe/xkemzm.safetensors<br />160</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/ane6p6.safetensors￼140">https://files.catbox.moe/ane6p6.safetensors<br />140</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/frshlg.safetensors￼120">https://files.catbox.moe/frshlg.safetensors<br />120</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/47hsmp.safetensors￼100">https://files.catbox.moe/47hsmp.safetensors<br />100</a> - <a target="_blank" rel="ugc" href="https://files.catbox.moe/qgbxs9.safetensors">https://files.catbox.moe/qgbxs9.safetensors</a></p>
modelVersions.availability: Public
modelVersions.baseModel: Other
modelVersions.baseModelType: Standard
modelVersions.downloadUrl: https://civitai.com/api/download/models/82680
modelVersions.downloaded: true
modelVersions.downloadUrl: https://civitai.com/api/download/models/82680
modelVersions.AutoV1: 6AE6860D
modelVersions.AutoV2: 1CB7A73E14
modelVersions.AutoV3: C3E1A57C80CE
modelVersions.BLAKE3: F29E517E5F376F7DBA2FF70D588F1704A50CF93F849A8789A754712A1B82E501
modelVersions.CRC32: 2A91951E
modelVersions.SHA256: 1CB7A73E142FEE20FA1C3C550709F296078EF411228F1FCF3676B443DA05AC5A
modelVersions.format: SafeTensor
modelVersions.name: DAdapt BA Test.safetensors
modelVersions.pickleScanMessage: No Pickle imports
modelVersions.pickleScanResult: Success
modelVersions.scannedAt: 2023-05-27T12:06:32.990Z
modelVersions.type: Model
modelVersions.virusScanResult: Success
modelVersions.hash: UnKd#v-;~q-;xuxuofInt7RjxuM{bbR*V@t7
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/9e96d6d4-d8aa-46b3-8829-1f6db92d2129/width=450/949597.jpeg
modelVersions.hash: UmJ[6bayaxt7~qfPayof?bayj[fQ_3ofj[j[
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/7e798a89-0291-4e42-8d23-43d808741bec/width=450/930764.jpeg
modelVersions.hash: U03us6tSV?t3xtofRkjYt7WBWAazxvj]WAfh
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/d8e09513-cbfa-41c6-9f98-5587c4d6f5b7/width=450/930763.jpeg
modelVersions.name: DAdaptation BA
modelVersions.publishedAt: 2023-05-27T12:05:24.061Z
name: LoRA Lazy DAdaptation Guide
tags: anime guide lora