<p>This is a SDXL LoRA trained on the ponyDiffusion_V6XL base model. (It doesn’t work on other models; however, you are free to try).</p><p></p><p>The intended effect is to create images which look like high quality (well lit) 3D renders.</p><p></p><p>The base model is already perfectly capable of generating images in a 3d style (using the tags <em>3d</em> and <em>blender</em>), however I wanted more of a certain style, and if possible, also more flexibility.</p><p> </p><p>For that, I tagged a certain list of keywords during captioning, in the hope to be able to specify certain styles. As quite a bit of images however had a mixture of those tags, the biggest influence is achieved by using multiple tags at once. However, even not using any tag at all has some influence.</p><p>To show an exaggerated, non-cherry-picked example (not using 3d and blender tags, and using source_anime, with seeds 1,2,3) this is the effect of the LoRA with the prompt:</p><pre><code>score_9,score_8_up,score_7_up, 1girls, big_breasts, sfw, selfie, female, light_skin, slim, crop_top, leggings, pink_hair, brown_eyes,  v_sign, peace_sign, living_room, source_anime, rating_safe</code></pre><p> </p><p><img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/8d4a65d8-181a-4c3d-9a2b-4484bab1152f/width=525/8d4a65d8-181a-4c3d-9a2b-4484bab1152f.jpeg" /></p><p><span style="color:rgb(134, 142, 150)">(View full resolution </span><a target="_blank" rel="ugc" href="https://files.catbox.moe/xll4lc.webp"><strong><span style="color:rgb(134, 142, 150)">here</span></strong></a><span style="color:rgb(134, 142, 150)">) </span></p><p></p><p>The tags specifically are (together with a description):</p><p> </p><ul><li><p><strong>RTX_on</strong> – This is what I intended to be the base tag, which was tagged for every image</p></li><li><p><strong>RTX_soft</strong> – Soft rendered images, and soft lighting</p></li><li><p><strong>RTX_flat</strong> – Used for 3D images which were rather flat, especially in skin texture (imagine Overwatch models, but rendered with Source Filmmaker and few lights)</p></li><li><p><strong>RTX_pt</strong> – Stands for “path tracing”, tagged in images with especially dominant lights and shadows, or when the the scene was very well lit (read realistic global illumination, ambient occlusion, indirect lights, accurate shadows, …)</p></li><li><p><strong>RTX_hairsim</strong> – Stands for “simulated hair”, I tagged a subset of images, where the hair was realistically simulated with many individual hairs. However I did not tag all of them, and seemingly insufficient, so this tag can be a bit stubborn</p></li><li><p><strong>RTX_texture</strong> – Similar to the opposite of RTX_flat, this tag was applied when realistic skin or fabric textures were used, or realistic fluids (sweat / water) were on skin, as the previous tag, this was not tagged throughout, so can also be a bit more inaccurate</p></li></ul><p> </p><p>While <em>RTX_texture</em> and <em>RTX_flat</em> seem like opposites, there were some source images where I tagged both. Speaking in video game terms, this was in cases where the skin didn’t really have an albedo texture, but it had a normal map and was lit in the correct angle, so that shadows were created.</p><p> </p><p>For a rough idea of what the individual tags do, I refer to the following example image.</p><p>That image is a matrix of all tags, where the columns have an increased weight compared to the row (to show the effect of the tag a bit more strongly). Always both tags are in the prompt, the column tags are first. I suggest opening the image externally.</p><p> </p><p> <img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/02f44d23-9d20-4f71-a30d-4a4bc33b00e2/width=525/02f44d23-9d20-4f71-a30d-4a4bc33b00e2.jpeg" /></p><p> <span style="color:rgb(134, 142, 150)">(View full resolution </span><a target="_blank" rel="ugc" href="https://files.catbox.moe/a6fp4v.webp"><strong><span style="color:rgb(134, 142, 150)">here</span></strong></a><span style="color:rgb(134, 142, 150)">) </span></p><p></p><p>Images were sourced from rule34, this means this LoRA could have worse effects on non-humanoid or realistic characteristics. (I did one or two example images, which looked fine (probably thanks to it being SDXL), but I suggest looking at sample images and reviews from other users, to get a better overview).</p><p> </p><h2 id="heading-28">Training</h2><p> </p><p>As always, I will add a little bit about the training.</p><p>This was the first time I trained on a non-base model, and the first style I tried. I am personally sufficiently happy with the results, although I would have hoped for them to be a bit better.</p><p>I selected 1250 high resolution base images from rule34. Due to my requirements of image feel and quality, this resulted in a good part of the images being from games like overwatch, Cyberpunk 2077 or similar. Many source images also had watermarks. This could increase the likelihood of creating characters looking like for example overwatch characters (when not prompting for specific characters), but also the likelihood of it generating watermarks.</p><p>For the collected images, I kept the original tags they had, but then added the above listed custom tags. RTX_on was tagged for almost every image besides very few exceptions. In addition to that, I added the relevant tag, when I felt like it fitted for the image. As most of the images had a certain base style (imagine for example a Cyberpunk 2077 screenshot), I decided to keep only the RTX_on tag on those images, I didn’t for example tag RTX_hairsim or RTX_texture. In retrospect (if I would do this again), I would also tag these details in all images.</p><p> </p><p>The training itself was done in kohya, I chose 4 repeats, batch size 6 and 30 epochs. This resulted in 25110 steps. As I was using booru tags, I enabled shuffled captions and kept the first 3 tokens for each caption. Additionally, as some images had tons of tags from the boorus, I increased the max token length to 150.</p><p>I tried out the Prodigy optimizer for the first time, so I chose my settings very similar to the settings which seemed to work for somebody else. If you want to know more about those settings, I really recommend watching this video: <a target="_blank" rel="ugc" href="https://www.youtube.com/watch?v=QpWacUWeqbE">https://www.youtube.com/watch?v=QpWacUWeqbE</a></p><p>During those 30 epochs, the LoRA did not overbake, so unlike previous LoRAs I made, I didn’t mix together multiple versions I had for better results. So, you can actually also look at the additional metadata in the safetensors this time around.</p><p>Training was done on rank 128 (both dimension and alpha), which was later resized to a target rank of 32.</p><p> </p><p>The overall training process took a bit more than 19 hours on a cloud RTX 4090 (using 23.5 GB peak VRAM).  </p><p> </p><p>If you have any more questions, feel free to ask me.</p><p> </p><h2 id="heading-29">License</h2><p> </p><p>As this model is trained on Pony Diffusion V6 XL, I decided to license the LoRA under a similar modified Fair AI Public License 1.0-SD (<a target="_blank" rel="ugc" href="https://freedevproject.org/faipl-1.0-sd/">https://freedevproject.org/faipl-1.0-sd/</a>) license.</p><p>The following modifications have been added to Fair AI Public License:</p><p> </p><p>You are not permitted to run inference of this model on websites or applications allowing any form of monetization (paid inference, faster tiers, etc.). This applies to any derivative models or model merges.</p><p> </p><p>Explicit permission for commercial inference is granted to CivitAi and Hugging Face.</p>
Public
Pony
https://civitai.com/api/download/models/321758
true
https://civitai.com/api/download/models/321758
639DA3E2
100229E777
4445EAD0D354
A9C59E4B84622C6EF733125982EB007A37C648DB2C32D2B092A1458DFB7620FA
459EC61B
100229E77772EA36E8137E557281B28FAF1E7859AE1BD2A30AE858B967443D6D
SafeTensor
DI_RTX_on_pony_diffusion_v6.safetensors
No Pickle imports
Success
2024-01-31T04:35:52.948Z
Model
Success
UFI;;p?IUaNf0f9ZkD-;4UZgmktS,,xax^i^
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/b9f2044d-892c-4c5d-bd04-c3c2766be2f6/width=450/6088021.jpeg
ULKATx=d*IEN0;j]0xOYGHt7R4NG0$Rk+ttR
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/0cd9d85d-408e-44e9-8a14-7de55e4b7901/width=450/6088080.jpeg
UEFYfa}Q};%%D%OqInohH?T1In4oBqIAV@Xn
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/82da77aa-161e-40fb-879b-4041a33a7992/width=450/6088225.jpeg
UIHBxk_MN0Md.4I-smM|D$Z~t7Iqo~E3R*SL
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/57135eef-d7cd-41e7-8771-a3fcb01011bf/width=450/6088230.jpeg
UEHxTp^6%gD%?^Di.8xZx^4:of%LM|R+Rjoz
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/947774c9-00c6-4ad8-9835-5bcaaf16efa6/width=450/6088093.jpeg
U8GbF*^P0001Kbof0gIp-qD$-q~DGF-pD$In
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/8c3bb5c6-1c0c-45c1-97b5-a6f50ba44d43/width=450/6088198.jpeg
UKHnsmwH0LJB_2E2D*bE?^%1IANGXlIpD*s.
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/f193a3ad-18c0-4bd5-a49e-4a146c5467ab/width=450/6088121.jpeg
UUFr*O4._NjGbwRjjFWVx]oLRjW=%NaeNFjs
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/d70e28aa-7e27-49cc-92f5-1584f0da204b/width=450/6088122.jpeg
UMFiMt?cE+N{x]-;r=R%x]x^s+ae.T%MWExZ
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/5683ab96-790c-4dd7-a95b-ffcfc2fd9900/width=450/6088200.jpeg
USIqH0E0IA-n?dIovztREUi{r:XB${aLbcad
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/c38a5594-63bc-4bfd-9672-f43ab77dcaa3/width=450/6088219.jpeg
ULHxsv-V.Tpd5[r=TKJVohslIUtSKQbvn$S$
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/725d4b32-f959-49ae-aef3-44359f36093e/width=450/6088239.jpeg
UAI#4A01~Atm,bV@o~jF~C-:0#kB0znN^4W;
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/747095f5-3448-4448-ad33-95b93dc4e827/width=450/6088240.jpeg
UKGuas?H594p~o-:4:9a?ZxZDiog%2IUofo}
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/90c3c00d-765d-4577-a30d-9c678d48f855/width=450/6088256.jpeg
UCD,WAKRE2=y7|WFIVOn~7nNt6xwImD+-okk
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/1465d091-7c1b-45be-ac53-2d9bc6421225/width=450/6088250.jpeg
UKI#W3?HE*oz0#56IA#l}[M{ozj?r?%NIpM{
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/705aa487-6f20-44be-b09e-f07154b6b43b/width=450/6088028.jpeg
UEHxW.-p_L?FTwOuT0-.L2_2$PSO?wNdD*Z~
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/c2bf34b3-7149-40bb-9785-6e681c4cf10a/width=450/6088296.jpeg
U7G8x~02AU~BL4-BNatR07n3vh0gD$%0$}a0
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/351d35a2-0fb9-4b86-b299-2dfec811ee43/width=450/6088268.jpeg
UcIhBQ,n?wNe.9jZo~ogJCt7R%s.%3S4s.WV
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/228d02a7-491e-41d0-86b3-46ed2c4f19a3/width=450/6088269.jpeg
UCHUFR?u2yEN?bs9I;Iq_MM{$%-:6-bbJCSh
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/e8583da6-e0d3-45d7-8023-0763fa69a21d/width=450/6088295.jpeg
UJI;k]^lVrw5}kg2?wxaENM|%NIoyENtv}IU
image
https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/762a73cf-7ec8-4e4c-9569-786340bc94fe/width=450/6088326.jpeg
DI_RTX_on_ponydiffusionv6
2024-01-31T04:41:38.429Z
RTX_on  RTX_soft RTX_flat RTX_texture RTX_pt RTX_hairsim
RTX On - PonyDiffusionXLV6
concept