All The LoRA FLUX Training Experiments I Have Done So Far
I have been using a machine on RunPod that has 8x RTX A6000 over a week now. I have done all the following experiments with each one being 3000 steps.
Latest configs shared on : https://www.patreon.com/posts/kohya-flux-lora-110879657
All findings, grids, and best configs and 1-click installers for Windows, RunPod and Massed Compute shared here : https://www.patreon.com/posts/kohya-flux-lora-110293257
Best configs got updated and total number of trainings reached to 104
You can see saved safetensors files in below repos. Click load more button at the end to see entire repo files
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part2/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part3/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part4/tree/main
https://huggingface.co/datasets/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part5/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_Final/tree/main
https://huggingface.co/MonsterMMORPG/FLUX_Kohya_SS_Massive_Research_Part6_T5_XXL/tree/main
All Tested Experiments List So Far
1) 16GB_Config.safetensors
2) 1e_04_bf16_128_rank-000170.safetensors
3) 1e_04_bf16_128_rank_full_bf16.safetensors
4) 1e_04_fp16_128_rank-000180.safetensors
5) 1e_04_fp16_128_rank_full_fp16-000180.safetensors
6) 1e_04_fp8_32_rank-000170.safetensors
7) 1e_04_fp8_bf16_accelerate_32_rank-000160.safetensors
8) 1e_04_fp8_bf16_accelerate_full_bf16_32_rank-000160.safetensors
9) 1e_04_fp8_full_fp16_32_rank-000170.safetensors
10) 2e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
11) 3e_04_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
12) 6e_05_best_additive_sigma.safetensors
13) 6e_05_best_additive_uniform.safetensors
14) 6e_05_best_max_grad_norm_0.safetensors
15) 6e_05_best_raw_sigma.safetensors
16) 6e_05_best_raw_sigmoid.safetensors
17) 6e_05_best_sigma_scaled_sigma.safetensors
18) 6e_05_best_with_reg_images.safetensors
19) 6e_05_bf16_128_rank_full_bf16.safetensors
20) 7e_05_bf16_128_rank_full_bf16.safetensors
21) 8e_05_bf16_128_rank_full_bf16.safetensors
22) 8e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
23) 9e_05_bf16_128_rank_full_bf16.safetensors
24) 9e_05_fp8_bf16_accelerate_full_bf16_32_rank.safetensors
25) Best_V4_T5_1e_05-000150.safetensors
26) Best_V4_T5_1e_05_TE_5e_05-000150.safetensors
27) Best_V4_T5_2e_05-000150.safetensors
28) Best_V4_T5_3e_05-000150.safetensors
29) Best_V4_T5_4e_05-000150.safetensors
30) Best_V4_T5_5e_05-000150.safetensors
31) Best_V4_T5_5e_05_With_Captions-000150.safetensors
32) Best_v1.safetensors
33) Best_v1_3e_5.safetensors
34) Best_v1_4e_5.safetensors
35) Best_v1_512px.safetensors
36) Best_v1_512px_Lowest_VRAM.safetensors
37) Best_v1_5e_5.safetensors
38) Best_v1_5e_5_max_grad_norm_0.safetensors
39) Best_v1_Lowest_VRAM.safetensors
40) Best_v1_T5_attention_mask.safetensors
41) Best_v1_cfg_3_5.safetensors
42) Best_v1_max_grad_norm_0.safetensors
43) Best_v1_reg_0_1_percent.safetensors
44) Best_v1_reg_prior_lost_0_3.safetensors
45) Best_v1_reg_prior_lost_0_6.safetensors
46) Best_v1_reg_prior_lost_1_0_apply_t5_attention_mask.safetensors
47) Best_v2.safetensors
48) Best_v2_1280.safetensors
49) Best_v2_1280_256rank.safetensors
50) Best_v2_1536.safetensors
51) Best_v2_1536_256rank.safetensors
52) Best_v2_2e_05_Clip_1e_04.safetensors
53) Best_v2_2e_05_Clip_3e_04.safetensors
54) Best_v2_2e_05_Clip_6e_05.safetensors
55) Best_v2_2e_05_Clip_8e_05.safetensors
56) Best_v2_4e_05_Clip_1e_04.safetensors
57) Best_v2_4e_05_Clip_3e_04.safetensors
58) Best_v2_4e_05_Clip_6e_05.safetensors
59) Best_v2_4e_05_Clip_8e_05.safetensors
60) Best_v2_5e_05_Clip_1e_05.safetensors
61) Best_v2_5e_05_Clip_5e_05.safetensors
62) Best_v2_FP8.safetensors
63) Best_v2_FP8_LowVRAM.safetensors
64) Best_v2_Flow_Shift_0_5.safetensors
65) Best_v2_Flow_Shift_1.safetensors
66) Best_v2_Rank16.safetensors
67) Best_v2_Rank256.safetensors
68) Best_v2_Rank32.safetensors
69) Best_v2_Rank4.safetensors
70) Best_v2_Rank64.safetensors
71) Best_v2_Rank8.safetensors
72) Best_v2_Shift_Sample_3e_04.safetensors
73) Best_v2_Shift_Sample_4e_04.safetensors
74) Best_v2_Shift_Sample_6e_04.safetensors
75) Best_v2_Shift_Sample_7e_04.safetensors
76) Best_v2_Split_QKV.safetensors
77) Best_v2_joycaption.safetensors
78) Best_v2_joycaption_Split_QKV.safetensors
79) Best_v2_only_OHWX.safetensors
80) Best_v2_shift.safetensors
81) Best_v2_shift_1e_05.safetensors
82) Best_v2_shift_2e_05.safetensors
83) Best_v2_shift_3e_05.safetensors
84) Best_v2_shift_4e_05.safetensors
85) Best_v2_shift_6e_05.safetensors
86) Best_v2_shift_9e_06.safetensors
87) Best_v2_shift_value_1.safetensors
88) Best_v3.safetensors
89) Best_v3_4e_05_TE_6e_04.safetensors
90) Best_v3_Clip_Skip_0.safetensors
91) Best_v3_Clip_Skip_2.safetensors
92) Best_v3_Discrete_Flow_1_0.safetensors
93) Best_v3_Discrete_Flow_3_0.safetensors
94) Best_v3_TE_0.safetensors
95) Best_v3_xFormers.safetensors
96) Best_v4.safetensors
97) Best_v4_Clip_Skip_1.safetensors
98) CivitAI_FLUX.safetensors
99) Rank_5_11700MB_14_62_Second_IT.safetensors
100) Rank_8_15300MB_4_53_Second_IT.safetensors
101) Rank_8_15400MB_3_81_Second_IT.safetensors
102) Rank_8_15450MB_3_80_Second_IT.safetensors
103) Rank_8_15600MB_4_89_Second_IT.safetensors
104) Rank_8_9500MB_10_00_Second_IT.safetensors
Comments (12)
Hello brother, I can't download the configuration file on huggingface. I need the author's consent. My name is he, a purple avatar. Please agree to my download request.
yes these are my personal trainings not open to download
Which configuration would you recommend for training a character model on an RTX 4090 so that it doesn’t take all day? My model is FP8 PixelWave
go for rank 4 it should be really fast , load fp16 version of the model it will auto cast
So which config ist the best for style? I will give it a try in runpod
for runpod go with Rank 1 - it is best - if it be slow go with rank 2
Can you share the settings to make kohya_ss train faster with multiple gpu's? I am currently training with 4 A6000's on Massed Compute but it seems slower than my actual 4090.
good idea i can share massed compute max speed
Do you have a one click installer for flux which downloads the models etc cheers
yes i have : FLUX Auto Model Downloader for Windows, RunPod and Massed Compute https://www.patreon.com/posts/110323512 zip file also has auto downloader for training needed models : https://www.patreon.com/posts/kohya-flux-lora-110293257
Thanks for all of your work and research. Incredibly helpful! Does your LORA research help you determine what DB fine tuning training will work best?
it will help me for sure. after lora hopefully full fine tuning / dreambooth next