You may have to make use of the gpu_memory_limit and/or lora_on_cpu config possibilities to stop functioning from memory. If you still operate outside of CUDA memory, you can make an effort to merge in method RAM https://www.asgdfx.com/