ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]

Is there any tutorial or post somewhere on how to install other checkpoints or LoRAS to use with Stream diffusion?

...Loading sd15 ControlNet model: control_v11p_sd21_depth.safetensors Failed to load ControlNet model https://huggingface.co/thibaud/controlnet-sd21/blob/main/control_v11p_sd21_depth.safetensors due to: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. Traceback (most recent call last): File "D:\AI TOUCH\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 637, in _load_model controlnet = ControlNetModel.from_single_file(controlnet_file, config_file=config_file, use_safetensors=True, local_files_only=clip_exists, torch_dtype=torch.float16).to(device=self.device, dtype=self.dtype) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\loaders\single_file.py", line 617, in from_single_file controlnet = download_controlnet_from_original_ckpt( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1846, in download_controlnet_from_original_ckpt controlnet = convert_controlnet_checkpoint( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1125, in convert_controlnet_checkpoint set_module_tensor_to_device(controlnet, param_name, "cpu", value=param) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\accelerate\utils\modeling.py", line 292, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect.

issue, when i start streaming with controlnet ON, show me this error: "ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. " help pleaseee

Hello, when start streaming it says in the console : " V2V + ControlNet are not supported and will be disabled. A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'. " Do you have an idea where this issue comes from and how I can resolve it ?

I can't do it on streaming. cmd doesn't load data. can you please help me

[issue] ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Loading pipeline components...: 0%| | 0/5 [00:00

how do I enable the additional settings for negative prompts?

Thank you so much for the controlnet tutorial. Would you also share how to use openpose in stream diffusion ? How to get the colorful skeleton form video source? Thanks in advance.

Hi all, I'm enjoying this tool, thanks DotSimulate. But I'd like to know if you have found models better than SD-TURBO; do you have any advice?

What would be the equivalent of img2img in comfy? I get good results when I send a video thru stream diffusion but I would like to increase the quality and fps, and I'm looking for the equivalent with stable diffusion / comfyui. Thanks !

same issue. fixed with pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 thanks dot!

lo voy a ver, Gracias Dot!!

Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Why prompt No internet connection detected。

is there only 512x512 ? or is there a way to have a better resolution?

# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages ( AND replace cu121 with cu118 if you have cuda11.8): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.

hi! i have this issue: ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Hello. Thank you for the amazing work! does 0.2.2 work on mac? Oh I just saw in changelog that you will update Mac-support. I wanna test that :)

Mac install guide?

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed+1

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed

Can you record a tutorial for a plug-in to load the civitai Lora model library? I downloaded several model libraries and loaded them in the models in the plug-in, but the effect I got was similar to the effect of not binding the plug-in. I'm not sure whether my model library is not running now. Is there any way to judge that my model library is being used normally?

Amaaazing work man! Very cool stuff. I’m curious, as I have a couple of GPU’s…. would there be a way for 2 frames to be processed at once on say a 4090 and 4080 setup? I’ve seen some chatter on this being the way to leverage them, in lieu of actual multi-gpu crunching on each frame. Appreciate it!

hello, im newbie, i already install the streamdiffusion 0.2.2, after install the step 1 and step 2, i try to run start stream, the command prompt stuck in last log here Loading pipeline components...: 0%| | 0/5 [00:00

When I click Download StreamDiffusion in Install, the message "No internet connection detected" is displayed.

Thank you for the great work! Where do you link to the DotUpdater tox?

[Issues] : I recieve a "Script Error" when I pulse the second Install step (Venv~) By the way, thanks for the great work!! [Edit] : 0.2.0 works well