StreamDiffusionTD v 0.1.1
Important Fix for NDI that prevents memory build up
( at the time of this post, I also uploaded v0.1.1 as the operator in the 0.1.0 TOX post )
Comments (41)
Hello, after installing your plugin and running stream diffusion, the application crashes. Do you know what could be the problem? The specific error message displayed right before the crash is as follows:UnicodeDecodeError: 'locale' codec can't decode byte 0xeFin position 0: decoding error The above exception was the direct cause of the foll winexception: Traceback (most recent call last): File "D: StreamDiffusion-main/odi-stream/main_ndi.py lin 505, in fire. Fire(main) File "D:\'StreamDiffusion-main/venw/lib/site-packages/fire/come.py line 143,inFire File "D:\StreamDiffusion main\'veny\'iib\'site packagss\fire/core.py", linعلى الما component_trace = Fire(component, args, parsed_flag_args, context name component, remaining args = CallAndUpdateTrace C File "D: StreamDiffusion-main\'venv\'lib\'site-packages\fire\core.py", line 693, in CallAndəpdateTrace component = fn(*varargs, **kwargs) File_"D:/StreamDiffusiçn-main/ndi-stream/main_ndi.pv",_line 410, in maiin provide ridi sender name = select ndi source (args.sendear) # Pass args.sender to the function File "D: StreamDiffusion-main/ndiastream/ndi_spout_utils.py", 1ine 221, in select_ndi_source if sender_name. TypeError: Unable to lower() in source.ndi_nams.lower (): convert function return value to a Python type! The signaturewas (argo: NDI1ib.NDI1ib. Source) -> str
i dont see anything in my settings tab in the streamdiffusion window. why could this be?
This is amazing loving it so far, just a question, I can't seem to get any SDXL and SDXL-Turbo models running from HuggingFace, only SD-Turbo Is there a way to use my own models locally with ComfyUI to do the StreamDiffusioning?
Love the TOX! It is still crazy to me, that image generation works in almost real-time on a laptop :o Is there a way to link the timeline-speed to the FPS of the SD Operator? (For getting a smooth video rendered).
Hi ! I'm enjoying this very much ! Is there a way to put weight on words like in ComfyUI ? The (words:1.5) formula dosen't do anything, brackets either.
I do not believe so as far as I know. That would be super helpful.
great work. running good but i would love to use the acceleration features. I think the venv istallas cuda 11.6 but xformers need 12. But what do i know :) ANyway, great work, keep it up!
as far as i know xformers doesnt help in accelerating inferences, perhaps you should have a look to a proper update of cuda to 11.8 inside venv alonside compliant Cudnn version to get tensorRT working as an accelerator (i m pretty new to all this and still trying too so it's just a suggest to help nothing more ^^) hope it ll help btw
hello. the base StreamDiffusionTD has nothing in most of the tabs, nothing in setup. only one button is in custom for download/update and nothing happens when clicking it. the base is signaled with several errors. what should i do?
hello I have this problem with my memory and model too. How can I allocate the memory and how can I download the model from hugging face? Should I download just the safetensors file and put in model? Traceback (most recent call last): File "C:\Users\matte\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 417, in _load_model pipe: StableDiffusionPipeline = StableDiffusionPipeline.from_pretrained( File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 864, in to module.to(device, dtype) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1160, in to return self._apply(convert) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) [Previous line repeated 7 more times] File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply param_applied = fn(param) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacty of 2.00 GiB of which 0 bytes is free. Of the allocated memory 1.70 GiB is allocated by PyTorch, and 53.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Model load has failed. Doesn't exist.
Hey . If your gpu has only 2 gigs of vram, that won't enough to load the model into memory unfortunately. minimum of 6 is recommended.
Hi, has anyone used the model stabilityai/sdxl-turbo? I'm getting error messages, and streaming is unsuccessful
sdxlturbo is not supported yet. That is something that the developers of StreamDiffusion have mentioned they plan to do, but I am unsure on the exact timeline.
HI, I can't log on to discord ... keeps saying my mails are already used. So that is a deadend for me. I keep running in to this error that I don't understand and the stream never starts : ==================================== Initializing NDI Streaming... Sender: 'StreamDiffusion_NDI_out' ==================================== Starting OSC server on 127.0.0.1:4567 `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden. `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden. `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden. C:\Users\******\streamdiff\StreamDiffusion\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
I don't see any errors ! just warnings. if it wasn't working, save/close all TD and cmd windows and reopen the TD project and try again !
Hi there! I've got this error while running setup.py install for ndi-python ... error error: subprocess-exited-with-error × Running setup.py install for ndi-python did not run successfully. │ exit code: 1 ╰─> [12 lines of output] running install C:\Users\User\Documents\StreamDiffusion\venv\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build\lib.win-amd64-cpython-311 creating build\lib.win-amd64-cpython-311\NDIlib copying NDIlib\__init__.py -> build\lib.win-amd64-cpython-311\NDIlib running build_ext cmake .. -DCMAKE_INSTALL_PREFIX=C:\Users\jleung\AppData\Local\Temp\pip-install-l4jeb7w6\ndi-python_918fa6417cb640af864d665da90d8060\build\install error: command 'cmake' failed: None [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> ndi-python note: This is an issue with the package mentioned above, not pip.
I've actually got streamdiffusion running locally with conda before so not too sure why it isn't working in TD through venv?
Great TOX!!! Anyway, do you know any Lora which works with this TOX? I download and attach some Lora, but nothing happen. I am not sure Lora file is collapsed or I have a wrong operation.
Both the Lora and the SD model have to both be 1.5 based and either full local file paths to the file or a huggingfaceid to download them.
Absolutely insane TOX. Thank you so much for piecing this together. Was up and running in like 10 minutes producing insane results at 17FPS. Pro tip - I started using Nvidia upscaler TOP downstream of streamdiffusion and the results are really crisp without sacrificing my streamdiffusion FPS. Again, absolutely insane TOX. You're the man!
thanks Amir! thats awesome haha. 10 minutes !!
no nothing! Also NDI-sdk is installed
ah. confused myself with your error. If it is saying there are no ndi device / source, perhaps try restarting or dragging in a new operator. And also make sure something is plugged in, although as far as I know , it should work with no input. Basically, the issue is that your ndi from Touchdesigner is not cooking/active for some reason.
Hi guys! Thanks @DotSimulate for your work! I still had an issue: after the settings and then clicking on pulse to start stream my cmd don't find any NDI device. In the Setup tab I did pulse on the Install NDI/Spout req only. How can I fix it?
When you run the install NDI/Spout, are there any errors or red text in the cmd window? Also make sure to NDI-sdk installed.
And on another computer I keep getting errors about python not found although I installed the latest and referenced in Path in TouchDesigner.
If you are able to shoot a screenshot over to me on discord, that is easiest for me or someone to help with !
Why are most of the generated pictures soft focus? compared to let's say ComfyUI or Automatic1111
I thought I responded to this ! not sure where that message disappeared to. Its mostly about the VAE that is running which is https://huggingface.co/madebyollin/taesd When this encodes and decodes the images a lot of details are lost. There are definitely other factors and it improves with more steps added.
Is there a way to slow it down or reduce the sensitivity so it's more consistent over time?
Adjusting the step sliders can help as well as smoothing out / simplifying the input. The model doesn't have any temporal understanding which makes it tricky and very prone to flicker.
Hi, I'm having this issue and I haven't been able to run it. This is the error on the CMD: D:\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\diffusers\configuration_utils.py:135: FutureWarning: Accessing config attribute `requires_safety_checker` directly via 'StableDiffusionPipeline' object attribute is deprecated. Please access 'requires_safety_checker' over 'StableDiffusionPipeline's config object instead, e.g. 'scheduler.config.requires_safety_checker'. deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False) Process Process-4: Traceback (most recent call last): File "D:\StreamDiffusion\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 417, in _load_model etc, etc finishes with NotImplementedError: Cannot copy out of meta tensor; no data! what should I do? thanks!🙏
Hey. I believe that this is an issue with the model path you are providing. There are a few others that have run into this as well and there might be info in troubleshoot-central help forum thing on my discord server. Feel free to post in there or shoot me a message over there if possible. Patreon works as well, but doesn't support screenshots so its a bit hard.
This is so freaking awesome! But... I think i found a little bug. I saved and closed my project while StreamDiffusion was still running. When I opened it back up this morning, the operator still thought SD was running and I couldn't start it from the operator.
Oh, amazing ! I can fix that in next version. For now, in the Setup page, you can manually turn Off the Streamactive toggle at any point if it thinks it is on when it is not.
Is it possible to support SDXL-turbo in addition to SD-turbo?
I tried different models and seems it only works with the sd_turbo based on 1.5. And if you get it from the modelID, otherwise says "no data"
Hi. Thank you for this! However, I have some questions: I have an A5000 with 24gb, and a Xeon 8-core 3.5Ghz. I noticed that the VRAM used is only 5gb VRAM, and the CPU is maxing at 80%. I cannot install xfromers nor tensorrt for some reason ( i guess i have to install them in the env?. Seems as if the system is only using the CPU somehow. I can get only 10fps at 512x512. Do you have any tips on how to use the full VRAM? Thanks again
Due to the way StreamDiffusion offloads the VAE to the CPU, it tends to be quite high on CPU usage. VRAM usage remains to be pretty low, with the distilled diffusion model taking only a few gigs of the gpu. I do believe this is an opportunity for optimization.
Would this also work on Mac M1? I am not able to getting up and running after looking at you tutorial... great project though, would love to try it out
currently only working on windows + nvidia graphics. Hopefully would like to have a non local option at some point, but need to figure out best way to do that.