StreamDiffusionTD 0.1.5 🖼️🚀
Solid update with a few more additions for working with models and a few bug fixes on installation !
If it was working for you before, there is no need to reinstall/upgrade the streamdiffusion repo. Load in the new tox and switch out the Basefolder, models, etc and you are all set.
Changes:
Installation Guide ! New 'View Instructions' pulse in Install page will open helpful instructions !!! check it out !!
'Lora' parameter page is renamed to 'Models' + more info in headers to make it clearer
- Moved 'Use local LCM/VAE models' pulse to the 'Models' page
Added Hugging Face Model Downloader
- downloads SD models and LoRA models from Huggingface. Warning : will block the TD process while downloading. which may take a minute for main SD models (only needs to be done once)
- Check model info pulse
Added a LoRA Dropdown for .safetensors files in 'StreamDiffusion/models/LoRA' folder. When model is selected, it will set that as the LoRA File Path in that block.
- LoRA .safetensors files will appear on the Lora Folder menu if they are moved to that folder (including subdirectories).
- Full file paths still work like previously in case model exists in other disk location
Better handling of 1.5 models using local .safetensors
- Any LCM / VAE or Main SD model used from huggingface will use the folder as the parameter.
- For LoRA's or main SD models using .safetensors files you must use the full file path to the .safetensors file.
btw - Errors turn operator color red now and warnings turn operator yellow. Check the logs (dat out4) or the Log Message at top of the Stream Settings pages for more info
Enjoy !
Comments (110)
Hey there, im getting also an error: Current directory: C:\Users\lumaia86\Documents\Touchdesigner\TEST2DOTSTIMULATE\StreamDiffusion Changed directory to: C:\Users\lumaia86\Documents\Touchdesigner\TEST2DOTSTIMULATE\StreamDiffusion Virtual environment already exists at: "C:/Users/lumaia86/Documents/Touchdesigner/TEST2DOTSTIMULATE/StreamDiffusion\venv" Attempting to activate virtual environment... El sistema no puede encontrar la ruta especificada. Failed to activate virtual environment. Please check the path and ensure the venv exists. Path to venv: "C:/Users/lumaia86/Documents/Touchdesigner/TEST2DOTSTIMULATE/StreamDiffusion\venv" VIRTUAL_ENV: "" Presione una tecla para continuar . . . Installing 'wheel' to ensure successful building of packages... C:\python-3.10.11-embed-amd64\python.exe: No module named pip Installing nvidia-pyindex to ensure access to NVIDIA-specific packages... C:\python-3.10.11-embed-amd64\python.exe: No module named pip Installing dependencies with pip from the activated virtual environment... C:\python-3.10.11-embed-amd64\python.exe: No module named pip C:\python-3.10.11-embed-amd64\python.exe: No module named pip C:\python-3.10.11-embed-amd64\python.exe: No module named pip Traceback (most recent call last): File "C:\Users\lumaia86\Documents\Touchdesigner\TEST2DOTSTIMULATE\StreamDiffusion\setup.py", line 4, in from setuptools import find_packages, setup ModuleNotFoundError: No module named 'setuptools' C:\python-3.10.11-embed-amd64\python.exe: No module named pip Installation Finished Presione una tecla para continuar . . .
Hey man, I've tried to get this working for a couple days, and no luck. When I get to "Install SD (venv + all req)", I get this error: × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [4 lines of output] Traceback (most recent call last): File "E:\StreamDiffusion\venv\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 15, in import json ModuleNotFoundError: No module named 'json' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.
Hi, I'm new to stream diffusion in TD and have several errors. the biggest issue is that the parameters are not shown on the parameters tab. I can only see "Download/Update SD" on the custom tab. I've downloaded the stable diffusion from github but in venv. should I reinstall it global?
Hey. I believe it may be because of the sequence parameters/ version mismatch. Make sure to be on TD version 2023 or later. 10k version or higher should show the parameters.
Hi, everything is working fine, but how can I change the resolution from 512x512 to another with tnsorrt acceleration? it gives an error (
currently the base resolution is the only thing ive managed to get working with tensorrt. I have been looking deeper into the tensorrt code recently, but current implementation is locked @ 512x512
Hi! I'm finding myself unable to run it after rebooting TD. I get a "base folder parameter must be set" error, despite the base folder parameter being set precisely to where it was last time. I usually have to run a full reinstall to get it working, which is far from ideal. To make matters worse, the download/update SD pulse is working very inconsistently all of a sudden, sometimes taking multiple minutes to work, and freezing TD while doing so. Any help or advice appreciated! Thanks so much in advance.
what are you setting the Basefolder parameter as? You shouldn't need to download/update SD ever after downloading the first time. thats an odd error. is a bit easier to troubleshoot via discord with screenshots or screen recording
Hi! I've tried your suggestion to plugged noise to the SD operator but its still not solving the problem, the output is still blank. how to solve this problem? or can i get an invite to discord pls? Thank You!
Thank you!
I'm trying to start stream but i get this on my CMD: "==================================== WARNING: NDI source containing 'StreamDiffusion-TD' not found. ==================================== ==================================== Available NDI sources: ==================================== Enter the number of the NDI source you want to use:" What should i do?
I am guessing you need to install ndi sdk . Will also need to restart after installation in order for the sources to be seen.
Hi! I've been following the instruction, but somehow i can't generate any image and only blank that showed up, how can I solve this problem? Thank you!
If you are in img2img mode (located in stream settings 2 page), then you will need to have something plugged into the operator and the project playing in order to see images.
Traceback (most recent call last): File "E:\SDIFFUSION\StreamDiffusion-main\streamdiffusionTD\main_ndi.py", line 7, in import NDIlib as ndi ModuleNotFoundError: No module named 'NDIlib' How should I solve this problem~~?
Make sure to install via the operator Install page. There are a few reqs that aren't in the main repo. also download / install ndi if you don't have it. will need to restart after installing for it to be recognized - https://ndi.video/download-ndi-sdk/
Hi, I had some failures. In my Stream_Settings page, the Log Message kept showing [started with ndi input using CMD], and an NDI QR code was output. If I click the Start Stream button, it brings up my CMD, but nothing is running in the CMD and it closes after a second. I don't understand what's going on.
there is a debug toggle in stream settings 2. Try turning that On and starting the stream again. My guess is that the install isn't fully setup. It is likely not finding a module or having trouble importing the external scripts. Check the install guide in the Install page for the correct requirements.
Error: Basefolder parameter must be set to '/StreamDiffusion'. 'path/to/StreamDiffusion/venv' not found. How can I fix that?
You will want to sett he Basefolder parameter to be the full folder path to the repo folder on disk. if it is saying the venv is not found, then either the venv doesn't exist or you don't have the folder correctly linked.
Hi, I'm trying to run two StreamDiffusionTD in a project, but I get the error: Unable to Create NDI Sender for Specified Name. The cause lies with the NDIOut1 Top. What should I change to resolve this error? Thanks in advance for your response!
Hey, I was able to install everything correctly, but when I hit Pulse on the stream, this window pops up for half a second and then disappears, without generating any image: TRACEBACK (MOST RECENT CALL LAST): File: "C/users/fede/Documents/Stream Difussion/StreamDiffuionTD/main:ndi.py, line 16, in from torchvision.transforms.functional import to_tensor ModuleNotFoundError: No module named `torchvision` Please help, I've been trying to install everything and make it work for days but there's always some kind of error popping up!
Hi, I'm trying to run two StreamDiffusionTD in a project, but I get the error: Unable to Create NDI Sender for Specified Name. The cause lies with the NDIOut1 Top. What should I change to resolve this error? Thanks in advance for your response!
It likely means you have another version of the operator open in a different project. You can also change the 'Stream Out Name (TO SD)' so that it is unique
hi, I would like to know where is the downloaded checkpoint files located? thanks
Any chance SDXS will be usable in Streamdiffusion? https://huggingface.co/IDKiro/sdxs-512-0.9
yep! We've got it working in the discord. also sdxl ! someone got 45 fps with SDXS
could not bulid wheels for nvidia-pyindex error
I installed it yesterday and it worked, but it doesn't work today because I called it in. Why is that?
what version TD do I need to run in order to make this work? I currently have updates in pro to 2022.33910. I ask because I see no messages on the 'Install' tab
It does use some new parameter types from production 2023 official release. I believe when opening it in 2022, several parameters do not appear like you are seeing.
Hi I want to use StreamDiffusion for video art - it's a beautiful tool - and would like to set a large resolution. I have not installed TensorRT due to the 512 x 512 resolution limitation of the model. But with Width & Height set on 1024 in Stream Settings 1, the Stream is continuously stopping and starting. How can I solve this problem?
There is a timer called timer_osc inside that checks if the stream is active. If you change the length of that timer, it shouldn't start/stop like that. Planning/have planned to expose this in the parameter menu somewhere
hello! i have downloaded your version of stream diffusion 0.1.5 and the installation of venv was succeded, but after i clicked "start stream" i have got this error "A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton' Config file path: C:\StreamDiffusionTD\StreamDiffusion\streamdiffusionTD\stream_config.json ==================================== StreamDiffusion: Starting the NDI capture and transmit script... ==================================== Model ID or Path: stabilityai/sd-turbo LCM LoRA ID: None VAE ID: None ==================================== ==================================== WARNING: NDI source containing 'StreamDiffusion-TD' not found. ==================================== ==================================== Available NDI sources: ==================================== Enter the number of the NDI source you want to use: " How can i fix this?
Same problem for me!
Hey! Is there some work around for getting this to work on apple silicon?
this is nvidia exlucive.
After clicking Start Stream it shows: Traceback (most recent call last): File "D:\StreamDiffusion-main\streamdiffusionTD\streamdiffusionTD\main_ndi.py", line 20, in from utils.viewer import receive_images ModuleNotFoundError: No module named 'utils' I've installed utils and it didn't work......
Hi! Thanks for your work! I tried SD-Turbo and Deliberate_v5 and both show the same 8 fps. Shouldn't SD-Turbo works faster than a normal 1.5 model? Maybe I'm doing something wrong
The 1.5 models automatically load the lcm lora that allows them to also run much faster. I get a few fps faster with sd-turbo, but 1/5 models are running fast ! also 1.5 models have a better license for usage that sd-turbo
It took me over 10 hours to get it to run successfully, what should I do? I don't know what the problem is now and I'm anxious and confused.
after I deployed streamdiffusion to TD, after I clicked on start stream Pulse, it still doesn't work and shows not streaming on stream info! After I've linked noise to streamdiffusionTD and I've turned on "stream active", now in the "streamdiffusionTD" box it shows something like this Anything is the same as the initialization screen in "noise".
The toggle for stream active should turn on / off automatically based on whether or not the model is streaming. Patreon doesn't make it easy to share screenshots but if you can shoot a direct message with more info, happy to help.
Hi, after I deployed streamdiffusion to TD, after I clicked on start stream Pulse, it still doesn't work and shows not streaming on stream info! After I've linked noise to streamdiffusionTD and I've turned on "stream active", now in the "streamdiffusionTD" box it shows something like this Anything is the same as the initialization screen in "noise".
Hi! Is it possible to generate with transparent background? I’m using webcam and threshold, but final generation goes with black background.
try Nvidia Background
Hi There! It's been working great on a single GPU, but I wanted to know if having multiple GPUs (like two 4090) would increase the performance, and can we integrate huggingface Accelerate (https://huggingface.co/docs/transformers/accelerate)?
How do I get NDI to start?Log Message Started with ndi input using CMD.
same thing happened to me! and the command window exited immediately....Have you fix it? :/
Hi there (: Everything worked well but the stream doesn`t start. Loading pipelines components 100% and then "Preapring Stream . . ." shows up but nothing happens, no fps count, no active stream..
make sure there is something plugged into the TD operator and that the TD project is playing actively. so that frames are being sent !
Hi ! What’s your discord server ?
sent dm !
any idea why when i plug the noise top and play around with its paramaters nothing happens ie nothing changes in the Stream Diffusion Operator ????!!!
im having some troubles trying to uso another sd model id, can you help me with this ?
Thank you for sharing. I would like to ask, there are negative prompt in youtube videos, but there are no negative prompt in 0.1.5? Can you answer it for me?plz.
Thanks for sharing this. is it possible to use other turbo check points like Lykon/dreamshaper-xl-v2-turbo ? what would be amazing .
is there a manual for StreamDiffusionTD?
There is an install guide in the Install Page. I plan to have more documentation on next version that I release.
Hi, thanks for creating the awesome tool! However, I am not able to run this in my PC due to this error "ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host" after NDI and OSC server starts and the program is trying to get model from Hugging face. I have turned off all firewalls or proxy settings on my PC but this issue persists. Do you have an idea of why it's happening? Also it would be great if you can invite me to Discord so that I can paste the full log there. Thanks in advance for your help!
https://discord.gg/mptfw6bD
Hi ~!Unfortunately after installing everything and press "Start stream" a cmd window appears and then goes away without giving any error message. Stream Info says "Not Streaming". Any idea of what could be the problem? I've also tried with the first version and I have the same problem... Thanks for your help and work!
In the Stream Settings 2 page, there is a toggle which you can turn on that should show the error message. it is likely something about a python library module not being found. theres a forum of troubleshooting questions on the discord if you are able to join. Or you can copy and paste the error to me in a direct message. I can check tomorrow.
Hey DotSimulate. Thanks for this awesome tool. Is there a way I could run more than one instance by specifying a second cuda device for the 2nd instance?
Hi. why can't we have the negative prompt?
Hello, when I press the Start Pulse I get " warning: NDI source containing 'StreamDiffusion-TD' not found." any ideas where I am going wrong ? I did a fresh install of this (CUDA 11.8)(Python 3.10.11), downloaded and set the base repo folder to my Local StreamDiffusion folder, installed the venv + require,etc. any help appreciated, many Thanks, Chris
Same issue! Asks me "Enter the number of the NDI source you want to use:" and no matter what I put there it closes CMD window and nothing happens. No stream
Thanks for uploading this! Could you help me？Traceback (most recent call last): File "G:\StreamDiffusion-main\streamdiffusionTD\main_ndi.py", line 6, in import cv2 ModuleNotFoundError: No module named 'cv2'
Hi, Im wondering how to use LORA Model in this pulgin? It doesn't work when I open the Use Lora button and add the path, I dont know what I did wrong... Could you please make a tutorial? big thanks!!
Lora doesn't work with sd-turbo currently due to how current model loading code skips the lora loading with sd-turbo specifically. But if using a 1.5 model, you can load a 1.5 lora.
Thanks for the informative video! Is there a way to run vid2vid?
Hello can I please get an invite to the discord.
hi can you invite me to discord？big thanks
this one should work for next 7 days ! https://discord.gg/9RVBZxDD
same problem as Andrea above... getting printout that loras are loaded but nothing works
if using sd-turbo, the model loading process skips the lora currently. This can be changed by unindenting a few lines in utils/wrapper.py so it doesn't skip lora_dict for sd-turbo models.
Hi, have you tested which LCM model runs faster after accessing it, and do you have a download link you can provide?
if using 1.5 model, then the LCM lora is automatically downloaded and used, which allows any 1.5 model to be LCM model. I haven't noticed any are particularly faster.
Hi! thank you it's amazing! Unfortunately I can't make the Loras work, don't understand why... they look like they're loaded on startup. Do have any insight? I downloaded them from civitai, they're all sd 1.5...
Does it print out the paths to them in the command window when you start the stream? I added some better printouts and there should be a line before 'preparing stream...' that shows if Lora is Loaded or not
Hey Lyell. This looks amazing. I don't see anything about Mac in the in install guide - is there a working version for Silicon ?
Not currently. The best option for that would be non local with runpod/docker. Hopefully maybe something I test this week? Theres been a few discussions on the discord about it
Hi! Thanks for your great work. Could you help me, when I specify another model locally in the 'SD Model / Path' property I got the next error on starting stream: "OSError: Cannot load model Models/sd_turbo.safetensors: model is not cached locally and an error occured while trying to fetch metadata from the Hub.".
or like this: "ValueError: The provided pretrained_model_name_or_path "StreamDiffusion/models/Model/sd_turbo.safetensors" is neither a valid local path nor a valid repo id. Please check the parameter."
Do you have some faq for installation troubleshooting?
discord ! but seems like you found the invite !
Hi, I'm having big problems installing it. Downloading it goes ok, but Install SD options brings up quite a big load of errors. I have read the guide and installed all the prerequisites. The problem starts with installing wheel, quite early Installing 'wheel' to ensure successful building of packages... Traceback (most recent call last): File "C:\Users\lukas\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\lukas\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\lukas\Dropbox\cloud_lab\TouchDesigner\DotSimulate plugins\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\pip\__main__.py", line 29, in from pip._internal.cli.main import main as _main File "C:\Users\lukas\Dropbox\cloud_lab\TouchDesigner\DotSimulate plugins\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\pip\_internal\__init__.py", line 3, in import pip._internal.utils.inject_securetransport # noqa ModuleNotFoundError: No module named 'pip._internal.utils' Installing nvidia-pyindex to ensure access to NVIDIA-specific packages...
has anyone ran into an issue where the next prompt after the 1rst one wont change and that you cant turn off your stream? Ive been having this issue with the 2 last releases
I answered in discord, but in case anyone hits this, then you like have a second operator or touchdesigner project or something using the osc port coming out of the operator. To fix: change the osc port numbers in the settings 2 page to be different close the cmd window manually then start stream again Also might get an error on the op 'Unable to create NDI sender name' which is similar and can be solved by changing the Stream Out Name in settings 2 page.
is there a way to adjust the resolution?
Yes, but will need to stop stream and start again at new resolution due to how the model is loaded and run. also, if using TensorRT, only 512x512 is supported.
Is there a way to use XL models?
Not currently with StreamDiffusion. That is on my list to look at.
Found there's a error while installing tensorRT, but it work in 0.1.4.....ut showned ERROR: Failed building wheel for tensorrt
Apologies if I didn't make this clearer. but there is no reason to installing again between operator versions unless I say so in the release notes. You just want to change the Basefolder in the new operator in order to link to your local installation and you are all set to start using it.
Looking forward to giving this a shot, doing a clean install. Super curious about the Hugging Face models, anybody have suggestions about their favorites?
Just in case, if StreamDiffusion was working previously, there is no changes on the python requirements during this update. it is just an operator upgrade. You shouldn't have to install more than once per computer.
First time upgrading tox, what exactly do I have to do? 0.1.4 is working great. Do I have to just replace old for new tox in .toe file?
Yep. unless I specifically note that there is new requirements, upgrading the operator should just be switching out the Basefolder parameter and any other settings from your setup. The Install pulses are for first setting up the repo. Once things are working, that installation will work for future updates.
Stuck on this forever while intalling Tesnorrt any help? Building wheels for collected packages: tensorrt Building wheel for tensorrt (pyproject.toml)
same
no error but still in "preparin stream..." for hours if img2img is on. In txt2img mode the stream starts without output
I have the same problem, did you fix it?
thnks man ;) the installation process now is ok but in img2img mode still not working, also in text2img the stream is active but i don t see "banana" output. My machine is windows 11, rtx 4070 and cuda 12.1. I m not sure if cuda is added to path
Are you getting an error with img2img mode? that should be something you can change back and forth on the fly. The step sliders determine the denoise. Higher step values will resemble the input image.
how do I view the entire installation instructions? I see how_to_install but it's an entirely baked bitmap... thanks!
Also having an issue reading the page after clicking 'pulse'.
Thank you!!
Thank you for this awesome tool
THANK YOU!