tal

Thanks for trying it out! Both the inpainting and outpainting -- the expansion -- worked better than I'd expected, though I dunno if that's exactly what M0oP0o's after.

Lots of odd artifacting, slow creation time and yes it had some issues with sailormoon.

It probably isn't worth the effort for most things, but one option might also be -- and I'm not saying that this will work well, but a thought -- using both. That is, if Bing Image Creator can generate images with content that you want but gets some details wrong and can't do inpainting, but Midjourney can do inpainting, it might be possible to take a Bing-generated image that's 90% of what you want and then inpaint the particular detail at issue using Midjourney. The inpainting will use the surrounding image as an input, so it should tend to try to generate similar image.

I'd guess that the problem is that an image generated with one model probably isn't going to be terribly stable in another model -- like, it probably won't converge on exactly the same thing -- but it might be that surrounding content is enough to hint it to do the right thing, if there's enough of that context.

I mean, that's basically -- for a limited case -- how AI upscaling works. It gets an image that the model didn't generate, and then it tries to generate a new image, albeit with only slight "pressure" to modify rather than retain the existing image.

It might produce total garbage, too, but might be worth an experiment.

What I'd probably try to do if I were doing this locally is to feed my starting image into the thing to generate prompt terms that my local model can use to generate a similar-looking image, and include those when doing inpainting, since those prompt terms will be adapted to trying to create a reasonably-similar image using the different model. On Automatic1111, there's an extension called Clip Interrogator that can do this ("image to text").

Searching online, it looks like Midjourney has similar functionality, the /describe command.

https://docs.midjourney.com/docs/describe

It's not magic -- I mean, end of the day, the model can only do what it's been trained on -- but I've found that to be helpful locally, since I'd bet that Bing and Midjourney expect different prompt terms for a given image.

Oh I also tried local generation (forgot the name) and wooooow is my local PC bad at pictures (clearly can’t be my lack of ability it setting it up).

Hmm. Well, that I've done. Like, was the problem that it was slow? I can believe it, but just as a sanity check, if you run on a CPU, pretty much everything is mind-bogglingly slow. Do you know if you were running it on a GPU, and if so, how much VRAM it has? And what you were using (like, Stable Diffusion 1.5, Stable Diffusion XL, Flux, etc?)

@M0oP0o@mander.xyz, as far as I can tell, you always use Bing Image Creator.

And as far as I can tell, @Thelsim@sh.itjust.works always uses Midjourney.

I don't use either. But as far as I know, neither service currently charges for generation of images. I don't know if there's some sort of different rate-limit that favors one over the other, or another reason to use Bing (perhaps Midjourney's model is intentionally not trained on Sailor Moon?), but I do believe that Midjourney can do a few things that Bing doesn't.

One of those is inpainting. Inpainting, for those who haven't used it, lets one start with an existing image, create a mask that specifies that only part of the image should be regenerated, and then regenerate that part of the image using a specified prompt (which might differ from the prompt used to generate the image as a whole). I know that Thelsim's used this feature before with Midjourney, because she once used it to update an image with some sort of poison witch image with hands over a green glowing pot, so I'm pretty sure that it's available to Midjourney general users.

I know that you recently expressed frustration with Bing's Image Creator's current functionality, wanted more.

Inpainting's time-consuming, but it can let a lot of images be rescued, rather than having to just re-reroll the whole image. Have you tried using Midjourney? Was there anything there that you found made it not acceptable?

it's just ComfyUI behaving this way. Nothing is loaded, no LLMs

Oh, okay, then I'm probably wrong on VRAM, then. On my system, it needs to actually run the nodes before the VRAM gets allocated. Sorry! I thought I had it...

Just realized that I was dyslexic with the ordering directive, which worked well with Flux apparently not understanding it, since I wound up with the desired ordering; I'd intended to write "left to right".

Not yet! One thing that AI generated images right now are not so good at is maintaining a consistent portrayal of a character from image to image, which is something you want for illustrating a story.

You might be able to do something like that with a 3d modeler to pose characters, generate a wireframe, and then feed that wireframe into ControlNet. Or if you have a huge corpus of existing images of a particular character portrayed in a particular way, you could maybe create new images with them in new situations. But without that, it's hard to go from a text description to many images portrayed in a consistent way. For one image, it works, and for some things, that's fine. But you'd have a hard time doing, say, a graphic novel that way.

I suspect that doing something like that is going to require having models that are actually working with 3D internal representations of the world, rather than 2D, at a bare minimum.

it starts flipping frames between the nodes and a different set of nodes.

Yeah, I don't know what would cause that. I use it in Firefox.

Maybe try opening it in Chromium or a private window to disable addons (if you have your Firefox install set up not to run addons in private windows?)

I'm still suspicious of resource consumption, either RAM or VRAM. I don't see another reason that you'd suddenly smack into problems when running ComfyUI.

I'm currently running ComfyUI and Firefox and some relatively-light other stuff, and I'm at 23GB RAM used (by processes, not disk caching), so I wouldn't expect that you'd be running into trouble on memory unless you've got some other hefty stuff going on. I run it on a 128GB RAM, 128GB paging NVMe machine, so I've got headroom, but I don't think that you'd need more than what you're running if you're generating stuff on the order of what I am.

goes investigating

Hmm. Currently all of my video memory (24GB) is being used, but I'm assuming that that's because Wayland is caching data or something there. I'm pretty sure that I remember having a lot of free VRAM at some point, though maybe that was in X.

considers

Let me kill off ComfyUI and see how much that frees up. Operating on the assumption that nothing immediately re-grabs the memory, that'd presumably give a ballpark for VRAM consumption.

tries

Hmm. That went down to 1GB for non-ComfyUI stuff like Wayland, so ComfyUI was eating all of that.

I don't know. Maybe it caches something.

experiments further

About 17GB (this number and others not excluding the 1GB for other stuff) while running, down to 15GB after the pass is complete. That was for a 1280x720 image, and I was loading the SwinIR upscaler; while not used, it might be resident in VRAM.

goes to set up a workflow without the upscaler to generate a 512x512 image

Hmm. 21GB while running. I'd guess that ComfyUI might be doing something to try to make use of all free VRAM, like, do more parallel processing.

Lemme try with a Stable Diffusion-based model (Realmixxl) instead of the Flux-based Newreality.

tries

About 10GB. Hmm.

kagis

https://old.reddit.com/r/comfyui/comments/1adhqgy/how_to_run_comfyui_with_mid_vram/

It sounds like ComfyUI also supports the --midvram and --lowvram flags, but that it's supposed to automatically select something reasonable based on your system. I dunno, haven't played with that myself.

tries --lowvram

I peak at about 14GB for ComfyUI at 512x512, was 13GB for most of generation.

tries 1280x720

Up to 15.7GB, down to 13.9GB after generation. No upscaling, just Newreality.

Hmm. So, based on that testing, I wouldn't be incredibly surprised if you might be exhausting your VRAM if you're running Flux on a GPU with 12GB. I'm guessing that it might be running dry on cards below 16GB (keeping in mind that it looks like other stuff is consuming about 1GB for me). I don't think I have a way to simulate running the card with less VRAM than it physically has to see what happens.

Keep in mind that I have no idea what kind of memory management is going on here. It could be that pytorch purges stuff if it's running low and doesn't actually need that much, so these numbers are too conservative. Or it could be that you really do need that much.

Here's a workflow (it generates a landscape painting, something I did a while back) using a Stable Diffusion XL-based model, Realmixxl (note: model and webpage includes NSFW content), which ran with what looked like maximum VRAM usage of about 10GB on my system using the attached workflow prompt/settings. You don't have to use Realmixxl, if you have another model, should be able to just choose that other one. But maybe try running it, see if those problems go away? Because if that works without issues, that'd make me suspicious that you're running dry on VRAM.

EDIT: Keep in mind that I'm not an expert on resource consumption on this, haven't read material about what requirements are, and there may be good material out there covering it. This is my ad-hoc, five-minutes-or-so-of testing; my own solution was mostly to just throw hardware at the problem, so I haven't spent a lot of time optimizing workflows for VRAM consumption.

EDIT2: Some of the systems (Automatic1111 I know, dunno about ComfyUI) are also capable, IIRC, of running at reduced precision, which can reduce VRAM usage on some GPUs (though it will affect the output slightly, won't perfectly reproduce a workflow), so I'm not saying that the numbers I give are hard lower limits; might be possible to configure a system to operate with less VRAM in some other ways. Like I said, I haven't spent a lot of time trying to drive down ComfyUI VRAM usage.

Full Size

UI: ComfyUI

Model: STOIQNewrealityFLUXSD_F1DAlpha

A cute, adorable, loveable, happy cave spider.

The image is an illustration.

When a model is initially being loaded, I see slowdown, but once that has happened, I don't. I see that with Automatic1111 as well. Once it's been loaded, though, I don't get that. I regularly do (non-GPU-using) stuff on another workspace when rendering, can't detect any slowdown.

So I don't know what might be the cause. Maybe memory exhaustion? A system that's paging like mad might do that, I guess.

As to an alternative, it depends on what you want to do.

If you've never done local GPU-based image generation, then Automatic1111 is probably the most-widely-used UI (albeit the oldest).

If you want to run Flux and Flux-derived models -- which I'm using to generate my above image -- I believe I recall reading that while Automatic1111 cannot run them -- and maybe that's changed, have not been monitoring the situation -- the Forge UI can do so as well. But I've never used it, so I can't provide any real guidance as to setup.

kagis

Yeah, looks like Automatic1111 can't do Flux:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/16311

And it looks like Forge can indeed run Flux:

https://sandner.art/flux1-in-forge-ui-setup-guide-with-sdsdxl-tips/

If you're short of VRAM or RAM or something, though, I don't know if Forge will do better than ComfyUI. I think that I might at least try to diagnose what is causing the issue first, as there are some things that can be done to reduce resource usage, like generating images at lower resolution and relying more-heavily on tile-based upscaling. With at least some of the systems, haven't played around with ComfyUI here, there are also some command-line options to reduce VRAM usage in exchange for longer compute time, like --medvram or --lowvram in Automatic1111.

I don't think that there's a platform-agnostic way to see VRAM usage. I use a Radeon card on Linux, and there, the radeontop command will show VRAM usage. But I don't know what tools one would use in, say, Windows to look up the same numbers.

On Linux, top will show regular memory usage, can hit "M" to sort by memory usage. I'm pretty out of date in Windows or MacOS -- probably Task Manager or mmc on Windows and maybe top on MacOS as well? You may know better then me if you're accustomed to that platform.

I can maybe try to give some better suggestions if you can list any of the OS being used, what GPU you're running it on, and if you can, how much VRAM and RAM is on the system and if you can determine how much is being used.

despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that's installed system wide and thus can't install pytorch.

Can you paste your commands and output?

If you want, maybe on !imageai@sh.itjust.works, since I think that people seeing how to get Automatic1111 set up might help others.

I've set it up myself, and I don't mind taking a stab at getting it working, especially if it might help get others over the hump to a local Automatic1111 installation.

venv nonsense

I mean, the fact that it isn't more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I'd call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.

Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I'd be fine with it.

And I don't do much Python development, so this isn't from a "Python awesome" standpoint.

I called Hillary "Hillary", but that's to distinguish her from Bill Clinton, who I called "Clinton".

Honestly, you have to be a very commonly-used name before I'm going to use a single name for general purposes at all rather than a full name, so the set of people who have the chance to get into the "one name club" is very small.