Only if they actually start performing at the level of OpenAI’s models. I’m not a fan of StableDiffusion, but at least their models work at general parity with private offerings. All the LLama derivatives and OpenAssistant stuff performs far below GPT-3.5 for everything I’ve tested.
In my experience, the threshold to be useful is much lower than GPT-3.5. These smaller models can "easily" be finetuned to achieve a comparable performance on a specific task. For example, I've achieved promising results for data summarisiation and image captioning (BLIP2-based) using Alpaca.
Also, server/hardware costs are still a limiting factor for running and finetuning the larger 33/65B Llama models. Especially, if they can only be used for personal toy projects.
I don’t use LLMs for anything image related, so I can’t speak to their value there, but almost all simpler NLP tasks are IMO better handled using other techniques that predate them. I’ve yet to see an example where fine-tuning is cheaper/more efficient/better performing than older solutions to these problems.
If older techniques work for you, there is of course no reason to switch to LLMs besides general curiousity or to explore what's possible already. That said, in my case I was enable to generate much more engaging text summaries of tabular data using a Llama derivative.
Llama itself performs comparably to GPT3.5 (at least 30/60g models), but the RLHF of chatgtp is much better than what the community has produced thus far, and it's tuned to work well without tinkering. There will be open source models with that level of fine tuning in the near future, at which point ChatGPT4 will mainly be superior for stuff like code that needs the best possible cohesion and accuracy.
I don't think there is a ready made alternative to Midjourney.
Midjourney is way more versatile than SD. If you start getting some fine tuned models on civitai, trained to do well some specific tasks, you can get comparable quality but I haven't seen a single model which is able to replace Midjourney.
Llama is no different, it has ok performance on generic queries but still far away from GPT3.5: if you start fine-tuning you can get good perf on specific tasks.
SD with ControlNet and some other open source plugins is far more flexible than MidJourney. It just has all the typical hurdles of OSS vs. commercial offerings. Default image quality in Midjourney is better in terms of its pedestrian aesthetic biases, but it’s not very interesting as an actual artistic tool. And I say this as someone who doesn’t like either service and used to be a commercial illustrator before moving into Data Science.
You're 100 percent right. People will say control bla bla bla and that's certainly true. You can get a lot more control with Stable Diffusion but like 99% of digital comics created with ai art use midjourney. One of the most control and versatility inclined use cases of generated art and midjourney is still easily winning. There's a reason for that.
> and midjourney is still easily winning. There’s a reason for that.
Sure, Midjourney is a centralized commercial service with a clear statement that you (as a paid user) own the images you create. While that doesn’t resolve all potential copyright issues (as there are still at least theoretical issues with the underlying dataset), if you doing something commercial with it like, say, a webcomic from which you derive income, its a lot simpler than dealing with the SD ecosystem where the plethora of models also have different stated usage restrictions, different suppliers (many of which are hobbyists) to keep track of, and more potential avenues of indirect copyright risk, as well. For some webcomics, even the base CreativeML Open RAIL-M license itself might be problematic.
This isn’t a technical or quality advantage, but its definitely an advantage that would very often tip the balance between two tools if both are minimally adequate to your task.
No, technically it’s all very impressive. My displeasure with them was there doing a Napster-style maneuver to force artists into accepting AI art generation
LAION is a German company and what StableDiffusion is doing seems to be covered under UrhG § 44b. If artists don't want their work inspected by bots they have the option to put a robots.txt on their site.
While this may very well be covered, I think the general problem in meatspace is that there was no advance notice given to exercise the option to place the proper robots.txt directives to opt out of having one’s artwork collected for model training before it happened, while still preserving the ability to have one’s artwork findable by search engines and the like. I’m sure there are more than a handful of people who have no idea that a robots.txt file can be used to prevent AI data collection—and some may even be surprised to learn the file that’s been used for search engine crawlers is also going to double for AI crawlers.
To push a bit further, there’s something that just feels particularly off about assuming everyone’s content is up for grabs unless the producers do the work to opt out. I think there’s an especially palpable bit of irony looking at it from the EU’s perspective—where cookies must be opt-in, but grabbing all your copyrighted material so companies can do whatever they like with it places the burden on the owner to opt-out. It just feels backward. Perhaps one should have to expressly opt-in to allowing their work to be accessible as training data. At least then there will be a clear signal that the producer of the work can’t later complain, as they willingly granted permission.
I wonder if these authors would have complained so loudly if they had known that other artists were looking at their output to learn how to create their own work? Absolutely none of them sprung from the womb, tablet in hand, to create their work ex nihilo, based on nothing other than their own entirely original thoughts.
Their art wasn't collected for model training. The #1 artist supposedly being copied in SD1.5 is not in the training data. Artists just don't know how the model works and think you need to put in an "art" image to get an "art" image out, but of course that's not true.
> and think you need to put in an "art" image to get an "art" image out, but of course that's not
While you don't need that to generate an image, it's something SD can actual do extremely well with ControlNet, Textual Inversion, LoRA, img2img and so on.
That's an area where things are going to get interesting in the future, as you can take any image, feed it into SD and produce hundreds of AI images from it. Very easily, without much effort and within minutes. The delineating line between original work and derivative becomes extremely blurry here, as what you are copying is not "the image", but just concepts within the image, that can be a pose, camera angle, scene layout, art style or really anything. You can "copy" it with as much variation as you want, you can remix it with other images, text prompts and so on. Where does "looking at reference" stop and "doing a copyright violation" start?
The spooky part with AI art that it stops images from being singular entities, with AI you can explode every piece into millions of possible variations. AI is so fast at generating art that a future where we could generate movies in real time might not be far away. It's already fast enough to produce images and text stories faster than you can consume them. There might be a fundamental shift in art consumption ahead of us.
Hm. That to me seems to be quite a badly written law. Is a copyright notice written in plain German in the website footer 'machine-readable'? Is there some definition of 'machine-readable' somewhere?
It's also far from clear to me whether a court would find training an LLM to constitute text and data mining 'for the purpose of gathering information, in particular regarding patterns, trends and correlations'.
> That to me seems to be quite a badly written law.
It's pretty normal for a law to not be specific on the technicalities so they don't have to update the law whenever the software changes. The de facto standard to prevent bots from scraping your sites has been robots.txt for almost 30 years.
If artists didn't mind Google scraping their images, putting them on their site, adding ads and making billions, I really don't see them having much of a justification to call out StableDiffusion for "stealing" their stuff. In general artists would be in a lot of trouble if taking stuff from the Internet would be outlawed, as that's where they get all their reference images from too.
Either way, I am sure we'll see quite a few lawsuits going forward, laws are always open to interpretation, especially when new technology archives. But long term I really see copyright in general being in a lot of trouble, since derivatives and remixes are becoming completely trivial with AI. Where does the original work stop and the copyright violation starts is being rather difficult to decide when you can just wander around latent space and create literally thousands of similar images in minutes, with as much or as little variation as you want.
My argument is that although robots.txt is a machine-readable way of asserting reservation of use, it's not the only machine-readable way, and the law does not seem to place a burden on the rights-holder to choose a particular 'machine-readable format'.
While a court would likely conclude that a watermark on an image is not 'machine-readable' (I say likely—OCR technology would however make it possible that a court could find that a watermark is machine readable), I would say that because the law does not require a specific method, I think it might be found that a copyright notice in the footer, or in an image caption, is indeed 'machine-readable'.
On balance, I agree that there's a lot of things we are woefully underprepared for coming up in the very near future on using tools in this way to generate art. The answer is not simply to try and lock up all the art away from the robots—but I don't know what the answer actually is.
None of this voids the terms of international copyright agreements and someone on Hacker News should know better than to claim that a robots.txt on a personal site would cover all instances of an image being scraped. I’m not saying that artists will necessarily come out on the winning end of this battle, but it’s also specious to claim that company says what they did is legal, therefore it is.
"Wait, wait, stop, -- I said stop! -- it turns out that, despite the lack of any legal basis for their opinion in any known jurisdiction, user 'bugglebeetle' on Hacker News disapproves of this activity. Better fold up our tents, boys. It was fun while it lasted."
What I mean is releasing a free service out into the world that allows anyone to effectively pirate an artist’s work. Their intention was obviously to be rewarded by established players for doing this bit of dirty work, forcing artists to accept terms they wouldn’t have otherwise.
> not a fan of StableDiffusion, but at least their models work at general parity with private offerings
I think you're being a bit generous there. Either I'm using it seriously wrong or SD can only generate vague blobs while Midjourney can make some proper stuff. It's a larger difference than GPT 3.5 vs GPT 4.
> Either I’m using it seriously wrong or SD can only generate vague blobs
You are definitely using it wrong, if the alternative is “SD can only generate vague blobs”. Even the base SD models are much better than that (though, the strength of the SD ecosystem is the availability of custom checkpoints, hypernetworks, LORAs, embdeddings, ControlNet, etc., not just the base models.)
Went back to do some more tests now, and funny enough I can actually get it to make decent stuff after realizing that it just completely sucks at below 512px (I was initially running it at 128x256 to speed up generation). I guess I should stop listening to advice from morons on reddit who said that lower res + upscaling works fine. Lol.
Not sure why there's even an option to go below 512.
SD isn't comparable to Midjourney. 99% of comics created with ai art use midjourney. One of the most glaring need cases for control and still nothing. There's a reason for that.
I have seen really convincing comics made with SD, much more convincing than any comics made with MJ, and the reason is really obvious. Models and LoRAs on CivitAI and Huggingface are really good, and the fact that MJ can generate slightly better images does not justify the total lack of control.
Never said you couldn't make impressive stuff with SD but feel free to share those comics.
Models on CivitAI are okay. Cool if you're looking for a certain style and/or want to create something that looks like the training images but style isn't everything.
Midjourney generates much better than "slightly better images" and the very fact you say this just tells me you've not even used the thing in any real capacity.
But no, MJ is not much better if you know how to use SD, although if what you did with SD was just put a prompt in a huggingface space, I can understand why you say that.
>I never said you can't do impressive things with SD, but feel free to share these comics.
I am arguing that they are better than any comics made with MJ, not that they are simply impressive, that's really the entire point. I know some on Pixiv, you can look them up if you want; I am not linking them for obvious reasons (to say they are NSFW is putting it mildly).
>But no, MJ is not much better if you know how to use SD, although if what you did with SD was just put a prompt in a huggingface space, I can understand why you say that.
I'm the person behind these - [url-redacted]
I think it's safe to say i know something about SD's capabilities.
>I am arguing that they are better than any comics made with MJ, not that they are simply impressive, that's really the entire point.
Sure that's why i'm asking you to link these comics that are supposedly better than anything Midjourney has ever produced. With a claim like that, i'm sure you understand wanting to see results.
>You can go look them up on Pixiv if you want, they host some; I am not linking them for obvious reasons (to say they are NSFW is putting it mildly).
So you can't link anything that isn't NSFW on pixiv? Lol, that just solidifies my point. Frankly if the best you can come up with is pseudo porn(or maybe not pseudo lol) on pixiv (i don't imagine any readers of that will care about the things i'm looking for) then that's not a very good look.
You seem surprise that porn brings innovation, but you shouldn't if there has to be someone obsessed with creating the best possible illustration, it is indeed a Pixiv user or more generally a user who wants to create porn of their favorite character; moreover, I know these comics not because I have a weird obsession with going to read comics that were created by an AI, I know them because they are good enough to have gone on trend as NSFW comics, whereas the comics made by MJ are known not because they are good comics but because they are made by MJ (so it's cool I guess), so I don't see how it can solidify your point of view ahah, if you can't control the generation every panel will look different, a collage of images, that's why the comics made by MJ seem to be known just because they are made by MJ and not because they are in the interest of others communities like NSFW comics on Pixiv. Also for this reason, I have not saved links to these posts, I found them randomly while browsing Pixiv, another reason why you should look for them yourself.
