1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means.
My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.
> 1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means.
Multimodality in a model That's between 4-7% the cost per token of OpenAI’s cheapest multimodal model is an important feature when you are talking about production use and not just economically unsustainable demos.
In preview, can't be used in production, they already rug-pulled people building on Gemini w/r/t cost and RPM, and they're pointedly not putting any RPM or cost on the page. (seriously, try finding info on cost, RPM, or release right now, you're linked in circles.)
Agree on OpenAI multimodal but it's sort of a stilted example itself, it's because OpenAI has a hole in its lineup - ex. Claude Haiku is multimodal, faster, and significantly cheaper than GPT 3.5.
Given the one comment, I wouldn't draw too many conclusions re: Anthropic.* SNR ratio on AI everywhere is low-ish, and optimizing for a low cost multimodal model is a business-directed need that's rather niche currently.
* I was very pleasantly surprised to see it ranking as high as #4, and its still top #50, in App Store.
A) This fills a gaping hole for cheap multimodal models, OpenAI doesn't have one
B) Anthropic's Haiku is a good choice.
You) wow A didn't know Anthropic. Goes to show power of brands, much like snack foods
B) Eh I wouldn't conclude anything from A. Its one comment. some people don't know what an Anthropic is because there's high interest in AI relative to interest in AI APIs. you can expect a low SNR ratio, even on HN
You) Stop personally attacking me! It's against the rules!!
I don't think its helpful to broaden the aspersions: you had a couple hours to cool off, and also got signal from the community that you went overboard.
It's clear I was talking to you, not about you, you can tell because it's a reply to you.
I do understand it would feel awful if I replied to you, ignoring you, and instead voicing to some anon 3rd person you don't know Anthropic pricing. Someone with looking to bully could possibly use that to say you are an amateur who doesn't know what you're talking about w.r.t to AI and only knows a brand name or two.
If anyone does that, let me know, I'll correct their bullying...pulls on sunglasses...with extreme prejudice.
1. The comment where you began getting upset and attacking people is grayed out, the only one in this thread.
2. The other person straightforwardly explaining to you I was replying to you, not talking about you. (which, I do understand why you're taking personally, but humbly, it isn't a rule violation or close, or "low-brow", it's well within bounds of adult conversation)
3. Me agreeing the reply was a reply, I don't know you or judge you, and if anyone does, I gave a nice playful comment telling you I'd fight for your honor.
I humbly suggest taking a step back from this thread: it can be incredibly frustrating when you feel you were wronged and no one is listening, I feel ya. I tried giving you an e-hug without being condescending with the sunglasses comment. We both agree its outside the bounds of HN to continue this. Yell at me on Twitter for being sneaky and lying and saying you're bad at AI and making low-brow comments, @jpohhhh.
Thanks for responding to that point - it's the one most on my mind.
So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context.
So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.
Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.
Yes but we can distinguish between embedding provided to customers, and internal embeddings. One is optimized for usage in certain types of applications, but the internal embeddings need to be optimized to support long contexts and are not constrained by the customer facing embeddings.
For example, with Open AI I believe it’s known that the internal dimension for Gpt3 was 12,288.
Are the same embeddings not used internally? I thought they were. Maybe I'm wrong about that.
Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?
For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement.
They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM.
There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.
So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.
Will compute allow that number to go up? Or is that an optimal number?
Definitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
> My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.
Is it possible to explain what this means in a way that somebody only roughly familiar with vectors and vector databases? Or recommend an article or further reading on the topic?
Essentially each token of a text occupies a point in a many-dimensional model that represents meaning, and LLMs predict the next token by modifying the last token with the context of all the tokens before it. Attention heads are basically a way of choosing which prior tokens are most relevant and adjusting the last token's point in vector-space accordingly.
>My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.
We are dealing with multi-headed attention, therefore we have multiple points per token. You can always increase the number of heads or the size of the key vector.
The token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding size, afaik.
Yeah it's not very good in practice, you can get a halfway decent demo out of it ("look I gave it 6.5 harry potters and it made an SVG map connecting characters with annotations!!"...some of the characters...spare annotations...cost $20). Just good enough to fool you a couple times when you try to make it work 10 times.
My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.