What do LLMs know about their own measure?
What do LLMs know about their own measure?
In a basic sense, all what makes an LLMs is the set of logits it produces as a response to a set of tokens; this set of logits defines a probability measure $\mathbb P$, which is particularly meaningful as a probability measure for pre-trained models, as it tries to _compress_ the language (something that fine-tuned models don't try to do), and as this is the phase from which the emergence of _understanding_ of the models come
In a basic sense, all what makes an LLMs is the set of logits it produces as a response to a set of tokens; this set of logits defines a probability measure
P, which is particularly meaningful as a probability measure for pre-trained models, as it tries to
compress the language (something that fine-tuned models don't try to do), and as this is the phase from which the emergence of
understanding of the models come
What is a very wrong idea about synthetic data?
What is a very wrong idea about synthetic data?
I saw a post on X that was saying something like 'those who think that using synthetic data for training LLMs have not understood the data processing inequality'
I saw a post on X that was saying something like 'those who think that using synthetic data for training LLMs have not understood the data processing inequality'
I think this is a (very) misleading take, because the data processing inequality abstracts away the compute needed to process data...
I think this is a (very) misleading take, because the data processing inequality abstracts away the compute needed to process data...
If one follows this reasoning, then one rapidly concludes that playing chess is trivial or that mathematics as a field is just the data of the basic axioms, or that thinking or dreaming about things without interacting with the real world is just pointless
If one follows this reasoning, then one rapidly concludes that playing chess is trivial or that mathematics as a field is just the data of the basic axioms, or that thinking or dreaming about things without interacting with the real world is just pointless
Ironically enough, this quote comes (of course) from a pretty good theoretically-minded researcher...
Ironically enough, this quote comes (of course) from a pretty good theoretically-minded researcher...
Understanding here means 'building efficient representations allowing the model to perfom powerful abstractions and useful predictions'
Understanding here means 'building efficient representations allowing the model to perfom powerful abstractions and useful predictions'
This measure $\mathbb P$ takes as input a sequence of tokens $s$ and assigns to it the probability $\exp (-\mathcal L_X[s])$, where $\mathcal L_X$ is the cross-entropy loss, defined by $\mathcal L_X[s]=\sum_{k=0}^n\ell _X(s_k:s_0\cdots s_{k-1})$, where$ $ $\ell_X$ is the log of the predicted probability for the token $s_k$
This measure
P takes as input a sequence of tokens
s and assigns to it the probability
exp(−LX[s]), where
LX is the cross-entropy loss, defined by
LX[s]=∑k=0nℓX(sk:s0⋯sk−1), where
ℓX is the log of the predicted probability for the token
sk$\ell_X(s_k:s_0\cdots s_{k-1})=\log(e^{\ell_{s_0, \ldots, s_{k-1}}[s_k]}/\sum_j e^{\ell_{s_0,\ldots,s_{k-1}]}[j]})=\ell_{s_0,\ldots,s_{k-1}}[s_k]-\log(\sum_j e^{\ell_{s_0,\ldots,s_{k-1}}[j]})$, where $\ell_{s_0,\ldots,s_k}$ is the vector the logits with context $s_0,\ldots,s_k$
ℓX(sk:s0⋯sk−1)=log(eℓs0,…,sk−1[sk]/∑jeℓs0,…,sk−1][j])=ℓs0,…,sk−1[sk]−log(∑jeℓs0,…,sk−1[j]), where
ℓs0,…,sk is the vector the logits with context
s0,…,sk The only thing we know how to do well _a priori_ is to sample from $\mathbb P$, but we for instance don't even know how to sample from $\exp(-\beta \mathcal L_X )$ for $\beta\neq 1$ (perhaps $\beta\to\infty$ would be particularly useful?)
The only thing we know how to do well
a priori is to sample from
P, but we for instance don't even know how to sample from
exp(−βLX) for
β=1 (perhaps
β→∞ would be particularly useful?)
Setting the temperature to $T\neq 1$ does not give the same result (e.g. it is obvious that $T=0$ will not give $\beta\to\infty$: we will typically end up with a not-so-likely sentence if we just maximize greedily the token probability as we sample progressively)
Setting the temperature to
T=1 does not give the same result (e.g. it is obvious that
T=0 will not give
β→∞: we will typically end up with a not-so-likely sentence if we just maximize greedily the token probability as we sample progressively)
There is a lot of nontrivial questions that could be asked about this measure that the model would not know how to answer by generating completions to them; we could call this _explicitly knowing_ versus _implicitly knowing_
There is a lot of nontrivial questions that could be asked about this measure that the model would not know how to answer by generating completions to them; we could call this explicitly knowing versus implicitly knowing
What is a thing we know implicitly but not explicitly?
What is a thing we know implicitly but not explicitly?
If we know mathematics, we can probably recognize a solution to a problem as being correct, but we may not be able to find it
If we know mathematics, we can probably recognize a solution to a problem as being correct, but we may not be able to find it
If we know the rules of chess, we could probably recognize that the solution to a chess problem is correct (e.g. if the claim is 'white mates in five', given enough time, we should be able to prove by hand that a proposed solution is correct, even if we are not very good at chess), though it could take us a very long time to verify it (it would still take substantially less time than finding the solution)
If we know the rules of chess, we could probably recognize that the solution to a chess problem is correct (e.g. if the claim is 'white mates in five', given enough time, we should be able to prove by hand that a proposed solution is correct, even if we are not very good at chess), though it could take us a very long time to verify it (it would still take substantially less time than finding the solution)
If we have a riddle to solve, we can probably recognize a solution with an explanation is correct, in spite of the fact that we don't know very well how to solve it a priori
If we have a riddle to solve, we can probably recognize a solution with an explanation is correct, in spite of the fact that we don't know very well how to solve it a priori
All the above problems are somehow reminiscent of the 'P vs NP' problems (things that are easy to check are not easy to find)... but beyond that class of problems, there are other differences between implicitly versus explicitly kwowing: given an idea, how does knowing it change our understanding of the world, i.e. how much better are we with it as we would be without it, how much less surprising would things be if we knew it compared to if we didn't know it?
All the above problems are somehow reminiscent of the 'P vs NP' problems (things that are easy to check are not easy to find)... but beyond that class of problems, there are other differences between implicitly versus explicitly kwowing: given an idea, how does knowing it change our understanding of the world, i.e. how much better are we with it as we would be without it, how much less surprising would things be if we knew it compared to if we didn't know it?
The Red-Blue Game
The Red-Blue Game
The red-blue game where we should find the pre-prompt based on the colors induced by it on the words of a sentence is particularly interesting, because it is fun and because it makes us think deeply about the way things are inter-related, about unusual correlations between pieces of a text; we end up finding that some things are remarkably dependent... it is a bit like if we get different reads on a text, with a fresh mind everytime...
The red-blue game where we should find the pre-prompt based on the colors induced by it on the words of a sentence is particularly interesting, because it is fun and because it makes us think deeply about the way things are inter-related, about unusual correlations between pieces of a text; we end up finding that some things are remarkably dependent... it is a bit like if we get different reads on a text, with a fresh mind everytime...
And the inverse problem of the red-blue game is something that lead us to think about the possible causes of observed consequences; learning to do this definitely has the favor of learning to performing Bayes-type inversion...
And the inverse problem of the red-blue game is something that lead us to think about the possible causes of observed consequences; learning to do this definitely has the favor of learning to performing Bayes-type inversion...
What problems can LLMs implicitly solve?
What problems can LLMs implicitly solve?
While LLMs do not seem to be able to explicitly know when they are making a correct mathematical statement, or they are not able to solve (yet) things like the Arc-AGI challenge, my intuition is sufficiently advanced LLMs can _implicitly_ recognize correct solutions over incorrect ones (at least in most cases)
While LLMs do not seem to be able to explicitly know when they are making a correct mathematical statement, or they are not able to solve (yet) things like the Arc-AGI challenge, my intuition is sufficiently advanced LLMs can implicitly recognize correct solutions over incorrect ones (at least in most cases)
Intuitively, I tend to think that an incorrect solution should _look slightly more like a stretch_ than a correct one... that the LLM would identify parts of the wrong discourse it makes if they were shown to it in isolation
Intuitively, I tend to think that an incorrect solution should look slightly more like a stretch than a correct one... that the LLM would identify parts of the wrong discourse it makes if they were shown to it in isolation
Generations of Models
Generations of Models
The idea is that xent closure would be used to construct an next-gen model from a base model, and then that one would transform that next-gen model into a next-next-gen model, etc...
The idea is that xent closure would be used to construct an next-gen model from a base model, and then that one would transform that next-gen model into a next-next-gen model, etc...
This chain of models could be expected to asymptotically converge to "something that's as aware about the answers to the xent questions as could be" (given a compute envelope, and certain constraints)
This chain of models could be expected to asymptotically converge to "something that's as aware about the answers to the xent questions as could be" (given a compute envelope, and certain constraints)
We can abstractly study the limit of that process, and refer to it as a 'xent-closure fixed point', somehow a little bit like a renormalization group fixed point
We can abstractly study the limit of that process, and refer to it as a 'xent-closure fixed point', somehow a little bit like a renormalization group fixed point
January 28th, 2025
January 28th, 2025
January 29th, 2025
January 29th, 2025
An important idea about xent closures, which was not clearly discussed above is the importance of models understanding abstract concepts like originality
An important idea about xent closures, which was not clearly discussed above is the importance of models understanding abstract concepts like originality
I would postulate that originality is something that definitely contains the notion that something is 'unexpected' (of course, it is not only 'unexpected', but that's part of it)...
I would postulate that originality is something that definitely contains the notion that something is 'unexpected' (of course, it is not only 'unexpected', but that's part of it)...
The 'unexpectedness' is definitely something that the model would 'implicitly know' from its xent loss, but that an explicit knowledge of xent could definitely help gauge
The 'unexpectedness' is definitely something that the model would 'implicitly know' from its xent loss, but that an explicit knowledge of xent could definitely help gauge
February 5th, 2025
February 5th, 2025
I think it is important to distinguish the xent games from the xent pre-training
I think it is important to distinguish the xent games from the xent pre-training
The pre-training idea would be to make the models better at _learning to play the games_
The pre-training idea would be to make the models better at learning to play the games
We should not seek to have particular originality in the making of the synthetic data related to the pre-training, we should just teach the models to know various xents for various models under various contexts... we should just teach the models to do that, and we should teach them red-blue inverse game questions; and in some sense, that's it
We should not seek to have particular originality in the making of the synthetic data related to the pre-training, we should just teach the models to know various xents for various models under various contexts... we should just teach the models to do that, and we should teach them red-blue inverse game questions; and in some sense, that's it
These things should help the models acquire the xent notions that would enable them to then play the games quite well; this way, they would understand what makes sense, and what doesn't
These things should help the models acquire the xent notions that would enable them to then play the games quite well; this way, they would understand what makes sense, and what doesn't
And our score would be the speed at which they would learn to play the game, either by reinforcement learning, or in zero-shot mode
And our score would be the speed at which they would learn to play the game, either by reinforcement learning, or in zero-shot mode
March 8th, 2025
March 8th, 2025
So... the game language syntax is now completed, with the five key instructions (the key instructions are the only ones allowed to produce side effects)
So... the game language syntax is now completed, with the five key instructions (the key instructions are the only ones allowed to produce side effects)
March 12th, 2025
March 12th, 2025
So, there was a game with a kind of arrow of time question (find something that makes the text maximally predictable, and yet would be a the same time minimally predictable from it... perhaps with an extra constraint that the entropy of that text should be minimal), and Vassilis very rightly commented that this would naturally lead to the factorization, if we were to start with a number
So, there was a game with a kind of arrow of time question (find something that makes the text maximally predictable, and yet would be a the same time minimally predictable from it... perhaps with an extra constraint that the entropy of that text should be minimal), and Vassilis very rightly commented that this would naturally lead to the factorization, if we were to start with a number
So, something that would be very interesting would be to find a setting like that that really spontaneously generates the factorization, and to see what else would come in different settings
So, something that would be very interesting would be to find a setting like that that really spontaneously generates the factorization, and to see what else would come in different settings
.