It seems to me this is only useful for self-improvement up to the point of accomplishing objectives and problems that humans have already clearly defined, solved and mapped. For example, "At checkpoints, a stronger evaluator can replace the old one if it performs better on trusted ground-truth examples." - if they're a trusted ground-truth, they must be rigorous. If we're attempting to solve problems that humans have not already solved, where do you get your ground-truth examples? It's not like the AI is going to be able to generate these for you if it has never seen a solution to the problem.
I can definitely see the argument that this allows us to train models faster and converge faster, because if you scale difficulty of evaluation alongside the learners capabilities, it spends a lot less time floudering around. It basically works out to be loss minimization through strategic ordering of the training data. Is that the goal here though? Or is the goal recursive self-improvement and solving problems that are currently outside of reach? Because it doesn't feel like the latter would be possible with this design.
For example, how do you quantify "the evaluation gets -harder- as the agent gets better". Harder, how? In what direction? Via what criteria or measure?
For tasks with no existing ground-truth data, one could try to use an adversarial objective on its own by making AI-generated outputs compete against each other, although this is beyond the scope of our experiments. Adapting our method to this setting is quite straightforward.
For now, we have looked into composite objectives trading off performance on the ground truth against the ability to reject generated samples produced in earlier epochs. For example, when using this system to co-evolve paper writers and reviewers, we reuse generated papers that were accepted by the reviewer of epoch t as adversarial samples in epoch t+1. Then, new reviewers are rewarded for rejecting AI-generated papers. You can have ground-truth performance account for x of the utility of a reviewer, with the adversarial objective then providing 1-x.
Here’s a paper by Floreano at EPFL from 1997 explicitly on Red Queen dynamics for creating neural networks for intelligent robot control.
There was lots of discussion of these ideas in the 1990s. In those days we trained very small NNs - tens of nodes - by evolving their weights and topologies. A run could take days on a workstation of the time.
This particular paper is about co-evolving predator and prey, where the behavior of each is the ‘evaluation’ of the other.
I wonder if this would work for generating algorithmic code for a town of NPC's in a RPG or city sim. The problem the teacher would be tasked to solve, in this case, would be to score NPC algorithms on whether they would lead to happiness, health, and wealth for their character. This would allow the generation of NPC algorithms to run at faster pace then would normally be possible if you had to simulate their lives for a day, a week, or a month just to see if their algorithm would be a success or not. And since NPC's are competing against each other, they would naturally need more sophisticated algorithms to succeed.
This does sound very interesting, and my immediate thought on this would be to have 2 or multiple agents in learning, where they continuously set a new bar among all of them. It might be that it will just converge towards them all becoming more similar as they would raise bar by what they know they perform better at than the other. So it might be necessary to find some heuristic to the measurement to avoid that route to be taken.
> The research team, which includes collaborators from NVIDIA and Flower Labs, have come up with a new method for recursive self-improving AI agents to continue improving themselves.
What happens if you apply the method to non-recursive self-improving AI agents? Can they continue improving themselves? Or does the recursive self improvement only recursively self improve AI agents which are themselves recursively self-improving?
The paper is explicitly an application of a very general evolutionary principle to the self-improving tree search algorithms popularised by the Huxley-Gödel Machine and Darwin-Gödel Machine.
We agree that the methods used have been extensively researched across a wide variety of domains.
18 comments:
It seems to me this is only useful for self-improvement up to the point of accomplishing objectives and problems that humans have already clearly defined, solved and mapped. For example, "At checkpoints, a stronger evaluator can replace the old one if it performs better on trusted ground-truth examples." - if they're a trusted ground-truth, they must be rigorous. If we're attempting to solve problems that humans have not already solved, where do you get your ground-truth examples? It's not like the AI is going to be able to generate these for you if it has never seen a solution to the problem.
I can definitely see the argument that this allows us to train models faster and converge faster, because if you scale difficulty of evaluation alongside the learners capabilities, it spends a lot less time floudering around. It basically works out to be loss minimization through strategic ordering of the training data. Is that the goal here though? Or is the goal recursive self-improvement and solving problems that are currently outside of reach? Because it doesn't feel like the latter would be possible with this design.
For example, how do you quantify "the evaluation gets -harder- as the agent gets better". Harder, how? In what direction? Via what criteria or measure?
For tasks with no existing ground-truth data, one could try to use an adversarial objective on its own by making AI-generated outputs compete against each other, although this is beyond the scope of our experiments. Adapting our method to this setting is quite straightforward.
For now, we have looked into composite objectives trading off performance on the ground truth against the ability to reject generated samples produced in earlier epochs. For example, when using this system to co-evolve paper writers and reviewers, we reuse generated papers that were accepted by the reviewer of epoch t as adversarial samples in epoch t+1. Then, new reviewers are rewarded for rejecting AI-generated papers. You can have ground-truth performance account for x of the utility of a reviewer, with the adversarial objective then providing 1-x.
Here’s a paper by Floreano at EPFL from 1997 explicitly on Red Queen dynamics for creating neural networks for intelligent robot control.
There was lots of discussion of these ideas in the 1990s. In those days we trained very small NNs - tens of nodes - by evolving their weights and topologies. A run could take days on a workstation of the time.
This particular paper is about co-evolving predator and prey, where the behavior of each is the ‘evaluation’ of the other.
https://infoscience.epfl.ch/entities/publication/a65d0679-68...
Funnily enough, Jürgen Schmidthuber invented / coined the term "Gödel Maschine" in 2003: https://arxiv.org/abs/cs/0309048
He'll probably have a field day over this.
OP's link: > Now the researchers have addressed this issue by having both the self-improving agent and the evaluator evolve together.
and your quote:
> This particular paper is about co-evolving predator and prey, where the behavior of each is the ‘evaluation’ of the other.
Both sound like the GAN approach that was popularized a decade ago and kinda the start of the "genAI" boom.
Yes, they apply the exact same principle to different algorithms and domains.
I wonder if this would work for generating algorithmic code for a town of NPC's in a RPG or city sim. The problem the teacher would be tasked to solve, in this case, would be to score NPC algorithms on whether they would lead to happiness, health, and wealth for their character. This would allow the generation of NPC algorithms to run at faster pace then would normally be possible if you had to simulate their lives for a day, a week, or a month just to see if their algorithm would be a success or not. And since NPC's are competing against each other, they would naturally need more sophisticated algorithms to succeed.
This does sound very interesting, and my immediate thought on this would be to have 2 or multiple agents in learning, where they continuously set a new bar among all of them. It might be that it will just converge towards them all becoming more similar as they would raise bar by what they know they perform better at than the other. So it might be necessary to find some heuristic to the measurement to avoid that route to be taken.
Have not read the paper yet, but this not sound like GAN applied to agent training?
Correct.
You're all going to die down here.
> The research team, which includes collaborators from NVIDIA and Flower Labs, have come up with a new method for recursive self-improving AI agents to continue improving themselves.
What happens if you apply the method to non-recursive self-improving AI agents? Can they continue improving themselves? Or does the recursive self improvement only recursively self improve AI agents which are themselves recursively self-improving?
This 'new' method was quite common in evolutionary computing in the 90's.
The paper is explicitly an application of a very general evolutionary principle to the self-improving tree search algorithms popularised by the Huxley-Gödel Machine and Darwin-Gödel Machine.
We agree that the methods used have been extensively researched across a wide variety of domains.
> "Instead of improving an agent against a fixed test, we let the evaluation evolve alongside the agent"
This quote should have been highlighted earlier in the article.
Yes, this is key
Yeah I think it is broadly applicable to tech as a whole, I mean any great startup is just really a counter positioned company to incumbents -
true. And all of us are finding RSI is a great lever to explore.