LLMs and rule-based search are a killer combo for planning tasks
AutoToS brings together LLMs and classic search algorithms, reducing the costs of planning tasks while providing 100% accuracy.
Scientists at IBM Research have presented Automated Thought of Search (AutoToS), an algorithm that enhances the planning abilities of LLMs while reducing the costs to a fraction of current methods.
There is growing interest in using LLMs for planning tasks. The most successful techniques, such as Tree of Thought, treat planning tasks as a search problems. They use LLMs as search algorithms that explore the solution space of the target problem. However, these techniques are slow and expensive due to the numerous calls they need to make to the LLMs. Moreover, due to the stochastic nature of LLMs, they can result in unstable results and violate the completeness and soundness requirements of search algorithms.
“Thought of Search,” a technique recently introduced by IBM Research, brings a different approach to search. Instead of doing the actual search ToS “thinks” of a search algorithm that can solve the problem and proposes the code for the successor function and the goal function. The proposed algorithm can then be plugged into any classic search mechanism.
ToS is much more efficient than previous LLM planning approaches, but it requires manual review and feedback on the proposed code.
AutoToS removes this bottleneck by using a multi-step process that gets the model propose, produce, evaluate, and correct the components of its search algorithm. The model uses its own reasoning abilities to vet its produced code, check it for completeness and soundness and make corrections to it. Once the code has been finalized, it can be run without further need for the model.
Experiments on various sizes of model families show that AutoToS can reach 100% accuracy with very few calls to the LLM. For example, the researchers were able to solve all 1,362 puzzles of the Game 24 dataset with an average of 2.2 calls to GPT-4, as opposed to around 100,000 calls required by other methods.
AutoToS shows the power of neurosymbolic AI systems and their practical uses for real-world applications.
For more on AutoToS and comments from its authors, read my article on VentureBeat
Read the paper on arXiv