This is not (or not just) about RAG. It is also not a manifesto… to the extent that the NewType Manifesto is one; it’s just something I wrote down that i forgot why i did…
Part One
The concept of search in artificial intelligence (AI) is rooted in the foundational ideas of Alan Turing, who envisioned computers using their internal states for data storage and manipulation. This groundbreaking notion set the stage for the development of algorithms designed to search for solutions in vast possibility spaces.
A core challenge in AI is computational complexity—the phenomenon where the time required to find a solution increases exponentially with the number of variables and constraints. This complexity is exemplified by the traveling salesman problem, where finding the shortest route among many cities becomes increasingly difficult as more cities are added. Heuristic methods like hill-climbing algorithms and Monte Carlo simulations (I LOVE METROPOLIS HASTINGS!!!!!!) were developed to address these challenges. These techniques reduce search time by identifying patterns within the problem space, though they do not guarantee optimal solutions due to their reliance on randomness and heuristics.
More sophisticated approaches, such as genetic algorithms and neural networks, mimic natural evolution and learning processes to enhance solution quality over time. However, these methods still face limitations when applied to complex, real-world scenarios that demand high levels of adaptability and creativity.
Part Two
Has the definition of “general intelligence” been blurry and all over the place? I don’t really know, and it’s against my personal clock to find out. I do remember one of the clearer definitions (you can argue it to be more narrow, but whatever) to be:
capable of learning any task from minimal examples.
Not sure about how everybody else learned, considering that I did not have adequate internet access until 16, but the way I see intelligence being acquired is through demonstration and imitation. Where to find the demonstrations? Search. Right now, the implications of scaling law and the (partially) corresponding synthetic data research emerged to be a promising direction to work on the things being searched from. More efficient sampling algorithms are also needed –– then again, one person can only do as much as he can. (don’t mind me trying tho)
GPT-4o added the below parts
Beyond technical efficiency, progress in AI prompts deeper reflections on human values and societal norms. As we design systems that increasingly act on our behalf, we must consider the broader implications of their decisions and actions.
Part Three
The search problem in AI research presents a compelling piece to the puzzle: how to create intelligent agents that efficiently navigate vast spaces of possibilities while minimizing computational demands. Continued progress in this area promises new discoveries and insights, pushing the boundaries of what AI can achieve.
Refining AI search techniques is not merely a technical endeavor but also an opportunity for philosophical reflection on morality and human existence. Disregarding the argument of balancing ego vs no ego, I find that mental priors benefit from being flexible on the basis of preserving agency.
Post-Script
For some reason, I find the first draft of this piece just casually lying in my notes, about twice as long as this refined draft. Frankly, some of the thoughts in the original draft were too far-fetched to be considered relevant here. It was also one short paragraph followed by another, as if I was taking shots in between. I do think search is an important problem, let it be algorithmic search or information retrieval.