
For the past month, I've been obsessed with a simple but dangerous problem: modern LLM-augmented search systems trade quality for latency.

Tools like Perplexity are fast, but that speed often comes at the cost of accuracy, trustworthiness, and depth. That's risky. If a system picks the wrong source or collapses nuance into a shallow answer, users can get misinformed quickly and confidently.

I wanted to build something different: a system where **accuracy and trustworthiness are first-class goals**, even if that means a little slower.

## The Early Missteps: RL Finetuning Adventures

Like many, my first instinct was to throw reinforcement learning at the problem.

I started experimenting with Qwen3, thinking its "thinking" in the chat template would make multi-turn RL fine-tuning hell. Models like Qwen3-4B-Instruct-2507 struggled with basic instruction following, they wouldn't diversify queries, and they ignored answer templates entirely.

Switching to the Qwen3.5 series didn't help much. Even larger models like Qwen-2.5-32B produced basic tool-calling errors. Training showed brief hope (reward climbing), but then collapsed and plateuaed. It became clear: **RL alone wasn't going to cut it without strong SFT foundations**.

That realization led to another roadblock: I needed thousands of high-quality supervised traces. But generating them was financially out of reach, especially since I'd need both SFT and RL runs, plus evals later. After some sunk GPU costs (and a few movies to clear my head), I stepped back.

## The Reset: Orchestration Over Training
The question I had to ask myself was: **why was I ashamed of just orchestrating GPT calls?**

Training your own model sounds glamorous, but orchestration is often the more pragmatic path. So I pivoted: instead of RL finetuning, I built a plain GPT-5 agent hooked up to web search via Exa AI.

At first, the outcome was disappointing. Exa's neural embedding retrieval often surfaced irrelevant or outdated pages, while their Gemini Flash summaries frequently mis-parsed key details. On top of that, GPT-5 would lazily call one or two searches, skim results, and declare itself done.

That failure was instructive: the problem wasn't them LLM per se, but the research process itself.

## Thinking Like a Human Researcher
When we research online, we don't stop at one search. We try multiple phrasings, open several tabs in parallel, read through entire articles, cross-check across sources, and then synthesize.

So why not replicate that exact workflow using an LLM and Python? A state machine.

### The State Machine
- Generate Queries Node:
  - Given the user query, reason about intent, fill gaps, and generate several diverse search queries.
  - Too many queries (5-6+) leads to superficial diversification. Too few (1-2) just restate the question.
  - The sweet spot: **3-4 well-reasoned queries**.
- Retrieve & Summarize Node:
  - Run all queries in parallel through Exa AI.
  - Rerank results with simple heuristics (sometimes even without LLM intervention).
  - Instead of relying on Exa's summaries, use **GPT-5-Nano to summarize full pages** relative to original query.
  - Each summary explicitly mentions the details it found relevant or not.
- Final Answer Node:
  - Take the user query + all snippets/summaries.
  - Reason over them to construct a grounded answer.
  - Depending on preference, highlight multiple corroborating sources or elevate one authoritative, recent source.
  - Always expose reasoning traces for transparency.
  - If information is insufficient, say “I don’t know”.

This pipeline mirrors how humans actually search: broad exploration, careful reading, then reasoned synthesis.

## The Role of Reasoning: Concise > Exhaustive
One surprising lesson was about reasoning depth.

- High reasoning effort: the model overthinks, tries to solve the whole problem in the `generate_queries` step, and skips the pipeline.
- Minimal reasoning: outputs shallow restatements, with no real expansion.
- Concise reasoning (“low” effort): the sweet spot. Enough reasoning to generate useful queries, but not so much that the system derails.

More test-time compute didn’t mean better results. Concise reasoning actually increased accuracy.

## Benchmarking Against SOTA
It's easy to claim you've built something better - but proof requires evals.

I choose the `vtllms/sealqa` benchmark, `seal-0` subset. It's 111 queries spanning diverse backgrounds. Before running experiments, I went through the queries myself just to get a sense of their difficulty.

What I found was shocking: SOTA agentic search systems perform abysmally here.

- OpenAI o3 and Gemini 2.5 Pro struggled, both under 20% accuracy.
- xAI Grok-4 topped out at just 20.7%.

My approach using GPT-5-mini as a strict judge, hit 38/111 = ~34% accuracy. That's a huge relative gain. Even if a few ground truths in the benchmark are debatable, the result is clear: **structured orchestration beats brute-force agentic search**.

![Benchmark Results](https://github.com/user-attachments/assets/2b53dad3-d26d-4729-93e2-0bce39f9a2c1)

## knobs, Controls, and Next Steps
One of the strengths of this system is how tunable it is:
- Adjust reasoning effort.
- Control number of search queries generated.
- Vary the number of pages retrieved per query.

There’s also a clear path to more accuracy:
- Add confidence estimation when constructing the final answer.
- If confidence is low, automatically trigger retries with fresh queries focused on knowledge gaps.
- This loop should push accuracy significantly higher across the benchmark.

## Closing Thoughts
In just 2-3 days of sys eng mode thought, no fine-tuning, no massive datasets, no expensive RL runs, we outperformed SOTA LLM search agents on a good benchmark.

The takeaway? Sometimes the best way forward isn't more training, but better orchestration. Mimicking how humans actually research: casting a wide net, reading carefully, and reasoning concisely. This goes a long way.

Transparency matters. Trustworthiness matters. And if that means a little more latency, that’s a tradeoff worth making.

For some transparency on my side, you can see the eval sheet here: [Think-a-Tron/atreya](https://github.com/Think-a-Tron/atreya)


Building a better LLM augmented Search Engine


So Deepmind dropped AlphaEvolve - a Gemini powered agent that uses LLMs to evolve better algorithms. Think matrix multiplication, fast attention... basically, a nerd paradise.

I was awestruck. Sat frozen for an hour. Then I did what any sane person would do.

Opened ChatGPT and asked, "how was it built?"

Crickets.

Turned out Deepmind didn't release any details. No code. No paper. Just buzzwords like "evolutionary search."

Fine. I'll do it myself. (Thanos aaaaaahhhhh)

## Descent into Genetic Programming

Found the Field Guide to Genetic Programming. Breezed through the essentials:
- Population of programs
- Fitness functions
- Tournament selection
- Crossover (ignored, obviously)
- Mutation (my true love, and also what AlphaEvolve used)

Also went down a rabbit hole on Pareto fronts and NSGA-II. Multi objective optimization sounded cool and made me feel like I was doing science.

## First Attempt: Fast Fourier Foolery
Whipped up a script to evolve Fast Fourier Transform implementations.
- Objective? Speed
- LLM? GPT-4.1
- Result? It got faster

...but only because it returned the input. No FFT. Just `return a`.

I stared blankly. Then punched myself in the face. Deserved.

## Second Attempt: Accuracy enters the chat
Added accuracy as a second objective. Now we're doing real evolution right?

Wrong.

Ran 50, 100, 200 generations. Nothing improved. The LLM was spitting garbage, sometimes not even valid python. Despite giving it function signatures and warm hugs.

Then I found the `prediction` parameter in the OpenAI's API. You can nudge the LLM to follow a previous example. So I fed it that last mutation. It helped. A bit.

But now it was mostly just rediscovering slightly better existing solutions. Maybe deepmind ran theirs for days. I ran mine for a few minutes and went to bed defeated.

## Can you see the Matrix?
New day, new dumb idea. Let's evolve matrix multiplication instead.

Started with GPT-4.1 making freeform mutations.

Results? Errors. Broken logic. Code that didn't even run. Great job, me.

So I enforced mutation strategies. Real ones:
- Reorder loops
- Apply loop tiling
- Add blocking
- Vectorize inner loop

Finally got some good mutations.

But guess what? Every candidate just used `np.dot` or `a@b`.

I had been reward-hacked hard.

Every single one of them discovered that if you just delegate to numpy, you win the benchmark. Technically correct. Ethically illegal.

So I banned the cheats. No `np.dot`, `a@b` and `np.einsum`. Just you, me, and raw loops.

And then it worked. Slowly, painfully, but it evolved a matrix multiplication routine faster than the seeds and didn't cheat.

I almost cried.

## Cursed Implementation Details
Here's what this Frankenstein system actually does:
- Maintains a population of matrix multiplication implementations
- Mutates them via GPT-4.1 using one of many predefined strategies
- Uses a multi-objective fitness function
  - Speed
  - Accuracy
- Runs NSGA style pareto selection with crowding distance to keep diversity
- Ages individuals and replaces them with mutated offspring + random immigrants
- Evaluates each candidate via importlib, runs it on benchmark sizes 128x128 and 256x256, and filters garbage.

Here's the code if you want to read it, clone it, run it, or regret your life choices: [GitHub](https://github.com/think-a-tron/evolve)

If you see me doing crossover next week or trying to evolve merge sort, please stop me. This rabbit hole has no end.

Kthnxbye.


Replicating AlphaEvolve


A few weeks ago, if you'd asked me what "advantage" means in reinforcement learning, I'd probably have just shrugged. Today, I'm casually dropping terms like Group Relative Policy Optimization (GRPO) as if I've known them forever—and somehow built a decent physics solver using Large Language Models (LLMs) along the way. All this happened over a couple of weekends and cost me less than what you'd spend on a pizza night.

### Step 1: Getting Friendly with RL

It all started with DeepSeek’s paper on **GRPO**, a clever trick that optimizes policy without a separate value network—basically, just pick the best from a bunch of actions and reinforce that. But first, I had to figure out what the heck "advantage" actually meant.

Enter Hado van Hasselt’s DeepMind RL lectures. Seriously, this guy's explanations are the gold standard—clear, practical, and somehow entertaining enough to keep my attention through math.

### Step 2: HuggingFace's TRL to the Rescue

Then came HuggingFace’s **TRL library**, packed with user-friendly trainers and some fancy plugins like vLLM for faster inference, plus Liger-kernels and Flash Attention for extra speed. Suddenly, everything felt doable—and fast.

### Step 3: The Initial Struggle Was Real

My first idea was to keep it simple, inspired by DeepSeek’s R1-Zero—solve a physics problem, get it right, earn a +1 reward. Sounds easy, right? Nope. Turns out my base model (Qwen3-1.7B) couldn't handle even the basics reliably, so it was back to the drawing board.

### Step 4: Fine-Tuning (and Some Hope)

To beef things up, I curated about 1,500 physics problems across multiple domains like electromagnetism, acoustics, and kinematics—you know, the usual physics suspects.

Next, I gave my struggling Qwen model a crash course in physics via supervised fine-tuning (SFT). In just about 20 minutes on a single H-100 GPU, the model’s performance drastically improved—like cramming before an exam and actually remembering stuff. Loss dropped impressively from 0.7 down to around 0.2-0.3.

### Step 5: One-Shot RL with GRPO (Magic Happens!)

With a slightly smarter model in hand, I moved onto RL—this time with the cool-sounding GRPO technique. Inspired by the idea of "one-shot RLVR" (sounds fancy, right?), I chose a medium-difficulty problem—not too tough, but enough to challenge the model.

Reward scores shot up dramatically, from a measly 0.05 to 0.8 within just 70 iterations. And just like that, over a weekend and about \$10 later (half on dataset prep, half on GPUs), my LLM physics solver was up and running.

### Check it Out

You can actually play with this physics solver now—it's on HuggingFace: [think-a-tron/raman-01-1.7B](https://huggingface.co/think-a-tron/raman-01-1.7B).


Physics and LLMs Can Be Friends?


Reinforcement Learning has emerged as a compelling and powerful method for training agents to make decisions, often surprising humans with moves that initially appear counterintuitive, risky, or outright baffling. Yet, these seemingly insightful choices often lead to exceptional outcomes, driven purely by an agent's relentless optimization of rewards.

But this raises a deeper, philosophical question: Are RL agents truly exhibiting intelligence, or are they simply highly efficient "reward hackers"?

When humans label something as "intelligent", we imply understanding, a grasp of cause-and-effect relationships, purposeful intent, and meaningful abstraction. An RL agent, however, has none of these. It doesn't understand what a "win" truly signifies beyond a numeric increase in its reward function. It doesn't possess intentions or desires in the human sense; it merely follows gradients in policy space, iteratively tuning its actions to maximize returns.

Yet, before dismissing these agents as merely mechanical, we must reflect: What exactly does it mean when we humans claim to "understand" something? Our sense of understanding is deeply tied to causal reasoning, semantics, and the context of accumulated knowledge. But fundamentally, isn't human cognition also driven by reward signals, dopamine hits that reinforce beneficial actions and behaviors? Aren't we, at our core, sophisticated reward hackers, shaped by millions of years of evolutionary pretraining?

Indeed, unlike RL agents who begin training from a random initialization, humans enter the world pretrained. Evolutionary pressures have finely tuned our instincts, biases, and intuitions over thousands of generations. We effortlessly generalize from minimal examples, reason by analogy, and rapidly adapt to new environments, skills RL agents must painstakingly acquire through immense trial and error.

This juxtaposition offers two intriguing possibilities:

First, perhaps RL agents aren't simplistic but merely at the earliest stages of a long evolutionary trajectory toward genuine intelligence. With additional structures, perhaps they could evolve layers of abstraction, theory of mind, and contextual understanding.

Second, and perhaps more unsettling, maybe humans aren't as profoundly intelligent as we like to believe. Perhaps our consciousness, emotions, and complex social structures are sophisticated forms of reward hacking, layers of storytelling and heuristics built atop basic reward-driven optimization.

Thus, the true tension lies not in whether RL agents are intelligent, but rather how we define intelligence itself. Are we setting the bar artificially high, or are we overlooking the simplicity underlying our own complexity.

In the end, reinforcement learning doesn't just challenge how we build intelligent machines, it compels us to reconsider the nature of intelligence altogether.


lab

Building a better LLM augmented Search Engine

Replicating AlphaEvolve

Physics and LLMs Can Be Friends?

You are just a fancy reward function, bro