Building a Smart Math Tutor With a Small Model
Everyone talks about making AI systems better by switching to larger models. My goal was to use a small CPU deployable model for usecases where compute is a big problem. So I decided to use a 2-billion parameter model. However, I faced several problems as the model wasn't able to retain any memory and would hallucinate in second prompt. Moreover, it struggled with the most basic math questions when they were worded cleverly. So I decided to improve that with Retrieval Augmented Generation and the results were pleasant.
Unlike common uses, I did not want another chatbot. I wanted to keep the project simple which is why I decided to store the knowledgebase beforehand. The goal of this project was to build an assistant that could help students of classes 1 - 6 learn basic maths concepts by asking them questions and building stories. For that a small model seemed to be enough, allowing me to avoid using a GPU altogether.
The model wasn't the bottleneck
This may seem crazy at first but the model wasn't the bottleneck. All it needed was some help to remember the context of the conversation and some clues to understand the query better. For that I vectorized the knowledgebase as well as user queries. This allowed the model to make connections between what it had (the knowledgebase) and what was asked (the query) by calculating similarity between the vectors. This was a game changer as the model was now able to understand queries much better and provide answers that were more related to what was asked.
My model of choice was Gemma-2B, which on its own wasn't terrible but it lacked reliability. But vectorizing everything alone made it much more predictable. Now the model could move past straight forward questions and tackle long form statements which it lacked before. Instead of replacing the model, I started asking a different question:
What information should the model stop trying to remember?
That question shaped the rest of the project.
Retrieval solved a different problem than I expected
I originally added Retrieval-Augmented Generation because I wanted better factual accuracy. That happened, but the bigger improvement was consistency. Every mathematical explanation now started from the same curated knowledge base instead of whatever the model happened to recall. The model stopped "guessing" definitions because it no longer had to. The knowledge base became the source of truth. That also meant updating educational content became much easier. Instead of retraining a model every time I wanted to improve explanations, I could simply update the documents and rebuild the vector index.
Conversation memory mattered more than retrieval
Retrieval fixed knowledge. Memory fixed conversations. Without conversation history, every question looked independent. A student might ask:
What are fractions?
followed by:
Can you explain that another way?
Without memory, the second question has almost no meaning. Passing the complete conversation history into every prompt made interactions feel much more natural. The model could reference earlier explanations instead of starting over every time. The improvement wasn't flashy, but it made the assistant feel significantly more coherent during longer conversations.
Prompt engineering became more about constraints than creativity
Before this project, I mostly thought of prompts as instructions. Building this system changed that perspective. The most useful prompts weren't the longest ones. They were the ones that clearly defined what the model should not do. Some of the most effective rules were surprisingly simple:
- Don't repeat the user's question.
- Don't invent numbers.
- Answer only what's being asked.
- Keep explanations concise.
- Use short stories only when they improve understanding.
None of these rules made the model smarter. They made it more disciplined. And discipline turned out to be just as valuable as intelligence.
Storytelling changed the interaction completely
One of the project goals was making mathematics feel less intimidating for younger students. Instead of explaining concepts through definitions, the assistant explained them through small stories. Fractions became pizza slices. Multiplication became groups of toys. Division became sharing chocolates among friends. The mathematics stayed the same but the presentation changed. That small design decision made responses feel far more approachable without sacrificing correctness.
Building evaluation early saved a lot of debugging later
One mistake I see in many AI projects is treating evaluation as something you add after everything works. I went in the opposite direction. The project included scripts for testing:
- embedding quality,
- retrieval accuracy,
- and multi-turn conversational consistency.
That made debugging much easier. Instead of asking "Why did the model answer badly?", I could narrow the problem down. Was the retrieval wrong? Was the prompt unclear? Or did the language model simply fail to reason with the correct context? Those are completely different problems, and they deserve different solutions.
Final thoughts
This project reinforced something I've started seeing across modern AI engineering. The quality of an AI application depends far less on the language model than people think. Retrieval, memory, evaluation, prompt design, and system architecture often determine whether an application feels reliable. Gemma-2B didn't suddenly become a larger model. It became part of a better system. And in practice, that's usually the more valuable improvement.
Future improvements
Right now the project can only be interacted through terminal which may seem daunting to new AI adopters. But I intend to build a user interface soon which will allow almost anyone to interract with it. Moreover, there are alot of valuable things I learned which will be fixed in near future.