Change the model or change the prompt?

I have been spending time lately on the different ways to make an LLM system better, so I’m documenting my understanding of the main techniques and where each one fits.

Say you have built something with an LLM. It works, but not well enough. The summarizer misses details that matter, or the agent keeps calling the wrong tool. What are our options?

Broadly, there are two. We can change the model itself, which is the fine-tuning family. Or we can change what we tell it, which is prompt engineering, though I’ll use the broader term text-based refinement here, since the interesting new methods go well beyond tweaking prompts by hand.

Changing the model (parametric learning)

An LLM’s knowledge lives in billions of numbers called parameters, or weights. Training means nudging those numbers, which is why this family of approaches is called parametric learning.

The first thought that comes to mind is supervised fine-tuning (SFT). Collect thousands of examples of the behavior you want, each one an input paired with the ideal output, and adjust the weights until the model imitates them reliably. It is apprenticeship by imitation: watch the master, copy the master.

The other option is reinforcement learning (RL), with methods like GRPO. Here we don’t show the model correct answers. We let it attempt the task, score each attempt, and shift the weights toward whatever scores well. Learning by trial, error and grades.

Both are powerful, but the costs are not as fixed as they sound. Full fine-tuning of a large model does need a big dataset and serious GPU time. Parameter-efficient techniques like LoRA change that math: they train small adapter layers on top of a frozen base model, which brings the data and compute requirements down dramatically (I wrote about LoRA adapters earlier). Many providers now offer fine-tuning as a hosted service too, so sitting behind an API no longer rules it out. RL still needs a reliable way to score attempts, and enough of them to learn from. The one thing that holds across the board is opacity: the improvement lands in the weights, billions of numbers that nobody can read.

Changing the instructions (text-based refinement)

The alternative is to keep the model frozen and improve the text around it: the prompts, the rules, the playbook the model follows.

If parametric learning is sending an employee away for months of retraining, text-based refinement is editing the employee handbook. Same person, sharper instructions.

Everyone already does this by hand. Try a prompt, watch it fail, reword it, try again. The interesting development is that this loop can now be automated, and the best automated versions learn from something richer than a score.

The score versus the explanation

Imagine you’re helping a new engineer get better at code reviews.

One approach is to score every review: 6 out of 10, 8 out of 10, 4 out of 10. After enough attempts, perhaps they slowly develop better instincts.

A more useful approach is to sit down with one review and say: “You caught the null check, but missed the race condition because you only followed the happy path. Next time, trace what happens when two requests update the same object.”

The score tells them how well they did. The explanation tells them how to improve.

Keep that distinction in mind. Parametric RL learns from scores. The most interesting text-based methods learn from explanations.

LLM-as-a-judge

Before anything can improve, something has to define what “better” means. An LLM-as-a-judge is simply an LLM used as an evaluator: given an output and a rubric, it returns a score, a critique, or both.

Note that a judge by itself improves nothing. It plays the role of a teacher grading an assignment, the red ink rather than the revision. The judge is also optional: feedback can come from unit tests, compiler errors, exact-match checks, or human comments.

TextGrad

Real LLM systems are rarely a single prompt. A research assistant might have one prompt that writes search queries, a retrieval step, and another prompt that composes the final answer. When the final answer is bad, which prompt is at fault?

TextGrad is an optimizer inspired by backpropagation, the algorithm used to train neural networks. It records how text flows through the pipeline, then carries criticism backward through it. If a judge says the answer lacks supporting evidence, TextGrad translates that into one critique for the answer-writing prompt and a different critique for the query-writing prompt. An LLM then revises each piece of text accordingly.

These “textual gradients” are natural-language suggestions, not numerical derivatives. The spirit is the same though: figure out which upstream component to blame, and route the criticism to it.

GEPA

GEPA (short for Genetic-Pareto) is a prompt optimizer that treats a failed attempt as a story worth reading, not just a number.

Say a retrieval agent gets a question wrong. A score of 0 tells us it failed. The execution trace, the record of what actually happened along the way, tells us much more:

• The first search found the right person but the wrong event.
• The second search merely rephrased the first.
• The retriever surfaced a useful source, but the final answer ignored it.

An LLM can read that trace and infer a general lesson: follow-up searches should use new entities discovered in earlier results instead of repeating the original query. GEPA writes that lesson back into the query-writing prompt and tries again.

The loop is roughly:

1. Run the current system on a few examples.
2. Reflect on its traces and feedback.
3. Rewrite one prompt in the system.
4. Test the new version and keep it if it helps.

This works for a single prompt, but it really shines in compound systems with several LLM calls, retrieval steps and tools, since GEPA can improve one module at a time instead of treating the whole pipeline as a black box.

The name describes the search strategy.

Genetic refers to evolution. GEPA maintains a population of prompt candidates. Reflection mutates one candidate, the new version inherits the rest of its parent’s instructions, and a merge step can combine improvements found along different lineages.

Pareto refers to selection. GEPA doesn’t only keep the candidate with the best average score. It also preserves candidates that are unusually good at something:

Candidate C can be discarded since A beats it on both examples. But A and B have different strengths, so GEPA keeps both alive. This prevents the optimizer from fixating on one early winner and getting stuck, and it lets useful specialist strategies survive long enough to contribute later.

Putting them side by side

Note the split in the “what changes” column. SFT and RL change the model. Everything else changes the words.

Which one should you use?

In the GEPA paper’s experiments across four tasks and two model families, GEPA outperformed both a GRPO fine-tuning setup and MIPROv2, an earlier automatic prompt optimizer, while sometimes using far fewer attempts. The Pareto strategy also beat greedily evolving only the single best prompt.

I’d treat those results as promising rather than universal. GEPA adds extra reflection calls, leans heavily on the quality of evaluator feedback, and was tested on a limited set of systems. With abundant data, updating weights may still be the better choice.

The honest answer is that it depends on more variables than data volume. The ones I find myself weighing:

1. Knowledge or behavior: If what’s missing is facts, especially facts that change often, retrieval (RAG) is usually the right tool and neither lever needs to move. Fine-tuning is better at teaching skills, style and format than at storing fresh information.
2. Rate of change: A fine-tuned model bakes in today’s behavior. If requirements or data shift every few weeks, retraining becomes a treadmill, while a prompt can be edited and shipped in an afternoon.
3. Auditability: An optimized prompt is a document. You can read it, diff it, and walk a reviewer through it. A weight update is a pile of numbers.
4. Economics: A long prompt is paid for on every single call, while fine-tuning moves that cost up front and makes each call cheaper. At high volume, the economics often decide ties.
5. Feedback richness: If failed attempts leave clues behind (compiler errors, missing documents, tool output, grading feedback), reflective methods like GEPA have material to work with. If all you get is a score, that advantage shrinks.

And whichever way you lean, build the eval first. Every technique in this post is steered by feedback, so the quality of your evaluation sets the ceiling on your optimization. Evals before opinions.

Conclusion

An execution trace isn’t just exhaust. It can be training data.

Parametric methods compress experience into weights you can’t read. Text-based methods compress it into instructions you can. You can open the optimized prompt, see the lesson the system learned, and decide whether you agree with it.

When an LLM can read its own attempts, understand the grader’s complaint, and rewrite the instruction that led it astray, prompt optimization starts to look less like wordsmithing and more like learning.

This is all for now, hopefully this was useful. I’d appreciate any feedback or comments.

References

1. Lakshya A. Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, 2025.
2. DSPy GEPA documentation.
3. Mert Yuksekgonul et al., Optimizing generative AI by backpropagating language model feedback, 2025.
4. Zhihong Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO), 2024.
5. Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023.