The prompt that stopped working

You spent an afternoon getting a prompt just right — the wording, the order of the instructions, the "think step by step" nudge tacked onto the end — and it worked. Then you pointed the same prompt at a different model, or the vendor shipped a routine update, and the accuracy quietly dropped ten points. Nothing in your code changed. The instructions you tuned by hand were never really an interface; they were a set of coordinates that happened to land in the right place for one specific model, on one specific day.

That's the situation most teams treat as normal, because it's the situation prompt engineering as commonly practiced puts you in. Phrases like "be concise" or "think step by step" or "you are an expert" aren't documented behaviors of a language model API — they're folklore, passed from model to model, discovered by trial and error, only loosely portable. They work because a model's training happened to reward them, not because the model exposes any contract that guarantees they'll keep working tomorrow, or on a different model entirely.

DSPy starts from a different premise: interacting with a language model is a programming problem, not a prompt-tuning problem. The bet is that you can separate what you want the model to do from how the model gets coaxed into doing it — and once those two things are separated, the "how" stops being something you hand-tune by staring at outputs and starts being something you can search, measure, and optimize the way you'd optimize any other piece of software.

That distinction — what versus how — is the spine of this entire course, and it starts with naming the problem precisely, because you can't fix a trap you haven't described.

Trial, error, and no signal

Here's what manual prompt engineering actually looks like, stripped of the marketing language. You write a prompt, run it against a handful of examples, and read the outputs. Something's off — the model wraps its answer in unwanted preamble, or misses a case you care about — so you reword a sentence, add an example, tighten an instruction, and run it again. You keep doing that until the outputs look good enough to ship.

The trouble is what "good enough" is actually measuring. You're eyeballing a handful of outputs, not scoring them against a metric, so there's no systematic signal telling you whether your last edit made things better or worse beyond the narrow slice of examples you happen to be staring at right now. A phrasing change that fixes the three cases in front of you can just as easily degrade twenty cases you didn't think to check, and you won't find out until a user does.

None of that hard-won tuning travels, either. The prompt you spent a week polishing for one model is calibrated to that model's particular quirks — its tokenizer, its instruction-following habits, the exact phrasing patterns its training happened to reward. Swap in a different model, or even a new version of the same one, and the wording that used to produce clean, well-formatted answers can start producing rambling ones, or start ignoring your formatting instructions altogether.

And every fix is a gamble against your own past work, because there's no metric tracking the full picture — tightening the prompt for a new edge case carries a real risk of quietly breaking something you already fixed two iterations ago, a regression you'll only discover the next time a user hits that older case. This is the trap: an activity that feels like engineering but has none of engineering's feedback loops, because the thing you're editing is a string, not a system. DSPy's answer isn't a bigger library of clever phrasings — it's four pieces that give this loop something it never had: a signature, a module, a metric, and an optimizer.

model = "gpt-4"  # tuned by hand for this specific model


def build_prompt(ticket_text):
    if model == "gpt-4":
        return (
            "You are a support triage assistant. Read the"
            " ticket and reply with exactly one word:"
            " URGENT, NORMAL, or LOW.\n\nTicket:"
            f" {ticket_text}\nUrgency:"
        )
    else:
        # a different phrasing was needed here
        return (
            f"Ticket: {ticket_text}\nHow urgent is this?"
            " One word.\nUrgency:"
        )


response = call_llm(
    build_prompt(
        "Production database is down, customers can't check"
        " out."
    )
)
# Works today: GPT-4 replies "URGENT". Swap the model, or
# its version, and it can start replying "This is an urgent
# issue." -- still correct in spirit, but now the downstream
# parser expecting a single word breaks, and nothing here
# tells you why.

Four pieces, one answer

A signature is the first piece, and it does the job a hand-written prompt used to do badly: it declares what goes in and what comes out, without saying a word about how to phrase the request. You might write a signature as simply as question -> answer, or something more specific like context, question -> answer, reasoning — and that's the entire contract. No "you are a helpful assistant," no "think step by step," just the shape of the input and the shape you expect back.

A module is what actually fulfills that contract, and DSPy gives you more than one strategy to choose from. dspy.Predict takes the signature and makes the most direct call possible — ask, get an answer. dspy.ChainOfThought takes the same signature and inserts a reasoning step before the answer, prompting the model to think before it commits. Swap one for the other and the contract doesn't change; only the strategy for satisfying it does.

A metric is where DSPy stops asking you to trust your eyes. It's ordinary code — a function that takes an example and the model's output and returns a score for whether that output was actually good. Once you have a metric, "good enough" stops being a feeling you get from skimming five outputs and becomes a number you can compute across however many examples you have.

And an optimizer — DSPy calls these teleprompters — is what turns that metric into leverage. Give it your program, your metric, and a handful of training examples, and it automatically searches for better instructions and better few-shot examples to slot into your modules, systematically trying variations and keeping what scores higher instead of you rewording a string and hoping. This is the part of the work that prompt engineering asked you to do by hand; DSPy asks a search procedure to do it instead. What ties these four pieces together into something you'd actually call a program, rather than four disconnected concepts, is that every one of them is a plain Python object — which is where composability comes in.

import dspy

# Signature: declares the contract, not the phrasing Module:
# dspy.Predict runs that signature against real input
triage = dspy.Predict("ticket_text -> urgency")
triage(
    ticket_text=(
        "Production database is down, customers can't check"
        " out."
    )
)
# Prediction(urgency='URGENT')


# Metric: turns "good enough" into a number -- urgency is a
# fixed label (URGENT/NORMAL/LOW), so exact match is the
# right call here. Free-text fields need a softer metric --
# that's Lesson 6.
def urgency_correct(example, prediction, trace=None):
    return example.urgency == prediction.urgency


# Optimizer: searches for better instructions and few-shot
# examples, scored by urgency_correct() across the training
# set
optimizer = dspy.BootstrapFewShot(metric=urgency_correct)
compiled_triage = optimizer.compile(
    triage, trainset=trainset
)

# Everything above runs once, offline -- by hand, in a
# notebook, or in CI. It never runs inside a live request.
compiled_triage.save("compiled_triage.json")

# In the app: no optimizer here, just load the compiled
# state onto a fresh, unoptimized module and run it
triage = dspy.Predict("ticket_text -> urgency")
triage.load("compiled_triage.json")
triage(
    ticket_text=(
        "Production database is down, customers can't check"
        " out."
    )
)

Notice the shape of that last part: the optimizer never appears again after compile() finishes. Compiling is a build-time step you run once against your training set; loading the result back into a plain module is the only thing your application does at request time. Lesson 10 covers that split in full.

Programs, not paragraphs

Here's the detail that makes "programming" the honest word for what you're doing in DSPy, rather than a marketing flourish: a DSPy program is a Python class. You write it by subclassing dspy.Module and defining a forward() method, in a pattern that will look immediately familiar if you've ever written a PyTorch model — layers become modules, and the forward pass becomes the flow of data through them.

Because a module is just an object, you can nest modules inside modules the same way you'd nest any other component. A retrieval step can feed its output into a reasoning step, which feeds its output into an answer-formatting step, and the class wiring them together is still readable top to bottom as ordinary control flow — no string concatenation, no template logic buried in a prompt file you have to reverse-engineer.

That composability is what actually earns the shift in vocabulary. Prompt engineering, at its core, is editing one long string and hoping the parts don't interfere with each other. Programming with DSPy is composing units that each have a defined contract — a signature in, a signature out — the same discipline you'd apply to any function or class in a codebase you intend to maintain.

And because it's genuinely Python, it gets everything Python gets for free. You can set a breakpoint inside forward() and step through exactly what each module received and returned. You can write a unit test that feeds a fixed input to one module in isolation and asserts on its output, without running the rest of the pipeline. Try doing that to a paragraph of prompt text. The pipeline is composed, testable, and legible — but everything so far has described intent, and what actually executes that intent, against which model, is a separate question entirely.

import dspy


class RetrieveThenAnswer(dspy.Module):
    def __init__(self):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=3)
        self.generate_answer = dspy.ChainOfThought(
            "context, question -> answer"
        )

    def forward(self, question):
        context = self.retrieve(question).passages
        prediction = self.generate_answer(
            context=context, question=question
        )
        return prediction


program = RetrieveThenAnswer()
program(
    question=(
        "What's our refund window for items damaged in"
        " transit?"
    )
)

The model is a setting, not a rewrite

Go back to where this lesson started: a prompt tuned by hand for one model that quietly breaks when you point it at another. That happens because a hand-written prompt bakes execution details — this model's preferred phrasing, this model's particular quirks — directly into the thing describing your intent. Change the model and you've changed the thing the prompt was calibrated against, without changing the prompt itself.

A DSPy program doesn't make that mistake, because it never mixed the two things together to begin with. Your signatures and modules describe what you want; the model you run them against is a separate, swappable setting, configured with a single line — dspy.configure(lm=...) — that sits outside the program itself. Point that line at a different model and the program's structure doesn't move.

What does move is what gets optimized for it. Re-run your optimizer against the new model and it searches again — new instructions, new few-shot examples — tuned to whatever that model actually responds well to, using the exact same metric and training examples you used before. You're not starting over from a blank page; you're re-running a search procedure against a new target.

That decoupling is worth more than convenience. It means the investment you make in a DSPy program — the signatures you defined, the modules you composed, the metric you wrote — isn't tied to any single vendor's model. Today's frontier model is not guaranteed to be next year's, and a program built this way survives that change; a hand-tuned prompt string typically doesn't. Put the four pieces together — signature, module, metric, optimizer — and you can already see the shape of the reframe this lesson opened with.

import dspy

triage = dspy.ChainOfThought("ticket_text -> urgency")

claude = dspy.LM("anthropic/claude-3-5-sonnet-20241022")
llama = dspy.LM("together_ai/meta-llama/Llama-3-70b")

dspy.configure(lm=claude)
triage(
    ticket_text=(
        "Production database is down, customers can't check"
        " out."
    )
)  # tuned for claude

dspy.configure(lm=llama)
triage(
    ticket_text=(
        "Production database is down, customers can't check"
        " out."
    )
)  # same program, new model

Programming, not prompting

Put the four pieces back together and the reframe from the start of this lesson stops being a slogan and starts being a description of an actual workflow. A signature declares the contract. A module picks a strategy for fulfilling it. A metric turns "is this good?" into a number instead of a feeling. And an optimizer uses that number to search for better instructions and examples automatically — doing the part of the job that used to be you, alone, rewording a string at midnight and hoping it holds.

None of this makes prompts disappear — a language model still needs instructions, still needs phrasing, still needs examples. What changes is who writes them and how they get improved. You describe intent in code; the optimizer handles the coaxing, guided by a metric you control and validated against examples you chose, and the result travels with you when the model underneath it changes.

That's the whole bet this course is asking you to take seriously: treat the model as a component you program against, not an oracle you negotiate with one clever phrase at a time. Everything from here is about learning to use the four pieces well, starting with the one you touch first in any DSPy program.

In the next lesson, we'll look closely at signatures — the declarative contract between your code and the language model, and the starting point for turning what you want into something DSPy can actually optimize.