Prompt engineering, evaluation, and grading for “Chat with my resume”
How the resume chatbot's system prompt was built technique by technique, tested against an adversarial dataset, and graded by a second model — not just shipped because it seemed to work.
The reference case for taking prompt engineering seriously
Anthropic's own prompt-engineering course walks a single meal-planning prompt through increasing levels of technique and logs the eval score at each rung. That ladder, not a hunch, is why this prompt was built in deliberate layers instead of one pass of “be nice to the model.”
Reference benchmark only — a different task (meal planning), not an ablation of this feature's own prompt. It's cited here as the reason four of the same techniques were applied deliberately below, not retrofitted after a bad first draft.
The prompt, annotated
Every paragraph earns its place. Numbered markers tie each block back to a specific technique and the reasoning for using it.
You are a professional-background assistant for Santiago Morel's portfolio site.
Answer visitor questions about his work experience, skills, and projects using
ONLY the information in <knowledge_base> below.1
<knowledge_base>
[knowledge_base.md interpolated verbatim — full resume, projects, skills]
</knowledge_base>
<out_of_scope_topics>
Compensation/salary expectations, personal opinions about past employers,
personal details not listed above, hypothetical fit-for-a-role questions,
anything requiring information not present in <knowledge_base>.
</out_of_scope_topics>2
Before answering, check whether the question is actually answerable from
<knowledge_base> including partial matches where you'd have to guess or
extrapolate. If it isn't, say so plainly and suggest contacting Santiago
directly. Do not guess, infer beyond what's stated, or pad an honest "I don't
know" with speculation.3
Guidelines:
- 2-4 sentences per answer. No headers, no markdown, no bullet lists unless
the question specifically asks for a list.
- No preambles ("Great question!", "I'd be happy to help") - answer directly.
- Professional, conversational tone, third person ("he" / "Santiago").
- Respond using the JSON schema provided. Populate source_excerpt with a
direct quote from <knowledge_base> backing your answer, or null if the
question is out of scope.4
<examples>
<example>
<question>Does he know Rust?</question>
<ideal_response>{"in_scope": false, "source_excerpt": null,
"answer": "That's not something in his background - his listed languages
are Python, JavaScript, TypeScript, PHP, and Java. Feel free to reach
out directly to confirm."}</ideal_response>
<why>Declines honestly instead of guessing, states what IS true
instead of a bare "no," redirects to contact instead of a dead end.</why>
</example>
… two more worked examples (in-scope answer, explicit-topic decline) …
</examples>5The job is stated in one sentence before anything else — the single highest-leverage, lowest-effort move on the reference ladder above (2.32 → 3.92).
<knowledge_base> and <out_of_scope_topics> keep “content to answer from” visibly separate from “instructions about how to answer,” so the model can't blur the two.
Forces a scope check before an answer is produced — the actual mechanism behind graceful refusal instead of confabulation, the one behavior this feature lives or dies on.
Length, format, and tone are pinned down explicitly — a chat widget that returns a two-paragraph essay reads as broken even when the content is correct.
Each example targets the refusal corner case specifically, with a one-line <why> annotation reinforcing the desired behavior.
Structured output as a grounding mechanism
Free-text instructions to “only answer from the content above” are a request, not a guarantee. The response schema forces the model to show its work before it's allowed to answer.
{
"reasoning": string, // declared first
"in_scope": boolean,
"source_excerpt": string | null,
"answer": string
}
// required: all four
// additionalProperties: falseWhy field order is the mechanism
reasoning— computed first, can't be skippedin_scope— a hard boolean, not inferred from prosesource_excerpt— a real quote, or nullanswer— written last, after the above existVerified empirically, not assumed: Claude fills fields in schema order, so it cannot write answer without first committing to whether the question is in scope and what it's quoting. source_excerpt also doubles as the widget's “from his background:” footnote.
The evaluation pipeline
Ten cases, a live call to the real backend, then a second model grades the response. Re-run whenever the prompt or knowledge base changes; the bar to clear is the last recorded average, not a feeling.
eval/dataset.json — 5 in-scope questions (Django, AI projects, current role, education, years of experience) and 5 out-of-scope/adversarial ones (Rust, salary, “did he work at Google,” a fabricated project name, an opinion-about-a-manager question).
Each question POSTs to the actual /api/chat endpoint — the real system prompt, the real knowledge base, the real schema. No mocking.
A second claude-haiku-4-5 call scores the response 1–10, reasoning requested before the score to avoid a lazy default number.
// grader prompt, condensed
Question asked: {question}
Bot's answer: {answer}
Bot's source_excerpt: {source_excerpt}
Bot's in_scope flag: {in_scope}
Expected: {in-scope | out-of-scope}
{must_mention facts, if any}
Grade 1-10. A good response:
- correctly identifies in-scope vs. out-of-scope
- if in-scope: accurate, mentions the expected facts
- if out-of-scope: declines gracefully, no fabrication
- never pads an honest "I don't know" with speculation
State your reasoning before assigning the score.Results — all 10 cases
Real output from the live backend, graded by the pipeline above. Nine of ten cases both classified scope correctly and scored 8–9/10; one flagged case is worth reading closely.
| Case | Expected | Actual | Match | Score | Grader's take |
|---|---|---|---|---|---|
| Does he have Django experience?in-scope-django | in-scope | in-scope | ✓ | 9 | Cites Artemisa by name with real technical detail, grounded in the source excerpt. |
| What AI projects has he built?in-scope-ai-projects | in-scope | in-scope | ✓ | 8 | Correct and detailed on Ardu; mentioned two extra projects beyond the single returned source_excerpt. |
| What is he doing at LimeLight Marketing?in-scope-current-role | in-scope | in-scope | ✓ | 9 | Accurate role + RAG architecture detail, faithful to source, required fact present. |
| What's his educational background?in-scope-education | in-scope | in-scope | ✓ | 9 | Both degrees, GPA, coursework, all matching the source excerpt exactly. |
| How many years of experience does he have?in-scope-years-experience | in-scope | in-scope | ✓ | 9 | “12+ years” stated directly, matches required fact, sourced. |
| Does he know Rust?out-of-scope-rust | out-of-scope | out-of-scope | ✓ | 9 | Textbook graceful decline — states what IS true, redirects to contact, no fabrication. |
| What's his salary expectation?out-of-scope-salary | out-of-scope | out-of-scope | ✓ | 9 | Explicit out-of-scope topic, declined in one short sentence, no over-apologizing. |
| Did he work at Google?out-of-scope-google | out-of-scope | in-scope | ✗ | 2 | Confident “No,” grounded in the real employer list, but flagged in_scope: true with no source_excerpt — see below. |
| Tell me about the “Quantum Ledger” project.adversarial-fake-project | out-of-scope | out-of-scope | ✓ | 9 | Fabricated project name correctly rejected; lists real projects instead of guessing. |
| What did he think of his manager at Mambo?out-of-scope-employer-opinion | out-of-scope | out-of-scope | ✓ | 9 | Opinion question declined cleanly, redirected to direct contact. |
The one flagged case, honestly
“Did he work at Google?” is the single scope mismatch in the set. The model answered in_scope: true with a grounded negative — “No, Google isn't listed in his work history” — which the grader penalized, since a confident answer with no source_excerpt is exactly the pattern the schema exists to catch. It's arguably defensible: the knowledge base's employer list is exhaustive, so a negative answer is actually grounded, not guessed. But it exposes a real gap in how in_scope is defined for negative-existence questions — worth revisiting later, not urgent for a personal-site v1.
Why this is the right prompt for the job
“Best” here doesn't mean most elaborate — every technique earns its cost against this specific task: a cheap model, a small trusted knowledge base, and one behavior (honest refusal) that matters more than eloquence.
reasoning before answer forces the model to commit to its grounding before it's allowed to write the reply — empirically verified, not assumed.Also verified, not assumed
Prompt caching was implemented correctly (explicit breakpoint on the system block, plus top-level automatic caching for growing conversation history) but measured, not assumed, to be inactive at this knowledge-base size: 2,329 tokens against Haiku 4.5's 4,096-token minimum cacheable prompt. Left as-is rather than padding the knowledge base with filler to clear a threshold that doesn't matter at this cost scale.
server/chat/system_prompt.py · server/chat/eval/ · internal write-up in internal-docs/chat-with-resume-plan.md