Jody
Allard

Our initial proposal was to adapt content guidelines into core criteria for an LLM-as-judge evaluation. The learnings reshaped the approach: too many criteria degrade output quality, and highly context-dependent rules are hard for models to evaluate reliably.

The final framework used a highly focused set of rule-based core criteria, a weighted evaluation strategy, and critical fail criteria to trigger human review — with an adaptable rubric for feature-specific evaluation post-launch.

We also identified a gap: no heuristic existed for the ethical use of AI in our content. We built one.

The work on match explanations became the foundation for how Thumbtack approaches generative AI content more broadly.

I worked closely with Legal and Creator Operations to ensure every message was accurate, compliant, and timed correctly. Tone moved from informational to increasingly urgent — but never alarmist.

I also negotiated with the support team to route creators directly to a support ticket form when funds were about to expire. This created a clear, actionable path for creators who genuinely couldn't resolve the issue themselves — and reduced creator frustration.

Every message named the specific amount at risk. Making the loss concrete was critical to driving action without manufactured panic.

I started with a Custom GPT on chatgpt.com — no infrastructure, just me and the system prompt. I spent weeks refining the instructions until the model reliably applied Thumbtack's guidelines, asked the right clarifying questions before writing emails, and always gave full rewrites instead of inline edits.

Once the prompt was right, I moved to building the plugin. I used Claude Code — an AI coding tool — to write all four files, describing the behavior I wanted in plain language rather than writing code myself. The whole thing runs on Cloudflare Workers to keep the API key secure.

This project is also how I learned that the skills content designers already have — defining constraints, writing to a brief, iterating on output — are exactly the skills that make someone good at building AI tools.