Case study / Intel / Validity Engage
Intel shows email marketers how their competitors' emails perform. I led product and design, and took it from a bundled add-on idea to 100+ customers.
Problem
No product on the market shows how a competitor's email program performs. Validity owned data that could, but the business had never sold anything in this space.
Bet
AI could turn a huge, messy data pipeline into a few trustworthy (and juicyExample insight“Lululemon uses shorter subject lines with personalization at the beginning, not the end, to drive a higher open rate than you.”) insights that marketers would pay 💰 to see.
Outcome
Intel became the hook in Validity Engage demos. Prospects ask to click into insights and request recordings to revisit them. 100+ customers since the June pilot, and 8 of 10 pilots converted.
Proving demand
Manual reports sold the pilots and told us to staff the team
Before any product existed, we pulled competitive reports by hand, polished them with AI, and sent them to prospects. The reports gave ideal-customer-profile companies a reason to join a pilot. They also showed that demand was real enough to justify a full team.
- Before launch Manual reports, polished with AI, sent to prospects
- Pilot prep Account managers generate reports themselves
- June 30 Pilot launch with 10 ICP customers
- October 1 General availability
The pivot
The first concept was a dashboard. Plot twist, everyone hated it.
My first concept was a number-heavy dashboard that left the user to do the interpreting. Then I showed two large customers a mock-up built around six insights, and their reaction settled it. Intel has been through three major design iterations since.
With a dashboard, the interface is the product. With insights, the hard part moves under the hood: turning huge volumes of data into something people trust.
Trust
A non-deterministic product needs a quality system, so we’re building one in three stages
Customers cannot rely on output we cannot consistently verify. The system grew with the product.
Humans first
Our professional services team already curated reports for customers, so they became the first line of defense. Pilot customers decided what was good enough.
A feedback loop
Sales and customer success flagged poor quality and thin data in Slack. An automation turned those notes into a digest. I triaged it into Jira and prioritized what was achievable on data quality.
Evals
- Now Build a golden set from insights customers liked and bookmarked
- Next Run evals in Mastra Studio on every model and prompt change
- Then Automate the runs with our QA engineers
The signal that we were on track came in demos. Customers started saying “click on this” and “ask the agent this,” and asked for recordings so they could revisit the insights.
Why evals became non-negotiable
Adding languages made insights slightly worse
When we expanded beyond English, our filters for unreliable data threw out more of it. Each insight lost 10 to 15% of its data points, which was enough to make it noticeably weaker.
That’s why every model or prompt change will have to prove it didn’t make insights worse.
Data
Reducing 150 million emails a day to real brands meant choosing what not to cover
I oversaw the data pipeline team’s work to dedupe and cull a very large intake down to real brands, then turn it into insights. The funnel below is illustrative and not to scale.
- Raw intake about 150M emails a day from owned traps and managed inboxes
- Deduplicate collapse repeats of the same send
- Cull to real brands Fortune 1000 first
- Classify in-house ML models by industry
- Ranked insights what the agent shows
Fortune 1000 first
Insights are only useful if they cover brands a marketer recognizes as competition. Big brands came first.
Industries with the deepest coverage
Our data is strongest where email is a primary channel: hotel and leisure, retail, e-commerce, and food and beverage. I built the in-house ML models for those industries first.
Broadcast before transactional and journeys
The models were trained mostly on public broadcast emails. Starting there meant insights we knew would be accurate. Transactional and journey emails come later.
Monetization
I proposed pricing from our costs. The pricing lead moved it to customer value
Intel sells as a platform fee plus usage-based credits, so every feature decision was also a pricing decision. I partnered with our pricing and packaging lead on what customers should spend credits on.
My first recommendation tied credit usage to token complexity and COGS, with a multiplier. She pushed back: usage had to make sense from the customer’s view of value.
Big feature swings
Capabilities customers readily see as valuable, and happily spend credits on.
Repeatable actions
Custom jobs and reports that run automatically and push to Slack or an MCP connector, which builds habitual usage.
Data quality, as its own stream
Much of our work improved data quality, and customers will not buy that directly. It needed to be planned and funded separately from features.
Results
The product found its market, and the team grew around it
Core team today: 2 engineers, 2 data science engineers and 1 product data scientist.
Hindsight
What I would do differently
Start evals before the pilots
The language expansion showed how fast quality can slip. A golden set from day one would have caught it sooner.
Learn what a great insight looks like earlier
In-product feedback is ambiguous. A low rating can mean a bad insight, a dislike of the feature, or an insight that missed the customer’s goals. I would build a feedback system early that separates those cases.
Bring pricing and packaging in sooner
Understanding how Intel would be sold earlier would have shaped what we built first.