01 Summary.
Learning that our data scientist users felt guilty when not being able to support ML engineers with infrastructure issues (especially on small data science teams) might seem like a foot note to others. But to me, it reads as crucial opportunity being missed. The timing was right. Roadmap planning for the next quarter was coming up, and there was space for riskier projects.
-
The Ask
Data Scientists and ML Engineers sit on opposite sides of a brittle handoff. When pipelines fail, neither has the full context the other needs.
-
The Approach
Start shifting our product philosophy. Instead of assuming that providing more technology will fix the user experience, move to create UX that supports users through difficult events like pipeline failures.
-
The Outcome
I designed an AI assistant that reads the logs and the docs, then suggests the appropriate fixes. Moreover, shifted the entire point of view of the product team.
Project Resources
1 Week
1 Designer
1 Workshop
1 Dev intern
1 ChatGPT model trained on our docs
02 Refrmaing our shared perspective
For years, there was a mad dash to create parity in technical capability between our web app and our command line. This focus on CLI parity had the unintended consequence of creating a preference towards ML engineers instead of creating a UX that supported an ecosystem of personas. That alienated a whole group of users, and blocked our land and expand business strategy. If our ML engineer persona can’t convince their closest partners, they wouldn’t convince other teams to adopt our product.
The business case
$10,000s/day Cost of running a pipeline
4-7 days Typical pipeline runtime
Important usage stats
100s sessions per week During pipeline creation or debugging
1-10 sessions per week During stable pipeline performance
Create a UX that resolves events, rather than a technology showcase.
The leadership team came in wanting an AI feature. Research said 10 out of 10 participants named miscommunication between Data Scientists and ML Engineers as a major issue — and the strongest emotional signal was Data Scientists feeling guilty, not stuck.
- A clean “ship an AI feature” brief
- The right user problem to solve with AI
- Outcome
- AI stayed the response — but the problem statement became “reduce the cost of a missed handoff,” not “add AI to the product.”
Internal Signals
- —Cross-functional team (Product + Dev + me) needed a problem statement that wasn’t “AI assistant” — otherwise scope creep would be infinite and success would be undefined.
- —One-week timeframe meant the problem statement had to be sharp enough to design and prototype in days. “Miscommunication between two specific roles” was that sharp.
- —Big-Ideas exercise later weighed AI against alternatives. That weighing only worked because the problem was framed as a communication gap, not a missing feature.
External Signals
- —10 / 10 interviews independently surfaced miscommunication. That kind of unanimity changes the brief — it means you’re looking at the disease, not a symptom.
- —Emotional language (“guilt”) came from Data Scientists unprompted. The pain wasn’t “I lack tools,” it was “I’m letting my teammate down.” That’s a relationship problem.
- —$10k/day & 4–7-day runs meant every missed handoff carried real dollars and real time. The business case for the reframe wrote itself.
It’s inefficient process, not technical capacity.
The team had just spent considerable resources deeply expanding our documentation, and implementing ChatGPT to parse the docs for our users. Our leadership team wanted to use our docs as a training manual for the product. The hope was that ML engineers and data scientists would create pipelines with fewer errors by becoming more educated about the product and how it works with its supporting ecosystem.
03 Grooming a BHAG
I got started with behavioral design with my very first design job in 2017 which was in the healthcare sector. The decision to rework the docs was correct, but has an unintended message of “this is how you get better at using the product.” What we need for our data scientist persona is experiences that say, “this is how we can support you in this moment.”
Run our workshop through a behavioral design lens, not a purely ideation one.
I used IBM’s Big Ideas exercise to weigh concepts — but I enriched it with the Behavioral Design Toolkit from the Illinois Institute of Technology. The pain we were solving was emotional (guilt, frustration), and a feature-cost-benefit lens alone would have missed that.
- A faster, lighter ideation cadence
- Concepts pressure-tested for behavior change
- Outcome
- AI assistant won as the “Big Idea” — not because it scored highest on features, but because it removed the cost of asking for help.
Internal Signals
- —Big Ideas alone tends to reward novelty and ambition. With one week to ship a PoC, ambition needed to be checked by “will users actually change behavior?”
- —Behavioral Design Toolkit gave us shared vocabulary for friction, motivation, and ability — concepts the engineering side of the team understood faster than UX jargon.
- —I’d run this combination before. Bringing a known method into a one-week project reduced facilitation risk.
External Signals
- —Guilt as the pain point: people don’t fix guilt with features, they fix it by lowering the social cost of asking. That insight maps directly to behavioral-design’s “reduce friction” lever.
- —Interview language kept landing on relationship verbs — “ping,” “interrupt,” “bother.” Those are behavioral signals, not feature requests.
- —Cross-domain analogues: developer tools that succeeded in similar spaces (Sentry, Datadog AI summaries) won on lowering the cost of asking the next question, not on raw analytical power.
The big hairy audacious goal.
When our users are in a state of stress — hemorrhaging tens of thousands of dollars a day, the last thing they need to hear is “get better at the product.” The workshop led to this idea of an AI assistant that parses the logs, the pipeline specifications, and historical performance data to identify why the pipeline isn’t working.
Trimmed, scoped, and achievable.
6 screens and a user flow on rails, that was the result. An “explain logs” button where users would already be during debugging invited them in to quickly get a summary of issues, and provided them the docs and actions that they could take to remedy the situation. This would shift the conversation between personas from “I don’t know how to fix this.” to “here’s what I know how to fix, and this is where I need your help.”
04 Proving it for real
With one week on the clock and a single developer as the only engineering partner, the team faced a choice that would shape what the eventual user test could actually measure: build something that looked real, or build something that behaved like the real thing. Design is mostly judgment, not pixels — and this was the highest-leverage judgment call left in the project.
Ship a coded proof-of-concept, not a Figma prototype.
A clickable Figma would have tested layout and copy. I argued for a fully-coded PoC with a RAG model trained on real internal docs — because the user behaviors we needed to study only show up when the AI’s answers feel real, not scripted.
- A polished, fast Figma walkthrough
- A test that surfaces actual AI behaviors
- Outcome
- Built end-to-end in a week. Now ready for in-depth user testing against the behavior-change spec, not just usability.
Internal Signals
- —Development partner was already capable of standing up a RAG model on the team’s docs. The marginal cost of “real” over “scripted” was small.
- —Edge-case planning token limits, model switching, chat history reversion — only matters if you’re testing a real assistant. A Figma can’t simulate any of it.
- —Trust-in-AI was the riskiest unknown. The team needed signal on whether participants would accept the AI’s answers, not whether they could tap a button.
External Signals
- —AI quality is non-linear: Figma can’t show that a hallucinated answer breaks trust for the next 10 real ones. A coded PoC reveals that immediately.
- —RAG-specific quirks citations, refusal language, suggested follow-ups — only render fidelity in a working system. Users react to them differently than to mocked UI strings.
- —Stakeholder demos: a working PoC is far easier to fund forward than a Figma. The next round of investment depends on what the team sees, not what they imagine.
Real answers, not scripted ones.
Token limits, model switching, and chat history reversion were all scoped into the build — not because they were glamorous, but because a coded PoC that couldn’t handle them wouldn’t have told the team anything real about whether users could trust the assistant’s answers.
05 Lessons learned for the next time.
This project didn’t get the ending most case studies aim for. The PoC
shipped on a Friday; within the week, the entire AI Solutions department —
including this team — was laid off. The user testing that would have
validated or challenged the behavior-change hypothesis never happened.
What I’d carry forward: the reframe and the behavioral design insight — that
guilt isn’t solved with features, it’s solved by lowering the social cost of
asking for help — were the most valuable outputs of this project, more valuable
than the prototype itself. If I’d known how little runway was left, I’d have
spent less of the week polishing the coded PoC and more time documenting the
research insight in a form that could travel independent of the team that produced
it. Good research outlives bad timing. The artifact didn’t get the chance to
prove itself, but the thinking behind it should have been built to survive the
team that built it.