AI Practice
AI Product Practice
Discovery, Evaluation & Responsible Deployment
My AI practice starts with three questions:
- How does this workflow really run?
- What does “good enough” mean here?
- Will the people using it trust the answer?
The case studies below are what happened when I asked them.
Feasibility, ethics & responsible deployment
Academic integrity and customer trust through workflow impact and significance thresholds. My work spans higher education, digital platforms, and public‑sector environments where the cost of getting it wrong lands on someone who did not choose the technology.
Model evaluation & significance
Evaluation measures how often the same answer comes back. Whether the input is a librarian or a language model, the discipline is identical: define “good enough,” know your significance threshold, and stop sampling when further data adds no value.
Case Studies
The first is what I've built. The three that follow are decisions I made about shipping — and not shipping.
Project Pinetree — Agentic AI, Built in the Open
Context: Project Pinetree is my self-directed operating framework for product, workflow and content management. It combines structured documentation, workflow design and AI-assisted analysis, so one maintained source produces every downstream document without version drift.
What shipped: The Pinetree CV Targeter, a public-facing MVP delivered within 48 hours from the same workflow patterns. It reads a CV against a job description and returns a targeted version, two fit scores, the gaps, and a record of every change. It runs entirely in the browser. The system prompt forbids inventing any fact, and every change is listed so you can check.
In progress: Further agentic workflows for analysis, sequencing and document generation, each held to the same test as the case studies that follow: understand the workflow first, agree what “good enough” means, and name where the model must not be trusted before anything ships.
Alpha release — known limitations are published, and feedback comes straight to me.
Try the Pinetree CV Targeter How it works, and what it does with your data See the Product Lab pipeline
Risk & Ethics — The Decision Not to Ship AI
Reading List & Discovery Platform — UK Higher Education
Context: Shortly after the public launch of GPT, my employer ran company‑wide AI training and a hack day to identify viable use cases. Two colleagues with doctorates in machine learning proved that hallucination risk was too high for academic users to trust the output.
What we found: Our senior AI engineer produced worked examples. Bibliographies and citation links were entirely fabricated — a critical failure mode for a product whose core function depends on citation accuracy.
Decision: An AI theme sat near the end of the roadmap, not yet planned. All three of us in product knew at once it had to wait. I made the call and secured firm agreement from my Head of Product: the tagging theme came first, as a hard prerequisite. Tagging gives every reading-list item consistent, library-controlled labels — verified data that any later AI feature could be grounded in and tested against.
Outcome: No AI shipped on unverified or fabricated data. A failure mode caught in a workshop, before any student saw it.
Product Judgement — AI‑Assisted Market Evaluation
Acquisition Workflow — Academic Content Distributor Integrations
Context: A major academic content aggregator announced a shift to a subscription model part‑way through our planned integration. I used AI to model downstream effects at speed — workflow disruption, cost exposure per institution, and cost/benefit under three pricing outcomes.
What I did: Stopped the integration. The modelling showed librarian acquisition workflows would be destabilised for a benefit that could evaporate at the supplier’s next pricing decision. AI compressed a multi‑week evaluation into days; the decision itself rested on product judgement and customer absorption capacity.
Outcome: A reversible decision made early, protecting customer workflows during supplier‑driven market uncertainty.
Evaluation & Significance — When Enough Is Enough
Discovery Programme — University Library Sector
Context: I ran 17 discovery calls on a single problem. Pure discovery needed fewer; I ran the full set because each call was also a chance to meet customers and thank people who volunteered their time.
Finding: No single call proved the need was widespread. The recurrence across calls did. That is prevalence — evidence you can prioritise a roadmap against.
Why this is an AI skill: Model evaluation follows the same logic. A prompt run ten times measures how often the same answer comes back.
Outcome: A deliberate, defended trade‑off — spending past the significance threshold for relationship value, with reasoning made explicit and agreed in advance.
How I Sequence AI Work
I work in this order: understand the existing workflow before proposing to change it; define what “good enough” means and how it will be measured; establish where the model must not be trusted; and only then design for adoption. The uncertainty that would kill the product gets reduced first.