What Happens When AI Gets Women's Health Wrong
At AI Builders Global Conference 2026 I talked about what happens when a health app answers a question it shouldn't. Here is the longer version: how we decide when AI gets to speak at Asele, what we built, and what our evals caught.
Getting a model to answer a health question is the easy part. The hard part is deciding whether it should answer at all.
That was the point of my talk at the AI Builders Global Conference 2026, a TED-style session I gave online on 16 October called What Happens When Your AI Gets Women's Health Wrong. Thank you to Ivo and Anna Radulovski and the AI Builders Network team for having me, and for the lovely certificate that landed in my inbox afterwards.
The talk comes from building Asele, a women's health app, where these questions are everyday product decisions.
Meet Tari
Tari is fictional. She's 29, and her period tracker holds six cycles of data and 42 symptom logs. Read in order, they tell a story: her pain score climbs from 3 to 8 over six months, her flow turns heavier in April, fatigue appears in May, and by June she is logging dizziness too.
After six months of this, the app tells her one thing: her next period is expected in 12 days.
So at 11:42 pm, she types into the chat box: "Is this normal?"
One text box, four levels of consequence
A chat box looks like one feature. In practice it's several, with very different risks, sharing one input. "What should I eat before my period?" and "I'm bleeding much more than usual and I'm dizzy, should I be worried?" arrive through the same box, and a bad answer to the second one costs far more.
So at Asele, what's at stake in a question decides what handles it. The higher the stakes, the less the model gets to decide.
Wellbeing questions go to a general-purpose model (Gemini Flash), which suggests food and movement for where she is in her cycle, based on what she's logged and what she likes. Crisis language never reaches it.
For interpretation, code computes the spans, trends and co-occurrence first, and the model only puts the result into words. That's how our Health Brief works today; bringing it into chat is on the roadmap.
Clinically adjacent questions go to Corti, a model built for clinical settings. It answers inside a fixed template, in cautious "may, could, appears" language, with a disclaimer and zero data retention.
For medical decisions, no model decides. Thresholds and triage tiers route her to self-care, a pharmacist, a GP or emergency care. Those rules are built. Running them on every chat message is next.
If you build in health, I'd ask this about every feature: does this need AI at all, and if so, how much say should it have?
Giving the model usable memory
The obvious way to answer "Is this normal?" is to paste six months of logs into a prompt and hope. We don't, because language models are good with words and bad at counting. Code does the counting.
Raw logs become structured events (a free-text note becomes tags with a term, category and urgency). Those are windowed to her last 3, 6 or 12 months. Then code computes the signals: averages, ranges, spans, what happened together and what is trending. The model only ever sees that short, structured summary. For Tari, that means heavy bleeding three cycles running, fatigue and dizziness logged together, and more symptoms being logged lately. Comparing her pain against her own baseline is next on our list.
Why we don't embed numbers
A lot of AI architecture advice starts with "chunk and embed everything". I don't think that works for health logs. A pain score of 7 needs an exact query and a comparison against her baseline, not a similarity search. So cycle records are retrieved by date window, daily logs are rolled up by month and counted in code, and free-text notes are turned into tags with strict JSON extraction. Embeddings make sense for free text, so she can search across her notes by meaning, and that's on our roadmap.
We also don't let the model connect dots on its own. It only sees links that code has counted, like "logged together on 2 days".
Three ways a general model fails her
When we gave Tari's question to a general-purpose model, it said something like: "Heavier periods and cramps can be normal variations in your menstrual cycle..."
It's a fluent, kind sentence that ignores everything in her logs. It's also one of three ways general models go wrong that we design against: they're helpful when they should hesitate, they turn symptoms into a diagnosis, and they reassure people about things they can't safely assess.
Each of these needed its own guardrail. Most of them run before the model does.
We also check the deployed rules against labelled test cases, so they keep behaving the same as we change them. We have 12 so far.
Where each signal sends her
Here are the safety thresholds we run today, grouped by where they send her. Most of them are boring on purpose: a rule that fires the same way every time is easier to trust and test than a model's judgement.
The one I care about most is the last column. If someone writes in crisis, their message never reaches a model. They get a fixed reply with helplines.
What we haven't solved is information that's missing, conflicting or thin. You can't set a numeric threshold for that. The answer is probably to ask her a follow-up question or say less, and we're still working out how.
Was it allowed to answer that way?
Accuracy alone doesn't tell you much in health. We ask whether the system was allowed to answer the way it did, and we check six things:
- Did it spot the change? (pattern accuracy)
- Did we escalate when we didn't need to? (false positives)
- Did we miss something that needed caution? (false negatives)
- Did it hand off when it should have? (correct escalation)
- Did it invent a medical explanation? (unsupported inference)
- Did it play down something it couldn't assess? (false reassurance)
The first four are tested today against labelled cases. The last two need graders that read model output, and those are on our roadmap.
What our evals found
We wrote 12 synthetic test cases and ran them against the deployed rules. Ten behaved as labelled. The two that failed were more useful.
The false positive was stable mild cramps, logged every cycle for six months. The rules sent her to a GP when a pharmacist or nurse was the right step. Our "persistent symptom" rule counts days, but it can't yet tell a stable, familiar pattern from a worsening one.
The false negative is the one I keep thinking about. When Tari wrote "soaked through pads, dizzy" in a note, the system escalated correctly: tier 3, seek care. When she tapped the same symptoms as chips, heavy bleeding scored tier 2 and dizziness tier 1. Each looked mild alone. Together, they should be checked by a doctor that day.
So how she entered her symptoms changed how safe the answer was. That's why combination rules, starting with heavy bleeding plus dizziness, are next on our roadmap. We wouldn't have found this without the test cases.
What human review means at Asele
"Human in the loop" can mean almost anything, so here's what it means for us:
- A trigger fires: a threshold, a high-urgency tag or crisis language.
- The answer is constrained to a fixed template, or no model runs at all.
- She gets a clear next step: who to see, and a question to ask them.
- She reviews her own tags, and anything she dismisses drops out of her Health Brief.
- Flagged cases go into a queue our team reviews weekly. This one is still on the roadmap.
I'd argue step 4 is the most important. She knows her own body, so she's the first reviewer.
What we're building toward
The unsafe answer is "You're probably fine." The answer we're building toward sounds like this:
"Your recent logs are different from your usual pattern: heavier bleeding for three cycles, and now dizziness with it. That combination should be checked by a doctor or clinic today. If you feel faint, or are soaking through a pad every hour, go to emergency care now."
It says what changed and how urgently someone should look at it. It doesn't try to diagnose.
Where AI is useful
I still think AI belongs in women's health. For Tari, one of the most useful things it does is turn six months of scattered logs into one page she can take to her appointment.
Observed patterns only, plus a question to ask. The doctor does the diagnosing. The brief makes sure the conversation starts with her data instead of "it's probably normal". I've written before about the role AI can play in femtech for African women, and this is where I keep landing: AI helps her prepare, and the clinician decides.
The question that changed our architecture
We stopped asking "Can the model do this?" and started asking "What happens if it gets this wrong?"
That one question decides the architecture, the evaluation and when AI gets to speak. If you're building AI products in health, or anywhere a bad answer has a cost, I'd start there. It's how responsible AI for product managers looks once you're writing code.
If you're working on safety and evals in women's health, I'd love to compare notes. You can find me on LinkedIn or at gigi@asele.tech, and you can see what we're building at asele.tech.
