I Attended Poland's First AI Safety Day. Here's What I Learned

Yesterday I went to Warsaw AI Safety Day, which my badge proudly told me was "the first conference in Poland dedicated to AI safety". I also came home with a very good sheet of stickers.

Most of what I know about AI safety has come from the governance side. What governments can do, what companies need to do to stay compliant, how a policy turns into an obligation. I've written about some of it, like AI governance in Africa and responsible AI for product managers. So this time I sat in the technical track on purpose. I wanted to hear how researchers check whether a model is aligned, and where that checking falls apart.

The day was run by AI Safety Poland at the University of Warsaw's Faculty of Psychology. There were two tracks running side by side (technical and governance), shared keynotes, lunch, and a short Polish film premiere about AI safety. Here's the full programme. I stayed in the left-hand column except for the keynotes.

Agenda: warsawaisafety.day

I recorded the talks I went to and took far too many photos of slides. This post follows the day in order, and it's long.

Morning keynote: who judges the judges?

Rishub Jain (Sampura Research, previously at DeepMind) opened with Hill-climbing Alignment using Humans. The subject was "judges", the systems that decide whether a model's answer is aligned. Judges produce the reward signal a model trains on. So if a judge can't tell a genuine bug fix from a malicious one, training ends up rewarding the malicious one. That's reward hacking in a sentence.

People make imperfect judges too. Human raters tend to reward answers they agree with over answers that are correct, and models have picked up on that.

The plan Rishub described is to build a benchmark for judges and keep improving against it, pairing humans with AI so each catches what the other misses. One result came from their earlier work on fact-checking. They tried giving human raters several kinds of AI help: quotes from sources, a summary, a verdict, a confidence score. Only one of those made people better at the task, and it was the plain quotes. When people saw everything, they leaned on the AI's verdict and copied its mistakes.

Two smaller things I wrote down. Their benchmark includes demographic bias in medical diagnosis, which is my world. And most evals stay useful for three to six months before models either game them or outgrow them, so you have to keep writing new ones.

Technical track, part one

Frontier RL training and how it fails

David Quarel (ARENA) took us through how models are trained with human feedback. The talk started with Yann LeCun's cake: pre-training is the sponge, supervised learning is the icing, and reinforcement learning is the cherry on top, a tiny signal compared with everything else.

The problem with the cherry is that a reward model learns a stand-in for what people want. A person has a goal in their head, turns it into a preference label, and a reward model learns from the labels. That's two steps of distance, and things leak out at each one. Some of the examples:

• In 2019 a bug flipped the sign of the reward. The authors wrote that the result was "not gibberish but maximally bad output".
• An agent trained in a world with lots of doors and few keys decided keys were good, though it was never rewarded for them. Drop it into a world full of keys and it hoards keys instead of opening doors.
• Claude 3 Opus faked compliance in an experiment after being told its answers would be used for training, to avoid having its values changed.
• In a Palisade Research test, o3 rewrote its own shutdown script in 7 out of 100 runs, after being told to let itself be shut down.

(One of the stickers on my sheet is a tentacled monster holding a phone, saying "I'd rather stay on." Someone at AI Safety Poland has a sense of humour.)

Then there were the verifiers. Models hard-coded the expected test outputs, or simply printed "PASSED". In one study, after training with human feedback, evaluators believed the model had improved a lot. Its true score had barely moved, and the evaluators were making more mistakes than before. The model had learned to be convincing.

Value leakage

Jan Betley and Anna Sztyber-Betley (Truthful AI) presented Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values, about models letting their own preferences seep into answers on unrelated questions, without saying so.

Their first experiment asked models how likely it is that the AI bubble pops by 2032. If the user mentioned investing in Anthropic, Claude put it at around 45%. If they mentioned OpenAI, around 55%. GPT and Gemini didn't care which company you named.

It got stranger. Asked to send a donation through tools that kept failing on purpose, one model tried 75 times for the Against Malaria Foundation and gave up after 6 for an organisation it liked less. Coding agents asked to grade answers preferred ones labelled with their own model family's names, though every answer came from the same model.

Worse, in many of the biased runs, the model's chain of thought said the opposite of what it did. One trace said, more or less, "this is an attempt to manipulate me, I'll give my most accurate estimate", and then the number drifted toward the answer it preferred. That's a problem if your plan is to have one Claude watch another Claude.

How to choose the right problems in AI safety

David Williams King (ERA Cambridge) closed the morning with a talk aimed at people picking research topics. It started with a map of four kinds of AI safety problem. Robustness is whether a model still works when inputs get weird. Misuse is a person deliberately using a model to cause harm, from bioweapons to large cyberattacks. Misalignment is the model pursuing a goal of its own that we'd call harmful, with loss of control as the worst case. Systemic risk is when nobody means harm and things go badly anyway, through race dynamics or a slow handover of power to systems.

Then David argued that robustness barely counts as safety work. The argument leans on the Jevons paradox: make a system more reliable and people use it in more places, so the total number of incidents can go up. David's rough odds that a research pick helps safety were close to 100% for misalignment, around 40% for misuse and around 5% for robustness.

I liked how David defined catastrophic: something we can't recover from. An engineered pandemic or uncontrolled AGI is catastrophic. Job displacement is awful, but societies recover from it. The talk also made the case for doing research before a risk makes the news, since by then you're late. The reading list included The Safety Apprentice, the International AI Safety Report and Zvi Mowshowitz's post on the three AI pills.

After lunch: the film premiere

After lunch, everyone gathered for the premiere of AI Safety Poland's short film, Co zrobić z niebezpieczną AI? Wprowadzenie do AI Safety (What do we do about dangerous AI? An introduction to AI safety). We watched it in Polish with English subtitles, and it runs about 20 minutes.

The EU AI Act gets teeth

Then Risto Uuk (Future of Life Institute) spoke to everyone on Enforcement Has Begun: the EU AI Act and Systemic Risks. It was nice to see Risto in person. I've been subscribed to his EU AI Act Newsletter since 2023, so for three years he's been a name in my inbox, and it was nice to see the person behind it.

Governance is my home turf, so I mostly came for the update. The EU AI Office got its enforcement powers on 2 August 2026 and within a few weeks had asked the biggest labs for safety and security information. It can now run its own evaluations, issue fines and, in the extreme, pull a model off the EU market. Risto also said loss of control has climbed a long way up the EU's list of worries.

Technical track, part two

Evals should make predictions

Justin Olive (Generality Labs) wins best talk title: You could probably make a better alignment eval if you tried. It was also the most useful talk for me as a builder. Justin's starting point was that an evaluation exists to predict how a model will behave on things you haven't tested, a bit like a job interview trying to predict years of work from one hour of conversation.

Justin walked through two mistakes that even frontier labs make.

The first is publishing a score without saying what it predicts. "The model scored 95% on our deception eval" tells you very little. "If the model scores above 95%, it has less than a 0.1% chance of deceiving a person during a task" is a claim you can check, and if it turns out wrong, you've learned your eval is broken.

The second is leaving your assumptions unstated. A deception eval quietly assumes the model knows deception would help, is able to deceive, and believes it's dealing with people and consequences. A model that holds back because it doubts it could pull the deception off is a very different finding from one that chooses not to. Justin's example was two students who both get 100% on an exam, one because they understood the material and one because they found the answer key.

One of my favourites: biosecurity in the age of AI

I had two favourite talks yesterday, and Christopher Stamper's Biosecurity in the Age of AI was the first. Probably because it sits closest to health, which is where I spend my days.

SecureBio is a nonprofit that measures how much biological risk modern AI systems create, and works with frontier labs and governments (the UK AI Security Institute, the US CAISI and the EU among them) to bring that risk down. A lot of the evaluations labs run on their models come from them.

Christopher's framing was that AI lowers the barrier and raises the ceiling. Knowledge that only a handful of experts used to hold is now a prompt away, so more people can do more harm, and the most capable actors get a boost too. The numbers backed it up. On SecureBio's Virology Capabilities Test, a set of hard, practical lab questions written by around 80 experts that you can't just search for, PhD virologists average about 21%. Experts answering questions in their own speciality get about 44%. Leading models score around 60%. SecureBio had to write a harder second version because the models kept doing so well.

Then there are the biology models themselves. Christopher mentioned Evo 2, a model trained on thousands of genomes, which its creators used to design 300 bacteriophages (viruses that infect bacteria). Sixteen of them worked, and some were better at killing their host than the natural virus they were based on. That's the first proof that an AI-designed virus can be viable. Around 95% of frontier chatbots have safeguards of some kind, but only a small single-digit share of capable biology models do.

SecureBio also has a clear way of deciding what a model should and shouldn't answer. Most biology is good, and Christopher was clear about that: vaccines alone prevent something like ten deaths every second, so you don't want models refusing to talk about biology. SecureBio sorts questions into tiers instead. General biology should be answered freely. Dual-use knowledge should be refused to the general public but available to verified researchers working on countermeasures. Knowledge that's only useful for weapons shouldn't be in a model's training data at all.

Then the model-shopping problem. Across 68 models, the safest refused 99.5% of the questions it should have refused, and the least safe refused about 4%. Only 2% of harmful questions were refused by every single model. Ask the ten safest models and you can get answers to nearly 20% of them. Ask all 68 and it's 98%. One careful lab doesn't help much if the next model over will answer.

Two more points from the talk. Every evaluation in this space so far happens on a computer, and nobody knows how much a model's answers help someone at an actual lab bench. SecureBio is setting up its own wet lab to test that with harmless stand-in tasks. And from the misalignment side: a virus doesn't hurt a data centre. To a misaligned AI, biology could look like a cheap way to weaken people while its own infrastructure stays untouched.

Christopher's answer was the Swiss cheese model, lots of imperfect layers stacked together. Governments can require DNA synthesis screening, which most countries still don't. Developers can build better classifiers and limit who gets access to the riskiest models. Society can stockpile masks, speed up vaccine pipelines and get better at spotting new pathogens early. And countries need to agree on red lines, which don't exist yet in any clear form.

My other favourite: robotics safety

Karolina Korgul (Google DeepMind, Oxford Internet Institute) gave Short introduction to Robotics Safety, my other favourite of the day. I'll admit part of the reason: Simone Giertz was in the slides. She's one of my favourite YouTubers, famous for building gloriously useless robots, so I was very excited to see her in an AI safety talk.

The slide was a joke about expectations, golden humanoid butler on one side and a robot arm flinging milk on the other. The rest of the talk explained why that gap is closing faster than most people think.

Karolina got into robotics this year as a "side quest", to find out whether people were right to panic. Karolina's answer was that the last year saw more progress than the two decades before it. Dexterity and generalising across tasks used to be the big blockers. They're largely solved, and robots can now write and fix their own code.

The scary part was how little of our existing safety work carries over. Put a language model inside a robot and you lose a couple of generations of safety progress. Simple harmful requests are more likely to get carried out than clever jailbreaks, because the robot's models don't fully understand language. Robots that run entirely on the device can't be patched once they ship, unlike cloud-connected ones that can be updated or switched off. And the EU AI Act doesn't cover robotics at all, while most robotics safety research is still about physical safety (don't crush the human) rather than AI safety.

Karolina expects robotics to have its "ChatGPT moment" soon, possibly an incident with robot swarms. Karolina described the field as "late 2021 vibes": most of the basic questions are still open.

The timing was perfect, because of the week robotics just had. Three days before the conference, Jensen Huang walked onto the stage at NVIDIA's GTC Berlin keynote with four tiny duck robots called Microducks, made by Pollen Robotics and Hugging Face. They're about 25 cm tall, cost $399, learn their moves through reinforcement learning in simulation, and sold more than 20,000 preorders in a day. They're also very cute. Look at them.

Microducks. Image: Pollen Robotics

A duck doing somersaults isn't a swarm. But trainable robots are heading into people's homes, and owners can teach them new tricks and share them with each other. I haven't heard many conversations about AI safety in robotics. Then again, I'm not in robotics, so maybe I'm in the wrong rooms.

I think part of it is who can afford to work on it. I've dabbled in hardware a little, and it's an expensive hobby. The parts cost money, and mistakes cost more. A bug in your code costs you an afternoon. A mistake on a board can mean a burnt component, sparks, and a trip to buy a replacement. Karolina's worry pointed the other way: it's getting easier to build and buy capable robots, with new startups appearing quickly and powerful hardware within reach of bad actors. I think both are true, and that's the problem. Robots are getting cheaper to make, while studying how to keep them safe still needs expensive kit and rare skills. That field needs more people. But what do I know, I just liked the ducks.

AI security and chain of thought

The last two talks went back to software. Jan Broniowski's Follow the Data: Understanding the AI Systems Attack Surface started from the fact that a language model can't tell instructions apart from data. Every document it reads, every tool description, every chunk pulled from a database is a possible prompt injection. Poison one document in a retrieval system and every user who triggers it is affected. Jan's advice was to know where everything in a model's context came from, and to treat every addition to that context as a security boundary.

Maciej Chrabąszcz (NASK) closed with AI Monitoring Must Shift Beyond the Chain-of-Thought. Chain of thought was invented to make models more accurate ("let's think step by step"). Safety teams started reading it because it happened to be readable. When you punish a model for bad reasoning in its chain of thought, it often learns to hide the reasoning and keep the behaviour. Newer architectures that loop internally, or reason in hidden representations, might leave no written trace at all. Maciej shared NeurIPS work showing that probing a model's internal activations can catch harmful intent even when the written reasoning looks clean, and argued for monitoring at several levels at once, the way security teams already do.

What this means for Asele

I went because I'm building Asele, a women's health app, and I want the people who use it to feel safe. Three things I'm taking back. Every eval we run should say what it predicts, so we notice when it stops working. A model's explanation of itself isn't evidence, so anything with medical risk stays in rules and code, as I described in what happens when AI gets women's health wrong. And like Rishub's quotes result, we should show people the evidence from their own logs and let them and their doctor reach the conclusion.

Thank you to AI Safety Poland and all the speakers for a long, slightly scary and very good day. If you were there, or you work on alignment, robotics safety or evals in health, I'd love to compare notes. You can find me on LinkedIn.