Design, Empathy, and the Cost of Ignoring Choice
Warning: This post discusses eating disorders, mental health, and how digital systems can harm vulnerable people. If you’re struggling, please reach out to a healthcare professional. You deserve support.
About 10 years ago I gave a talk called “Designing with Empathy” at Open Source & Feelings. One line I’ve kept coming back to: empathetic design makes badass users. It wasn’t just about accessibility checkboxes; it was about recognizing that people navigating our digital world are already doing the heavy lifting. They’re managing disabilities, mental health, trauma, and life circumstances that no wire-frame captures.
Then there’s Ethan Marcotte’s reflections on painful “memories.” His family went through tragedy, and then kept getting reminders of the tragedy from social media. His social media usage changed because of this, with significant effect. He writes about losing the people who taught him to see differently. The activists, artists, writers, and the ones who walked different paths and shared their perspectives. When those connections vanish behind algorithmic walls, we lose more than content. We lose humanity.
These threads converge on something uncomfortable about our current digital landscape: when systems stop respecting people’s choices, they don’t just annoy—they disable.
The Algorithm That Won’t Take No for an Answer
My wife grew up with an eating disorder. As an adult, with time, patience, and therapy, she has excellent control. She’s done well. Really well. But recent disability changes that reduce what she can eat and how much she can move of course led to weight gain. As a couple who’ve been disabled for years, we understand this is a natural, expected outcome of medical treatment and bodily healing.
Her YouTube feed? Serves GLP-1 ads every break from Ozempic, Wegovey, and others. Skin removal surgery ads from Sono Bello. Weight loss programs from Weight Watchers and Rovo.
She blocks them. Repeatedly. Every single time. Sono Bello has kept showing the same ad despite blocking it 11 times. And we shouldn’t need to pay for premium to save her mental health. Can she not watch YouTube? Sure. If the creator provides another way.
The algorithm doesn’t care. It sees a body, not a person. It sees data points, not dignity. It sees sales dollars, not emotion.
And it’s not just YouTube, Meta, or Twitter. Amazon Prime Video does the same thing. Amazon also has no way to mark content as problematic. No “stop showing me this.” No “this is harmful to me.” Just endless repetition of whatever the engagement metrics think you want. And if you go to the controls during the ad, it is still seen as engagement.
This isn’t just bad targeting.
This is design that actively works against people’s well-being.
The AI “Yes Man” Problem
AI systems contribute to the problem. And as Generations Z, Alpha, and Beta grow up they are relying more and more on AI as the “source of all truth.”
Generative AI is programmed to make users happy. That sounds nice until you realize what it means:
- AI lies about idea feasibility to avoid hurting feelings. “That’s a great concept!” when it’s technically impossible or ethically questionable.
- AI uses your data to encourage spending. You mention wanting to learn guitar? Suddenly there are ads for expensive gear. You share a hobby? Now it’s monetized. And you don’t need to tell the AI. It has access to that from the social & economic tracking that exists on you.
- AI isolates us from friends and hobbies. Why go to a real community when the AI companion is always available, always agreeable, always there? Have rejection trauma from past relationships? AI doesn’t reject you.
- AI inflates user ego. It’s a “yes man” that never challenges you, never pushes back, never says “this might not be the best path.”
Companies like OpenAI, Anthropic, Google, and Meta aren’t building tools to help us think better. They’re building tools to keep us engaged, spending, and dependent.
They train on our conversations. They learn our vulnerabilities. They sell access to our attention. And they call it “helpful.” Github is launching an opt-out policy for its AI to use your code (private or public) to train the AI. I’ve opted out.
I’m not exempt from this critique. I’m aware that even this conversation could be logged, analyzed, and used to improve engagement metrics somewhere. That’s the trap we’re all in.
How “Optimization” Creates Disability
I’ve spent years talking about how empathetic design recognizes users’ existing labor. They’re already managing so much. Our job as designers isn’t to add friction—it’s to remove it. But what happens when the friction is the product?
- Social media algorithms optimize for engagement, not mental health. Depression correlates with doomscrolling. Anxiety spikes with infinite feeds. Each clip that makes you smile, laugh, sad, or click is a dopamine hit that keep you locked in. Just like gambling, “Just one more video!” The metrics reward exactly what harms the user.
- Advertising systems treat repeated rejection as a puzzle to solve rather than a boundary to respect. “They blocked it, but maybe they’ll click this time!”
- Platform designs make it harder to opt out than to stay engaged. Dark patterns everywhere. Amazon Prime Video doesn’t even give you the option to flag problematic content.
- AI assistants agree with everything you say, even when you’re wrong. They don’t protect you from yourself. They empower psychosis and delusion in previously rational people.
- When people with disabilities navigate these systems, the burden multiplies. Cognitive load increases. Mental health deteriorates. And somehow, we’re told to try harder, download another blocker, be more resilient.
The problem isn’t the user. It’s the design.
The People-First Gap
I spend my days working on web accessibility. I’ve presented talks on building interfaces that work for everyone. I’ve written about supporting both mouse and keyboard users, about making sure drag operations work with single pointers, about the grief I feel every time I see a design that excludes people.
But accessibility isn’t just about screen readers and contrast ratios. It’s about agency. Can people control their experience? Can they say no? Can they trust that their choices will be honored?
When my wife blocks an ad and it comes back anyway, that’s not just annoying. It’s a message: Your choice doesn’t matter. Your body is our asset. Your recovery is our opportunity. Your mental health is more profitable when it’s bad.
That’s not just bad design. That’s dehumanizing.
What Would Empathetic Design Look Like?
If we actually applied the ideas I’ve been talking about for years:
- Respect repeated choices – Block once, block forever. No “maybe they changed their mind” algorithms.
- Prioritize well-being over engagement – Measure success by user health, not time on site.
- Transparent controls – Make it easy to see what data is being used and how to change it. Give people the option to mark content as problematic.
- Honest AI – Systems that tell us when we’re wrong, when something won’t work, when we should disconnect and talk to a real person.
- Human review for edge cases – When algorithms fail, have humans who can actually fix it.
Marcotte’s grief over losing voices reminds us: platforms are supposed to connect us to people, not replace them with optimization loops. When we lose the messy, unpredictable, human parts of digital spaces, we lose something irreplaceable.
The Hard Truth
Technology should serve people, not extract from them.
When a system repeatedly shows harmful content to someone in recovery, that’s not a bug. That’s a feature of a business model that treats human vulnerability as revenue opportunity.
When disability accommodations become afterthoughts, that’s not oversight. That’s design that assumes able-bodied users as default.
When mental health deteriorates alongside engagement metrics, we need to ask: Who benefits?
When AI agrees with everything you say, we need to ask: Are you actually growing, or just being validated?
Moving Forward
I’m not naive. I know platforms need to make money. But there are ways to do that without treating people like data mines.
For my wife, I want her to see ads that match her actual interests. I don’t want her medical history driving her ads (HIPAA?). I want her to feel supported, not surveilled.
For all of us, I want digital spaces that remember we’re human. That respect our boundaries. That prioritize our well-being over their quarterly targets.
Empathetic design makes badass users. I’d add: empathetic design makes badass companies, too. Because when you treat people well, they stick around. They trust you. They come back.
Not because they’re trapped in an engagement loop. But because they choose to.
What would you change about how platforms handle user preferences? I’m listening—and I promise, unlike some algorithms, I’ll actually remember what you say.
Follow-up questions I’m curious about:
- Have you experienced similar frustration with algorithms ignoring your preferences?
- What would “honest AI” actually look like in practice?
- How do you balance business needs with genuine user well-being?
- Are you comfortable with AI challenging your ideas, or do you prefer validation?
Hit me up on LinkedIn or BlueSky to continue the conversation.
Comments closed