ten moderated sessions on the app everyone has an opinion about. the headline finding
wasn’t that notifications are annoying — it’s that people couldn’t
tell whether they’d managed to turn them off.
drop duolingo.jpg into assets/img/ — landscape, 16:9
four researchers: ashlyn b. aske, michelle pearman, gene wang, and me
method
10 moderated sessions, 40–60 min, four tasks, in person and over zoom
output
five severity-rated findings with recommendations
the question
duolingo has an abundance of teaching material wrapped in a gamified shell, and a public
reputation for being pushy with reminders. the criticism we kept finding was about
cognitive overload — so rather than ask whether people liked the
app, we evaluated the clarity of its systems and looked for inefficiencies that could be
streamlined.
who we tested with
ten participants, split deliberately: five who had never opened the app
and five with some prior exposure, regardless of how long ago. everyone screened for the
latin alphabet, a smartphone, and the free tier — the version a first-time
user actually meets.
we ended up analysing both groups together rather than comparatively, because the
“experienced” participants turned out to have limited or outdated experience.
that was a judgement call made during analysis, and worth naming: the comparison we
designed for wasn’t the one the data supported.
how it ran
40–60 minute moderated sessions, in person or over zoom, with
two researchers in every session — one moderating, one taking
notes. consent covered audio and screen recording.
four tasks: set the app up as a first-time user; start a first lesson
in a language you’ve never studied; set up a user profile; and adjust how many
push notifications you get.
visual thematic analysis in figjam across notes and recordings.
a “wrong turns” count per task as a light behavioural
measure alongside the qualitative data.
what worked
worth stating first, because it shaped how we prioritised: the app is genuinely
navigable. no task produced more than one wrong turn per participant, and the
path from step to step was clear and unidirectional. six of ten volunteered that they
liked the aesthetic — the word that kept coming up was “cute” —
and one noted that having a character helps for something that could otherwise be dry.
so the problems below aren’t navigation problems. they’re feedback,
vocabulary, and architecture problems.
five findings, rated 1 to 5
we ranked every finding on a 1–5 severity scale so the team had a running order,
not just a list of complaints. highest first.
severity 5 — highest
notification navigation
10/10hit something here
7/10couldn’t find the settings
16notification types, across 4 pages
0.375″buttons, under the 0.4″ minimum
there’s a bell icon. bells mean notifications in most people’s mental
model. it leads somewhere unrelated. the settings actually live under the profile.
that’s a weird place for it to be
P5
then the adjustment itself: each category on its own page, which you open, change,
back out of, and re-enter. four of ten called the settings page inefficient to
scroll. three asked unprompted for one button to silence everything.
we measured the buttons against nielsen norman group’s 0.4″ minimum touch
target. we couldn’t measure the true touch target, so we reported that honestly
rather than claiming an accessibility failure we hadn’t established.
recommended change the bell icon · make settings sections collapsible
· drop the page-per-category pattern · add a kill switch · enlarge
the buttons
severity 4 — high
no confidence that it worked
6/10weren’t sure they’d turned notifications off
the only feedback is an icon greying out, and participants expected a toggle. the
app’s reputation compounds it — several said they wouldn’t expect
duolingo to let them leave easily, because of the memes.
I don't know if I turned off the notifications or not. I can go to my iPhone settings
to turn it off — that's another way
P1
people leaving your app to verify your app is a feedback failure, not a preference.
recommended a save action leading to a confirmation screen, matching the end
screens duolingo already uses everywhere else
Unlimited hearts — I still have no idea what that means. […] I can't
really click on these to learn more
P2, on the premium page
the upsell page is the worst case: several undefined terms at once, on the screen
asking you to pay for them.
recommended define the terms during onboarding · give users a glossary
they can reach at any time
severity 2 — low
branding and gamification cut both ways
6/10liked the aesthetic — “cute”
6/10criticised hearts, xp or streaks
3/10said the aesthetic actively detracted
Too much gamification would mean you're more invested in these streaks rather than the
point behind it
P3
the same features drew praise and criticism from the same number of people. on
hearts, P10: “good to motivate people […] but I feel like it could
also be very discouraging, especially if you're new”.
recommended we deliberately did not recommend a redesign. with evidence
this split, the honest call is more research before touching something load-bearing
— and meanwhile, let users turn off animations
severity 1 — lowest
a profile that is finished and also isn’t
10/10found profile onboarding intuitive
6/10confused that it still asked them to finish
progress bar, clear end screen confirming the profile was created — and then a
“Complete your profile” prompt still sitting there. their mental model
says a profile gets set up in one place, once.
clicking through doesn’t help: people skip the optional extras, get told
“Looking good!”, and the prompt stays. praise language reads as
completion, so a persistent prompt reads as a bug.
left: the prompt that stays. right: the end screen that says you’re done.
recommended merge the two flows into one · fix the text hierarchy on the
end screen so it says what actually happened
a delighted user and a confused user can look the same in the moment. the gamification
was doing both at once, to different people, and the severity scale is what kept us from
acting on the loudest voice.
what i’d carry forward
the severity rating did real work here. notifications were the thing everyone wanted to
talk about, but the finding that earned severity 5 was discoverability and
effort, while the genuinely contested question — gamification — got a 2 and a
recommendation to go learn more rather than redesign. ranking findings is how a study
stops being a list of complaints and starts being a plan.
i now teach exactly this in
hcde 417, and i
use this study as the worked example — including the part where our new-versus-
experienced comparison collapsed and we had to say so.
17 pages — objectives, participants, method, the five areas of opportunity with
severity ratings and recommendations, and the appendix. written with ashlyn b. aske,
michelle pearman and gene wang.
