Vol. 3 Testing ballots for usability
The top 10 guidelines for conducting usability tests of ballots come from two main sources. The first is a group of documents put together into the LEO Usability Testing Kit developed by the Usability and Voting Project of the Usability Professionals’ Association. (LEO stands for local election official.)
The second source is the years of experience the team behind the Field Guides has conducting usability tests and working with counties and states to help them make ballots, forms, and web sites work better for all citizens.
What is usability testing?
Usability testing is a tool for learning where people interacting with a design – such as a ballot – encounter frustration, and translating what you see and hear to make a better design that will eliminate those frustrations.
At its essence, usability testing is a simple technique: Watch and listen to people who are like your voters as they use a design as they normally would (or as close to normal as you can get). You can probably already see how this is different from conducting surveys or focus groups.
No. 01
Testing helps ensure that voters can vote the way they intend.
Why should you test?
When it is easy for voters to use a ballot, they are more likely to vote as they intend. That means fewer lost votes, which means wider margins (generally), which means fewer ballots are contested if there is a recount. All of which adds up to better elections for everyone.
No. 02
Put together a ballot, pick an interviewer, find voters, and find a place to watch them use the ballot.
What do you need?
It’s really simple. You don’t need recording equipment, but you might want to take notes. So, a clipboard can be handy. And you probably want an envelope or a folder to hold the ballots (or other materials) from the test sessions.
No. 03
Test when you know what is going to be on the ballot or when something has changed.
When should you test?
Test ballots to improve the ballot design and to understand training issues for election workers when:
- you have a good idea what will be on the ballot for the next election.
- something major has changed, such as new legislation.
- something happens that may cause the overall layout to change, such as removing a candidate or a question.
No. 04
Usability testing answers questions about how and why voters will use the ballot.
Know why you are conducting a usability test.
Usability tests can answer questions like these:
- How easily and successfully do voters mark the ballot?
- What mistakes do voters make in marking the ballot?
- How close is the marked ballot to how they said they intended to vote?
No. 05
Start testing early with a few voters trying out the first versions of the ballot, one at a time.
Who is needed to run a test?
Voters
How many: 12 to 15 one at a time
Interviewer
How many: 1, you or someone who didn’t design the ballot
Helper / note taker
How many: 1 or 2, someone who can help you review
Observers
How many: 1 or 2 from citizen groups (you can turn them into helpers, too)
No. 06
Test with what you have available. Test again when you have the final version.
What do you test?
When you do your first usability test, you might want to practice on somebody else’s ballot. That way, you won’t feel so bad when you test yours.
Otherwise, you can test:
- mock-ups or early versions
- ballots from the last election
- a nearly-final ballot
No. 07
Voters for your usability tests are everywhere. Go to them.
Where should you test?
Choose a place where you can find voters:
- library branches
- farmers’ markets
- street fairs
- parks and plazas
- village offices, town halls, city hall, county buildings
- kitchen tables, food courts, pancake breakfasts, fish fries
No. 08
Follow these steps to run each session of a usability test.
What happens in a usability test?
- Introduce the session.
- Go over what will happen.
- Give instructions.
- Give the voter the ballot.
- Watch the voter vote.
- Listen for questions (don’t answer them) and comments (write them down).
- When they are done voting, ask the voter to walk you through what they did and why.
- Thank the voter profusely.
No. 09
Watch and listen. Don’t teach. Don’t help.
What is the role of the interviewer?
As the interviewer, you guide the voter through the session, watch what the voter does, and take notes (if you can).
Do not help the voter use the ballot. (Well, until after you have finished learning what you need to learn.)
Ask open-ended questions, like, “How did that go?” Follow up with a statement like, “Tell me about how you marked the ballot.”
No. 10
Watch for mistakes, listen to questions, look for hesitations.
What should you look for?
Did the voters:
- Ask for help with instructions or using the ballot?
- Ask questions? (If so, what questions.)
- Make comments? (Again, note what they say.)
- Take out reading glasses or lean way in?
- Mark the ballot incorrectly?
- Have trouble moving through the ballot?
- Seem confused, puzzled, or frustrated?
Bonus
Review what you saw and heard. Tally the types of problems voters had.
What do you do with what you find out?
Look at what parts of the ballot caused questions, comments, mistakes, or requests for help.
This should tell you what is confusing to voters, what is unclear to voters, and why. It should also tell you what might need instructions or a different heading or label.