Vol. 107 Conducting multilingual usability testing

Just like material written and designed for English speakers, material in other languages requires feedback from the people who use it. No matter how much experience you have with usability testing, adding multiple languages requires some additional steps and considerations about how to run the test sessions.

Before you begin multilingual testing, you need to:

Know what languages you want to test
Know what voter information or other election materials you want to test
Have a plan for translating them, or already have materials in the target languages

If you are new to either usability testing or language access or are still working out the above information, we have resources to help you get started.

– Usability testing resources include a Usability Testing Kit with all the basics, and many articles on our website
– Language access resources include a guide to setting up your program and tips for translations

This workbook will help you through the process of testing materials in multiple languages, and provide you with tools to run usability testing from pre-planning to reporting on what you learned.

With this workbook, you’ll build on your existing knowledge of usability testing, learn ways to run a multi-language test, and be ready to welcome voters with material that works for them and their language community.

What’s in this workbook:

Section 1. Planning the usability test

Section 2. Choosing a moderator for the test

Section 3. Translating the materials you will need for testing

Section 4. Recruiting test participants

Section 5. Running sessions in multiple languages

Section 6. Analyzing the usability test results

Section 7. Concluding the project

Please note that this workbook is a work in progress. We’re still developing this approach and will refine it as we learn more about testing in multiple languages. If you have more thoughts please reach out to us. We look forward to hearing feedback from the people who use it.

Section 01

Planning the usability test

Every usability test requires two things: participants and a place to run the test. Finding participants for a usability test can often require planning or connecting with your community in advance. We recommend making these connections early so that when you are ready to recruit participants, you can do so more easily. If you’re just starting off thinking about where to find participants, community groups are your ally. Here are the steps we suggest starting with:

Reach out to community organizations in your area
Look at data from surrounding institutions, like schools, hospitals, libraries to find out what languages they provide. Similar types of institutions may be providing services for the same communities. Just like you, there are people in these organizations who are thinking about how to provide language services.

How to plan for usability testing and translation

We put together a worksheet that will get you started on planning. Feel free to customize to your needs.

Planning worksheet

What you need	Options for how to provide it	How will you do it? What will it cost?
Translation for what you are testing	Will the translation be done in-house, or will you use a translation service?
Translation for testing materials such as signs, moderator’s guide and any forms	Will the translation be done by the moderator or by your in-house or external moderator
Moderators to run the test in language	Do you have someone on staff who speaks the target language for the testing, or will you hire someone for the testing?
Participant gifts	Will you pay a small “incentive” to the participants, or offer them a memento from the elections office?
Space for testing	Where will you conduct the sessions? Is there a rental fee for the space or a donation to a local community group?

>> Download this planning worksheet (Google Doc)

Section 02

Choosing a moderator for the test

Choosing a moderator is one of the most important components of a successful testing session. A moderator facilitates the conversation with a voter during usability testing. They set the tone for the session by asking questions and listening to what people are saying about the material. After the session, they help ensure that voters’ feedback makes it into the final design. In some cases, a moderator might need to translate all the materials into the target language. This includes any signs or posts inviting participants, forms the participants might complete, and the moderator’s guide. Keep this in mind as you hire and think about compensation.

Ideally, you would have two people who speak the target language conduct the session: A moderator and a notetaker to record the session. However, one person can moderate and take notes during the session at the same time if necessary.

If someone on your team can speak and write in the target language and feels comfortable speaking and listening to people in that language, they can moderate and lead testing for the target language. But if you need to do testing in a language not spoken in your office, you’ll need to hire a moderator or two.

Skills to consider when hiring a moderator

We recommend looking for people with a combination of:

Proficiency in usability testing
Community engagement experience
Fluency in the target language (written and spoken)

This combination of skills might not always be available within your office. Not a problem! We’ve conducted some projects where we hired freelance moderators without direct usability testing experience. In these cases, we often hire moderators with related experience, such as those with a background in community outreach, researchers in other fields, or students in these fields, and we focus on training them in how to run the test.

Some places to find moderators include your own social media, local job boards, local college’s language or design departments, and community groups. Reach out to other departments within your county. They might have relevant experience or know where to connect with potential participants from the community groups.

Below, we’ve included a sample job description for a moderator. Feel free to adjust the information to fit your needs.

Help us test voter education materials in [language]

We are looking for a Research Assistant fluent in [target language] and English to moderate usability testing on [material]. The project focuses on creating [target language] material that is accurate and culturally sensitive.

We are holding feedback sessions with [target language] speakers in [location] with the aim of gaining meaningful feedback on the material’s final design from community members.

Responsibilities:

Engage in thoughtful conversations in [target language] with community members to gather feedback on newly designed election forms and materials

Plan usability testing sessions in collaboration with our team, managing participant recruitment, scheduling, and logistics

Facilitate research sessions by providing guidance to participants during tasks and asking insightful questions to uncover insights

Translate materials from English into [target language] and research notes into English

Analyze data to identify patterns and trends that inform the usability and effectiveness of election materials

Report findings to elections office staff

Optional: Support work to recruit participants, such as translating ads or notices to post and managing communications with possible participants for scheduling and reminders

Qualifications:

[Target language] fluency in reading, writing, and speaking (Required)

Experience in usability testing, qualitative research, or an adjacent field such as Public Policy, Anthropology, or Psychology

Experience working in communities with low civic participation

Time required:

We anticipate this work, including preparation, translation, conducting the feedback sessions and debriefing, to take between [X] days

The feedback sessions will be conducted on [when]

Other work will be done on your own schedule, arranged to meet the project schedule.

Pay: $X

How to Apply:

If you are interested, please send an email listing your qualifications and why you’re interested in the position to [NAME]

>> Download an editable version of this job description (Google Doc)

Section 03

Training the moderator

When working with moderators without direct usability testing experience, we recommend providing additional training. This includes communicating a comprehensive rundown of the project’s context, data collection methodologies, and outlining the procedures for data analysis.

Help moderators understand the overall impact of the project

We always emphasize the importance of the moderator within a community of voters, and how their work helps people with access to voting. By providing these details, moderators will be well-informed and deeply invested in the project’s outcome.

For example, if we are redesigning mail-in envelopes for a state, it is important to emphasize to moderators how it will directly affect an entire state and its voters. Spending time going through these elements ensures that moderators are prepared to contribute to the usability testing process, even if they are new to the process.

Train your moderators on core skills

Develop a training procedure that emphasizes the skills necessary to observe behaviors, ask pertinent questions, and take notes during the testing session. These detailed notes will provide invaluable information about how people interact with the material and inform the next phase of the project. They are especially important if the moderator is the only person on the team who speaks the test language.

Your training should include a few practice runs before speaking to real participants. These practice runs can be in English and work to align goals for everyone involved.

Section 04

Recruiting participants

Recruiting the right participants from the target language community will ensure your usability tests are insightful to the information you’re looking to learn. So where do you find participants? As we mentioned earlier, it’s important to think about this step early in the process as it can take time. Here are the ways we’ve found to be most effective.

Partner with community organizations

A good way to find participants to test materials is to talk with community organizations about what you’re doing and ask if they can help with recruiting. We also understand that some election offices don’t have an existing relationship with community organizations – yet. We recommend election offices initiate in-person contact at community events or meetings. We’ve created many of our relationships by dropping in at community events. In the past, we’ve gone to community gyms and activity centers, language schools, and senior centers. Sometimes community organizations will be interested in similar topics like voter civic engagement and will actively want to support your usability test and overall goals.

Community organizations can also help host usability sessions, or let you know the best time to find participants in community centers like the library.

For example, when recruiting participants for previous usability tests:

An elementary school let us set up tables to talk to parents picking up kids.
A community center invited us to come to an event and set up a table in the lobby.
An ESL center let us know their class times, so we could catch people before or after class.

Recruiting participants in advance for scheduled sessions

One way to meet with participants is to have people sign up for scheduled sessions. In this type of testing, you can set up your own schedule with open time slots and have participants choose what works best for them. Sometimes community organizations can help host the event, or help you recruit participants by inviting their community members to sign up.

Recruit participants through community connections, social media (either paid ads or free posts), flyers in public places like libraries or community boards, or location-specific listservs like county social media pages, Craigslist, or NextDoor.

A challenge to scheduled interviews is that not everyone will show up. In this case, you might have to pivot to intercept testing. You can also plan to mix participants you find on-site with scheduled sessions.

Find participants through intercept testing

Another alternative is approaching people in a public area and asking if they have time to look through some material. This is known as intercept testing. Plan intercept testing in places with a lot of foot traffic, such as libraries, parks, grocery stores, and community centers.

For multilingual intercept testing, it’s important to approach people in-language and give a brief introduction to what you’re doing and why it’s important. It’s helpful to bring a translated sign about your test so people can see which languages you will be testing in. You should wear badges or include your office name and logo, so people know who you are. For example, in the past, to get people’s attention we used a sign that said, “Help us make voting easier for everyone. We’re not selling anything. We’re not representing any political campaigns.” Make sure all your signage is translated.

Rejection is part of the process of intercept testing. The more you approach people, the less scary it is and the more comfortable you become with your introduction. Before you know it, you’ll feel the rush of engaging with people and talking over the material you’ve designed and translated. For more information, read this quick article about how to prepare and run intercepts.

Section 05

Running the sessions

Be ready to talk to participants in either English or the target language and be ready for the possibility of going back and forth between languages. Some participants might feel comfortable talking about certain topics in one language and other topics in the other language. We like to encourage people to go with what feels most comfortable to them. We often have rich conversations about translation and what a word or phrase might mean by switching between languages.

Translating notes from the sessions

Moderators have their individual preferences when it comes to translating their notes from sessions. Some moderators prefer to take notes in the language spoken during the session and then translate at the end of the session. Others might translate simultaneously during the sessions. There is no right or wrong approach, but there are distinct advantages and challenges associated with each method.

Translating at the end of a session

Translating at the end of a session allows a moderator to take notes in the same language they are speaking, avoiding translation fatigue. The challenge to this approach is that it leaves a lot of translation work after each session. And, if the moderator needed clarification about a term or an idea expressed during a session, the participant isn’t around to help better translate their opinions into English. If you’re planning on taking this approach, we recommend scheduling sessions with enough time in between each session for the moderator to gather and record their initial thoughts, revisit any translation issues, and take a much-needed break.

Translating during sessions

Translating during sessions gives the team immediate insights into a session and allows moderators the opportunity to address any misunderstanding promptly and ensure that a participant’s feedback is translated accurately. This approach is also helpful if the participant is bilingual and switches back and forth between languages.

These sessions can be demanding because they require constant switching between languages and the translation of nuanced concepts. The mental effort required to maintain accurate translation throughout multiple sessions results in translation fatigue. To help mitigate this, be sure to leave short breaks between sessions. A good approach is to hold rapid debriefs after each session where you go through notes to make sure everyone agrees on them.

We’ve been in a situation where we had a line of interested participants waiting to take part. We opted for real-time translation so that we could hear from everyone who was interested in being part of the testing. The moderator talked to participant after participant while translating concepts and taking notes for each session. Since translations were ready after every session, we were able to look at participant feedback immediately and make any necessary adjustments on the same day. If, in this case, the moderator waited to translate everything at the end of the day, we would not be able to see all notes as a team that same day and address issues on the spot.

In another situation where we had a large group of people who wanted to be part of the test, we worked with pairs of participants. This is not an approach for an inexperienced moderator, but it allowed the two participants to talk to each other, working through the form together, while we listened and asked questions of both of them.

Both approaches work well. Choose an approach that takes the moderator’s experience, level of comfort with live translation, your preference for the test, and any time restraints.

Section 06

Analyzing what you learned and updating the materials

Now that you’ve gone out and tested the material with different language communities, it’s time to identify a few common themes.

Analysis for multilingual testing differs in a few ways. If you have tested in multiple languages at once, say, Spanish, Chinese, and English, you’ll first want to run an analysis on each language individually. Treat each language as its own test. Once you’ve identified the major themes of each language, you can look at everything together and see if there are any similarities between languages.

You are looking for:

Places where the translation is unclear. In the first analysis, you’ll want to focus on identifying all of those points.
General usability problems in understanding or using the materials. Here you are looking for the same insights you would have for the English materials and any nuances of the experience that might be different for the language community – for example, based on their history, or because they are new voters.

Then, you can adjust your translation process based on what you learned and update any issues relating to the design of the material.

For more on how to do analysis for usability testing, check out this training on ElectionTools.org.

Section 07

Concluding the project

The goal of usability testing is to improve the materials being tested. After all the testing is done and you have analyzed what you learned, it’s time to share it with your team. For each insight or recommendation from the usability test, you and the team have to decide how to act on it.

In addition to applying what you learn to one form or other material, you have probably learned things that apply to other materials or voter education

Log the experience from this test into your language access plan so you build your body of knowledge

You should include:

Translation insights that build your word bank or glossary

Which words, phrases and sentences worked?
Update your word bank with words that worked best during testing to communicate with translation vendors for your next translation job.
Identify words, phrases and sentences that confused people.

Design insights that can be used on other materials

Which design layouts worked best to help people read or use the material?
Are there differences for people who speak different languages or have different prior election experiences?
If the material is bilingual or multilingual, could people find the information in their own language easily?
Is there anything else you learned about the information design that could be part of the design standards in your offices.

Resources and procedures for future testing

Which recruiting methods worked and which ones didn’t?
Is the location one you might want to use again?
Did you make any new contacts for voter outreach and education in the community?
What worked well (or not) in your approach to running the session and handling translations?
What worked well (or not) in finding and training a moderator? Is this a person you might work with again?

Documenting this will help your office repeat the process with other material, and keep refining your process with each project.

Multilingual testing is an important part of language access and voter outreach

Multilingual testing will help make sure you are communicating to your voters in a language and design that resonates with them. Now that all the materials have been tested by people who speak different languages and hold various experiences of the voting process, you’ll be able to clearly identify which words and phrases are effective and which ones aren’t. You’ll also be able to better understand the language community you serve, and learn about ways to meet them where they’re at. This becomes crucial when considering how to inform voters about the availability of materials in their language.

Reach out to us if you have any questions along the way.