This is one in a series of blog posts from the 2014 class of MJ Bear Fellows, three journalists under 30 who are expanding the boundaries of digital news. Apply for the 2015 fellowship here.
Note: This blog is the second in a two-part series by Fellow Anika Anand, Director of Product at Chalkbeat, an education news network based in New York City. Her first post, published June 2, chronicled ChalkBeat’s need to develop a robust audience research project with a small team. That discussion is continued here.
As we planned for our project, I couldn’t find many good resources on how small newsrooms can do quick, effective audience research.
Below, we share a few of our research methods and takeaways in hopes that other newsrooms will chime in with their own suggestions. It is important to note we did not hire a professional researcher for this project, and these tools were not meant to be as detailed as a proper scientific survey. That said, we received useful feedback for better understanding our audience.
Illustration by Flickr user Giovana Milanezi
E-mail survey of Chalkbeat readers
Goal:
To learn who our existing readers are, whether they believe we are serving them well and, if not, how we can better serve them.
What we did:
We created a 50+ question survey and offered $10 Amazon gift cards to the first 100 respondents. We distributed it first through our daily e-mail newsletter because we know those subscribers are our most engaged readers. Then we posted about the survey on our website, our Facebook page and our Twitter feed. We received about 300 responses.
How we’d do it differently/best practices:
Some of the responses were half-completed. We couldn’t tell if the respondents were all real people, and if so, whether they were only motivated by the chance to receive a gift card. To address this, If we sent out a survey again, I’d limit it to 15 questions maximum. Also, rather than offer the gift cards again, I’d think of other incentives to get our readers excited. Maybe a shoutout in our newsletter or on Twitter or Facebook? Or a free drink ticket at our next event?
Cost:
We paid to upgrade our SurveyMonkey account, but we could have done this for free if we had decreased the number of questions or used other free services, like Google surveys or Typeform. The 100 Amazon gift cards cost a total of $1,000.
E-mail survey of non-Chalkbeat readers
Goal:
To learn how we could reach new readers and grow our audience.
What we did:
We purchased a SurveyMonkey sample of educators and parents who live in New York. That meant that SurveyMonkey was responsible for sending out the survey and guaranteeing we got enough responses. (We ended up with 184 total from both groups).
How we’d do it differently/best practices:
This was really easy and worked well. We used roughly the same survey with a few tweaks to account for the fact that this group of people likely didn’t know who we were. But it’s important to keep the questions as similar as possible so you can compare answers between readers and non-readers.
Cost:
With support from a grant, we paid a little more than $1,000 for this. But we had 39 questions, which was way too many. Decrease those and you’ll pay less, and I personally think it’s a worthwhile expense.
Phone call interviews with influencers
Goal:
To learn if key education players (local advocates, policy and lawmakers) thought Chalkbeat was giving them the information they needed to better inform their work.
What we did:
We conducted phone interviews with 13 people whom we considered to be education influencers in New York City over two weeks.
How we’d do it differently/best practices:
Thirteen was a good number to give us a sense of what influencers thought of Chalkbeat. Just make sure that the group you end up with represents a diverse audience (for us that meant policymakers, lawmakers, advocates, parents, etc.)
We scheduled 30-minute calls with people, knowing that our calls may get cut short. We also prioritized the most important questions from our long list so that if we had less time to talk, we hit the important stuff. Ideally every interview should have two people on the call: one asking questions and one taking notes. The person taking notes should timestamp important things that are said so that it’s easy to reference quotes afterward.
We used Zoom to create an audio recording of the calls, and we made sure the person on the other line knew that we were recording them.
Finally, all of the calls were off the record. We also told them if we shared their insights with external audiences, we would just identify them as a “policymaker” or “educator” in New York.
Cost:
Staff time. Otherwise, free!
In-person focus groups
Goal:
To interview everyday readers and learn how to build a community around them.
What we did:
We conducted three different focus groups in three days, at 90 minutes each. Our community editor went into our MailChimp e-mail list and reached out to people who were the “most engaged,” i.e. who opened our newsletter nearly every single day. Then she made sure she had a diverse mix, including parent, teacher, advocate, researcher. Our goal was 30 people total, about 10 for each group. Out of 56 contacted, we ended up with seven people for group one, 11 people for group two and nine people for the third group.
How we’d do it differently/best practices:
We found it helped to record the whole conversation. Have at least one person taking notes and time-stamping noteworthy things that were said.
Also, encourage your staff to attend! While one person should lead the conversation, it’s great to have your staff pipe in with smart follow-up questions based on their knowledge of a story or beat. It’s also great for them to hear directly from readers.
We provided dinner for everyone. This was one small way to make them feel invested in coming. (Only three people canceled at the last minute, which is pretty good for New York).
At the last minute, we debated whether to do actual activities or to just have a free-flowing conversation. We decided to go with the latter because this was our first time doing any sort of focus group with our readers and wanted to learn as much as we could rather than potentially limit the conversation. But in the future, we definitely would be better served by doing more structured, hands-on activities to learn more about a specific product idea.
Cost:
We bought sandwiches for everyone, which cost about $10 per person.
Fellow Anika Anand is Director of Product at Chalkbeat and was previously a reporter for GothamSchools. She has been published on California Watch, MSNBC.com, Salon, the New York Daily News and others. She graduated from the CUNY Graduate School of Journalism with a Master’s degree in business and economics reporting and studied journalism at the University of North Carolina at Chapel Hill.