Measuring community success

How do you know if a community is succeeding? Is the Toronto community a success?

My Toronto community wrangling efforts have not had a corporate master. It has allowed the community to evolve without the tooling or reporting that is often present in corporate activities. The meme used to seed the community has evolved but has always been about the altruistic value of having a strong community. It was built on the premise of participation, i.e., participation equates to citizenship.

The premise is that everyone attending must present something. This is great, it ensures participation.   2005/9/30 – BarCampToronto

It’s funny, because we then immediately broke with participation for DemoCampToronto2.

Do I have to have a demo?

Nope! You are welcome to come along and give feedback to the people who do (or just watch and absorb if you don’t feel like talking).

But we had a very clear mission:

we want to continue to develop an active startup community in Toronto.

The idea being that for startups in Toronto we need a common meeting place. DemoCamp is about bringing together a community. “The community is the framework” is the underlying mantra. By brining people together who have shared interests new things can develop. New relationships. New companies. New ideas. New employees. It has been about meeting people, making connections, learning, sharing, finding inspiration and generally getting excited about all of the cool things going on in Toronto.

In the process the number of people attending the events has grown. From 26 attendees at DemoCampToronto1, to over 400 at DemoCampToronto17. We’ve had corporate sponsors. We have pitches. We’ve had a lot of really cool demos, presentations and discussions. DemoCamp has happened in spite of a corporate mission. The costs were low enough that it could fly under the radar, there weren’t metrics associated with the $400 sponsorships. As the event grew, we just grew the number of sponsors. It was pretty straight forward. As the event has grown the costs have changed. The venues have changed. The presentation format has changed. But because it’s been about having an event where people in the Toronto community can show what they’ve been working on; meet others doing interesting things; and knowing the event was for the community.

Measuring Success

For everyone that thought this was going to be about the future of DemoCamp, sorry to disappoint you, but I’ve started to think about DemoCamp as a microcosm for community and why companies get involved.

It’s best when the events are directly aligned with the marketing efforts of the organizations involved. FacebookCampToronto is hosted by Refresh Partners and Trapeze Media and sponsored by Facebook. Refresh Partners is a social media marketing firm. Hosting events like FacebookCamp Toronto enable them to demonstrate their expertise and influence in the market place. It helps establish thought leadership and market expertise. And hopefully it should be pretty obvious why Facebook, would sponsor events like FacebookCamp or Facebook Developers Garage. The idea is the value of Facebook as a platform increases as the number of developers, the number of applications and the number of users increase – similar to Metcalfe’s Law describing the value of the a telecommunications network. One of the reasons for focusing on growing the number of attendees at DemoCamp and embracing outside groups (“it’s the derivatives that matter most”), has been Reed’s Law, being technology agnositic has allowed the group to grow and while it enables the formation of smaller subgroups, the overall utility of being a DemoCamp or TorCamp participant increases with the number of people.

Measuring success for the events and the tools is pretty straight forward.

  • Unique Attendees or Visitors
  • Registrations or New Memberships
  • Presentation/Demo Submissions
  • Attendee Loyalty (aka Turnover)

These are very similar metrics to the ROI dimensions reported in OCRN’s Online Community ROI [PDF – 1.28Mb] report and Online Community Metrics [PDF – 455kb].

OCRN Online Community Metrics – Question 12: Which community metrics do you track?

  Very Important Moderately Important Not Important No Collected
Unique Visitors 62% 26% 12% 10%
New Member Registrations 62% 21% 17% 12%
Page Views 57% 26% 17% 12%
Retention / Attrition 50% 22% 5% 22%
Member Loyalty 46% 12% 0% 41%
Member Satisfaction 44% 12% 2% 41%
Most Active Members 43% 40% 5% 14%
Top Searches 40% 30% 2% 30%
Message Posts 38% 48% 14% 12%
Conversion 38% 18% 5% 40%
Advertising Performance 32% 20% 0% 49%
Influencer/Evangelist Identification 32% 29% 2% 37%
Member Lifecycle 28% 22% 2% 50%
First Time Contributors 24% 27% 10% 39%
Content Rating 22% 22% 2% 52%
Ratio: Unregistered to Registered Visitors 17% 34% 12% 37%
Ratio: Page Views Per Post 15% 39% 12% 34%
Reputation Changes 13% 28% 5% 54%
Ratio: Posts Per Thread 12% 28% 22% 38%
Content Tagging 12% 24% 2% 62%
Comments per Blog Post 11% 26% 13% 50%
Ratio: Searches Per Post 10% 12% 20% 58%
Podcasts & Video (linked to Uploaded) 8% 33% 0% 59%
Member Blog Posts 5% 31% 8% 56%
Size of Networks/Buddy lists 2% 22% 5% 70%
         

These metrics apply to online communities, but can be modified and interpreted for real world communities. Bill Johnston provides an update with data of 150 survey respondents from February 2008.

Top ranking metrics

  • Traffic patterns & statistics
  • Community member engagement
  • Unique number of visitors
  • New Member registrations
  • Member Satisfaction
  • Product feedback and/or ideation for R&D

We haven’t collected these statistics. However, since we started using EventBrite to collect registrations it should be a lot easier to begin to apply them to the community. We could look at message traffic in the Google Group and on the Skype Chat. Alvin Chin has reported that the sense of membership and emotional connection with a community is strongly correlated to the number of acquaintances. Perhaps it’s the number of member replies on Twitter. Or the number of shared friends on Facebook, Twitter, MySpace, Plaxo, MSN or other.

Tara and I talked about the health of communities and community metrics at FooCamp. Tara has done a great job setting up a wiki to track Community Metrics. Here are the metrics Tara has used in the past to measure community health:

  • the rate of attrition, especially with new members (I think it is really telling when you drive traffic that doesn’t stick around – you will have to really examine whether you are offering something of need)
  • the average length of time it takes for a newbie to become a regular contributor
  • number of referrals (strength of positive word of mouth)
  • multiple community crossover – if your members are part of many communities, how do they interact with your site? Flickr photos? Twittering? Etc.?
  • the number giving as well as the receiving actions – eg. readers receive, posters are giving (advice, knowledge, etc.). PopSugar has a neat reward system built in for this with their gifting for contributions in the community
  • community participation in gardening, policing and keeping the community a nicer place (eg. people who click on the ‘report this as spam’, people who edit the wiki for better layout, etc.)
  • number of apps built off of your API (if you have one) – a good ‘number’ measure as the number of apps usually correlate with your social capital

What would be the metric you’d like to see DemoCamp success measured against?

11 thoughts on “Measuring community success”

  1. Good post David. I think you can probably use the standard &quot;customer lifecycle&quot; big 4 categories espoused by web metrics guru <a href="http://www.webanalyticsdemystified.com/&quot; target="_blank">Eric T. Peterson</a> and others in terms of Reach, Acquisition, Conversion, and Retention when it comes to further classifying or focusing your metrics on what really matters. An initial scan of the metrics you've got listed shows representation in each of the 4 categories, with perhaps an increased importance placed on the acquisition side of things in terms of growing a community – an obvious metric when a community is young and growing… The maturity of a community matters in terms of what you measure as well — the longer it exists, the more conversion (call it engagement in this case? getting the desired behaviours from your community members is key) and retention become important. How long can we keep people engaged doing great things before people drift away or stop participating? <br />
    <br />
    And like all metrics, it's not about the number per se, it's about the change of the number over time and what direction its heading. Good stuff.

  2. Good post David. I think you can probably use the standard “customer lifecycle” big 4 categories espoused by web metrics guru Eric T. Peterson and others in terms of Reach, Acquisition, Conversion, and Retention when it comes to further classifying or focusing your metrics on what really matters. An initial scan of the metrics you’ve got listed shows representation in each of the 4 categories, with perhaps an increased importance placed on the acquisition side of things in terms of growing a community – an obvious metric when a community is young and growing… The maturity of a community matters in terms of what you measure as well — the longer it exists, the more conversion (call it engagement in this case? getting the desired behaviours from your community members is key) and retention become important. How long can we keep people engaged doing great things before people drift away or stop participating?

    And like all metrics, it’s not about the number per se, it’s about the change of the number over time and what direction its heading. Good stuff.

  3. What about outcomes-based measurements? Are we building the community just to feel like we belong somewhere, or are we trying to change the city? I suspect the latter.<br />
    <br />
    Of course, it's a community, so amongst all of us we probably have many objectives. If I may, I'll take a wild stab, though, and say over all we're trying to build Toronto into a generator of world-class Internet culture, which includes business, art, media, academic work, civic change, etc., although I'd say we've focused more on business, media, and civic change.<br />
    <br />
    So, perhaps we could ask (over a long term):<br />
    <br />
    * how much economic activity have we generated as a group? (hard, diffuse)<br />
    <br />
    * how many successful, sustainable companies have arisen from TorCamp? (better, easier)<br />
    <br />
    * how many wealthy individuals have exited from TorCamp that have gone on to mentor/fund the next generation of startups? (I suspect this is the key concept that has led to other tech regions exploding in growth.)<br />
    <br />
    * how much impact we've had on 'discourse' outside of the city? (feels like tracking PR)<br />
    <br />
    * how many people come to Toronto to learn about new concepts for living on the Internet? (saying we built something like the Toronto International Film Festival, but for Internet culture).<br />
    <br />
    That's about as far as I've thought through this, which isn't very far. I'm put the most thought on the business side, as you might tell.<br />
    <br />
    Anyway, if we think about Big Change first, then community metrics like adoption rates, churn, bonds become more specific, focused, and interesting. <br />
    <br />
    How many of the city's leading bloggers, digital artists, tech companies are active participants?<br />
    <br />
    How many jobs, grants, commissions have been created and sourced through the community?<br />
    <br />
    How many thought leaders have emerged from within the community and propelled to world stages outside the city? (i.e. the more Torontonians behind podiums around the world, the more people around the world will come here to find out what's happening.)

  4. What about outcomes-based measurements? Are we building the community just to feel like we belong somewhere, or are we trying to change the city? I suspect the latter.

    Of course, it’s a community, so amongst all of us we probably have many objectives. If I may, I’ll take a wild stab, though, and say over all we’re trying to build Toronto into a generator of world-class Internet culture, which includes business, art, media, academic work, civic change, etc., although I’d say we’ve focused more on business, media, and civic change.

    So, perhaps we could ask (over a long term):

    * how much economic activity have we generated as a group? (hard, diffuse)

    * how many successful, sustainable companies have arisen from TorCamp? (better, easier)

    * how many wealthy individuals have exited from TorCamp that have gone on to mentor/fund the next generation of startups? (I suspect this is the key concept that has led to other tech regions exploding in growth.)

    * how much impact we’ve had on ‘discourse’ outside of the city? (feels like tracking PR)

    * how many people come to Toronto to learn about new concepts for living on the Internet? (saying we built something like the Toronto International Film Festival, but for Internet culture).

    That’s about as far as I’ve thought through this, which isn’t very far. I’m put the most thought on the business side, as you might tell.

    Anyway, if we think about Big Change first, then community metrics like adoption rates, churn, bonds become more specific, focused, and interesting.

    How many of the city’s leading bloggers, digital artists, tech companies are active participants?

    How many jobs, grants, commissions have been created and sourced through the community?

    How many thought leaders have emerged from within the community and propelled to world stages outside the city? (i.e. the more Torontonians behind podiums around the world, the more people around the world will come here to find out what’s happening.)

Comments are closed.