How to Surface the Best Ideas

Chris Beard

9 responses

One significant part of Mozilla Labs will be the Ideas Area where we will encourages contribution of great ideas from the community. Currently, there is a moderately active newsgroup (mozilla.wishlist) where ideas are flowing. For Labs, I was thinking of something a bit more structured. But how much structure?

Two aspects that I would like feedback on:

1) How to track and categorize the ideas?

A forum/newsgroup or linear list structure is hard to navigate, so I was thinking of creating browseable categories. Under each category would appear Idea items and discussion around each of those. Should the categories map identically to what’s currently on Add-ons (See “All Categories”) with a few more. Should we use tags instead so that items can appear in multiple categories?

2) How to have the best/most popular ideas rise to the top?

I know Mozilla is known for having a collaborative process that surfaces the best ideas. How does this work? Give me your thoughts. Shall we use voting? 1-10 rating for ideas? Use a process like digg/bury from digg.com? Give each community member 1000 points and have them allocate their points across their favorite ideas? Simply track the level of discussion and activity around a topic area? Other?

9 responses

  1. Sam Hasler wrote on :

    1) Categorization: I would have thought tagging was the obvious choice.

    2) Popularity: This is where I think you really need get things right. The choice of voting system can make or break how useful the results are.

    (Before I get to my main point I’d like to say why not allow voting for categories/tags themselves. This could yeild interesting results, such as areas where users want to see improvement but no suitable ideas exits.)

    (Also, it would be good if you can track trends over time so that you can tell when a bad idea has been revised so that it is now considered a good idea. This might not be evident from the overal statistics. I’m thinking of how online shops show feedback for sellers with totals for a month, quater, year, and all time. Except keep them really simple, use sparklines! (see sparkline.org) )

    “Use a process like digg/bury from digg.com?”
    “1-10 rating for ideas?”

    These don’t take into account why people voted against an idea. Is it really a bad idea, or is it just badly designed / ugly but otherwise a good idea. Is the problem it’s trying to solve a valid problem but the solution way off mark / unimplementable.

    I think that often a simple Yes or No isn’t enough to really guage the level of support for an idea. It doesn’t allow you to prioritise a non-obvious but really good idea that only a few people have seen.

    With ratings the results are almost always dispalyed as an average, which hides a lot of the detail. Which is better an idea with 2 votes of 10 or one with 20 votes of 9? I’d argue the latter but how would you prioritise it. What if it has 10 votes of 10 and 5 votes of 1, how do you represent that. I’d like to see a graph of all the results, but keep it small, again use sparklines! (see sparkline.org)

    Also, ratings aren’t always used in the same way by everyone. Are the lower half still positive votes or not? (You could indicate that with the design of the voting interface)

    Better than a rating might be a “Gradient Of Agreement” where you give specific meanings to each level of support and show the tally for each level. (See http://twiki.org/cgi-bin/view/Codev/GradientsOfAgreement for more)

    Why not try to classify *why* ideas are good or bad, particularly why ideas are bad so that you can matchmake them to people who might be able to help. Is it ugly? let the artists/designers look at it. Is it unimplementable? let the code geniuses look at it. Does it have useability problems? let the useability experts look at it. Is it just a plain bad idea? then if so many people want it what’s the real problem it’s trying to solve? (not sure who that last one should go to)

    “Simply track the level of discussion and activity around a topic area?”

    I think you should track this as it can be useful but only combined with other measurements. It’s hard to say what the discussion means: is it a very good idea or a very bad one? Good ideas might also be so obvious or well put that no discussion is needed. So you might want to look for popular ideas that haven’t been discussed, or haven’t been discussed recently.

    “Give each community member 1000 points and have them allocate their points across their favorite ideas?”

    I’ve always had a problem with this approach. It assumes that everyone is equally able to prioritise well. While on an individual basis it might mean that you can priority one idea over another by giving it more votes, (e.g. vote for 2 ideas with 200/800 split) this prioritisation is lost when you amalgamate the votes with everyone elses.

    The more involved you are with the site, the better you are going to be at deciding what ideas are good and what are bad, and the more likely you are to spread your votes over several ideas, whereas casual visitors may not understand the full implications of their vote and will use it all up on ideas that may not be good (e.g. not implementable).

    It’s true that any system will suffer from this problem but I think this system in particular would have it’s results badly squewed by it.

    Which is the better idea? Both have 2000 points but one has 2 peoples votes of 1000 each and another has 20 peoples votes of 100 each. I’d argue the latter but how are you going to show this?

  2. Walter wrote on :

    I now this is not the spot, but I don’t feel like registering first ;o)
    I’d like to see my bookmarks ordered on a start-page like page. So my home-page could be my bookmarks-page.
    This page could show much more information (title, url, description, times visited, etc) and provide a better overview of your links (wich I lack from the side-bar).

  3. Sebastian wrote on :

    Hi!

    I suggest using tags to categorize ideas. This would be more flexible and the target audience would be perfectly ok with that. Tag clouds could then show more clearly where progress is going on.

    For rating, I would not use something like Digg, because it’s used by a too small number of users. I would use a more old-school approach – a five-star-rating bar.
    The project page itself should stress two types of contents primarily: comments and the idea. Not technical stuff or whatever, but something that people can “feel”. For projects that create a visible feature, there should be screenshots as soon as possible, for new APIs, a basic documentation should be online. It needn’t be complete, and everybody will understand that it can change every day, but having something to see makes people much more likely to comment and discuss.

    For discussions/comments, I would suggest a very easy tree-structured forum. Comments like in a blog is not flexible enough, it makes it difficult to have cocnversations. A full-featured forum like MozillaZine is too complicated. Something in between would be great.

  4. Sam Black wrote on :

    A digg-like system, I think, would require the least effort on the part of users and would work pretty well. The only thing you’d have to worry about is spamming.

    I also agree with funThomas — the most popular extensions are a pretty good guide to the best ideas in the firefox community.

  5. neil wrote on :

    Why not just use MediaWiki? It has an excellent content tracking and categorization system, which is very flexible and well tested.

  6. Mark wrote on :

    Not trying to simply go with what’s popular, but a digg-like system seems to make the most sense.
    I think the limited points distribution would be problematic. I don’t think this approach would have the intended effect as people could abuse the system to get around the limitation (signing up for multiple accounts) and limiting votes goes against the whole paradigm of encouraging innovation (if I’m active in the community I get punished?).
    Also, this system seems to have failed in bugzilla as the suite and Core (and others?) allow for 10000 votes.

    The 1-10 rating system is too complex. This would require a highly comitted and disciplined community. Users willing to debate if something is a 6 or a 7. A part of digg’s strength is it’s simplicity. It’s either good, and worth a click, or it’s not worth my time to bother with. When I see a set of assessment decisions like this, I don’t even want to bother.

    Having a view to display the most discussed ideas today/this week/month, could be beneficial, but may be deceptive in evaluating how good an idea is.

    Other thoughts on the matter:

    I think it’s important that users be allowed/encouraged to post media casting vision for their ideas. This could look like pictures, slide shows, flash, movies, etc. This does a couple things. It allows people to communicate beyond text: the next great idea for empowering browsers with a particular visual feedback might get passed over because people just don’t “get it” from reading text. Also, some people stink at textual articulation. This also may force users to hash through an idea to discover that it doesn’t work, or to develop it even further by being forced to make it work visually.

    Another thought is that in following discussion, other users may want to refine the idea so allow them to upload media as well.

    What about branching an idea? A user takes an idea and modifies it or adds to it and posts it back to the pool for voting. Just a thought

    One fear that comes to mind is users who attempt to abuse the system because they have a pet idea/bug that they’ve been touting for years and they want to see in the browser.

    Sites, like digg, which require user participation have historically had a very small number of voters and an even smaller number of submitters with most users simply watching, so being intentional about jump-starting the site is important.

  7. funTomas wrote on :

    Damn, I forgot to mention this fantastic voting page. Definitely check it out!

  8. funTomas wrote on :

    Why not to harness current information available? In AMO, you can find popularity ranking of extensions. MoCo has stated, it’ll consider popular extensions to integrate them into FF. Regarding the point 2, I think bugzilla provides you with sufficient voting system so again, why not to harness it too?

  9. Cameron wrote on :

    The categories on AMO aren’t particularly wonderful, you’d be best thinking up a few of your own. And tags are much better than directories.