Two Projects

Community Aggregator

This is something I’ve really wanted to build for a while. A community aggregator. So you search for whatever, maybe pastel drawing and the site would serve up a list of the highest rated online communities that focus on pastels. Traditional search engines fail to accomplish this because they do not look for community, they look for keywords, in-edges and the such. But community has different metrics and should have the input of the community itself. The motivation behind this is that people really want to feel like a part of a community and online communities are valid ways of fulfilling this need, and this proposal can make finding the right one easy.

There are numerous technical and academic challenges here.

1. Identify what metrics can indicate a online community. And along the same lines, what is an online community. This part of the project might be one of the longest. I plan to put myself in a few self-indicated communities and observe behavior, find computer science-related metrics, and find a definition of community. Yes, this is a primarily anthropological venture, and has elements of ethnography involved. I will be posting updates about this, but keeping most of my records in notebooks – it just feels right, I can share the notebooks later if it’s of interest.

2. The next step will be to test web crawling software on the metrics I’ve discovered.  To do this I’m going to need to familiarize myself with the software and web crawling and then apply the skills I learn to a few sites of my choosing. I am going to try and do this in Javascript, but I might need to go to Java if I want to be able to scale easily (use MapReduce and such). If the language/software doesn’t work or the metrics I found are not working, I will repeat the necessary steps.

3. Once I’m comfortable with crawling and I’m convinced my metrics work, I will unleash the crawler on the web (within limits of course) and build a database of communities. This step will probably need money or resources that I don’t have atm, but we will cross that bridge when we come to it haha.

4. The final step is to create a web interface for this, which is almost a trivial technical task next to web crawling. The site needs to integrate the database, make it searchable for a variety of parameters (activeness, size, focus), and make these communities votable/commentable by users so that people can review bad communities and such.

This is definitely a long-term project, and might not be done for years, that’s okay, each step is a valuable learning experience and something that sounds fun atm. So I’m going to start searching online communities and choose like 3-5 to do an ethnography of – which involves becoming a part of each one and participating and such, as well as taking measurements of the infrastructure.


One of my friends from last Summer, Shengwei, texted me about a week ago and asked if I wanted to meet up and discuss an opportunity. I happily obliged and we met up on campus. After some chat, he went into a pitch about a website/app that would “use the internet to get off the internet”. Sounds kinda cool right? It involves users signing up to meet and chat with other users about whatever the “talker” wants to talk about (the reciprocating party is the “talkee”). He also mentioned what I think is an awesome idea, and that is to allow people such as professors to give talks and have them be open to the other users. Now maybe alarm bells are going off in your head – what if some weirdo shows up to these talks. I had the same thought, so I proposed that we only allow people with .edu emails to sign up – which eliminates at least the dangerous weird kind of people from showing up.

I think this idea has a lot of potential, especially marketed as an educational, social app that allows people to talk about interesting things they actually like in physical space (as opposed to online space). Maybe I’m being overly optimistic, but I don’t think that it is a terrible idea, people might really use it (if we limit the user-group, maybe only Penn first).

So the reason Shengwei brought me in was because he was wondering if I wanted to make an MVP of this idea which, after the pitch and a bit of discussion, I was happy to do.

Now here is a bit of transparency. For one of my classes, I’m creating a webapp that has a lot of similar features to FaceTalkToMe, definitely not exactly the same, but close enough that I feel safe using parts of it for both projects. How’s that for an awesome class?

So these are the two projects (outside of schoolwork, and other commitments) that I will be focusing on. And I’ll try to keep you updated as I go. What do you think? Bound to fail? Brilliant ideas? I guess we will see!

This entry was posted in Anthropology, Computer Science. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s