Superstructures in Social Networks

"We can save memory by storing the year as two digits instead of four" ~80's programmer
"640K ought to be enough [memory] for anyone" ~ Bill Gates, 1981

You always have to think of the consequences of your design...
Or, as one of my favourite webcomics, xkcd, puts it:

The most recent reincarnation of the limits of software design has surfaced was Facebook's notorious 5000 friend limit. This is well above Dunbar's number (~150), a proposed theoretical natural limit to the number of useful social connections we can have.

Now I'm sure a Facebook software designer would argue (rightly) that having 5000 friends is probably some form of inappropriate use of Facebook (if not outright abuse), but it turns out that is exactly what happened when popular web personalities tried to use Facebook as a tool to connect with their readership. With sky rocking success, suddenly Facebook's 5000 friend limit was a lot closer than people originally thought. Although most intentions were good (to keep closer contact with their fan bases) it turns out their success was too much for the Facebook frame work.

Certainly, most of us who imagined internet fame didn't conceive of this type of double edged sword.

I was recently speaking with a colleague who is a lead software developer for a popular Facebook application who was describing the processing inefficiencies that managers don't seem to understand when they design systems. Recopying entire databases, poor message protocols etc. Even with the most efficient code, there was that nasty problem of becoming 'popular', and suddenly experiencing exponential network growth. In a system like Facebook, where connections are always 'two-way', it exacerbates the problem.

What does that mean exactly? Unlike Twitter, where connections are generally one way (I follow you, but you don't HAVE to follow me), Facebook connections are all bidirectional - There is one connection shared between us. However, Twitter, each connection is one way and as such, you can have one popular node (such as a Barack Obama) with many connections in without as many connections out. What does this translate into?

I can follow Barack, but I'm pretty sure he doesn't care too much to follow me.

This simple assumption dramatically cuts down on the memory, processing and bandwidth requirements needed to provide the same level of basic service. But even, if the technology will permit it, are our minds too "primitive" to keep up? Or is it just physically impossible to reply and stay current at that volume on such an intimate level (versus the traditional mass communication channel models)?

I'm quite surprised that Facebook servers don't burst into flame each time someone logs into their website. At the time of writing, my Facebook boasts exactly 450 friends. From what I can see on my friends connections, this is hardly staggering. If you asked me to design a search algorithm that identified the latest activity of 450 people and then sorted them in order of time and relevance on my homepage in real time (a few seconds loading and transmission delay), I might have a heart attack (or a very large consulting contract). However, it becomes very clear (especially with their previous incarnation of "More / Less stories about..." design) that there is an inherent (unstated) hierarchy of friends. I would almost expect it to be a sort of "page rank of friends" that helps Facebook efficiently sort interesting stories for you.