The other field Facebook wants to revolutionize - Fortune Tech
The social networking giant is leading a consortium aiming to make data centers cheaper and more efficient.
FORTUNE — Facebook is known for creating the most popular social networking tool, not designing hardware. But the company has taken a do-it-yourself approach to building out its data centers and the servers and racks that fill them. The result? Data centers that are 38% more efficient and 24% cheaper than average, according to Frank Frankovsky, director of hardware design and supply chain at Facebook.
In the hopes of driving the cost down further, Facebook has even “open sourced” its designs — making it possible for anyone to contribute to (and replicate) what its engineers have built. Last week, as most of the business world speculated on the social networking site’s upcoming IPO, Facebook held a conference for its Open Compute Project, a consortium that now includes the likes of Hewlett-Packard (HPQ), Dell (DELL) and AMD (AMD). We caught up with Frankovsky to find out more about Facebook’s open source strategy and what’s next for the Open Compute Project.
FORTUNE: Why did you start the Open Compute Project?
Frankovsky: When we designed and built our first data center, we exceeded even some of our own internal goals. And we immediately thought it would be unnatural not to share this because we’ve all benefited so much from open source software – like the infrastructure software we’ve built our business on. This is why our software engineers can focus on innovation every day, on making the world more connected. We don’t need to go and reinvent an operating system. So we thought, let’s go and open source the hardware space so that we can give back too. Also, no single company is ever going to have all of the best brainpower in the entire industry under one roof. By open sourcing, you can get the industry’s best brainpower focused together. You get a bunch of great ideas, and it accelerates the pace of innovation.
A lot of companies fight standardization and commoditization. How have traditional suppliers reacted to Open Compute?
While the initial reaction might have been resistance, these are great innovation companies and they know that at some point in order to remain competitive and successful you have to reinvent yourself.
Are there any other efforts out there to open source data center hardware?
We have partnerships with a whole bunch of other projects , but we are specifically focused on the hardware design in the data center, and to my knowledge there are no other projects specifically around this. The old method is to keep all your cards close to your chest without sharing. The biggest project that inspired me and all of us at Facebook to get involved is the open source operating system Linux and the impact it had on the market. We want to have a similar impact on hardware.
Are there technologies that you won’t “open source” and share with others?
We think really, really carefully about what we open source. We’ve shared how we pick data center sites. But when we open sourced our data center blueprints we didn’t include the main point of entry for fiber runs—we felt it was a security issue. So there are some things like that that we don’t put out in the open. But that’s really because we need to defend ourselves and our end users. The thing we won’t open source are the key innovations we have in the application space. Those are the unique things that differentiate Facebook and the reason more than 900 million people come to Facebook. Intel is one of the founding members of the Open Compute Project. It also happens to have one of the richest IP portfolios in the industry. Intel’s engineers have made significant contributions [to Open Compute] but we wouldn’t expect them to share how they design CPUs.
A lot of people would be surprised to know that some of what you’ve done with your designs is actually simplifying and taking away capabilities. Can you explain?
I don’t think anyone would argue that putting a bunch of plastic logos in front of a server is a good idea. But sometimes simple is actually really hard. People sometimes overcomplicate things. When you look at a design it might look really elegant because it’s got all kinds of whiz-bang features. But when you step back and ask how you can do this with minimum components, sometimes making it simple is really the hard part. Some of the most successful mobile devices don’t look like they do anything when you pull them out of the box; it’s a flat screen with just one button. But when you turn it on and it does exactly what you ask it to do, then you really understand the beauty and simplicity of the design. You don’t see the engineering efforts that went into making it simple.
So what’s next for the Open Compute Project?
We’re getting a lot of traction. Most of it is in data center and server design, and we’ve extended it to [server] racks. The storage space is something you’ll see heat up, and there’s also a lot of interest in networking. But a lot of the activity in the coming six months is going to be around storage—how open source storage really changes the market. Hopefully it will let companies choose the best of breed from both hardware and software. In the future there will be a smaller number of larger data center operators because of the trend towards cloud computing. We’ve reached an inflection point where things can get a little more standardized. What’s exciting about the future is that we can now apply the brain power to new and unique requirements in computing.
via tech.fortune.cnn.com
10 great and powerful Linux commands you may want to know | Unixmen
Command line is more powerful because you can do lot with them, you can tell your computer exactly what you want and get the appropriate answer.
That is why most of us use the terminal at least once a day. Today we will see another List of useful commands, i`m sure that most of you didn`t hear about them before. Lets see .
1- Run the last command as Root
sudo !!2- Save a file you edited in vim/vi without the needed permissions
:w !sudo tee %If you opened a file for edit and when saving you noticed that you forgot to open file as root, the command above is the solution.
3- Runs previous command but replacing : ”foo” by “bar”
^foo^barIf you did run a long command and you noticed that you made a mistake, to correct the command you can replace only the mistake by the correct word without the need to run the whole command again.
Another alternative to the above command is:
!!:gs/foo/barAs opposed to ^foo^bar, which only replaces the first occurrence of foo, this one changes every occurrence
4- Short and elegant way to copy or backup a single file before you edit it.
cp filename{,.bak}5- mtr, better than traceroute and has ping combined
mtr unixmen.commtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.
6- Empty a file without removing it
> file.txt7- Execute command without saving it in the history
<space>command8- Clear a terminal screen
ctrl-l9- List of commands you use most often
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head10- Get your external IP address.
curl ifconfig.meThis is all for today, enjoy
Colourlovers Buys Forrst to Build an Etsy for Digital Bling | PandoDaily
Creative community Colourlovers has acquired another creative community called Forrst in a bid to create the preeminent place on the Web where creative types can come together, make pretty things, help each other get better at making pretty things, share pretty things, and ultimately monetize them. The combined site wants to make great design simple and accessible and create a lucrative business doing that.
As someone who works with words, and usually words that are just black on white, the concept was all a bit esoteric to me, until I went on Colourlovers to check out what the site actually does. Hours flew by as I got consumed in what I could only call “playing” with color and patterns in a way I never had before.
I mean that as a huge compliment to the site. Colourlovers makes the art of creating colors and patterns come alive, even for non-designers like me, who’d never had the urge to “create” a color in a world where there are already millions of them. No matter how intimidated by design you are, you can quickly find yourself creating colors, pallattes, patterns, and shapes on Colourlovers and start looking at design in a completely different way. Imagine what people with actual talent can do with these tools.
I’m clearly not alone getting sucked into the site: Nearly 1.5 million registered users have shared more than five million colors, two million palettes, two million patterns, and nearly 160,000 templates. “It helps people discover their inner designers,” says CEO Darius “Bubs” Monsef IV. “We want to give you the tools to go from inspiration to execution and eventually monetization.” (I originally linked “Bubs” to the lovable, addled crack-head from The Wire, who is the only other “Bubs” I know. Monsef didn’t find it funny, so I linked to his Twitter account instead.)
Colourlovers has been around for seven years. Forrst is also a designer and developer community but is younger, smaller, and somewhat different. It is more focused on professional and career development and less on tools. The original concept was something like Tumblr, where developers and designers could share cool UIs and interesting libraries of fonts and design experiences, says founder Kyle Bragger.
Bragger hacked it together as a side project, while he was working on a wine social network with the infamous Gary Vaynerchuk. It started to take off, and Bragger wanted to devote more time to it, so Vaynerchuk invested along with a round from 500Startups. The total raised was about $205,000. Forrst was always sort of a kindred spirit to Colourlovers, and Bragger is happy to have more resources to help connect and inspire designers around the world.
Beyond all the feel-good prettiness of the site is a potentially disruptive vision of a Creative Market, which has the tongue-in-cheek tagline, “Handcrafted, mouse made.” The vision is an Etsy of individual made digital goods where people can browse and find all the different creative elements they need to build a site, whether it’s art, fonts, icons, templates, shapes — you name it. It takes the idea of stock art and applies to all aspects of designs. It hasn’t launched yet, but there are about 30,000 people on the list, waiting to see what it looks like, Monsef says. (Add your name to the list here.)
Colourlovers demystifies a lot of the design process, and whenever something is demystifying a process there are people who love it and hate it. But my guess is a lot of the community will love where Colourlovers and Forrst are taking them, because it’s a path towards making more money off of their creations the same way platforms like Facebook and the various app stores opened up the market for individual developers.
It’s a somewhat amorphous vision. At some level how do you “sell” a color or a shape? But it’s also an idea whose time has come nonetheless. Design has long been the unsung hero of the Web. Sure, it’s the first thing to get blamed if a site sucks. And in rare cases, the design is so great that it’s a site’s main differentiator. But usually design does its best work when it goes unnoticed: When it’s so pleasing that you focus on the site or app itself. It’s like a day with clean air or a beautiful blue sky.
Good design is simply table stakes for a site being successful these days, that means even basic sites without a top designer on staff need to up their design game. Colourlovers is seeking to build a business around making that simple. Expect more details on this ambitious vision in the future.
(I found that adorable “egghead” pattern on Colourlovers by Mariette. Seriously, there’s some awesome stuff on there.)
Cache them if you can | High Performance Web Sites
“The fastest HTTP request is the one not made.”
I always smile when I hear a web performance speaker say this. I forget who said it first, but I’ve heard it numerous times at conferences and meetups over the past few years. It’s true! Caching is critical for making web pages faster. I’ve written extensively about caching:
- Call to improve browser caching
- (lack of) Caching for iPhone Home Screen Apps
- Redirect caching deep dive
- Mobile cache file sizes
- Improving app cache
- Storager case study: Bing, Google
- App cache & localStorage survey
- HTTP Archive: max-age
Things are getting better – but not quickly enough. The chart below from the HTTP Archive shows that the percentage of resources that are cacheable has increased 10% during the past year (from 42% to 46%). Over that same time the number of requests per page has increased 12% and total transfer size has increased 24% (chart).
Perhaps it’s hard to make progress on caching because the problem doesn’t belong to a single group – responsibility spans website owners, third party content providers, and browser developers. One thing is certain – we have to do a better job when it comes to caching.
I’ve gathered some compelling statistics over the past few weeks that illuminate problems with caching and point to some next steps. Here are the highlights:
- 55% of resources don’t specify a max-age value
- 46% of the resources without any max-age remained unchanged over a 2 week period
- some of the most popular resources on the Web are only cacheable for an hour or two
- 40-60% of daily users to your site don’t have your resources in their cache
- 30% of users have a full cache
- for users with a full cache, the median time to fill their cache is 4 hours of active browsing
Read on to understand the full story.
My kingdom for a max-age header
Many of the caching articles I’ve written address issues such as size & space limitations, bugs with less common HTTP headers, and outdated purging logic. These are critical areas to focus on. But the basic function of caching hinges on websites specifying caching headers for their resources. This is typically done using max-age in the Cache-Control response header. This example specifies that a response can be read from cache for 1 year:
Cache-Control: max-age=31536000Since you’re reading this blog post you probably already use max-age, but the following chart from the HTTP Archive shows that 55% of resources don’t specify a max-age value. This translates to 45 of the average website’s 81 resources needing a HTTP request even for repeat visits.
Missing max-age != dynamic
Why do 55% of resources have no caching information? Having looked at caching headers across thousands of websites my first guess is lack of awareness – many website owners simply don’t know about the benefits of caching. An alternative explanation might be that many resources are dynamic (JSON, ads, beacons, etc.) and shouldn’t be cached. Which is the bigger cause – lack of awareness or dynamic resources? Luckily we can quantify the dynamicness of these uncacheable resources using data from the HTTP Archive.
The HTTP Archive analyzes the world’s top ~50K web pages on the 1st and 15th of the month and records the HTTP headers for every resource. Using this history it’s possible to go back in time and quantify how many of today’s resources without any max-age value were identical in previous crawls. The data for the chart above (showing 55% of resources with no max-age) was gathered on Feb 15 2012. The chart below shows the percentage of those uncacheable resources that were identical in the previous crawl on Feb 1 2012. We can go back even further and see how many were identical in both the Feb 1 2012 and the Jan 15 2012 crawls. (The HTTP Archive doesn’t save response bodies so the determination of “identical” is based on the resource having the exact same URL, Last-Modified, ETag, and Content-Length.)
46% of the resources without any max-age remained unchanged over a 2 week period. This works out to 21 resources per page that could have been read from cache without any HTTP request but weren’t. Over a 1 month period 38% are unchanged – 17 resources per page.
This is a significant missed opportunity. Here are some popular websites and the number of resources that were unchanged for 1 month but did not specify max-age:
- http://www.toyota.jp/ – 172 resources without max-age & unchanged for 1 month
- http://www.sfgate.com/ – 133
- http://www.hasbro.com/ – 122
- http://www.rakuten.co.jp/ – 113
- http://www.ieee.org/ – 97
- http://www.elmundo.es/ – 80
- http://www.nih.gov/ – 76
- http://www.frys.com/ – 68
- http://www.foodnetwork.com/ – 66
- http://www.irs.gov/ – 58
- http://www.ca.gov/ – 53
- http://www.oracle.com/ – 52
- http://www.blackberry.com/ – 50
Recalling that “the fastest HTTP request is the one not made”, this is a lot of unnecessary HTTP traffic. I can’t prove it, but I strongly believe this is not intentional – it’s just a lack of awareness. The chart below reinforces this belief – it shows the percentage of resources (both cacheable and uncacheable) that remain unchanged starting from Feb 15 2012 and going back for one year.
The percentage of resources that are unchanged is nearly the same when looking at all resources as it is for only uncacheable resources: 44% vs. 46% going back 2 weeks and 35% vs. 38% going back 1 month. Given this similarity in “dynamicness” it’s likely that the absence of max-age has nothing to do with the resources themselves and is instead caused by website owners overlooking this best practice.
3rd party content
If a website owner doesn’t make their resources cacheable, they’re just hurting themselves (and their users). But if a 3rd party content provider doesn’t have good caching behavior it impacts all the websites that embed that content. This is both bad a good. It’s bad in that one uncacheable 3rd party resource can impact multiple sites. The good part is that shifting 3rd party content to adopt good caching practices also has a magnified effect.
So how are we doing when it comes to caching 3rd party content? Below is a list of the top 30 most-used resources according to the HTTP Archive. These are the resources that were used the most across the world’s top 50K web pages. The max-age value (in hours) is also shown.
- http://www.google-analytics.com/ga.js (2 hours)
- (8760 hours)
- http://pagead2.googlesyndication.com/pagead/js/r20120208/r20110914/show_ads_impl.js (336 hours)
- http://pagead2.googlesyndication.com/pagead/render_ads.js (336 hours)
- http://pagead2.googlesyndication.com/pagead/show_ads.js (1 hour)
- https://apis.google.com/_/apps-static/_/js/gapi/gcm_ppb,googleapis_client,plusone/[…] (720 hours)
- http://pagead2.googlesyndication.com/pagead/osd.js (24 hours)
- http://pagead2.googlesyndication.com/pagead/expansion_embed.js (24 hours)
- https://apis.google.com/js/plusone.js (1 hour)
- http://googleads.g.doubleclick.net/pagead/drt/s?safe=on (1 hour)
- (3825 hours)
- http://connect.facebook.net/rsrc.php/v1/yQ/r/f3KaqM7xIBg.swf (164 hours)
- https://ssl.gstatic.com/s2/oz/images/stars/po/Publisher/sprite2.png (8760 hours)
- https://apis.google.com/_/apps-static/_/js/gapi/googleapis_client,iframes_styles[…] (720 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yv/r/ZSM9MGjuEiO.js (8742 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/qP7Pvs6bhpP.js (8699 hours)
- https://plusone.google.com/_/apps-static/_/ss/plusone/[…] (720 hours)
- http://b.scorecardresearch.com/beacon.js (336 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/lP_Rtwh3P-S.css (8710 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yA/r/TSn6F7aukNQ.js (8760 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yk/r/Wm4bpxemaRU.js (8702 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yZ/r/TtnIy6IhDUq.js (8699 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yy/r/0wf7ewMoKC2.css (8699 hours)
- http://static.ak.fbcdn.net/rsrc.php/v1/yO/r/H0ip1JFN_jB.js (8760 hours)
- http://platform.twitter.com/widgets/hub.1329256447.html (87659 hours)
- (8699 hours)
- http://platform.twitter.com/widgets.js (1 hour)
- https://plusone.google.com/_/apps-static/_/js/plusone/[…] (720 hours)
- http://pagead2.googlesyndication.com/pagead/js/graphics.js (24 hours)
- http://s0.2mdn.net/879366/flashwrite_1_2.js (720 hours)
There are some interesting patterns.
- simple URLs have short cache times – Some resources have very short cache times, e.g., ga.js (1), show_ads.js (5), and twitter.com/widgets.js (27). Most of the URLs for these resources are very simple (no querystring or URL “fingerprints”) because these resource URLs are part of the snippet that website owners paste into their page. These “bootstrap” resources are given short cache times because there’s no way for the resource URL to be changed if there’s an emergency fix – instead the cached resource has to expire in order for the emergency update to be retrieved.
- long URLs have long cache times – Many 3rd party “bootstrap” scripts dynamically load other resources. These code-generated URLs are typically long and complicated because they contain some unique fingerprinting, e.g., http://pagead2.googlesyndication.com/pagead/js/r20120208/r20110914/show_ads_impl.js (3) and http://platform.twitter.com/widgets/hub.>1329256447.html (25). If there’s an emergency change to one of these resources, the fingerprint in the bootstrap script can be modified so that a new URL is requested. Therefore, these fingerprinted resources can have long cache times because there’s no need to rev them in the case of an emergency fix.
- where’s Facebook’s like button? – Facebook’s like.php and likebox.php are also hugely popular but aren’t in this list because the URL contains a querystring that differs across every website. Those resources have an even more aggressive expiration policy compared to other bootstrap resources – they use
no-cache, no-store, must-revalidate. Once the like[box] bootstrap resource is loaded, it loads the other required resources: lP_Rtwh3P-S.css (19), TSn6F7aukNQ.js (20), etc. Those resources have long URLs and long cache times because they’re generated by code, as explained in the previous bullet.- short caching resources are often async – The fact that bootstrap scripts have short cache times is good for getting emergency updates, but is bad for performance because they generate many Conditional GET requests on subsequent requests. We all know that scripts block pages from loading, so these Conditional GET requests can have a significant impact on the user experience. Luckily, some 3rd party content providers are aware of this and offer async snippets for loading these bootstrap scripts mitigating the impact of their short cache times. This is true for ga.js (1), plusone.js (9), twitter.com/widgets.js (27), and Facebook’s like[box].php.
These extremely popular 3rd party snippets are in pretty good shape, but as we get out of the top widgets we quickly find that these good caching patterns degrade. In addition, more 3rd party providers need to support async snippets.
Cache sizes are too small
In January 2007 Tenni Theurer and I ran an experiment at Yahoo! to estimate how many users had a primed cache. The methodology was to embed a transparent 1×1 image in the page with an expiration date in the past. If users had the expired image in their cache the browser would issue a Conditional GET request and receive a 304 response (primed cache). Otherwise they’d get a 200 response (empty cache). I was surprised to see that 40-60% of daily users to the site didn’t have the site’s resources in their cache and 20% of page views were done without the site’s resources in the cache.
Numerous factors contribute to this high rate of unique users missing the site’s resources in their cache, but I believe the primary reason is small cache sizes. Browsers have increased the size of their caches since this experiment was run, but not enough. It’s hard to test browser cache size. Blaze.io’s article Understanding Mobile Cache Sizes shows results from their testing. Here are the max cache sizes I found for browsers on my MacBook Air. (Some browsers set the cache size based on available disk space, so let me mention that my drive is 250 GB and has 54 GB available.) I did some testing and searching to find max cache sizes for my mobile devices and IE.
- Chrome: 320 MB
- Internet Explorer 9: 250 MB
- Firefox 11: 830 MB (shown in about:cache)
- Opera 11: 20 MB (shown in Preferences | Advanced | History)
- iPhone 4, iOS 5.1: 30-35 MB (based on testing)
- Galaxy Nexus: 18 MB (based on testing)
I’m surprised that Firefox 11 has such a large cache size – that’s almost close to what I want. All the others are (way) too small. 18-35 MB on my mobile devices?! I have seven movies on my iPhone – I’d gladly trade Iron Man 2 (1.82 GB) for more cache space.
Caching in the real world
In order to justify increasing browser cache sizes we need some statistics on how many real users overflow their cache. This topic came up at last month’s Velocity Summit where we had representatives from Chrome, Internet Explorer, Firefox, Opera, and Silk. (Safari was invited but didn’t show up.) Will Chan from the Chrome team (working on SPDY) followed-up with this post on Chromium cache metrics from Windows Chrome. These are the most informative real user cache statistics I’ve ever seen. I strongly encourage you to read his article.
Some of the takeaways include:
- ~30% of users have a full cache (capped at 320 MB)
- for users with a full cache, the median time to fill their cache is 4 hours of active browsing (20 hours of clock time)
- 7% of users clear their cache at least once per week
- 19% of users experience “fatal cache corruption” at least once per week thus clearing their cache
The last stat about cache corruption is interesting – I appreciate the honesty. The IE 9 team experienced something similar. In IE 7&8 the cache was capped at 50 MB based on tests showing increasing the cache size didn’t improve the cache hit rate. They revisited this surprising result in IE9 and found that larger cache sizes actually did improve the cache hit rate:
In IE9, we took a much closer look at our cache behaviors to better understand our surprising finding that larger caches were rarely improving our hit rate. We found a number of functional problems related to what IE treats as cacheable and how the cache cleanup algorithm works. After fixing these issues, we found larger cache sizes were again resulting in better hit rates, and as a result, we’ve changed our default cache size algorithm to provide a larger default cache.
Will mentions that Chrome’s 320 MB cap should be revisited. 30% seems like a low percentage for full caches, but could be accounted for by users that aren’t very active and active users that only visit a small number of websites (for example, just Gmail and Facebook). If possible I’d like to see these full cache statistics correlated with activity. It’s likely that user who account for the biggest percentage of web visits are more likely to have a full cache, and thus experience slower page load times.
Next steps
First, much of the data for this post came from the HTTP Archive, so I’d like to thank our sponsors: Google, Mozilla, New Relic, O’Reilly Media, Etsy, Strangeloop, dynaTrace Software, and Torbit.
The data presented here suggest a few areas to focus on:
Website owners need to increase their use of a Cache-Control max-age, and the max-age times need to be longer. 38% of resources were unchanged over a 1 month period, and yet only 11% of resources have a max-age value that high. Most resources, even if they change, can be refreshed by including a fingerprint in the URL specified in the HTML document. Only bootstrap scripts from 3rd parties should have short cache times (hours). Truly dynamic responses (JSON, etc.) should specify must-revalidate. A year from now rather than seeing 55% of resources without any max-age value we should see 55% cacheable for a month or more.
3rd party content providers need wider adoption of the caching and async behavior shown by the top Google, Twitter, and Facebook snippets.
Browser developers stand to bring the biggest improvements to caching. Increasing cache sizes is a likely win, especially for mobile devices. Data correlating cache sizes and user activity is needed. More intelligence around purging algorithms, such as IE 9′s prioritization based on mime type, will help when the cache fills up. More focus on personalization (what are the sites I visit most often?) would also create a faster user experience when users go to their favorite websites.
It’s great that the number of resources with caching headers grew 10% over the last year, but that just isn’t enough progress. We should really expect to double the number of resources that can be read from cache over the coming year. Just think about all those HTTP requests that can be avoided!
How Reddit ranking algorithms work - amix.dk
This is a follow up post to How Hacker News ranking algorithm works. This time around I will examine how Reddit’s default story and comment rankings work. Reddit’s algorithms are fairly simple to understand and to implement and in this post I’ll dig deeper into them.
The first part of this post will focus on story ranking, i.e. how are Reddit stories ranked? The second part of this post will focus on comment ranking, which does not use the same ranking as stories (unlike Hacker News), Reddit’s comment ranking algorithm is quite interesting and the idea guy behind it is Randall Munroe (the author of xkcd).
Digging into the story ranking code
Reddit is open sourced and the code is freely available. Reddit is implemented in Python and their code is located here. Their sorting algorithms are implemented in Pyrex, which is a language to write Python C extensions. They have used Pyrex for speed reasons. I have rewritten their Pyrex implementation into pure Python since it’s easier to read.
The default story algorithm called the hot ranking is implemented like this:
#Rewritten code from /r2/r2/lib/db/_sorts.pyx from datetime import datetime, timedelta from math import log epoch = datetime(1970, 1, 1) def epoch_seconds(date): """Returns the number of seconds from the epoch to date.""" td = date - epoch return td.days * 86400 + td.seconds + (float(td.microseconds) / 1000000) def score(ups, downs): return ups - downs def hot(ups, downs, date): """The hot formula. Should match the equivalent function in postgres.""" s = score(ups, downs) order = log(max(abs(s), 1), 10) sign = 1 if s > 0 else -1 if s < 0 else 0 seconds = epoch_seconds(date) - 1134028003 return round(order + sign * seconds / 45000, 7)In mathematical notation the hot algorithm looks like this (I have this from SEOmoz, but I doubt they are the author of this):
Effects of submission time
Following things can be said about submission time related to story ranking:
- Submission time has a big impact on the ranking and the algorithm will rank newer stories higher than older
- The score won’t decrease as time goes by, but newer stories will get a higher score than older. This is a different approach than the Hacker News’s algorithm which decreases the score as time goes by
Here is a visualization of the score for a story that has same amount of up and downvotes, but different submission time:
The logarithm scale
Reddit’s hot ranking uses the logarithm function to weight the first votes higher than the rest. Generally this applies:
- The first 10 upvotes have the same weight as the next 100 upvotes which have the same weight as the next 1000 etc…
Here is a visualization:
Without using the logarithm scale the score would look like this:
Effects of downvotes
Reddit is one of the few sites that has downvotes. As you can read in the code a story’s “score” is defined to be:
- up_votes - down_votes
The meaning of this can be visualized like this:
This has a big impact for stories that get a lot of upvotes and downvotes (e.g. controversial stories) as they will get a lower ranking than stories that just get upvotes. This could explain why kittens (and other non-controversial stories) rank so high :)Conclusion of Reddit’s story ranking
- Submission time is a very important parameter, generally newer stories will rank higher than older
- The first 10 upvotes count as high as the next 100. E.g. a story that has 10 upvotes and a story that has 50 upvotes will have a similar ranking
- Controversial stories that get similar amounts of upvotes and downvotes will get a low ranking compared to stories that mainly get upvotes
How Reddit’s comment ranking works
Randall Munroe of xkcd is the idea guy behind Reddit’s best ranking. He has written a great blog post about it:
You should read his blog post as it explains the algorithm in a very understandable way. The outline of his blog post is following:
- Using the hot algorithm for comments isn’t that smart since it seems to be heavily biased toward comments posted early
- In a comment system you want to rank the best comments highest regardless of their submission time
- A solution for this has been found in 1927 by Edwin B. Wilson and it’s called “Wilson score interval”, Wilson’s score interval can be made into “the confidence sort”
- The confidence sort treats the vote count as a statistical sampling of a hypothetical full vote by everyone - like in an opinion poll
- How Not To Sort By Average Rating outlines the confidence ranking in higher detail, definitely recommended reading!
Digging into the comment ranking code
The confidence sort algorithm is implemented in _sorts.pyx, I have rewritten their Pyrex implementation into pure Python (do also note that I have removed their caching optimization):
#Rewritten code from /r2/r2/lib/db/_sorts.pyx from math import sqrt def _confidence(ups, downs): n = ups + downs if n == 0: return 0 z = 1.0 #1.0 = 85%, 1.6 = 95% phat = float(ups) / n return sqrt(phat+z*z/(2*n)-z*((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n) def confidence(ups, downs): if ups + downs == 0: return 0 else: return _confidence(ups, downs)The confidence sort uses Wilson score interval and the mathematical notation looks like this:
In the above formula the parameters are defined in a following way:
- p is the observed fraction of positive ratings
- n is the total number of ratings
- zα/2 is the (1-α/2) quantile of the standard normal distribution
Let’s summarize the above in a following manner:
- The confidence sort treats the vote count as a statistical sampling of a hypothetical full vote by everyone
- The confidence sort gives a comment a provisional ranking that it is 85% sure it will get to
- The more votes, the closer the 85% confidence score gets to the actual score
- Wilson’s interval has good properties for a small number of trials and/or an extreme probability
Randall has a great example of how the confidence sort ranks comments in his blog post:
If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there’s not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes — figuring that by the time it’s also gotten 40 upvotes, it’s almost certain it will have fewer than 20 downvotes. And the best part is that if it’s wrong (which it is 15% of the time), it will quickly get more data, since the comment with less data is near the top.
Effects of submission time: there are none!
The great thing about the confidence sort is that submission time is irrelevant (much unlike the hot sort or Hacker News’s ranking algorithm). Comments are ranked by confidence and by data sampling - - i.e. the more votes a comment gets the more accurate its score will become.
Visualization
Let’s visualize the confidence sort and see how it ranks comments. We can use Randall’s example:
As you can see the confidence sort does not care about how many votes a comment have received, but about how many upvotes it has compared to the total number of votes and to the sampling size!Application outside of ranking
Like Evan Miller notes Wilson’s score interval has applications outside of ranking. He lists 3 examples:
- Detect spam/abuse: What percentage of people who see this item will mark it as spam?
- Create a “best of” list: What percentage of people who see this item will mark it as “best of”?
- Create a “Most emailed” list: What percentage of people who see this page will click “Email”?
To use it you only need two things:
- the total number of ratings/samplings
- the positive number of ratings/samplings
Given how powerful and simple this is, it’s amazing that most sites today use the naive ways to rank their content. This includes billion dollar companies like Amazon.com, which define Average rating = (Positive ratings) / (Total ratings).
Conclusion
I hope you have found this useful and leave comments if you have any questions or remarks.
Happy hacking as always :)
Related
- Reddit’s comment ranking algorithm, discusses a bug that Reddit’s implementation has
Remarkably simple and powerful.
Linus Torvalds: The King of Geeks (And Dad of 3) | Wired Enterprise
The license plate on Linus Torvalds’ Mercedes SLK convertible says it all. The frame running around the outside of the plate reads “Mr. Linux. King of Geeks.” But the plate itself says “Dad of 3.”
If you meet Linus Torvalds, he comes off as a mild-mannered, down-to-earth Finnish-American. He lives with his wife Tove, three kids, a cat, a dog, a snake, a goldfish, a bunny and a pet rat in a comfortable 6,000 square foot home just north of Portland’s tony Lake Oswego neighborhood. The house is yellow — his favorite color — and so’s the Mercedes.
But he’s not really like any of his neighbors. He drives his Mercedes fast, slamming the car into gear and flooring it. There’s no coaxing, no hesitation. Either the hammer is down, or the car is at rest. And he has an abnormal number of stuffed penguins on his mantle.
He leads a double-life. He’s the kind of guy who plays poker with the guys for a $20 buy-in every couple of weeks. But at the same time, he in charge of Linux, a truly remarkable open-source software development project that over the past two decades has shaken Microsoft and provided the building blocks for internet giants such as Google, Facebook and Amazon.
Linus Torvalds has reached middle age, and so has Linux. Nowadays, it’s easy to take both of them for granted. But both are still going strong — very strong. Linus still runs the Linux kernel with his unique brand of no-nonsense attitude. Two weeks ago, he called the makers of SUSE Linux morons because of the operating system’s security requirements. And Linux? It’s everywhere. Next week, Red Hat will become the first $1 billion open source company.
Linux began life as an underdog project. Torvalds started it while he was a student at the University of Helsinki because he wanted to improve Unix on his Intel 386 computer. But it soon became an antidote not only to the massive Unix servers built by the likes of Digital Equipment Corp and Sun Microsystems, but to Microsoft’s Windows operating system.
Throughout the ’90s and on into the next decade, the fight was fierce on both fronts, but now, so many of the battles are won. DEC and Sun don’t exist anymore. And Microsoft is playing quite nicely with Linux and other open source tools. Linux isn’t the hot-button topic it was once was. It’s just plain successful.
More than 8,000 developers have contributed to the Linux kernel in the past seven years, according to the Linux Foundation. And it has even become a standard operating system on custom-built consumer devices. You can find it on everything from inflight entertainment systems to streaming video players to Google’s Android phones. “It became the plumbing,” says Jeremy Allison, a Google engineer who speaks frequently on the topic of open source and is himself a lead developer with another coding project, called Samba.
And Linus became a dad. But what a dad he is.
Linus Torvalds Meets Robert Downey, Jr.
Red Hat can thank Linus for reaching $1 billion in annual revenue. And Linus can thank Red Hat for his yellow house in Portland. Prior to its initial public offering in 1999, Red Hat gave Torvalds what turned out to be about $1 million in stock. But Torvalds says that it was his only big Linux payout. Stock that he was awarded from Transmeta and another Linux startup, VA Systems, wasn’t worth very much by the time he was allowed to sell it.
Still, Torvalds’ life is pretty darned good by geek standards. He gets paid by the non-profit Linux Foundation to manage the open source software that he loves and — when he wants to — can fly around the world to talk about it. He has the freedom to pursue his other passion: diving. Last week, Torvalds and his friend Dirk Hohndel spent a few days in the 40 degree waters of the Hood Canal, helping to dive-certify six native American geoduck hunters, and Torvalds has even started writing open-source dive-log software.
Last year, Intel invited Torvalds and Tove to a pre-Oscar party in Hollywood, where he rubbed elbows with the likes of Robert Downey Jr., who didn’t know who he was, and Mad Men star Jon Hamm, who did.
Does he have any regrets? “Not at all,” he says. “Quite the opposite, actually. I’m very happy with feeling that I’ve done the right thing.” He adds: “I mean, if I’d started a company, that wouldn’t have been because I wanted to start a company. I concentrated on the technical side because that’s what I wanted to do.”
And that’s good news for just about every big internet company, along with the startups that aspire to displace them, because they love to use Linux.
The Linux Storm
Linux worked because three powerful forces just happened to converge. First, Linux started just as Intel’s processors were getting ready for prime time. Long before company employees were sneaking iPads and smartphones into the office, there were Linux freaks sneaking Intel machines into corporations to build prototype new programs and build cheap websites and file and print servers.
The second force was the GNU General Public License. In the 1980s, the Unix makers had done well, but they’d kept a lot of their best technology to themselves. This had been good for business, but in the long run it was bad for Unix. By 1991, there were many incompatible versions of Unix. But Linux’s license dictated that anybody who made changes had to share them. That’s kept the project from splitting apart, and it ensured that any really good software gets used by everybody.
But the third factor was Torvalds himself, who has put his personal stamp on the Linux in a way that is rare in the open-source world.
When Linus Torvalds moved to the U.S. in the late 1990s, the Linux hype was at its peak. And Linux’s creator was a particularly effective spokesman for the open-source revolution. He worked for an interestingly secretive chip startup called Transmeta — it fizzled out in the post-dot com implosion — but as long as reporters didn’t ask about Transmeta itself, Torvalds was the kind of guy who would speak his mind, apparently unconcerned with who he might happen to piss off.
Torvalds became the perfect foil to the monopolistic, unlikable, Bill Gates. He was low key, unassuming, a regular guy who was into computers just for fun. That was the name of his surprisingly readable autobiography, written in 2002 with journalist David Diamond — a book that Torvalds says he never thinks about today.
Torvalds is still doing things just for fun. He’s a free operator who pulls no punches in technical online discussions, but he’s not a blowhard. It’s enough to give him geek credibility, but to keep him from alienating the smart people. The makers of SUSE Linux know what we’re talking about.
The Job Offer From Steve Jobs
That passion to make the right design choice is still what drives Torvalds, even as Linux enters its comfortable middle age. “Linus, the person, certainly like all of us, he’s gotten older,” says Dirk Hohndel, the diving-buddy of Torvalds’ who also happens to be chief Linux and open-source technologist at Intel. “But Linus, the god of Linux, has not changed at all. He is still the same fiery aggressive, flaming wild, determined true-believer — the person who really knows exactly what he wants.”
Torvalds may have been a foil to Gates, but Linux’s creator probably has more in common with Steve Jobs. Torvalds leads the Linux project, not so much by writing code, but by arbitrating disputes and making the technical decisions that keep the project moving in the right direction. And that’s ability is similar to Jobs’ fanatical attention to design detail, says Google’s Allison.
“Jobs had this wonderful design sense of taste. He created these beautiful products that everybody loved,” he says. “Linus has engineering taste, and that’s the thing that kind of makes him special. He can look at all these potentially competing solutions and cut through the bullshit and say, no this is the right one to choose.”
“He’s good at that,” Allison adds. “It means he’s a dick sometimes, but he’s good at it.”
Torvalds has never met Bill Gates, but around 2000, when he was still working at Transmeta, he met Steve Jobs. Jobs invited him to Apple’s Cupertino campus and tried to hire him. “Unix for the biggest user base: that was the pitch,” says Torvalds. The condition: He’d have to drop Linux development. “He wanted me to work at Apple doing non-Linux things,” he said. That was a non-starter for Torvalds. Besides, he hated Mac OS’s Mach kernel.
“I said no,” Torvalds remembers.
Jobs He’s Not
But the Jobs-Torvalds analogy breaks down pretty quickly. Jobs was fabulously wealthy, dated celebrities, and didn’t write cute things about his kids on his license plate. In fact, he didn’t even use license plates. And when he had a software problem at Apple, he didn’t sit down and write an amazing new program that solved the issue. Torvalds has done that kind of thing.
On the day Wired visited Torvalds last month, his slightly obsessive attention to detail was on full display. Torvalds quickly invited us in and immediately starting making espresso after espresso in his modern kitchen. His employer, the Linux Foundation, had just bought him a brand new $3,000 Jura espresso maker, and he and his wife Tove were concerned that something is wrong.
Tove had been complaining about a metallic aftertaste, and Torvalds thought it may be a problem too. He handed over an espresso, asking: “Do you taste it?”
To us, the bitter, creamy espresso tasted like it came from a fine coffee shop.
Torvalds kept making coffees all morning, leaving us jittery and awake as we settled down to talk in the billiard room next to the modest unadorned home office that is the nerve center for Linux. It’s above the three-car garage. The kind of place where your typical suburban dad would keep his guitar collection and rock out with his buddies over a couple of beers on a Thursday night. Instead, Torvalds spends most days here — alone — managing what is surely the most important open-source software project in the planet.
As Torvalds himself concedes, Linux’s stormy days of fighting over big issues are largely behind it. “I am personally way less open to radical new redesigns,” he says. “We’ve done the radical redesigns to the point where most of the things we do, we do for damned good reason, and doing something radically different would be just stupid.”
Linux hosts an annual event called the Linux Kernel Summit, and to hear Torvalds describe it, it sounds almost like a synod of medieval theologians. “We had the most boring two-hour session on key-signing each others keys,” he remembers. “Boy that was not fun.”
In fact, Linux’s creator doesn’t really even like to talk about technology. He’d rather write. “I think it’s so much easier to be very precise in what you write and give code examples and stuff like that,” he says. “I actually think it’s very annoying to talk technology face-to-face. You can’t write down the code.”
He’d rather talk about politics. Or scuba diving. Or the state of the public school system. Or the taste of coffee.
From YC Rejection to 10,000 Users in 1 Month | Codiqa
My co-founder Ben Sperry and I applied for the W2012 Y Combinator funding round last fall with Codiqa, our jQuery Mobile developer tools and services company based in Madison and Milwaukee, Wisconsin.
I interviewed with YC for the W2010 round with my last startup, which ultimately failed. It was a really great experience and I had a blast, even though we didn’t get funded. Being able to talk with really brilliant people in the orange room, meeting and pitching the partners, and exploring the tech capital of the world was an experience I will always cherish.
This time around, I was really confident we’d at least get an interview for Codiqa, so we submitted early and iterated on the application after that. A week before the application deadline we got a message on Hacker News from Robbert Morris with a question for us to clarify something on our application. I took this as a sign this would be the last time they checked the application. Too bad, since we made a ton of changes after that which might not have been reviewed (if that is true, it’s an argument against submitting early).
My good friend Ryan Oldenburg, now lead Android developer at Hipmunk, put me in touch with Steve Huffman, co-founder of Hipmunk and Reddit, for some much needed feedback on our application. He basically said that we needed to do a better job explaining why jQuery Mobile matters. In the end, we didn’t do a good enough job of that, and I think our other answers lacked substance and confidence (and were too MBA sounding). On Halloween night we received an email saying we were rejected, had a moment of sorrow, and moved on.
Not long after, the jQuery Mobile project started following our updates and expressed interest in what we were building. The fact that Codiqa is 100% dedicated to jQuery Mobile, we believe, set it apart from any other mobile app building tool. We began talking with Todd Parker, leader of the jQuery Mobile project, who was instrumental in giving us early feedback as we developed Codiqa, from its evolution as a desktop application to a web-based service.
We launched our private beta of Codiqa in January of 2012 to about 500 users. For two months we worked feverishly to build our MVP, fixing bugs and adding necessary features. We opened publicly in late February with an announcement to our mailing list and Twitter followers.
In February we approached Todd with a proposal to put a free, embedded version of Codiqa directly on the jQuery Mobile homepage. He loved the idea, and on the night of February 27th we went live directly on jquerymobile.com (scroll down to play with a basic version of Codiqa).
Since then, Codiqa has been growing beyond our wildest imaginations. By the end of this week we will have over 10,000 users, up from about 1,000 just a month ago. The excitement surrounding jQuery Mobile is huge, and we have people from all over the world using it. Here is a graph of total registered users over the last three months:
Here is a graph of new accounts per day over the last three months:
Codiqa launched with a freemium model: we have a basic builder that only allows HTML exporting, no saving on codiqa.com or using our sharing and theming tools. We try to upsell through saving your app instead of just downloading the HTML. Right now we are not satisfied with our conversion rate from free to paid. We are going to run some experiments, but we’d love to hear your ideas (NOTE: We just made a change which seems to be really improving this!). Here is a graph of free users and users with a plan (we just launched, so almost all users are still in their trial period):
And the conversion rate from free to paid:
We hit a major milestone yesterday: our first successful charge after a 30-day trial ended. We have about 1400 users in a free trial right now so we expect to successfully convert and charge a number of them to bring in some revenue.
For the next few months, our plan is to focus on our “hot dog” (see Rework) – the builder we’ve developed. We are going to be adding more components, fixing bugs, and updating our pricing to better reflect the use-cases we’ve heard from our customers. We are going to be launching an alpha version of our desktop builder in a month or so and are going to spend more time blogging and sharing our expertise with our users.
Codiqa has given both Ben and I some really amazing opportunities. When I think back on my two years since graduating college, nothing has grown my career nearly as much as building Codiqa. I left my 9-5 job two months ago. Since quitting and then launching Codiqa, I’ve been approached to lead a chapter in the jQuery Mobile Cookbook, a project lead by appendTo and published by O’Reilly Media. I also started doing part-time consulting to pay the bills with appendTo and some of the best Javascript and jQuery developers in the world. I’ve learned a ton from them already. Both Ben and I have been approached numerous times with employment offers since we launched. It seems like starting a business is a great way to quickly boost your career. My only regret was not starting on it sooner!
We are really excited for the next few months. We want to be very transparent about our growth as we bootstrap Codiqa. We would love to hear your feedback or any ideas you have on how we can grow Codiqa! Please leave a comment below if you have something to add. Stay tuned for more posts as we build Codiqa!
“Boy CEO” Mark Zuckerberg’s Two Smartest Projects Were Growing Facebook And Growing Up | Fast Company
From studying leaders he admired to taking elocution lessons, Zuckerberg made his evolution into a world-class CEO a personal project. Photo by Martin Schoeller/AugustIt was a minor meta moment, the perfect inside joke to kick off a September day that was otherwise all business. The occasion was f8 2011, the erratically scheduled, mostly annual conference for Facebook developers and social-media innovators, a gathering that now has a pilgrimage-like quality for the Facebook faithful. It is one of the few opportunities for legions of Mark Zuckerberg fans viewing the event live online to observe their spotlight-averse hero perform a rite native to the CEO species: the keynote address. Ladies and gentlemen, Mark Zuckerberg. The whoops turned to laughter almost immediately; it took only a few seconds for the assembled engineers, designers, brand stewards, marketing mavens, nonprofiteers, pundits, bloggers, and investors to realize that they were being punked.
On stage was the comedian Andy Samberg, fully in character as “Zuck Dawg” in a hoodie, jeans, and Adidas sandals. “I want to start by focusing on some key issues,” he said. “The first is the importance of authentic identity. I …” he paused, hand over heart, “… am Mark Zuckerberg.” It was a delicious moment for the Facebook staff, now 3,200 strong. For them, it’s always been about identity. Since Facebook’s February 2004 launch, the company has succeeded because hundreds of millions of people—slowly at first and then in crashing herds—became comfortable sharing their true selves on the site. It is precisely that authenticity that makes Facebook matter to its 845 million users. If Marshall McLuhan had lived long enough to have a Facebook profile, his status might read thus: The medium isn’t just the message; the medium has become us.
Zuckerberg’s bet was that Facebook’s guiding essence, the Hacker Way, could be baked into a new style of management for a new type of company.But the moment belonged first and foremost to Zuckerberg, who for years has had his own identity problem: “boy CEO.” Young, arrogant, and awkward—no one believed that Zuckerberg could survive the adult swim of real business, and thanks to his depiction in The Social Network, some folks will forever see him as the fatally flawed psychopathic robot nerd looking to steal your code, your personal data, your girlfriend. “I don’t think about it … much,” he once told me when I asked him how he handles all the noise, measuring his words as he always does. “I understand why people need to have these dialogues, to ask these questions. We have so much to do here, we don’t think about it if we don’t have to.”
I first met Zuckerberg and his colleagues five years ago, when Facebook had just 19 million users and was on the verge of opening up its platform to outside developers. Looking back on more than 400 hours of reporting with Facebook staffers, investors, and people in the site’s ecosystem, including a visit in late December, plus more than seven hours of one-on-one interviews with Zuckerberg, one fact is clear: The only thing that could have derailed Facebook’s climb to Internet domination was the inexperience of a young CEO. He had never held a proper job before and, by virtue of his own ballsy negotiations, could not be ousted from his position. (Facebook, citing IPO-quiet-period restrictions, declined to make Zuckerberg available for this story.) But what was largely interpreted as control freakery in service of a bigger exit strategy turned out to be a real vision. “So many businesses get worried about looking like they might make a mistake, they become afraid to take any risk,” he told me after the company moved into its first grown-up tech campus, on Palo Alto’s California Avenue, in 2009. “Companies are set up so that people judge each other on failure. I’m not going to get fired if we have a bad year. Or a bad five years. I don’t have to worry about making things look good if they’re not. I can actually set up the company to create value.”
You Know What’s Cool? A Billion Dollars
Nine of the big winners when Facebook goes public.MARK ZUCKERBERG
Founder, chairman, and CEO
Share value*
$28.4 billionJIM BREYER
The venture capitalist led an early $12.7 million funding round.
Share value:
$11.4 billionDUSTIN MOSKOVITZ
Cofounder and first CTO
Share value:
$7.6 billionSEAN PARKER
The entrepreneur was Facebook’s founding president.
Share value:
$4 billionPETER THIEL
The PayPal cofounder was Facebook’s first angel investor.
Share value:
$2.5 billionDAVID CHOE
The artist received stock options for murals painted at Facebook HQ in 2005.
Share value:
$200 millionSHERYL SANDBERG
COO, Facebook
Share value:
$100 million
(Plus an additional 39 million in restricted stock units worth more than $1.1 billion)CHRIS HUGHES
Cofounder and first unofficial spokesperson
Share value:
$100 millionDONALD GRAHAM
The Washington Post Co. CEO has sat on Facebook’s board since 2008.
Share value:
$45 million*Based on a $100 billion valuationThis February, as part of his effort to ensure that this remain true, Zuckerberg asked investors to back a company in which he will retain 57% of the voting stock. He outlined the company’s guiding principle, which he calls the Hacker Way, in a personal letter to potential shareholders that accompanied the IPO filing. This is the idea that gives Facebook its identity—as a company that questions assumptions, moves fast, takes risks, shares information, and learns from other smart people. Nowhere does this manifest itself more clearly than in the company’s regular hackathons, extended coding sessions where employees race to invent new products. “What you’ll hear over and over and over again is ‘why?’ ” says HR chief Lori Goler of a culture filled with millennials (average age: 28) who question the purpose of every feature and expect a logical answer.
Turning that “we’re all coding together in one big room, and we get great ideas and move fast because anyone can walk up to anyone else” ethos into a business required the young CEO to turn his hacker sights on himself. An experiential learner, Zuckerberg transformed himself with astonishing discipline into a CEO worthy of the company he was building. “Look, we were so young,” Zuckerberg told me back in 2007. “When we first got here [to Silicon Valley], we knew that there was so much we just didn’t know.” He was 22 years old when he made that poignant observation. He had arrived in Palo Alto when he was 20.
Zuckerberg is one of the few CEOs in history to come to significant power without his personality fully formed, and he was smart enough to take himself on as a project. His maturity as a CEO and Facebook’s open culture are the result of what can be considered the longest hackathon in history.
My first visit to Facebook, in February 2007, started as a typical one. I was to begin and end the day with a one-on-one with Zuckerberg, with a series of get-to-know-the-company meetings in between. He arrived 20 minutes late for our first meeting, holding a paper bowl of Cheerios and looking more like an overworked paperboy than a new-media mogul. (Later that day, then-COO Owen Van Natta, a Valley veteran and early Facebook “adult,” would roll his eyes and tell me, “The kids eat way too much cereal around here.”) Zuckerberg confessed that he’d been up early to “work on something” and had fallen back asleep. He appeared to be telling the truth: Sleep creases surrounded his red eyes, and he was wearing the same thing he’d worn at the Fast Company photo shoot the day before. He was an odd mix of friendly, quirky post-teenager and philosopher king in training. He would talk about his love of Guitar Hero and Chicken McNuggets, yet he’d always return to the ideas that openness and connecting people were all that really mattered to him, and that he thought Facebook could change the world. Zuckerberg had already faced numerous tests as the site grew, from opening it up to any person over the age of 13 to jumping into the crowded digital photo-sharing market with a simply designed product (tag your friends!) that blew the others away.
Zuckerberg’s most important lesson as a “boy CEO” came from Facebook’s first flush of popularity in corporate America. He spent a lot of time in 2006 talking to the likes of Viacom and Yahoo, both of whom were kicking the tires on acquiring Facebook for up to $1 billion. The mogulizing had taken him away from the company, which was burning through cash; his absence sent waves of discontent through a staff that didn’t know what its leader was thinking. Are we selling? Not selling? Raising money? “What were we going to do, not take the meetings?” recalls Facebook cofounder Dustin Moskovitz, defending his friend. “We were learning about the world by talking to these people.”
But Zuckerberg got the message. “I needed to be more open,” he told me. Encouraged by legendary Silicon Valley recruiter Robin Reed, he hired an executive coach to help him identify and hone the essential skills of running a fast-growing company. He began to study and evaluate the successful people and companies around him, tapping them for insider lessons in leadership. “He is a sponge for process—in a way I’ve rarely seen,” Accel partner and early Facebook investor Jim Breyer told me. Zuckerberg instituted regular all-hands meetings so people could hear directly from him what was happening, and he began to tackle the tough issues of organizational design and personal accountability. (One of Sheryl Sandberg’s first great acts as COO was to hold a public forum exploring women’s issues, including their scant numbers in the engineering ranks, with Zuckerberg’s support.)
As public interest in Facebook grew, Zuckerberg had to master grace under the judgmental glare of the public spotlight—amplified in large measure by Facebook’s own success as a platform to share information. He sometimes seemed like a boy trying on the role of a CEO. He overrelied on jargon and talking points during public presentations, and he exhibited anxiety, even in front of audiences of his peers, making him seem shifty, fragile, and untested. His first appearance on the Today show took so much out of him that he pushed back that day’s meetings to walk the New York City streets and decompress. “I’m trying to get Dustin to do more media so I don’t have to do it as much,” he told me, recounting the story later. “It’s not the most fun thing.” Though the protective cocoon that formed around Zuckerberg and his young cohorts (Moskovitz and fellow cofounder Adam D’Angelo had little interest in speaking to the public) had the unfortunate effect of obscuring his more heartfelt motives, it provided much-needed room for him to work on the product and gave him time to prepare for the crucibles to come.
Back at headquarters, the young Zuckerberg could be his true self and could help his company define its own true self as it grew. In 2007, MySpace was the dominant social network, with Facebook but one of many upstart competitors. Zuckerberg needed the smartest people; to hire them, he had to make the case that Facebook was their best bet. When Zuckerberg and I circled back that first day we met, I sat in while a fairly sophisticated HR team updated its CEO on hiring. Zuckerberg ran the meeting with a good-natured crispness. Facebook’s early-recruiting efforts focused on employee referrals, which were a good way to create a pre-vetted band of brothers. “Oh, that guy?” Zuckerberg said as they ran through the list of names. “He taught me and D’Angelo at Exeter!” The hiring strategy netted essential employees such as Andrew Bosworth, who had taught Zuckerberg at Harvard and is now the company’s director of engineering. (He’s also the one who later invented the company’s all-important BootCamp program, where new hires learn the history of Facebook’s code.)
They knew they were going to run out of former teaching assistants to hire. The company set up a recruiting program that deeply involved even rank-and-file engineers in the process of finding their future peers. All had interviewing duties. The normally reticent and overworked programmers did campus visits, attended tech meetups, and even traveled to a little event in Austin called South by Southwest (which was explained in detail to Zuckerberg). Knowing that their efforts were important and appreciated, they took on the recruiting effort with unalloyed enthusiasm.
Two things made all this effort remarkable and essential to Facebook’s success. For starters, the team built the first of many tools designed to help everyone work together efficiently. They cobbled together a wiki that let everyone share feedback, recommendations about candidates, and ideas of how to persuade the undecided to fall their way. The wiki made the lives of the recruiting team infinitely easier. To this day, regular employees are critical to finding and wooing potential hires. More important, perhaps, the team approached every hire with an eye on the future. “The people we hired were capable of solving the problems we knew were coming,” Bosworth explains, launching into a high-level riff on cognition theory and communication biases before boiling it back down. “You have to be prepared to jump in, make stuff, and grow.”
When I visited with Zuckerberg in late 2009, almost three years after our first meeting, he was more seasoned and yet very much the same. This was the year he wore a tie every day, to telegraph that it was a serious year for the company. As always, he had a good story to tell, this one about bumping into Intel’s bellicose former CEO Andy Grove, who was visiting an executive at Facebook’s new headquarters. Zuckerberg had been studying the history of Intel’s strategy, and after they were introduced, Grove offered some unsolicited feedback. “I said something about what we were trying to do,” recalled Zuckerberg, “not just trying to build the biggest business, but do things that were really good. Then Andy said …” and Zuckerberg modulated his voice to mimic the septuagenarian Hungarian-American’s, “ ’Oh, that’s the biggest bullshit.’ ” Zuckerberg laughed, at the memory and his own impression. “Andy went on to say, ‘All these companies pretend that they’re trying to do something good and really they just need to be competing and killing each other.’ ” Zuckerberg wiped his eyes. “I totally like him. He yells at me no matter what we’re talking about.”
Though grateful for the feedback, Zuckerberg didn’t change course. He was still exceptionally focused on Facebook’s culture. As the company and service grew—it had 1,200 employees and 400 million users around the time we met in 2009—he and his colleagues worried endlessly about the death-by-meeting blues. Facebook had grown into 135,000 square feet in Palo Alto and many locations around the world. It was a quarter-life crisis in the making, the sinking realization that you can’t stick it to the man if you become The Man. In Facebook’s world, Google had become The Man. Engineers there checked in code, then waited as it disappeared for days, weeks, even months. Tales of the company’s bureaucracy were becoming legend—especially at a company loaded up with Google refugees. “You feel like you have to make a choice at some point,” said Mike Schroepfer, Facebook’s VP of engineering. “Will the system be reliable or will the innovation be fast?”
The Hacker Way was designed to sidestep this Faustian bargain; Zuckerberg’s bet was that the guiding essence of Facebook could be baked into a new type of management system for a new type of company. The philosophy respects efficiency above all else. And that could be applied beyond engineering. “Can we take what used to take 10 clicks for someone to get the information they need and reduce it to three?” Zuckerberg told me, recounting a conversation he had with an engineer running the tools group about a better system for the customer-service team. “It saves time over thousands of operations. What can we do with that time?”
Facebook is a company designed by millennials for millennials. “As we like to say, ‘Pixels talk,’ ” says Joey Flynn, one of the designers of Timeline. “You can do anything here if you can prove it.”Everything about how professionals interact and communicate was up for grabs. “We were born out of a mission,” explains Goler, “so any process we have must serve a clear purpose. Since we started with none, we really thought everything through.” The only thing that mattered: Help people do their work faster. Nothing was too sacred. “Email is poorly designed and useless,” reported Zuckerberg, citing a study the company had conducted. “Most subject lines are ‘hi,’ ‘hey,’ or left blank. What’s that tell you?” Instead, a series of internal tools evolved to let people communicate in a way that was more informal and more natural to the projects they worked on, such as a quick acknowledgment-badge system simply called “thanks.” The company then embraced a comprehensive feedback tool called Rypple, much of which was built and evolved within Facebook, with engineering teams as guinea pigs. (It has since been acquired by Salesforce.com.) Gone are the workflow management systems of a manufacturing age. Instead, says Rypple cofounder Daniel Debow, the software created a social environment where people and projects can keep in touch in an easier way. “We’re just amplifying existing behaviors—like texting, posting on walls, and looking at photos—that help people communicate more efficiently in ways that they already do.”
Inside the New Facebook Headquarters
The social-networking giant moved into its new Menlo Park, California, digs last December. Here are never-before-seen images from the expansive new campus.“What should reviews look like?” asked Molly Graham, then the head of culture and engagement at Facebook, citing another standard management practice that was up for, well, review. “We struggled hard. In the end we developed a system that’s meant to fairly reward people for their contributions to the company and is meant to help people grow.” The company encourages employees to form teams around projects of passionate interest, a natural way to craft a nontraditional career path showcasing competence, not brandishing credentials. “As we like to say, ‘Pixels talk,’ ” says Joey Flynn, a product designer on Timeline. “You can do anything here if you can prove it.” The company delivers promotions (and bonuses) twice a year. For millennials, who have grown up with the constant micro-interactions of pokes, badges, texts, tweets, and wall posts, the system fits their need for feedback and validation. As Graham points out, “This is a company designed by millennials for millennials.”
The company does still make traditional calls—the era of riding RipStiks down the hall, for example, came to an end when an intern broke his wrist. But for an idea that has turned into a company, Facebook has done a remarkable job of using its collaborative philosophy to develop the workforce it had into the innovators it needed. Back in 2007, Matt Cohler, Facebook employee No. 5 (and currently a venture capitalist who invested in both D’Angelo’s and Moskovitz’s startups), put a very flat, bare-bones management structure in place. There were few vice presidents, for example, and Zuckerberg had only five direct reports. “We were determined to keep things as flat as possible,” Cohler told me. “The harder we make it for people to invent together, the faster we fall behind.”
When I last visited Facebook in December, employees were packing up “the Bunker,” as they call their old digs, in preparation for a move to a 1-million-square-foot campus in Menlo Park. Sitting amid the packed boxes and lightbulbs with some A-players, including Flynn and engineer Josh Wiseman, it became clear that the foundation that Cohler had put in place had held up under the weight of rapid, enormous growth. One of our group was former Google superstar Lucy Zhang, who decided to come to Facebook in 2011 when it bought her group-messaging startup, Beluga. “I left Google because I couldn’t take enough risks there,” she said unironically. “Here, I can really do things.”
At the end of my first visit, back in 2007, Zuckerberg spent the last hour quizzing me about what I had picked up about the company. He asked me about the themes that we’d talked about in the morning, particularly openness. “Did you find that to be true?” he asked me. “How did you know? What were people saying? How did they talk about the culture? Like, specifically?” It was the first of many times he’s turned the table on me, and one of the best ways a non-Facebook employee can feel what it’s like to have assumptions dissected by one of the sharpest minds in tech. He nodded as I spoke, listened, laughed at my impressions of his friends. But what he wanted to know was simple: Could my experience confirm what he hoped was true of his fledgling company?
And then he gave me a piece of advice, meant for my writing of the Facebook story. But it serves just as well as the underlying force guiding Facebook and Zuckerberg himself: “It’s iterative, right?” he said. “You’ll write it, then next year you’ll write another story, and another, and eventually, the story will be the way you want it.”
Related:
Designing Happiness
The Morning After
Facebook IPO Players ClubA version of this article appears in the April 2012 issue of Fast Company
Megaupload Seizure Order “Null and Void” Says High Court | TorrentFreak
In another astonishing development in the Megaupload saga, a judge in New Zealand’s High Court has declared the order used to seize Kim Dotcom’s assets as “null and void”. The blunder, which occurred because the police applied for the wrong type of court order, means that the Megaupload founder could have his property returned.
Just when it seemed that the handling of the Megaupload case couldn’t get any more controversial, a development from New Zealand has taken things to the next level.
Following the raids on Kim Dotcom’s mansion in January, police seized millions of dollars worth of property belong to the Megaupload founder. But thanks to a police blunder, he could now see all of those assets returned.
On Friday, Justice Judith Potter in the High Court declared the order used to seize Dotcom’s property “null and void” after it was discovered that the police had acted under a court order that should have never been granted.
The error dates back to January when the police applied for the order granting them permission to seize Dotcom’s property. Rather than applying for an interim restraining order, the Police Commissioner applied for a foreign restraining order instead, one which did not give Dotcom a chance to mount a defense.
According to New Zealand Herald, on January 30th prosecution lawyer Anne Toohey wrote to the court explaining that the wrong order had been applied for and detailed five errors with the application.
Justice Potter said that police commissioner Peter Marshall tried to correct the error by applying for the correct order after the raids were completed and retrospectively adding the items already seized.
Although the correct order was eventually granted albeit on a temporary basis, Potter said she will soon rule on whether the “procedural error” will result in Dotcom having his property returned.
The Crown is arguing that since the new order was granted the earlier error no longer matters, but Dotcom’s legal team framed it rather differently by describing the seizure of assets as “unlawful”.
Whether the assets are returned will rest on Dotcom’s legal team showing a lack of “good faith” in connection with the blunder. A hearing to decide if the assets will be returned will take place next week.
Mario is hard, and that’s mathematically official - 14 March 2012 - New Scientist
IF YOU have ever struggled to complete classic Nintendo games, don’t feel bad - they are officially difficult.
An analysis of the computational complexity of video games, including those in the Mario and Legend of Zelda series, proves that many of them belong to a class of mathematical problems called NP-hard. This means that for a given game level, it can be very difficult to work out whether it is possible for a player to reach the end. The results suggest that some hard problems could be solved by playing a game.
The commercial versions of games are designed to be doable, of course, but Erik Demaine of the Massachusetts Institute of Technology and colleagues studied versions without that constraint. “We are working within the same rules, but we are free to design the levels however we like,” he says.
The team transformed each game into a type of logical puzzle called the Boolean satisfiability problem. This asks whether the variables in a collection of logical statements can be chosen to make all the statements true, or whether the statements inevitably contradict each other.
For each game, the team constructed sections of a level that force players to choose one of two paths - equivalent to assigning variables in the Boolean satisfiability problem. Arrangements of enemies and power-ups are equivalent to logical statements. If they allow a level to be completed, that is equivalent to all the statements in the Boolean problem being true; if they make the level impossible, that is equivalent to a contradiction.
Many of the games in the Mario, Donkey Kong, Legend of Zelda, Metroid and Pokémon series prove to be NP-hard. That means deciding whether a player can complete them is at least as hard as the hardest problems in NP, a complexity class involved in the tantalising problem of P versus NP (see “Million-dollar proof”). Not every game in each series was included in the proof, as they follow different rules.
For Mario, the team also prove that the games are NP-complete, an additional property with important consequences. Many difficult problems can be converted into any problem in the NP-complete category. Then if you can solve the NP-complete problem - say, by completing a level of Mario - you have solved the original problem too.
That includes the travelling salesman problem - finding the shortest route between a series of points - which is of real interest in the field of logistics, and also the knapsack problem, used in deciding how to allocate resources. So theoretically you could convert an example of either problem into a Mario level, and play the game to solve it. That approach would be fun, says Demaine, although it would probably be simpler to solve the satisfiability problem directly.
The results offer mixed news for game designers, says Giovanni Viglietta, a computer scientist at the University of Pisa, Italy, who was not involved in the research. If it is an NP-hard problem to decide whether a level can be successfully navigated, there is no easy way for a designer to check this. But it does ensure that playing the game is interesting, as players can’t easily decide if they are heading the right way or straying into an impassable area of the level. “The game demands some creativity and ingenuity,” Viglietta says.


























Just when it seemed that the handling of the Megaupload case couldn’t get any more controversial, a development from New Zealand has taken things to the next level.