The other field Facebook wants to revolutionize - Fortune Tech
The social networking giant is leading a consortium aiming to make data centers cheaper and more efficient.
FORTUNE — Facebook is known for creating the most popular social networking tool, not designing hardware. But the company has taken a do-it-yourself approach to building out its data centers and the servers and racks that fill them. The result? Data centers that are 38% more efficient and 24% cheaper than average, according to Frank Frankovsky, director of hardware design and supply chain at Facebook.
In the hopes of driving the cost down further, Facebook has even “open sourced” its designs — making it possible for anyone to contribute to (and replicate) what its engineers have built. Last week, as most of the business world speculated on the social networking site’s upcoming IPO, Facebook held a conference for its Open Compute Project, a consortium that now includes the likes of Hewlett-Packard (HPQ), Dell (DELL) and AMD (AMD). We caught up with Frankovsky to find out more about Facebook’s open source strategy and what’s next for the Open Compute Project.
FORTUNE: Why did you start the Open Compute Project?
Frankovsky: When we designed and built our first data center, we exceeded even some of our own internal goals. And we immediately thought it would be unnatural not to share this because we’ve all benefited so much from open source software – like the infrastructure software we’ve built our business on. This is why our software engineers can focus on innovation every day, on making the world more connected. We don’t need to go and reinvent an operating system. So we thought, let’s go and open source the hardware space so that we can give back too. Also, no single company is ever going to have all of the best brainpower in the entire industry under one roof. By open sourcing, you can get the industry’s best brainpower focused together. You get a bunch of great ideas, and it accelerates the pace of innovation.
A lot of companies fight standardization and commoditization. How have traditional suppliers reacted to Open Compute?
While the initial reaction might have been resistance, these are great innovation companies and they know that at some point in order to remain competitive and successful you have to reinvent yourself.
Are there any other efforts out there to open source data center hardware?
We have partnerships with a whole bunch of other projects , but we are specifically focused on the hardware design in the data center, and to my knowledge there are no other projects specifically around this. The old method is to keep all your cards close to your chest without sharing. The biggest project that inspired me and all of us at Facebook to get involved is the open source operating system Linux and the impact it had on the market. We want to have a similar impact on hardware.
Are there technologies that you won’t “open source” and share with others?
We think really, really carefully about what we open source. We’ve shared how we pick data center sites. But when we open sourced our data center blueprints we didn’t include the main point of entry for fiber runs—we felt it was a security issue. So there are some things like that that we don’t put out in the open. But that’s really because we need to defend ourselves and our end users. The thing we won’t open source are the key innovations we have in the application space. Those are the unique things that differentiate Facebook and the reason more than 900 million people come to Facebook. Intel is one of the founding members of the Open Compute Project. It also happens to have one of the richest IP portfolios in the industry. Intel’s engineers have made significant contributions [to Open Compute] but we wouldn’t expect them to share how they design CPUs.
A lot of people would be surprised to know that some of what you’ve done with your designs is actually simplifying and taking away capabilities. Can you explain?
I don’t think anyone would argue that putting a bunch of plastic logos in front of a server is a good idea. But sometimes simple is actually really hard. People sometimes overcomplicate things. When you look at a design it might look really elegant because it’s got all kinds of whiz-bang features. But when you step back and ask how you can do this with minimum components, sometimes making it simple is really the hard part. Some of the most successful mobile devices don’t look like they do anything when you pull them out of the box; it’s a flat screen with just one button. But when you turn it on and it does exactly what you ask it to do, then you really understand the beauty and simplicity of the design. You don’t see the engineering efforts that went into making it simple.
So what’s next for the Open Compute Project?
We’re getting a lot of traction. Most of it is in data center and server design, and we’ve extended it to [server] racks. The storage space is something you’ll see heat up, and there’s also a lot of interest in networking. But a lot of the activity in the coming six months is going to be around storage—how open source storage really changes the market. Hopefully it will let companies choose the best of breed from both hardware and software. In the future there will be a smaller number of larger data center operators because of the trend towards cloud computing. We’ve reached an inflection point where things can get a little more standardized. What’s exciting about the future is that we can now apply the brain power to new and unique requirements in computing.
“Boy CEO” Mark Zuckerberg’s Two Smartest Projects Were Growing Facebook And Growing Up | Fast Company
From studying leaders he admired to taking elocution lessons, Zuckerberg made his evolution into a world-class CEO a personal project. Photo by Martin Schoeller/August
It was a minor meta moment, the perfect inside joke to kick off a September day that was otherwise all business. The occasion was f8 2011, the erratically scheduled, mostly annual conference for Facebook developers and social-media innovators, a gathering that now has a pilgrimage-like quality for the Facebook faithful. It is one of the few opportunities for legions of Mark Zuckerberg fans viewing the event live online to observe their spotlight-averse hero perform a rite native to the CEO species: the keynote address. Ladies and gentlemen, Mark Zuckerberg. The whoops turned to laughter almost immediately; it took only a few seconds for the assembled engineers, designers, brand stewards, marketing mavens, nonprofiteers, pundits, bloggers, and investors to realize that they were being punked.
On stage was the comedian Andy Samberg, fully in character as “Zuck Dawg” in a hoodie, jeans, and Adidas sandals. “I want to start by focusing on some key issues,” he said. “The first is the importance of authentic identity. I …” he paused, hand over heart, “… am Mark Zuckerberg.” It was a delicious moment for the Facebook staff, now 3,200 strong. For them, it’s always been about identity. Since Facebook’s February 2004 launch, the company has succeeded because hundreds of millions of people—slowly at first and then in crashing herds—became comfortable sharing their true selves on the site. It is precisely that authenticity that makes Facebook matter to its 845 million users. If Marshall McLuhan had lived long enough to have a Facebook profile, his status might read thus: The medium isn’t just the message; the medium has become us.Zuckerberg’s bet was that Facebook’s guiding essence, the Hacker Way, could be baked into a new style of management for a new type of company.
But the moment belonged first and foremost to Zuckerberg, who for years has had his own identity problem: “boy CEO.” Young, arrogant, and awkward—no one believed that Zuckerberg could survive the adult swim of real business, and thanks to his depiction in The Social Network, some folks will forever see him as the fatally flawed psychopathic robot nerd looking to steal your code, your personal data, your girlfriend. “I don’t think about it … much,” he once told me when I asked him how he handles all the noise, measuring his words as he always does. “I understand why people need to have these dialogues, to ask these questions. We have so much to do here, we don’t think about it if we don’t have to.”
I first met Zuckerberg and his colleagues five years ago, when Facebook had just 19 million users and was on the verge of opening up its platform to outside developers. Looking back on more than 400 hours of reporting with Facebook staffers, investors, and people in the site’s ecosystem, including a visit in late December, plus more than seven hours of one-on-one interviews with Zuckerberg, one fact is clear: The only thing that could have derailed Facebook’s climb to Internet domination was the inexperience of a young CEO. He had never held a proper job before and, by virtue of his own ballsy negotiations, could not be ousted from his position. (Facebook, citing IPO-quiet-period restrictions, declined to make Zuckerberg available for this story.) But what was largely interpreted as control freakery in service of a bigger exit strategy turned out to be a real vision. “So many businesses get worried about looking like they might make a mistake, they become afraid to take any risk,” he told me after the company moved into its first grown-up tech campus, on Palo Alto’s California Avenue, in 2009. “Companies are set up so that people judge each other on failure. I’m not going to get fired if we have a bad year. Or a bad five years. I don’t have to worry about making things look good if they’re not. I can actually set up the company to create value.”
You Know What’s Cool? A Billion DollarsNine of the big winners when Facebook goes public.
Founder, chairman, and CEO
The venture capitalist led an early $12.7 million funding round.
Cofounder and first CTO
The entrepreneur was Facebook’s founding president.
The PayPal cofounder was Facebook’s first angel investor.
The artist received stock options for murals painted at Facebook HQ in 2005.
(Plus an additional 39 million in restricted stock units worth more than $1.1 billion)
Cofounder and first unofficial spokesperson
The Washington Post Co. CEO has sat on Facebook’s board since 2008.
$45 million*Based on a $100 billion valuation
This February, as part of his effort to ensure that this remain true, Zuckerberg asked investors to back a company in which he will retain 57% of the voting stock. He outlined the company’s guiding principle, which he calls the Hacker Way, in a personal letter to potential shareholders that accompanied the IPO filing. This is the idea that gives Facebook its identity—as a company that questions assumptions, moves fast, takes risks, shares information, and learns from other smart people. Nowhere does this manifest itself more clearly than in the company’s regular hackathons, extended coding sessions where employees race to invent new products. “What you’ll hear over and over and over again is ‘why?’ ” says HR chief Lori Goler of a culture filled with millennials (average age: 28) who question the purpose of every feature and expect a logical answer.
Turning that “we’re all coding together in one big room, and we get great ideas and move fast because anyone can walk up to anyone else” ethos into a business required the young CEO to turn his hacker sights on himself. An experiential learner, Zuckerberg transformed himself with astonishing discipline into a CEO worthy of the company he was building. “Look, we were so young,” Zuckerberg told me back in 2007. “When we first got here [to Silicon Valley], we knew that there was so much we just didn’t know.” He was 22 years old when he made that poignant observation. He had arrived in Palo Alto when he was 20.
Zuckerberg is one of the few CEOs in history to come to significant power without his personality fully formed, and he was smart enough to take himself on as a project. His maturity as a CEO and Facebook’s open culture are the result of what can be considered the longest hackathon in history.
My first visit to Facebook, in February 2007, started as a typical one. I was to begin and end the day with a one-on-one with Zuckerberg, with a series of get-to-know-the-company meetings in between. He arrived 20 minutes late for our first meeting, holding a paper bowl of Cheerios and looking more like an overworked paperboy than a new-media mogul. (Later that day, then-COO Owen Van Natta, a Valley veteran and early Facebook “adult,” would roll his eyes and tell me, “The kids eat way too much cereal around here.”) Zuckerberg confessed that he’d been up early to “work on something” and had fallen back asleep. He appeared to be telling the truth: Sleep creases surrounded his red eyes, and he was wearing the same thing he’d worn at the Fast Company photo shoot the day before. He was an odd mix of friendly, quirky post-teenager and philosopher king in training. He would talk about his love of Guitar Hero and Chicken McNuggets, yet he’d always return to the ideas that openness and connecting people were all that really mattered to him, and that he thought Facebook could change the world. Zuckerberg had already faced numerous tests as the site grew, from opening it up to any person over the age of 13 to jumping into the crowded digital photo-sharing market with a simply designed product (tag your friends!) that blew the others away.
Zuckerberg’s most important lesson as a “boy CEO” came from Facebook’s first flush of popularity in corporate America. He spent a lot of time in 2006 talking to the likes of Viacom and Yahoo, both of whom were kicking the tires on acquiring Facebook for up to $1 billion. The mogulizing had taken him away from the company, which was burning through cash; his absence sent waves of discontent through a staff that didn’t know what its leader was thinking. Are we selling? Not selling? Raising money? “What were we going to do, not take the meetings?” recalls Facebook cofounder Dustin Moskovitz, defending his friend. “We were learning about the world by talking to these people.”
But Zuckerberg got the message. “I needed to be more open,” he told me. Encouraged by legendary Silicon Valley recruiter Robin Reed, he hired an executive coach to help him identify and hone the essential skills of running a fast-growing company. He began to study and evaluate the successful people and companies around him, tapping them for insider lessons in leadership. “He is a sponge for process—in a way I’ve rarely seen,” Accel partner and early Facebook investor Jim Breyer told me. Zuckerberg instituted regular all-hands meetings so people could hear directly from him what was happening, and he began to tackle the tough issues of organizational design and personal accountability. (One of Sheryl Sandberg’s first great acts as COO was to hold a public forum exploring women’s issues, including their scant numbers in the engineering ranks, with Zuckerberg’s support.)
As public interest in Facebook grew, Zuckerberg had to master grace under the judgmental glare of the public spotlight—amplified in large measure by Facebook’s own success as a platform to share information. He sometimes seemed like a boy trying on the role of a CEO. He overrelied on jargon and talking points during public presentations, and he exhibited anxiety, even in front of audiences of his peers, making him seem shifty, fragile, and untested. His first appearance on the Today show took so much out of him that he pushed back that day’s meetings to walk the New York City streets and decompress. “I’m trying to get Dustin to do more media so I don’t have to do it as much,” he told me, recounting the story later. “It’s not the most fun thing.” Though the protective cocoon that formed around Zuckerberg and his young cohorts (Moskovitz and fellow cofounder Adam D’Angelo had little interest in speaking to the public) had the unfortunate effect of obscuring his more heartfelt motives, it provided much-needed room for him to work on the product and gave him time to prepare for the crucibles to come.
Back at headquarters, the young Zuckerberg could be his true self and could help his company define its own true self as it grew. In 2007, MySpace was the dominant social network, with Facebook but one of many upstart competitors. Zuckerberg needed the smartest people; to hire them, he had to make the case that Facebook was their best bet. When Zuckerberg and I circled back that first day we met, I sat in while a fairly sophisticated HR team updated its CEO on hiring. Zuckerberg ran the meeting with a good-natured crispness. Facebook’s early-recruiting efforts focused on employee referrals, which were a good way to create a pre-vetted band of brothers. “Oh, that guy?” Zuckerberg said as they ran through the list of names. “He taught me and D’Angelo at Exeter!” The hiring strategy netted essential employees such as Andrew Bosworth, who had taught Zuckerberg at Harvard and is now the company’s director of engineering. (He’s also the one who later invented the company’s all-important BootCamp program, where new hires learn the history of Facebook’s code.)
They knew they were going to run out of former teaching assistants to hire. The company set up a recruiting program that deeply involved even rank-and-file engineers in the process of finding their future peers. All had interviewing duties. The normally reticent and overworked programmers did campus visits, attended tech meetups, and even traveled to a little event in Austin called South by Southwest (which was explained in detail to Zuckerberg). Knowing that their efforts were important and appreciated, they took on the recruiting effort with unalloyed enthusiasm.
Two things made all this effort remarkable and essential to Facebook’s success. For starters, the team built the first of many tools designed to help everyone work together efficiently. They cobbled together a wiki that let everyone share feedback, recommendations about candidates, and ideas of how to persuade the undecided to fall their way. The wiki made the lives of the recruiting team infinitely easier. To this day, regular employees are critical to finding and wooing potential hires. More important, perhaps, the team approached every hire with an eye on the future. “The people we hired were capable of solving the problems we knew were coming,” Bosworth explains, launching into a high-level riff on cognition theory and communication biases before boiling it back down. “You have to be prepared to jump in, make stuff, and grow.”
When I visited with Zuckerberg in late 2009, almost three years after our first meeting, he was more seasoned and yet very much the same. This was the year he wore a tie every day, to telegraph that it was a serious year for the company. As always, he had a good story to tell, this one about bumping into Intel’s bellicose former CEO Andy Grove, who was visiting an executive at Facebook’s new headquarters. Zuckerberg had been studying the history of Intel’s strategy, and after they were introduced, Grove offered some unsolicited feedback. “I said something about what we were trying to do,” recalled Zuckerberg, “not just trying to build the biggest business, but do things that were really good. Then Andy said …” and Zuckerberg modulated his voice to mimic the septuagenarian Hungarian-American’s, “ ’Oh, that’s the biggest bullshit.’ ” Zuckerberg laughed, at the memory and his own impression. “Andy went on to say, ‘All these companies pretend that they’re trying to do something good and really they just need to be competing and killing each other.’ ” Zuckerberg wiped his eyes. “I totally like him. He yells at me no matter what we’re talking about.”
Though grateful for the feedback, Zuckerberg didn’t change course. He was still exceptionally focused on Facebook’s culture. As the company and service grew—it had 1,200 employees and 400 million users around the time we met in 2009—he and his colleagues worried endlessly about the death-by-meeting blues. Facebook had grown into 135,000 square feet in Palo Alto and many locations around the world. It was a quarter-life crisis in the making, the sinking realization that you can’t stick it to the man if you become The Man. In Facebook’s world, Google had become The Man. Engineers there checked in code, then waited as it disappeared for days, weeks, even months. Tales of the company’s bureaucracy were becoming legend—especially at a company loaded up with Google refugees. “You feel like you have to make a choice at some point,” said Mike Schroepfer, Facebook’s VP of engineering. “Will the system be reliable or will the innovation be fast?”
The Hacker Way was designed to sidestep this Faustian bargain; Zuckerberg’s bet was that the guiding essence of Facebook could be baked into a new type of management system for a new type of company. The philosophy respects efficiency above all else. And that could be applied beyond engineering. “Can we take what used to take 10 clicks for someone to get the information they need and reduce it to three?” Zuckerberg told me, recounting a conversation he had with an engineer running the tools group about a better system for the customer-service team. “It saves time over thousands of operations. What can we do with that time?”Facebook is a company designed by millennials for millennials. “As we like to say, ‘Pixels talk,’ ” says Joey Flynn, one of the designers of Timeline. “You can do anything here if you can prove it.”
Everything about how professionals interact and communicate was up for grabs. “We were born out of a mission,” explains Goler, “so any process we have must serve a clear purpose. Since we started with none, we really thought everything through.” The only thing that mattered: Help people do their work faster. Nothing was too sacred. “Email is poorly designed and useless,” reported Zuckerberg, citing a study the company had conducted. “Most subject lines are ‘hi,’ ‘hey,’ or left blank. What’s that tell you?” Instead, a series of internal tools evolved to let people communicate in a way that was more informal and more natural to the projects they worked on, such as a quick acknowledgment-badge system simply called “thanks.” The company then embraced a comprehensive feedback tool called Rypple, much of which was built and evolved within Facebook, with engineering teams as guinea pigs. (It has since been acquired by Salesforce.com.) Gone are the workflow management systems of a manufacturing age. Instead, says Rypple cofounder Daniel Debow, the software created a social environment where people and projects can keep in touch in an easier way. “We’re just amplifying existing behaviors—like texting, posting on walls, and looking at photos—that help people communicate more efficiently in ways that they already do.”
Inside the New Facebook HeadquartersThe social-networking giant moved into its new Menlo Park, California, digs last December. Here are never-before-seen images from the expansive new campus.
“What should reviews look like?” asked Molly Graham, then the head of culture and engagement at Facebook, citing another standard management practice that was up for, well, review. “We struggled hard. In the end we developed a system that’s meant to fairly reward people for their contributions to the company and is meant to help people grow.” The company encourages employees to form teams around projects of passionate interest, a natural way to craft a nontraditional career path showcasing competence, not brandishing credentials. “As we like to say, ‘Pixels talk,’ ” says Joey Flynn, a product designer on Timeline. “You can do anything here if you can prove it.” The company delivers promotions (and bonuses) twice a year. For millennials, who have grown up with the constant micro-interactions of pokes, badges, texts, tweets, and wall posts, the system fits their need for feedback and validation. As Graham points out, “This is a company designed by millennials for millennials.”
The company does still make traditional calls—the era of riding RipStiks down the hall, for example, came to an end when an intern broke his wrist. But for an idea that has turned into a company, Facebook has done a remarkable job of using its collaborative philosophy to develop the workforce it had into the innovators it needed. Back in 2007, Matt Cohler, Facebook employee No. 5 (and currently a venture capitalist who invested in both D’Angelo’s and Moskovitz’s startups), put a very flat, bare-bones management structure in place. There were few vice presidents, for example, and Zuckerberg had only five direct reports. “We were determined to keep things as flat as possible,” Cohler told me. “The harder we make it for people to invent together, the faster we fall behind.”
When I last visited Facebook in December, employees were packing up “the Bunker,” as they call their old digs, in preparation for a move to a 1-million-square-foot campus in Menlo Park. Sitting amid the packed boxes and lightbulbs with some A-players, including Flynn and engineer Josh Wiseman, it became clear that the foundation that Cohler had put in place had held up under the weight of rapid, enormous growth. One of our group was former Google superstar Lucy Zhang, who decided to come to Facebook in 2011 when it bought her group-messaging startup, Beluga. “I left Google because I couldn’t take enough risks there,” she said unironically. “Here, I can really do things.”
At the end of my first visit, back in 2007, Zuckerberg spent the last hour quizzing me about what I had picked up about the company. He asked me about the themes that we’d talked about in the morning, particularly openness. “Did you find that to be true?” he asked me. “How did you know? What were people saying? How did they talk about the culture? Like, specifically?” It was the first of many times he’s turned the table on me, and one of the best ways a non-Facebook employee can feel what it’s like to have assumptions dissected by one of the sharpest minds in tech. He nodded as I spoke, listened, laughed at my impressions of his friends. But what he wanted to know was simple: Could my experience confirm what he hoped was true of his fledgling company?
And then he gave me a piece of advice, meant for my writing of the Facebook story. But it serves just as well as the underlying force guiding Facebook and Zuckerberg himself: “It’s iterative, right?” he said. “You’ll write it, then next year you’ll write another story, and another, and eventually, the story will be the way you want it.”
A version of this article appears in the April 2012 issue of Fast Company
Letting Hackers Compete, Facebook Eyes New Talent - Technology Review
As it readies for an IPO, the social network puts engineers, not HR, in charge of a global search for young programmers.
Late this January, some 75,000 people around the planet sat in front of their computers and pondered how to make anagrams from a bowl of alphabet soup. They were participants in the Hacker Cup, an international programming battle that Facebook organized to help it find the brightest young software engineers before competitors like Google do.
After three more rounds of brain teasers, Facebook will fly the top 25 coders to its head office in Menlo Park, for an adrenaline-soaked finale this March that will award the champion $5,000. In return, Facebook gets a shot at hiring the stars discovered along the way.
“I’m in an all-out land grab for talent,” says Jocelyn Goldfein, Facebook’s director of engineering and most senior woman on its technical staff. The social network builds almost all of its own software, and young, smart coders are the company’s most critical asset as it manages the comments, photos, and “likes” of more than 800 million users. “We are in uncharted waters every day,” says Goldfein. “What’s great about young people is that they don’t know what’s impossible, so they try crazy things and lead us to be the first to make them work.”
Google and many other companies are chasing the same code slingers as Facebook, causing salaries to shoot up. Average salaries for technology professionals in Silicon Valley rose 5.2 percent in 2011 to break the $100,000 barrier, while pay rose just 2 percent nationally, according to a recent salary survey. One graduating college senior, posting anonymously on the Web, claimed that Facebook offered a $100,000 salary, a $50,000 signing bonus, and $120,000 in stock options. Facebook declined to comment.
According to the prospectus filed in connection with Facebook’s planned initial public offering of stock, the company’s headcount jumped from 2,127 to 3,200 full-time employees in 2011. Unlike some large companies, Facebook does not leave recruiting programmers to its human resources department. “The HR departments are in one building and engineering is in another,” says Goldfein. “Recruitment sits with us.”
The best hiring strategies simultaneously test skills and advertise Facebook’s internal culture, which Goldfein says values “clever workarounds that shortcut complexity.” In addition to the Hacker Cup and a series of similar “Camp Hackathon” contests that tour U.S. colleges, there’s a set of fiendishly tricky online puzzles that Facebook maintains online. Solving them with sufficient style can net a phone call from a recruiter. “This is a way to say that if you’re brilliant we don’t care where you worked and if you have a college degree,” says Goldfein.
All that reinforces Facebook’s status as a cool place to work. On Glassdoor, a job information site, Facebook leads technology companies in a ranking by employees of the best workplaces. In another survey that asked workers under 40 where they would most like to get a job, Facebook placed third, behind Google and Apple. Increasingly, other large technology companies aren’t even the stiffest competition for talent, says Rusty Rueff, a board member at Glassdoor. Many talented young people in Silicon Valley are finding that investors and startup accelerator programs will back them to go it alone and found their own companies.
One consequence is that technology companies are buying startups simply as a way to hire their twentysomething founders. Another is that companies aren’t hiring for specific jobs. Facebook puts new hires through a six-week boot camp where they rotate through projects, choosing one that suits them best. “Facebook and other companies doing this are saying, ‘You can work for us and still be entrepreneurial and create your own thing,’” Rueff says.
Although the coder competition looks like a fun and free-wheeling meritocracy, it also reflects problems in the U.S. education system. Very few women participate, and most of the winners are from overseas. “Facebook [is] aggressively going to other countries because there aren’t enough skilled people in the U.S.,” says Goldfein.
Of the 2011 Hacker Cup winners, all three were foreign men 26 or younger. Facebook hired the second-place finisher. The first-place winner was already employed by Google.
Facebook’s Top Cop: Joe Sullivan - Forbes
This story appears in the March 12, 2012 issue of Forbes magazine.
If Facebook were a country, it would be the third largest in the world and Joe Sullivan would be head of Homeland Security.
His actual title is chief security officer. The “terrorists” he’s up against include the “Koobface gang,” a quintet of Russians who unleashed a worm that turned Facebookers’ computers into enslaved bots; the spammers who flooded the site with violent and pornographic images in December; scammers who trick Facebook users into clicking links and filling out surveys for the swindlers’ profit; pedophiles using the site to make contact with minors; and scrapers who inappropriately raid Facebook for users’ valuable personal information. These scoundrels include those who use malicious apps, hackers and an amateur porn purveyor who matches profile pages to private nudie photos submitted by vengeful exes—making it easy to contact, harass and “poke” the unwitting and involuntary porn stars.
The dirt Facebook holds on its users makes it as attractive to cops as to criminals. Among Sullivan’s responsibilities are daily decisions about how much user information to give to law enforcement when it comes calling. And, as a digital nation’s DHS, Sullivan and his team actively police the site for user data worth volunteering to the authorities. Still, he says, “we err on the side of not sharing and have picked quite a few fights over the years.”
Users may have constitutional rights against unreasonable searches by the state, but the only Facebook Constitution is the company’s dense terms of service agreement. It focuses on prohibitions for users, such as bullying, creating fake accounts or uploading images of violence or nudity, as well as Facebook’s rights to intellectual property uploaded to the site. It doesn’t spell out when Facebook may dive into data for policing purposes or hand it over to the authorities.
Should Facebook give users a Miranda warning before they sign up—that anything they post and do on the site can and will be used against them? The company gives law enforcement “basic subscriber information” on requests accompanied by subpoenas: a user’s name, e-mail address and IP address (which reveals approximate location). Sullivan insists that everything else—photos, status updates, private messages, friend lists, group memberships, pokes and all the rest—requires a warrant.
Sullivan, 43, usually wears the “Mark Zuckerberg uniform” at the office: gray hoodie, sneakers, jeans. With longish light-brown hair and gray-speckled goatee, he looks more like a bouncer at a country music bar than an ex-federal prosecutor, let alone the guy responsible for safeguarding and investigating Facebook’s 845 million users.
Most of his security team is based at headquarters in Menlo Park, Calif. and sits at clusters of desks close enough to take dead aim at one another with Nerf darts. Broken roughly into five parts, the team has 10 people review new features being launched, 8 monitor the site for bugs and privacy flaws, 25 handle requests for user information from law enforcement, and a few build criminal and civil cases against those who misbehave on the network; the rest are handling security situations as they arise and acting as digital bodyguards protecting Facebook staffers (“We have someone trying to hack an employee’s account every day,” says Sullivan). If you include the physical security guards who patrol Facebook headquarters, Sullivan’s team numbers 70 people.
It’s a big kingdom to police, populated with mundane and highly personal information about its subjects. Its value, shaping up to be $100 billion when the company goes public later this year, depends on keeping the populace happy and safe—from overprobing law officials, as well as from predators.
THE OLDEST OF SEVEN CHILDREN, Sullivan grew up in Cambridge, Mass. He describes his father as a painter and sculptor, and his mother as a schoolteacher who wrote mystery stories about a nun who was a private eye. “So I rebelled and went to law school,” he says. (A Google search revealed that the apple did not fall so very far from the tree, though. Sullivan’s mother was a CIA analyst in Russia in the 1960s before she settled down to start a family.)
Sullivan got his law degree at the University of Miami in 1993. A self-described early adopter, he was the first of his friends to get a computer and an e-mail account. In his first job at the Department of Justice in Miami, he convinced his superiors that the office should have an Internet connection.
He has been riding the Internet crime wave since 1997, when he moved to Las Vegas as a federal prosecutor. When the DOJ started a computer crime program, recruiting one prosecutor in every office to work on cybercrime cases, he volunteered and began working on early eBay fraud and software piracy cases. After Bob Mueller, now director of the FBI, started recruiting a high-tech team to work in the DOJ’s Silicon Valley office in 1999, Sullivan jumped at the chance, putting him at the center of cybercrime during the Internet boom. In 2002 he went to eBay, where his security detail included the units PayPal and Skype. That’s when he had to make a fundamental shift in his thinking—not just how best to prosecute criminals but also how much information to hold back from authorities to protect the rights of customers.
“With Skype we’d tell law enforcement to go through Luxembourg, and good luck with that,” says Sullivan now. “But with eBay, if you were law enforcement investigating a seller, you didn’t even need a subpoena. You could just ask for it on your letterhead and we would hand it over. Back then some people were just putting money in envelopes, sending it to eBay sellers and hoping to get their products. There needed to be an expectation that sellers were being scrutinized.”
Sullivan says the experience of looking through different legal lenses in terms of what to give to law enforcement was “really helpful” when he came to Facebook in 2008, “where expectation of privacy is paramount and our philosophy has to be the Skype policy.” He claims that “99.9% of the time” when Facebook resists a request, the government backs down.
While Sullivan appreciates the nuances around privacy in the context of free expression and communication, he appears to have little tolerance for claims to privacy when it comes to either fraud or the treatment of children. With the rise of Facebook credits—the site’s monetary system, which requires users to use virtual dollars to buy goods in games and apps on the site—he will likely adopt the eBay approach. Those dealing in Facebook dollars can expect to be closely scrutinized.
IN DECEMBER THE RAPIDLY EXPANDING Facebook moved from Palo Alto to Menlo Park, into Sun Microsystems’ old headquarters, once known as “Sun Quentin,” after the notorious Marin County bayside prison. The sprawling campus is still under construction around us on this February morning, with workers carrying ladders and bulldozers preparing the intrabuilding walkways for food carts and play areas. Since employees can’t use the central paths, there are dozens of bikes outside each building for use on the paved “Hacker Way” road that circles the campus. “Even when they’re finished, it won’t look too sculpted,” says Sullivan, gazing out the windows of Building 18 at construction equipment. “The unfinished look of our campus is a cultural thing.”
Inside, the walls bear a passing resemblance to the scrapbook feel of profile pages. Prints from the videogame Donkey Kong and scrawled messages from visitors (many who thank Facebook for enabling them to “stalk” the man or woman who eventually became their spouse) hang alongside the security team’s “scalps”—photos and investigation details for spammers, hackers and pedophiles hunted down and kicked off the site. The conference room names in the security building are mash-ups of music artists and security threats, such as “Alicia Keylogger.” Sullivan gestures at ten people sitting at a row of desks, who smile shyly in our direction.
“They handle requests from law enforcement,” he explains. The security team has five other members based in Dublin, Ireland who speak every European language and field government requests internationally. “Claudio, for example, speaks to every police officer in Italy and answers any question they might have about Facebook. We’re very careful about the information we share, but that doesn’t mean we can’t help them understand the situation that they’ve never dealt with before.”
WikiLeaks’ Julian Assange has called Facebook the world’s perfect spying machine, with access to 40% of the world’s two billion Internet users. A 24-year-old Austrian law student recently took advantage of Europe’s “right to access” law—which forces companies to provide all information they have on a citizen upon request—to get his Facebook file. After three years on the site it ran to an incredible 1,222 pages long.
Sullivan scoffs at the spying machine characterization. “We don’t have a data pipeline to the CIA,” he says. “If people had horrible experiences, they would stop using Facebook.”
That echoes a sentiment expressed in the company’s recent S-1 filing to go public: “Any number of factors could potentially negatively affect user retention, growth and engagement, including if there are changes in user sentiment about the quality or usefulness of our products or concerns related to privacy and sharing, safety, security or other factors.”
Law enforcement, as well as civil litigants, increasingly rely on third-party companies like Facebook as a source of evidence in criminal investigations and lawsuits. It’s the nature of the overexposed age that we make much more information about ourselves readily available and easily discoverable.
Sullivan goes over his most recent weekly security report to Facebook executives, in which he highlights significant incidents of ne’er-do-wells getting poked by the security team. Sullivan notes that Florida police called Facebook’s 24-hour emergency hotline for law enforcement the previous week to request help locating a two-week-old baby who had been abducted from its mother. When law enforcement calls the hotline in a life-or-death emergency, Facebook waives basic legal requirements and hands over information to authorities without making them go through official channels. In this case it provided authorities with the IP address and location information for the last sign-in for the Facebook user suspected of abducting the child. The baby was recovered 30 minutes later.
In another incident, proactive policing by Facebook’s security team led to a potential pedophile being fingered. The site employs algorithms to detect suspicious behavior and bring it to the attention of Sullivan’s group. “We found that a youth pastor and children’s sports coach in Indiana was using fake accounts to try to engage with kids on our site,” he says. “So we called the FBI in Indiana and sent them his information.”
While a youth pastor reaching out to young people doesn’t seem particularly nefarious, Sullivan suggests that his use of fake accounts to do so, as well as the content of his communications, was disturbing enough to warrant police involvement.
Users may often forget they are constantly watched on the site, if not by actual people, then by algorithms. Last year Facebook adopted a Microsoft program called PhotoDNA, which scans every picture uploaded to the site to see if it matches known child porn images compiled by the FBI’s National Crime Information Center. “Our list of child porn images is actually much longer than the FBI’s,” says Sullivan. “Every time we find something new—through a user report or flagging on a keyword—we manually review the user album to see if there are other images that should be added to the list, and then we add them to our library. We’re exploring how to share our library with others.”
For years the site has had back-end algorithms to weed out fake spam accounts and monitor kid-adult interactions. Lotharios, beware: “If you’re sending friend requests that trend to 80% female, that’s a red flag, or if you change your birth date a lot—under and above the 18 threshold,” says Sullivan. “Our site integrity team has built engines to feed in characteristics, and they start hunting people down. When you have single concrete rules, it’s easy for people to figure them out, but with machine learning, it’s evolving all the time.”
Sometimes in ways that most Facebook members probably aren’t aware of. Sullivan’s team determines when cops get to dig into users’ accounts but also has the power to do its own snooping to prevent miscreants from abusing the site, as well as to turn those users over to authorities. The Constitution protects us against unreasonable searches by the feds, requiring them, for example, to get a search warrant from a judge to riffle through our digital homes just as they do for our physical ones. But when it comes to our privacy rights from the companies that store our data? That’s more complicated.
Privacy—or the lack of it—is the biggest complaint about Facebook. Constant tinkering with privacy settings has prompted instant pushback from users and retreat by the company. After an investigation by the Federal Trade Commission into unfair and deceptive practices, the site is now subject to privacy audits every two years. That hasn’t stopped a high recidivism rate. Recently, Facebook’s switching all users over from their exiting profile pages to “Timelines” that expose activity from years earlier by migrating it to the user’s front page provoked an outcry from privacy advocates.
SINCE JOINING FACEBOOK Sullivan has brought a more confrontational approach to security on the site. While many companies focus on deterrence and eliminating threats, Sullivan also wants to pursue malefactors. “Joe is aggressive about going after bad guys,” says Alex Rice, a member of his security team. “I attribute that to his prosecutor background.”
Sullivan complains that law enforcement is too focused on intellectual-property crimes and takes little interest in malware and spam cases, a growing headache for social sites. So Facebook has taken matters into its own hands, pursuing suspects in civil court and in the court of public opinion.
“A lot of companies stop at playing defense, like credit card companies—they invest a lot in fraud detection and prevention, but they’re not bringing civil actions,” says Sullivan. “We spend a lot of time trying to figure out who’s sitting on the other side of cybercrime.”
One frequent scam involves tricking users into filling out surveys or visiting websites that generate profit for marketing firms, inducing them with promises of lurid images, often involving Justin Bieber. Facebook’s lawyers hand out cease-and-desist letters like candy. When a site called IsAnyoneUp began taking screenshots of users’ pages to post alongside naked photos of them, Facebook sent the purveyor a letter, shut down his account and took away his ability to install the “like” button on his site. (That still hasn’t stopped him.) It has also taken to court dozens of spammers, as well as advertising and marketing outfits, under the CAN-SPAM Act, and has been awarded more than $1 billion in judgments.
“Security is so high on e-mail now,” says Dirk Kollberg, a security researcher for Sophos in Wiesbaden, Germany. “Everyone knows to look out for spam and viruses there. But people are not educated about avoiding these things on social media, so that’s where the criminals are migrating.”
So, taking the law into its own cyberhands, Facebook is outing them. Collaborating with Sophos, the company fingered five Russians behind the Koobface worm that infected hundreds of thousands of computers and generated at least $6 million in criminal gains for its creators. When users clicked on “YOU HAVE TO WATCH THIS CRAZY VIDEO” Facebook posts, they were instructed to download an update to their software. The infected computers became unwitting slaves of a botnet ring run by the Koobface gang. They profited by hijacking Web searches to send users to rogue sites and bombarding users with ads run by other cybercriminals.
Sullivan went the extra step because he thought he had to. With Sophos he tracked digital bread crumbs to expose the guys responsible for Koobface (an anagram for Facebook). They gave their evidence to the FBI and waited for it to make a move. After over a year of inaction, though, they took a vigilante approach, exposing the gang members in the New York Times after a security blogger blew the whistle on one member, thus alerting the group they were being pursued. Facebook and Sophos detailed how they tracked them down using IP fingerprints, Foursquare check-ins, Twitter activity, friend lists on a Russian social networking site and Flickr photos that showed the gang vacationing in Europe. “It’s not about monitoring the users,” says Kollberg, who participated in the Koobface sting, “but producing security for users.”
Sometimes Facebook goes too far—then pulls back. While Sullivan won’t be specific, he cites the hypothetical case of teens using Facebook in a “spammy but borderline legal way”—say, by mass inviting people to events. In such instances his team usually doesn’t turn the offenders over to authorities but instead calls their mothers.
Unrestricted by constitutional restraints, “lawyers at Facebook and Google and Microsoft have more power over the future of privacy and free expression than any king or president or Supreme Court justice,” writes legal scholar Jeffrey Rosen.
Sullivan has harnessed crowdsourcing in service to vigilantism. He won a budget to institute a Facebook bug bounty program, where independent sleuths can earn $500 or more for identifying (and keeping secret) security and privacy flaws on the site. “We have a very small security team,” he says. “So we’re trying to turn our users into patrol guards.”
Facebook has made that reporting process easy for users, including a “report” button on every piece of content that appears on the site, so that users can mark it as “spam or a scam,” “nudity,” “violence,” “hate speech” or a number of other categories. It has helped the company nab lots of high-profile bad guys. When a Chicago man posted a photo of his toddler bound and gagged with duct tape in December, captioning it, “This is wut happens wen my baby hits me back,” the photo was flagged. He was reported to authorities and charged with aggravated domestic battery.
But the community watch program can also backfire, especially when it tries to turn a miscreant into a do-gooder. In December a user discovered an Achilles’ heel in Facebook’s security and decided to go public with it, showing how you could expose a user’s private photos by reporting one of their public photos as “abusive”; Facebook then offered the user’s other private photos to flag any that were similarly abusive. The person who discovered the flaw posted it to the Bodybuilding.com message forum and included some of Mark Zuckerberg’s private photos exposed in this way. “My hope is that he just didn’t know there was a place he could go to make money for reporting a bug like that,” says Sullivan.
HOW DO YOU MAKE A SAFER, more law-abiding place without creating a stifling surveillance cyberstate? Now that Facebook has biometric face prints for hundreds of millions of users, what will it do when law enforcement comes with a photo of a criminal suspect and asks for an identification?
“We’d insist on a court order, and we’d fight it as far in the court system as we could go,” says Sullivan, adding that Facebook gets thousands of calls and e-mails from authorities each week. “Recently a government agency wanted us to start logging information we don’t log. We told them we wouldn’t start logging that piece of data because we don’t need it to provide a good product. We talked to our general counsel. The law is not black-and-white. That agency thinks they can compel us to. We told them to go to court. They haven’t done that yet.”
Still, the Fourth Amendment against unreasonable searches and seizures can’t shield against these requests because of the so-called third party doctrine, which says the information you knowingly provide to a third party loses its privacy protections, making it much easier for the government to get your phone, banking and Internet records. In a recent Supreme Court decision Justice Sonia Sotomayor suggested the doctrine is “ill-suited to the digital age, in which people reveal a great deal of information about themselves to third parties in the course of carrying out mundane tasks.” She suggested the doctrine be rethought.
The 1986 Electronic Communications Privacy Act puts up extra legal barriers around e-mail and other communication “content.” It requires cops to get a warrant to get access, rather than simply asking companies to hand it over. But the feds have been pushing for easier access and looser regulations.
In such a fluid legal climate, how much can we trust sites like Facebook to safeguard our digital diaries that we willingly surrender to them? For Sullivan it’s a matter of protecting users and the integrity of the site—and hunting down the bad guys, a vestige of his days at the DOJ. “As a prosecutor, you feel like you’re always on the side of right.”
Will Facebook become a police state?
Facebook Pulls Back Curtain on ‘Timeline’ | Wired Enterprise | Wired.com
Eight hundred million Facebook profiles will soon get Timeline.
Over the coming week, Facebook will officially roll out the latest addition to its world-spanning social network: a Timeline that maps out each profile as a series of chronological events. And for the average user, it will appear as if by magic.
But Timeline, Facebook’s biggest interface change in recent memory, is the end result of a six-month effort not only to create a new piece of software, but to find a way of quickly serving that software to an audience of 800 million.
Serkan Piantino — the man who will run Facebook’s New York engineering office, set to open in 2012 — oversaw this sweeping project, and as the company prepared to launch the new interface, he sat down with Wired at Facebook’s Palo Alto, California, headquarters to give us the timeline for his Timeline.
It all started with a hack called Memories. This spring, Facebook held one of its famous hackathons — a caffeine-fueled, all-night coding bender where the only rule is that you can’t work on anything you’d normally work on during the day — and Memories was one of the slap-dash software creations that appeared the next morning.
The goal of these hackathons is to get developers thinking about a new type of application that can really hook Facebook’s users, and Memories had potential. The idea was simple: It called up Facebook photos from a particular year or other date range. The company tested it quietly among a few members, and it quickly caught on. People enjoyed going back in time to peruse, well, themselves.
New applications tend to evolve organically inside the company. Like Google, Facebook still clings to a kind of startup mentality. Piantino and a handful of engineers started to develop the Memories idea on their own, and eventually, the project gained enough steam that CEO Mark Zuckerberg kicked it up the priority list and management gave Piantino 15 to 20 of Facebook’s top engineers, pulled from various teams across the company. “There are things that Zuck always thinks are core to Facebook and things that we have to get right. News Feed was one of them,” Piantino says. “A new profile always gets priority.”
The goal was to provide each Facebook user with a rapid-fire summary of all the best things that happened to them over the course of a year. On a site of Facebook’s size, this sort of thing quickly gets complicated. The company stores over a trillion rows of indexed data, covering status updates and other “events” — and this doesn’t include photos and additional data that turns up on the site. Depending on your number of friends and their activity, a click to your News Feed can call up as many as 10,000 stories that account for about 8 megabytes of space. And according to Piantino, all this occurs in about seven-tenths of a second.
He and his team knew that to keep up with Facebook’s meteoric growth and the success of News Feed, they had to provide similar speeds with Timeline.
Memories Doesn’t Use Memory
The trouble is that Timeline must dig far deeper for information. News Feed is primarily a game of memory management. It’s looking for recent information. But Timeline looks into the past.
Timeline data is recorded on disks, not memory. The average hard disk spins at about 10,000 rpm, and moving the head across the disk takes about 5 milliseconds. But when you have hundreds of millions of users accessing data across tens of thousands of machines, management gets very complicated, very quickly. Those milliseconds add up, and so do the costs.
To keep the service fast, Facebook moved to a system that allows it to fetch the data for each Timeline reload with a single seek of the disk. In other words, all Timeline information for a particular user is stored on one disk “stripe” — or at least that’s the goal. “We had to go out on a limb with the idea of how we were going to build this,” Piantino said, remembering his initial meeting with Facebook’s capacity team — the employees who actually buy and deploy the servers.
Facebook management wanted Timeline rolled out quickly enough that Piantino and team had to make some sweeping assumptions about its requirements — and they couldn’t be wrong. Piantino deemed this meeting the “forcing function” of Timeline’s creation. Piantino likens Timeline’s hardware setup to a jet engine, not necessarily because of its speed, but because it’s designed to do one task extremely well.
In describing Timeline’s software infrastructure, Piantino will only go so far. But he does say that one of the keys to the system was that Facebook locates the aggregation code — the stuff that sorts through a user’s Timeline information — on the same machine as the data. “If you can ship your aggregation code to the box itself, that’s easier than using a network link,” he says. “We’re using the CPU for aggregation and the disks and input-output system for MySQL.”
Yes, Timeline uses MySQL, not Hadoop Hbase (which Facebook uses in other parts of its site) or some other NoSQL database. Whereas NoSQL databases are meant to spread vast amounts of unstructured data across vast array of machines, MySQL is relational database designed to organize data in neat rows and columns on a single machine. But MySQL can be “sharded” across many machines, and that’s what Facebook does.
“A lot of people are surprised that for this shiny new thing for Facebook, we’re using MySQL,” Piantino says. “We treat [MySQL] as a generic engine for data manipulation. We use it as a storage engine. And it’s really efficient.”
In 2008, Piantino saw a presentation by an engineer from InnoDB — an outfit that does storage engines for MySQL. He remembers thinking that if he was ever trying to solve the problem of finding data on disk, there “wasn’t a chance” he’d come up with a better way than the engine InnoDB had built for MySQL.
Piantino points out that Timeline fundamentally deals with ordered data — where ordering is its most important quality. The connection to other Facebook “events” is secondary. This is different from “graphed data,” which lets you quickly traverse different kinds of information — from comments on a picture to geo-location. News Feed is a graphed product. Timeline is a log product.
The proof is in the pudding. According to Piantino, Timeline streams 5 megabytes off disk and aggregates in about 120 milliseconds.
Timeline “dark launched” in July. This means that for the last five months, every time someone clicked on their profile, Facebook not only accessed its existing databases, it opened the Timeline databases for writes as well. In essence, Timeline was running under the covers, so the team could monitor loads, update code, check for bugs, and, yes, start storing the data.
To release Timeline, they decided to buck Facebook’s usual practice of releasing half-baked features or betas. At Facebook, the profile is sacred, so they ignored one of the company’s unofficial mottos: “Done is Better Than Good.” That said, the company released the new look to developers at its user conference in September, and with so many other savvy netizens pretending to be developers, Facebook already has over a million users with a new Timeline.
Facebook shares some secrets on making MySQL scale — Cloud Computing News
When you’re storing every transaction for 800 million users and handling more than 60 million queries per second, your database environment had better be something special. Many readers might see these numbers and think NoSQL, but Facebook held a Tech Talk on Monday night explaining how it built a MySQL environment capable of handling everything the company needs in terms of scale, performance and availability.
Over the summer, I reported on Michael Stonebraker’s stance that Facebook is trapped in a MySQL “fate worse than death” because of its reliance on an outdated database paired with a complex sharding and caching strategy (read the comments and this follow-up post for a bevy of opinions on the validity of Stonebraker’s stance on SQL). Facebook declined an official comment at the time, but last night’s night talk proved to me that Stonebraker (and I) might have been wrong.
Keeping up with performance
Kicking off the event, Facebook’s Domas Mituzas shared some stats that illustrate the importance of its MySQL user database:
- MySQL handles pretty much every user interaction: likes, shares, status updates, alerts, requests, etc.
- Facebook has 800 million users; 500 million of them visit the site daily.
- 350 million mobile users are constantly pushing and pulling status updates
- 7 million applications and web sites are integrated into the Facebook platform
- User data sets are made even larger by taking into account both scope and time
And, as Mituzas pointed out, everything on Facebook is social, so every action has a ripple effect that spreads beyond that specific user. “It’s not just about me accessing some object,” he said. “It’s also about analyzing and ranking through that include all my friends’ activities.” The result (although Mituzas noted these numbers are somewhat outdated) is 60 million queries per second, and nearly 4 million row changes per second.
Facebook shards, or splits its database into numerous distinct sections, because of the sheer volume of the data it stores (a number it doesn’t share), but it caches extensively in order to write all these transactions in a hurry. In fact, most queries (more than 90 percent) never hit the database at all but only touch the cache layer. Facebook relies heavily on the open-source memcached MySQL caching tool, as well as it custom-built Flashcache module for caching data on solid-state drives.
Keeping up with scale
Speaking of drives, and hardware generally, Facebook’s Mark Konetchy took the stage after Mituzas to share some data points on the growth of Facebook’s MySQL infrastructure. Although he made sure to point out that the “buzzkills at legal” won’t let him share actual numbers, he was able to point to 3x server growth across all data centers over the past two years, 7x growth in raw user data, and 20x growth in all user data (which includes replicated data). The median data-set size per physical host has increased almost 5x since Jan. 2010, and maximum data-set size per host has increased 10x.
Konetchy credits the ability to store so much more data per host on software-performance improvements made by Facebook’s MySQL team, as well as on better server technology. Facebook’s MySQL user database is composed of approximately 60 percent hard disk drives, 20 percent SSDs and 10 percent hybrid HDD-plus-SSD servers running Flashcache.
However, Facebook wants to buy fewer servers while still improving MySQL performance. Looking forward, Konetchy said some primary objectives are to automate the splitting of large data sets onto underutilized hardware, to improve MySQL compression and to move more data to the Hadoop-based HBase data store when appropriate. NoSQL databases such as HBase (which powers Facebook Messages) weren’t really around when Facebook built its MySQL environment, so there likely are unstructured or semistructured data currently in MySQL that are better suited for HBase.
With all this growth, why MySQL?
The logical question when one sees rampant growth and performance requirements like this is “Why stick with MySQL?”. As Stonebraker pointed out over the summer, both NoSQL and NewSQL are arguably better suited to large-scale web applications than is MySQL. Perhaps, but Facebook begs to differ.
Facebook’s Mark Callaghan, who spent eight years as a “principal member of the technical staff” at Oracle , explained that using open-source software lets Facebook operate with “orders of magnitude” more machines than people, which means lots of money saved on software licenses and lots of time put into working on new features (many of which, including the rather-cool Online Schema Change, are discussed in the talk).
Additionally, he said, the patch and update cycles at companies like Oracle are far slower than what Facebook can get by working on issues internally and with an open-source community. The same holds true for general support issues, which Facebook can resolve itself in hours instead of waiting days for commercial support.
On the performance front, Callaghan noted, Facebook might find some interesting things if large vendors allowed it to benchmark their products. But they won’t, and they won’t let Facebook publish the results, so MySQL it is. Plus, he said, you actually can tune MySQL to perform very fast per node if you know what you’re doing — and Facebook has the best MySQL team around. That also helps keep costs down because it requires fewer servers.
Callaghan was more open to using NoSQL databases, but said they’re still not quite ready for primetime, especially for mission-critical workloads such as Facebook’s user database. The implementations just aren’t as mature, he said, and there are no published cases of NoSQL databases operating at the scale of Facebook’s MySQL database. And, Callaghan noted, the HBase engineering team at Facebook is quite a bit larger than the MySQL engineering team, suggesting that tuning HBase to meet Facebook’s needs is more resource-intensive process than is tuning MySQL at this point.
The whole debate about Facebook and MySQL was never really about whether it should be using it, but rather about how much work it has put into MySQL to make it work at Facebook scale. The answer, clearly, is a lot, but Facebook seems to have it down to an art at this point, and everyone appears pretty content with what they have in place and how they plan to improve it. It doesn’t seem like a fate worse than death, and if it had to start from scratch, I don’t get the impression Facebook would do too much differently, even with the new database offerings available today.
Facebook Will File IPO as Early as April 2012 [REPORT]
Facebook will file its long-anticipated IPO between April and June 2012, according to a report from The Wall Street Journal on Monday.
The report, which cites “people familiar with the matter,” says that Facebook is considering raising $10 billion in an IPO that could value it at more than $100 billion. This is consistent with a report in June that used the same eye-popping number of zeros to describe Facebook’s expected valuation. If realized, the valuation would make Facebook’s IPO one of the largest in history — more than four times as big as Google’s $23 billion IPO in 2004.
While the first rumors of Facebook’s impending IPO predicted that the company would go public during the first quarter of 2012, recent reports had suggested that the offering had been pushed back to “September or later.”
The Wall Street Journal‘s sources cautioned that Facebook has not made any final decisions in its internal discussions about the timing of its filing and that market conditions will ultimately determine how much money the company seeks and the value of the company.
Mashable has contacted Facebook for comment and will update this article with any additional information.
Thai crackdown on Facebook remarks on king
Penalties …. Thai people face charges if they insult the king, queen, heir or regent. Photo: AFP
BANGKOK: Thailand has warned users of Facebook that they could face prosecution under harsh lese-majeste laws if they press ”share” or ”like” on images or articles considered unflattering to the Thai monarchy.
The prosecution of a Thai-born US citizen who has pleaded guilty to translating a banned biography of King Bhumibol Adulyadej has signalled that authorities are also targeting lese-majeste offences committed overseas.
Thailand’s Information and Communications Technology Minister, Anudith Nakornthap, says that even though Facebook clicks of ”like” or ”share” are only done to show support for messages, they could violate laws that carry sentences of three to 15 years jail for each charge.
Authorities in Thailand have asked Facebook to delete more than 10,000 pages of content as computer technicians in Bangkok scour the internet for royal insults. ”We have informed Facebook and sought their assistance in deleting content which is offensive to our monarchy,” Mr Anudith said.
Under Thai law, people face lese-majeste charges if they insult the king, queen, heir or regent. Even repeating details of an alleged offence is illegal.
The Computer Crimes Act also carries five-year jail terms for digital dissemination of information that threatens the security of the country or violates the ”peace and concord or good morals of the people”.
Public criticism of King Bhumibol, the world’s longest-serving monarch, is rare in Thailand. In the past four years, authorities have blocked more than 70,000 internet pages, most for insults to the monarchy, officials say.
The Asian Human Rights Commission has expressed ”grave concern” over the latest conviction and sentence of a person for lese-majeste, Ampon Tangnoppakul, 61, a retired truck driver and grandfather suffering from cancer. He wept this week after being sentenced to 20 years jail for sending ”vulgar” text messages judged to be insulting to Queen Sirikit.
Human rights groups estimate that more than 300 lese-majeste charges that have been laid in Thailand since 2006. They include Australian English-language teacher Harry Nicolaides, who was sentenced to six years in jail, commuted to three years on pleading guilty, over a book hardly anyone read. Only 50 copies were published. He received a royal pardon in February 2009 and was deported.
A Thai-born US citizen, Joe Gordon, 55, was detained in Thailand in May for translating the book about King Bhumibol in the US where he had lived for 30 years. He had returned to Thailand temporarily for medical treatment. Gordon, who pleaded guilty to a lese-majeste charge in October hoping for a lenient sentence, is awaiting sentencing.
The webmaster of the Thai website Prachatai, Chiranuch Premchaiporn, is on trial over comments posted on the site that were deemed insulting to the monarchy. Ms Chiranuch, 44, told the court she regularly read through the thousands of items posted daily on the message board and deleted potentially offensive messages when she found them. Prosecutors allege she did not act quickly enough.
Ms Chiranuch was recently awarded the Hellman-Hammett award given annually to writers or activists for expressing freedom of expression.
Google vs. Facebook on Privacy and Security
Whether you use Social Networks for games, video and photos, or just to re-connect with old friends, you should be aware of how your Personally Identifiable Information (PII) is protected. This infographic details several of the ways Google and Facebook handle Privacy and Security.