Cluster Computing and MapReduce

Lecture 1: Introducing Mapreduce and Cluster Computing
Distributed systems overview, review of synchronization and networking.



Lecture 2: The MapReduce programming model.
Overview of the MapReduce programming model.



Lecture 3: Distributed File Systems
Overview of distributed file systems with attention to the Google File System.




Lecture 4: Clustering Algorithms
Types of clustering algorithms, MapReduce implementations of K-Means and Canopy Clustering




Lecture 5: Graph Algorithms
Graph representations, distributed Pagerank, distributed Dijkstra.





Just as people are social animals, computers are social machines—the more, the merrier. Twenty or thirty years ago, large, centralized mainframes sat alone in sheltered bunkers in computer science departments and government offices alike, choking for hours on mere megabytes of data. Even with recent advances in server technology, large, centralized machines are still struggling to cope with today’s modern computational challenges, which now involve terabytes of data and processing requirements well beyond a single CPU (or two, or four, or eight). One computer just won’t hack it; these days, to support a new paradigm of massively parallel systems architecture, we need to break the machine out of its bunker and give it some friends.

In this age of “Internet-scale” computing, the new, evolving problems faced by computer science students and researchers require a new, evolving set of skills. It’s no longer enough to program one machine well; to tackle tomorrow’s challenges, students need to be able to program thousands of machines to manage massive amounts of data in the blink of an eye. This is how I, along with my good friend and mentor Ed Lazowska of the University of Washington’s CSE department, started to think about CS curricula and the obstacles to teaching a practical and authentic approach to massively parallel computing.

It's no easy feat. Teaching these methods effectively requires access to huge clusters and innovative new approaches to curricula. That's why we are pleased to announce the successful implementation of our Academic Cluster Computing Initiative pilot program at a handful of schools, including the University of Washington, Carnegie-Mellon University, Massachusetts Institute of Technology, Stanford University, the University of California at Berkeley and the University of Maryland. This pilot extends our expertise in large scale systems to strong undergraduate programs at the pilot schools, allowing individual students to take advantage of the hundreds of processors being made available. As the pilot progresses, we'll work with our technology partner IBM to shake the bugs out of the system so that we can expand the program to include more educators and academic researchers.

The future of computing is already taking shape on campuses today, and Google and IBM are thrilled to help inspire a new generation of computer scientists to think big. All of the course material developed by UW as well as other tools and resources to facilitate teaching this cutting- edge technology is available at http://code.google.com/edu. If you're a student wondering just what this sort of thing means for you, check out the five-part video lecture series (originally offered to Google Engineering interns) that introduces some of the fundamental concepts of large-scale cluster computing.

Earlier this year, the University of Washington partnered with Google to develop and implement a course to teach large-scale distributed computing based on MapReduce and the Google File System (GFS). The goal of developing the course was to expose students to the methods needed to address the problems associated with hundreds (or thousands) of computers processing huge datasets ranging into terabytes. I was excited to take the first version of the class, and stoked to serve as a TA in the second round.

But you can't program air, so Google provided a cluster computing environment to get us started. And since computers can't program themselves (yet?), UW provided the most essential component: students with sweet ideas for a huge cluster. After learning the ropes with these new tools, students finished the course by producing an impressive array of final projects, including an n-body simulator, a bot to perform Bayesian analysis on Wikipedia edits to search for spam, and an RSS aggregator that clustered news articles by geographic location and displayed them using the Google Maps API. Check out Geozette.

We are looking at ways to encourage other universities to get similar classes going, so we've also published the course material that was used at the University of Washington on Google Code for Educators. You're more than welcome to check out the Google Summer Intern video lectures on MapReduce, GFS, and parallelizing algorithms for large scale data processing. This summer I've been working on exposing these educational resources and other tools so that anyone can work on and think about cool distributed computing problems without the overhead of installing his or her own cluster. In that vein, we've released a virtual machine containing a pre-configured single node instance of Hadoop that has the same interface as a full cluster without any of the overhead. Feel free to give it a whirl.

We're happy to be able to expose students and researchers to the tools Googlers use everyday to tackle enormous computing challenges, and we hope that this work will encourage others to take advantage of the incredible potential of modern, highly parallel computing. Virtually all of this material is Creative Commons licensed, and we encourage educators to remix it, build upon it, and discuss it in the Google Code for Educators Forum.

Lastly, a quick shout out to the other interns who helped out on our team this summer: Aaron Kimball, Christophe Taton, Kuang Chen, and Kat Townsend. I'll miss you guys!

Resources:
http://google-code-updates.blogspot.com/2007/09/uw-and-google-teaching-in-parallel.html
http://code.google.com/edu/
http://code.google.com/edu/content/submissions/mapreduce-minilecture/listing.html
http://googleblog.blogspot.com/2007/10/let-thousand-servers-bloom.html

YouTube Scalability

This talk will discuss some of the scalability challenges that have arisen during YouTube's short but extraordinary history. YouTube has grown incredibly rapidly despite having had only a handful of people responsible for scaling the site. Topics of discussion will include hardware scalability, software scalability, and database scalability.

Speaker: Cuong Do Cuong is currently an engineering manager at YouTube/Google. He was part of the engineering team that scaled the YouTube software and hardware infrastructure from its infancy to its current scale. Prior to YouTube/Google, he held various software development and management positions at PayPal and Inktomi.



Google Tech Talks June 23, 2007

100 Web Apps for Everything You Will Possibly Need

Running a business for yourself means you have to be inventive and always on the lookout for a new and better way to get things done. Innovation junkies, take note: the Internet has a lot to offer. From invoicing to marketing, these are tools that freelancers need to know about.

Organization

If you’re busy with lots of client work, it’s easy for things to get out of hand. Don’t let your work get away from you; organize information and projects with these tools.

1. Backpack: Get your projects organized by using Backpack. Create to-do lists, notes, files, images and a calendar with reminders that can be sent via email or to your mobile device.
2. Central Desktop: Collaborate, communicate and share files with clients and coworkers using Central Desktop.
3. iOrganize: With iOrganize, freelancers can organize work by keeping notes, ideas and bookmarks in one place.
4. Viapoint: Viapoint makes it easy for freelancers to store emails and files by client or project.
5. Stikkit: Use Stikkit’s “little yellow notes that think” to keep in touch, plan and collaborate with clients and coworkers.
6. Webnote: Webnote users can take notes using a web browser, then save and return to the notes on any computer. This is especially helpful if you’re working on-site with a client and have to use a computer other than your own.
7. Netvibes: Spend less time searching and more time working by customizing your browser’s start page with email, feeds, messaging, job boards and much, much more.

Calendars & To-Do Lists

Client meetings, important events and a never-ending list of things to do can wear you down if you can’t make them manageable. Use these handy calendars and to-do lists to keep your schedule from taking over your business.

8. Remember The Milk: Remember The Milk reminds you to take care of important tasks, so you’ll never forget when a project is due.
9. CalendarHub: Use CalendarHub’s web-based calendar to keep track of deadlines and set goals for your business.
10. Google Calendar: Google Calendar is a web based tool that allows users to organize their schedule, so you’ll always know exactly what you need to be working on.
11. Planzo: Planzo’s online calendar keeps freelancers connected to events and things to do. It lets you share your events just about anywhere, get a daily digest and receive text message reminders.
12. Spongecell: Spongecell is an online calendar made for freelancers with lots of meetings and engagements. Plan events, spread the word and allow guests to add content to your calendar.
13. Neptune: Neptune’s web based to do list tool helps you keep track of the things you need to take care of every day. It lets you email yourself new tasks, get an email report every morning and upload files to store with your projects.
14. Ta-da List: When you’re busy with lots of projects, it can be hard to keep track of what you’ve accomplished. Make web-based ta-da lists for yourself or other people, then share them and check items off as you go.

Your Money

Money is what keeps your freelance business going, but managing it can be tedious and time consuming. Sure, it’s fun to see money come in, but does anyone really like sending out invoices? Use these tools to make the process of managing your income easier and more enjoyable.

15. Wesabe: Use Wesabe to keep tabs on where the money in your business goes, helping you to make better financial decisions.
16. InstaCalc: InstaCalc is a web based calculator with lots of bells and whistles including spreadsheet capabilities, unit conversions and programming commands. It’s great for freelancers because you can send clients links to any of your calculations, put a calculator on your website or create charts and graphs from your calculations.
17. XE: If you’re working with an overseas client, you may need to handle foreign currencies. Use XE’s currency converter for accurate calculations with up-to-the-minute currency rates.
18. Dimewise: Use Dimewise’s web-based convenience to manage your business transactions wherever you are.
19. FreshBooks: FreshBooks offers a program for “painless billing,” so you’ll never have to spend hours sending client invoices out. Designed with service-based businesses in mind, this software provides a method to manage and send invoices, handle work orders and generate reports with ease.

Storage

Do you have too many client files clogging up your hard drive? Use these services to take a load off.

20. openonmy: openonmy’s website offers storage for files up to 1GB. These files are made to be accessible from any computer, so you can open them up even when on-site with a client.
21. Xdrive: Xdrive offers 5GB of online storage. Use their service to share files with your clients and coworkers.
22. YouSendIt: Have you ever had to clean out your inbox just so you’d have room to send out an email? YouSendIt puts and end to that trouble by sending files up to 2GB to your clients.
23. Flickr: Save space on your hard drive by uploading your photos to Flickr. It’s a great way to store, share and organize photos for your business.
24. Box: Create an online file-sharing location for you and your clients on Box.
25. MediaTemple: Keep your important client files safe on MediaTemple’s servers. They offer innovative hosting solutions for all kinds of websites.
26. DivShare: Use DivShare’s file hosting service to email files to clients or store them in a folder for shared access later.

Project Management & Productivity

Do you wonder where all of your time goes? Do your clients want frequent status updates? Keep tabs on your time and projects with these tools.

27. Harvest: Harvest offers web-based time tracking software with simplicity. Track your progress and inform your clients using Harvest’s reports.
28. Side Job Track: Side Job Track, designed with freelancers in mind, provides software that lets you track and manage project information. Features include job tracking, invoicing and reporting.
29. Basecamp: Basecamp offers a way to improve project communication. Use it to keep your teammates and clients informed about what’s going on with your projects.
30. ConceptShare: ConceptShare provides online design collaboration. Invite clients and coworkers to view your design and make comments on your work.
31. ProjectStat.us: Instead of fielding constant calls about the status of a project, let your customers view the status of their project online with ProjectStat.us.

Writing & Design Tools

Being creative can get expensive. Purchasing icons, stock photos, book publishing and the like can add up fast and eat into your profit margin. Check out these free and inexpensive tools designed to make the lives of freelance writers and designers cheaper and easier.

32. Writeboard: Have you ever had a client or coworker revise a document and end up wiping out your work? Use Writeboard to share and collaborate while saving separate versions each step of the way.
33. Lulu: Lulu gives fledgling writers an easy way to get published. Each product is printed as it’s ordered, so you don’t have to build an expensive inventory.
34. MailBuild: Built for web designers, MailBuild allows users to set up a template for client email campaigns.
35. MyFonts: Do you want to see how fonts will look on your design before committing to a purchase? MyFonts provides a source for trying out and buying new fonts.
36. IconBuffet: Find and trade stock icons for your web designs on IconBuffet.
37. stock.xchng: Add photos to your writing or web design at low or no cost. stock.xchng provides tons of stock photos, many of them royalty-free.
38. Google Docs & Spreadsheets: Don’t bother spending hard-earned money for the latest version of MS Word. Create, upload, share and edit word and spreadsheet documents online using Google Docs & Spreadsheets.
39. MorgueFile: MorgueFile’s high resolution photos are has-beens in the stock photography world, but that doesn’t mean they won’t do a great job spicing up your work.

Security & Privacy

Don’t let your work get stolen or compromised. Can you imagine how your business would suffer if you lost files due to a virus? Or worse, if the financial information of your clients got exposed to spyware? These services help you stay safe and secure.

40. SpamSieve: Fighting spam takes time away from your business and opens your system up to intruders. Use SpamSieve to keep spam out of your Mac email client.
41. Escrow: Don’t get burned by clients that make fraudulent payments. Use Escrow to protect yourself and your clients from payment fraud.
42. Cloudmark Desktop: Cloudmark protects your Outlook inbox from spam, phishing and viruses, so you’ll never have to worry about compromising business files when reading your email.
43. PayPal: Give your clients a way to pay without sharing their financial information by using PayPal.
44. Moneybookers: Open your business to the worldwide market safely. Use Moneybookers for secure worldwide payment and acceptance.
45. Spamato: Keep your business email secure by using Spamato with Outlook, Thunderbird or Mozilla Mail.
46. Spybot Search & Destroy: Make sure your confidential client information is safe from prying eyes by scanning for spyware with Spybot Search & Destroy.
47. AVG Anti-Virus Free Edition: Protect your business files with a good antivirus program. AVG Anti-Virus offers a free solution for protecting your computer from viruses.

Mobility & Contact

When running a business, it’s essential that you’re able to keep in contact with clients, vendors, coworkers and other important acquaintances. Use these tools to communicate with ease.

48. Campfire: Campfire is a web-based chat tool that’s ideal for online meetings with clients or coworkers.
49. Meebo: Don’t miss out on a client just because you don’t use the same instant messaging tool. Get access to every major messaging service on Meebo’s website without having to download anything.
50. FaxZERO: Online communication is popular, but sometimes you just have to send a fax. But who wants to invest in a fax machine and phone line for something that happens only on a rare occasion? Instead, use FaxZERO to send a fax anywhere in the US or Canada for free.
51. eBuddy: Don’t miss out on important messages while you’re away from your base of operations. Log on to eBuddy to sign on to online messengers using your mobile device.
52. K7: FaxZERO lets you send out a fax for free online, but what if a client wants to fax something to you? Enlist the help of K7, a service that assigns users a phone number that accepts fax and voicemail messages, which are then sent to your email.
53. GoToMeeting: Use GoToMeeting to keep in touch with clients and coworkers via online meetings.
54. LogMeIn: Don’t let your business suffer because you can’t take your computer on the road. Use LogMeIn to get access to your computer’s desktop anywhere.

Marketing & Networking

You provide a great service and offer awesome rates, so why isn’t the world knocking down your door with business? Perhaps it’s because they have no idea you exist. Get your name out there and find new clients with these tools.

55. LinkedIn: LinkedIn offers online networking at its best. Find opportunities and contacts based on your work and the people you already know.
56. askCHARITY: Take advantage of askCHARITY’s database of key media contacts to get the word out about your business.
57. Craigslist: Craigslist is an invaluable tool for any freelancer. Find clients, sell your work, buy supplies, network and more using Craigslist.
58. Coroflot: Use Coroflot to post an online portfolio and find design jobs.
59. ProfessionalOnTheWeb: ProfessionalOnTheWeb hosts a directory of portfolios. Make sure yours is there when clients search for help.
60. Elance: Use Elance to get connected with clients that need your services on a project.
61. 37signals Gig Board: Use the 37signals Gig Board to find freelance jobs in programming, design and more.
62. WebProJobs: Find full-time and freelance web professional jobs on WebProJobs.
63. Job Pile: Job Pile aggregates popular freelance job boards, so you can spend time working instead of searching.
64. YouTube: Use YouTube’s viral video to get the word out about your creativity.
65. CafePress: CafePress offers on-demand printing for promotional items as well as a place to sell your design.
66. Spot Runner: Use Spot Runner to build a TV advertising campaign.

Business & Legal

Paperwork isn’t always fun, but it is necessary. Forms and agreements can provide legal protection and help you avoid disputes down the road. Check out these tools that help you protect yourself and spend less time on the boring stuff.

67. MyNewCompany: Get legal and tax protection for your freelance business by making things official. Use MyNewCompany to incorporate or form an LLC online.
68. Help Me Work: Get the stability of corporate life while still working as an independent consultant. Help Me Work takes care of taxes, client billing, paychecks, benefits, retirement plans and more.
69. Designers Toolbox: Get set up with all the legal forms your design business may need with Designers Toolbox.
70. Creative Commons: Use Creative Commons to specify how much or how little legal restrictions you want your work to carry.
71. AIGA: Protect your business with AIGA’s standard form of agreement for design services.
72. Nolo: If you’ve got a legal question about your business, head to Nolo. Experts there offer legal advice for independent contractors and consultants.
73. Creative Public: Head to Creative Public for forms, contracts, pricing guides and more for your design business.

Client Contact & Feedback

Your clients are what keep you in business, so it’s important to check in with them and make sure they’re satisfied. Use these programs to keep in touch with your clients and find out what they have to say about your work.

74. Breeze: Send out email campaigns to your clients with Breeze’s easy tool.
75. Wufoo: Use Woofoo to make forms, surveys and invitations for your customers with ease.
76. Relenta CRM: Keep track of customer relationships using email, contact, document and activity management with Relenta.
77. Highrise: Don’t lose track of your customers: keep in touch with Highrise’s online contact management system.
78. ScratchnScribble: Use ScratchnScribble’s service to have handwritten or printed greeting cards sent to your clients.

Website Tools

Clients are looking for your business online. Will you be there? Check out these applications to make sure your business has an excellent online presence.

79. Big Cartel: Set up an online store to sell your products using Big Cartel’s service.
80. WordPress: Blog about your business and industry with WordPress.
81. Inblogit: If you’re a design professional, your blog should be attractive. Use Inblogit for blog functionality with more flexbility in design.
82. Icebrrg: Create web forms with Icebrrg so that your website can generate customer inquiry and feedback.
83. XHTMLized: If you’re too busy to build a website yourself, use XHTMLized. They take your design and turn it into a web page that is browser and search engine friendly.
84. XHTMLiT: XHTMLit offers freelancers another time-saving solution for converting design into HTML.
85. Userplane: Make your website interactive for your clients by bundling chat, messaging, video, search and live presence with Userplane.
86. Ning: Let your clients network with each other by creating your own Ning social network.
87. ExpressionEngine: Impress your customers by having a website with lots of features. Use ExpressionEngine to publish just about anything on your website.
88. FlashDen: Build a great-looking website that your clients will want to visit. Buy Flash, audio, video and fonts to make your website look good at FlashDen.

Printing & Packaging

When your products look good, your business does too. Presentation makes a difference, so be sure to make a good impression by using these innovative printing and packaging tools.

89. Jewelboxing: Impress your customers with Jewelboxing’s customized, professional-grade DVD and CD packages.
90. Qoop: Qoop prints everything freelancers might need: business cards, apparel, promotional items, posters and more.
91. Moo: Moo prints note cards and mini cards from your photos and design. These can be used for promotional materials or business cards.

Tools to Give & Take

These tools pack a double-whammy of functionality. Use them to get the resources you need or as an alternative revenue stream.

92. Google AdWords: Use Google AdWords to advertise your business or make money by putting AdWords on your website.
93. Prosper: Using Prosper’s people-to-people lending website, you can borrow money for business expenses from real people. If you have extra cash, use Prosper to earn interest by lending to others.
94. Text Link Ads: Generate interest in your website and attract clients with Text Link Ads or earn money by selling space for others on your site.
95. PayPerPost: Use PayPerPost to have a blogger write about your product or service. Alternately, you can earn money by writing about someone else’s business.
96. Guruza: Find answers to your business questions or earn money by giving expert advice on Guruza.

Miscellaneous

From office suites to creative stimulation, these tools have a lot to offer for freelancers. Check out these applications that cover anything and everything else you might need for your business.

97. Zoho: Zoho’s Office Suite includes a variety of software solutions for freelancers: a word processor, spreadsheet, presentation tool, wiki writer, notebook, project management, CRM solution, database creator, calendar, web conferencing, email and chat.
98. Veetro: Veetro offers another all-in-one solution that specializes in the tools freelancers need the most. This program combines money, document, task and customer management with email marketing, reporting and blog publishing.
99. Work: Are you clueless about what it takes to run a business in your industry? Work offers guides that show you how.
100. Pandora: Get your creative juices flowing with Pandora’s fully customizable internet radio.

Visit our StumbleUpon Blog for more technology and productivity tips

You could read the original article by clicking here