TheEdge10 : Big Data is Here - Hadoop to the Rescue

Editor's Notes

1. Lets say we have a huge log file that we need to analyzeOr that we have large amount of data that we need to store – what shall we do?Add more CPUs or memory to single machine (node?)Or add more nodes to the system?
2. Talk about commodity hardware"Commodity" hardware is hardware that is easily and affordably available. A device that is said to use "commodity hardware" is one that uses components that were previously available or designed and are thus not necessarily unique to that device. Unfortunately, at some point there won’t be a big enoughmachine available for the larger data sets. More importantly, the high-end machinesare not cost effective for many applications. For example, a machine with four timesthe power of a standard PC costs a lot more than putting four such PCs in a cluster.Spreading and dividing the data between many machines will provide a much higher throughput - distributed software application
3. 1. Because it runs on commodity hardware2. With X2 machines it will run close to X2 faster4. Talk about it being mature and very popular (then move to next slide)
4. http://wiki.apache.org/hadoop/PoweredByStories: New york times – converted 11 million articles from TIFF images to PDFTwitter - We use Hadoop to store and process tweets, log files
5. http://hadoopblog.blogspot.com/2010/05/facebook-has-worlds-largest-hadoop.html
6. Cutting is also the founder of Apache Lucene, popular text search library
7. Although we’ve spoken about key and values we have yet to discuss their typesSpeak about why java serialization is bad , about pluggable frameworks and about using predefined hadoop.http://tmrp.javaeye.com/blog/552696
8. Those 2 static classes will reside in a single file. Those inner classses are independent – during job execution the Mapper and Reducer are replicated and run in various nodes in different JVMs
9. Executing 2 jobs manually is possible, but its more convenient to automate itThe input of the 2nd will be the output of the firstJobClient.runJob() is blocking.
10. Useful for writing simple short programs that are rapidly developed in scriptsOr writing programs that can take advantage of non-Java libraries
11. Useful for writing simple short programs that are rapidly developed in scriptsOr writing programs that can take advantage of non-Java libraries
12. it becomes necessary to partition it across a number of separate machines. Filesystems that managethe storage across a network of machines are called distributed filesystems. Since theyare network-based, all the complications of network programming kick in, thus makingdistributed filesystems more complex than regular disk filesystems. For example, oneof the biggest challenges is making the filesystem tolerate node failure without suffering data loss.
13. Lets say you have a 100TB in a file – HDFS abstracts the complexity and give you the illusion that you're dealing with a single fileVery large files“Very large” in this context means files that are hundreds of megabytes, gigabytes,or terabytes in size. There are Hadoop clusters running today that store petabytesof data.*Streaming data accessHDFS is built around the idea that the most efficient data processing pattern is awrite-once, read-many-times pattern. A dataset is typically generated or copiedfrom source, then various analyses are performed on that dataset over time. Eachanalysis will involve a large proportion, if not all, of the dataset, so the time to readthe whole dataset is more important than the latency in reading the first record.Commodity hardwareHadoop doesn’t require expensive, highly reliable hardware to run on. It’s designedto run on clusters of commodity hardware (commonly available hardware availablefrom multiple vendors†) for which the chance of node failure across the cluster ishigh, at least for large clusters. HDFS is designed to carry on working without anoticeable interruption to the user in the face of such failure
14. Some of them exist only on one server and some across all servers
15. The most important daemonMake sure that the server hosting the NameNode will not store any data locally or perform any computations for a MapReduce program
16. Constantly reports to the namenode – informs the namenode of which blocks it is currently storingDatanodes also poll the namenode to provide information regarding local changes as well as receiving instructions to create move or delete blocks
17. all blocks in a file except the last block are the same size
18. One important aspect of this design is that the client contacts datanodes directly toretrieve data and is guided by the namenode to the best datanode for each block. Thisdesign allows HDFS to scale to a large number of concurrent clients, since the datatraffic is spread across all the datanodes in the cluster. The namenode meanwhile merelyhas to service block location requests (which it stores in memory, making them veryefficient) and does not, for example, serve data, which would quickly become a bot-tleneck as the number of clients grew.
19. One per clusterTakes snapshots by communicating with the namenode at interval specified by configuration
20. Relaunch possibly on a different node – up to num of retries
21. If JobTrackerfails to receive a message from a TaskTracker it assumes failure and submit the task to other nodes
22. -help to look for help
23. Local mode is to assist debugging and create the logicUsually the local mode and the Pseudo do mode will work on a subset of dataPseudo is “cluster of one”Switching is easy
24. If there are bugs that sometimes cause a task to hand or slow down then relying on the speculative execution to avoid these problems is unwise and won’t work reliably since the same bugs are likely to affect the speculative task
25. You may have a few large data processing jobs that occasionally take advantageof hundreds of nodes, but those same nodes will sit idle the rest of the time.You may be new to Hadoop and want to get familiar with it first before investing ina dedicated cluster. You may own a startup that needs to conserve cash and wantsto avoid the capital expense of a Hadoop cluster. In these and other situations, itmakes more sense to rent a cluster of machines rather than buy it.You can rent computing and storage services from AWS on demand as your requirement scales.As of this writing, renting a compute unit with the equivalent power of a 1.0 GHz32-bit Opteron with 1.7 GB RAM and 160 GB disk storage costs $0.10(varies if its windows or Unix instance) per hour. Using acluster of 100 such machines for an hour will cost a measly $10!
26. Supported operating systems onEC2 include more than six variants of Linux, plus Windows Server and OpenSolaris.Other images include one of the operating systems plus pre-installed software, such asdatabase server, Apache HTTP server, Java application server, and others. AWS offerspreconfigured images of Hadoop running on Linux,
27. Load users, Load pagesFilter by ageJoin by nameGroup on URLCount click, sort clicks , get top 5
28. 1.Under the covers pig turns the tranformations into a series of MapReduce jobs but we as programmersAre unaware of this which allows us to focus on the data rather than the nature of execution2. Pig Latin is a data flow pgoramming language where SQL is a declarative programming languate. Ping Latin program is a tep by step set of perations on an input SQL statements is a a set of statements taken together and produce output3. A script file – also pluginPigPen for eclipse exists
29. HBase – uses HDFS as the underlying file system. Supports of billions of rows and millions of columns
30. HBase – uses HDFS as the underlying file system. Supports of billions of rows and millions of columns
31. HBase – uses HDFS as the underlying file system. Supports of billions of rows and millions of columns
32. HBase – uses HDFS as the underlying file system. Supports of billions of rows and millions of columns