Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd Logo
Upload

Welcome to Scribd!

  • 35 views

    12 CodigoNetbeans

    Uploaded by

    Miguel Angel
    The document provides instructions for installing Hadoop on NetBeans, creating a Java project called "countword" that uses Hadoop libraries to count word frequencies, compiling the program, and then executing it on Hadoop to count words in input files. The program takes the input and output directories as arguments and uses mappers to tokenize the text, reducers to sum word counts, and outputs the results.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    12 CodigoNetbeans

    Uploaded by

    Miguel Angel
    35 views5 pages
    The document provides instructions for installing Hadoop on NetBeans, creating a Java project called "countword" that uses Hadoop libraries to count word frequencies, compiling the program, and then executing it on Hadoop to count words in input files. The program takes the input and output directories as arguments and uses mappers to tokenize the text, reducers to sum word counts, and outputs the results.

    Original Description:

    12 CodigoNetbeans

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document provides instructions for installing Hadoop on NetBeans, creating a Java project called "countword" that uses Hadoop libraries to count word frequencies, compiling the program, and then executing it on Hadoop to count words in input files. The program takes the input and output directories as arguments and uses mappers to tokenize the text, reducers to sum word counts, and outputs the results.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    35 views5 pages

    12 CodigoNetbeans

    Uploaded by

    Miguel Angel
    The document provides instructions for installing Hadoop on NetBeans, creating a Java project called "countword" that uses Hadoop libraries to count word frequencies, compiling the program, and then executing it on Hadoop to count words in input files. The program takes the input and output directories as arguments and uses mappers to tokenize the text, reducers to sum word counts, and outputs the results.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 5

    Descargar

    https://netbeans.org/downloads/8.0.1/

    Java EE

    Instalar

    File/New Project

    Next

    Project Name: countword

    Project Location: C:\Users\Usuario\Desktop\ejemplos (en el escritorio en un carpeta


    ejemplos)

    Create Main Class: countword.Countword

    Finish

    Dentro de la Ventana Proyectos hacer click derecho en el icono de countword

    Hacer click en “Properties”

    Click en “Libraries”

    Click en “Add JAR/Folder”

    carpeta:"hadoop-2.8.0/share/hadoop/common"

    seleccionar:"hadoop-common-2.8.0.jar"

    Open

    Add JAR/Folder

    carpeta:"hadoop-2.8.0/share/hadoop/mapreduce"

    seleccionar:"hadoop-mapreduce-client-core-2.8.0.jar"

    seleccionar:"hadoop-mapreduce-client-common-2.8.0.jar"

    seleccionar:"hadoop-mapreduce-client-jobclient-2.8.0.jar"

    Copiar el código en la clase Countword


    package countword;

    import java.io.IOException;
    import java.util.StringTokenizer;

    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.lib.input.FileSplit;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.util.GenericOptionsParser;

    public class Countword {


    public static class TokenizerMapper
    extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);


    private Text word = new Text();

    public void map(Object key, Text value, Context context


    ) throws IOException, InterruptedException {
    String filename = ((FileSplit)context.getInputSplit()).getPath().getName();

    StringTokenizer itr = new StringTokenizer(value.toString());


    while (itr.hasMoreTokens()) {
    word.set(itr.nextToken().toLowerCase().replaceAll("[^a-z]+","") +" "+ filename); // get rid
    of speicial char
    context.write(word, one);
    }
    }
    }

    public static class IntSumReducer


    extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,


    Context context
    ) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
    sum += val.get();
    }
    result.set(sum);
    context.write(key, result);
    }
    }

    public static void main(String[] args) throws Exception {


    Configuration conf = new Configuration();

    Job job = new Job(conf, "word count");


    job.setJarByClass(Countword.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[1]));
    FileOutputFormat.setOutputPath(job, new Path(args[2]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
    }

    Compilar el programa.
    Tendrás el archivo

    Copiarlo a c:/
    Ejemplo 1

    cd/
    hadoop dfsadmin -safemode leave
    hadoop fs -mkdir /input_dir
    hadoop fs -put C:/input_file.txt /input_dir
    hadoop fs -ls /input_dir/
    hadoop dfs -cat /input_dir/input_file.txt

    hadoop jar C:/countword.jar Countword /input_dir /output_dir

    Ejemplo 2

    cd/
    hadoop dfsadmin -safemode leave
    hadoop fs -mkdir /input_dir
    hadoop fs -put C:/shake.txt /input_dir
    hadoop fs -ls /input_dir/
    hadoop dfs -cat /input_dir/shake.txt

    hadoop jar C:/countword.jar Countword /input_dir /output_dir

    You might also like