Home > Software > BIGDATA > HADOOP
Interview Questions   Tutorials   Discussions   Programs   Videos   Discussion   

HADOOP - esProc is newly developed big data processing tool which is easier and faster in terms of developing efficiency and performance.

asked Li October 29, 2013 03:14 AM  

esProc is newly developed big data processing tool which is easier and faster in terms of developing efficiency and performance.


1 Answers

answered By Li   0  

Hadoop is based on the MapReduce framework for big data development and computation. Everything seems to be well if the computing task is simple. However, issues appear for those a little bit more complex computations. The poor development efficiency will bring more and more serious impacts with the growing difficulty of problem. One of the commonest computations is the "associative computing".

         For example, in HDFS, there are 2 files holding the client data and the order data respectively, and the customerID is the associated field between them. How to perform the associated computation to add the client name to the order list?

         The normal method is to input 2 source files first. Process each row of data in Map according to the file name. If the data is from Order, then mark the foreign key with ”O” to form the combined key; If the data is from Customer, then mark it with ”C”. After being processed with Map, the data is partitioned on keys, and then grouped and sorted on combined keys. Lastly, combine the result in the reduce and output. It is said that the below code is quite common:

public static class JMapper extends Mapper { //mark every row with "O" or "C" according to file name @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String pathName = ((FileSplit) context.getInputSplit()).getPath().toString(); if (pathName.contains("order.txt")) {//identify order by file name String values[] = value.toString().split("\t"); TextPair tp = new TextPair(new Text(values[1]), new Text("O"));//mark with "O" context.write(tp, new Text(values[0] + "\t" + values[2])); } if (pathName.contains("customer.txt")) {//identify customer by file name String values[] = value.toString().split("\t"); TextPair tp = new TextPair(new Text(values[0]), new Text("C"));//mark with "C" context.write(tp, new Text(values[1])); } } } public static class JPartitioner extends Partitioner { //partition by key, i.e. customerID @Override public int getPartition(TextPair key, Text value, int numParititon) { return Math.abs(key.getFirst().hashCode() * 127) % numParititon; } } public static class JComparator extends WritableComparator { //group by muti-key public JComparator() { super(TextPair.class, true); } @SuppressWarnings("unchecked") public int compare(WritableComparable a, WritableComparable b) { TextPair t1 = (TextPair) a; TextPair t2 = (TextPair) b; return t1.getFirst().compareTo(t2.getFirst()); } } public static class JReduce extends Reducer { //merge and output protected void reduce(TextPair key, Iterable values, Context context) throws IOException,InterruptedException { Text pid = key.getFirst(); String desc = values.iterator().next().toString(); while (values.iterator().hasNext()) { context.write(pid, new Text(values.iterator().next().toString() + "\t" + desc)); } } } public class TextPair implements WritableComparable { //make muti-key private Text first; private Text second; public TextPair() { set(new Text(), new Text()); } public TextPair(String first, String second) { set(new Text(first), new Text(second)); } public TextPair(Text first, Text second) { set(first, second); } public void set(Text first, Text second) { this.first = first; this.second = second; } public Text getFirst() { return first; } public Text getSecond() { return second; } public void write(DataOutput out) throws IOException { first.write(out); second.write(out); } public void readFields(DataInput in) throws IOException { first.readFields(in); second.readFields(in); } public int compareTo(TextPair tp) { int cmp = first.compareTo(tp.first); if (cmp != 0) { return cmp; } return second.compareTo(tp.second); } } public static void main(String agrs[]) throws IOException, InterruptedException, ClassNotFoundException { //job entrance Configuration conf = new Configuration(); GenericOptionsParser parser = new GenericOptionsParser(conf, agrs); String[] otherArgs = parser.getRemainingArgs(); if (agrs.length < 3) { System.err.println("Usage: J "); System.exit(2); } Job job = new Job(conf, "J"); job.setJarByClass(J.class);//Join class job.setMapperClass(JMapper.class);//Map class job.setMapOutputKeyClass(TextPair.class);//Map output key class job.setMapOutputValueClass(Text.class);//Map output value class job.setPartitionerClass(JPartitioner.class);//partition class job.setGroupingComparatorClass(JComparator.class);//condition group class after partition job.setReducerClass(Example_Join_01_Reduce.class);//reduce class job.setOutputKeyClass(Text.class);//reduce output key class job.setOutputValueClass(Text.class);//reduce ouput value class FileInputFormat.addInputPath(job, new Path(otherArgs[0]));//one of source files FileInputFormat.addInputPath(job, new Path(otherArgs[1]));//another file FileOutputFormat.setOutputPath(job, new Path(otherArgs[2]));//output path System.exit(job.waitForCompletion(true) ? 0 : 1);//run until job ends }

As can be seen above, to implement the associative computing, since programmers are unable to use the raw data directly, they have to compose some complex codes to handle the tag, bypass the original framework of MapReduce, and design and compute the associative relation between data from the bottom layer. Obviously, handling such computations by this way requires those programmers with strong programming skills. Plus, it is quite time-consuming, and there is no guarantee on computational efficiency. The above case is just the simplest kind of associated computation. As you can image, if using MapReduce for multi-table association or the associative computing with complex business logics, then the degree of complexity will rise in a geometric ratio. The difficulty and development efficiency becomes nearly unbearable.

         In fact, the associative computing itself is common and by no means complex. The reason of the apparent difficulty is that MapReduce is not specialized enough in a certain sector though it has strong universality. Similarly, developing via MapReduce is also quite inefficient when it comes to the ordered computations like year-on-year comparison and median operation, and the align or enum grouping.

         Although Hadoop has packaged Hive/Pig and other advanced solutions with MapReduce, these solutions on one hand is not powerful enough, on other hand, they only offer the rather simple and basic queries. To complete the business logics involving complex procedure, the hard coding is still unavoidable.


         Then, what can we do to boost the development efficiency for Hadoop? esProc is quite a good choice!

         esProc is a pure Java parallel computation framework with the focus on boosting the capability of Hadoop and the basic ability to improve the development efficiency of Hadoop programmers.

         Still the above example, esProc solution is shown below:

Main program:

Sub program:

As can be seen, esProc code is very intuitive and straightforward. In the Main program, the number of tasks are specified and the parameters for each task are prepared. Tasks are allocated to 4 computational nodes. The results of task computation are merged and written to the file. The Sub programmer is equivalent to the Map procedure of MapReduce: intercept a range of data to join, and return the result to Main.  Intuitive, straightforward, and concise - that's the impression that esProc syntax gives us.

         In fact, the associative computing is usually not the ultimate computational goal. The true business value is usually realized via summarizing in groups, sorting, and filtering after associative computing. For esProc, the whole procedure of all these computations can be written in a same graph of script, while MapReduce requires arranging another class for it. It is evident that MapReduce is far less efficient than esProc.

         esProc is a scripting language specialized for big data, offering the true set data type, easy for algorithm design from user perspective, and effortless to implement the complex business logics of clients. In addition, esProc supports the ordered set for arbitrary access to the member of set and perform the serial-number-related computation. The set of set can be used to represent the complex grouping style easily, for example, the equal grouping, align grouping, and enum grouping. Users can operate on the single record in as same way of operating on an object. esProc scripts is written and presented in a grid. By this way, the intermediate result can be referenced without definition. To add convenience, the complete code editing and debugging mechanism is provided. In short, esProc can be regarded as a dynamic set-lized language which has something in common with R language, and offers native support for distributed parallel computation from the core. esProc programmers are benefited from the efficient parallel computation of esProc while still having the simple syntax of R. esProc is designed for data computing and optimized for big data processing. By working with HDFS, esProc can improve the development and computation efficiency of Hadoop significantly.

         The framework of MapReduce restricts its use to the simple computation on the single data set, for example, searching. Due to this, it is not a good choice to use MapReduce for associative computing, ordered computation, and alike algorithms. esProc and other specialized computation scripts are essential to improving the development efficiency for such computations in Hadoop.

   add comment

Your answer

Join with account you already have



Ready to start your tutorial with us? That's great! Send us an email and we will get back to you as soon as possible!