Hadoop Pig Interview Questions and Answers

8/24/2013 No Comment

Apache Pig Interview Questions and Answers. 

If you are planning to pursue a career in Hadoop, then you can expect some PIG interview Questions. 

Explain what is PIG in Big Data?
Answer : PIG is nothing but a platform for analyzing large data sets that consist of high level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. PIG’s infrastructure layer consists of a compiler that produces sequence of MapReduce Programs. Pig was originally developed at Yahoo Research around 2006 for researchers to have an ad-hoc way of creating and executing map-reduce jobs on very large data sets. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets

Explain PIG's language layer an its properties?
Answer  : Pig’s language layer currently consists of a textual language called Pig Latin, which has the following key properties:
Ease of programming. Pig is intended to make complex tasks comprised of multiple interrelated data transformations that are explicitly encoded as data flow sequences easy to write, understand, and maintain.
Optimization opportunities. The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency.
Extensible. Users can create their own functions to do special-purpose processing.

What is bag in PIG?
Answer : A bag is a representation of one of the data models in Pig. A bag is nothing but a collection of tuples in an unordered form, this collection might hold possible duplicate tuples or records. Bags are used to store collections while grouping. The size of bag is the size of the local disk, this means that the size of the bag is limited. When the bag is full, then Pig will spill this bag into local disk and keep only some parts of the bag in memory. There is no necessity that the complete bag should fit into memory. We represent bags with “{}”

Explain what is the difference between logical and physical plans in PIG?
Pig undergoes some steps when a Pig Latin Script is converted into MapReduce jobs. After performing the basic parsing and semantic checking, it produces a logical plan. The logical plan describes the logical operators that have to be executed by Pig during execution. After this, Pig produces a physical plan. The physical plan describes the physical operators that are needed to execute the script.

What is the difference between PIG and SQL
Answer : The differences between Pig and SQL include Pig's usage of lazy evaluation, Pig's usage for ETL, Pig's ability to store data at any point during a pipeline, Pig's explicit declaration of execution plans, and Pig's support for pipeline splits.
  • Whereas, it has been argued DBMSs are substantially faster than the MapReduce system once the data is loaded, but that loading the data takes considerably longer in the database systems. It has also been argued RDBMSs offer out of the box support for column-storage, working with compressed data, indexes for efficient random data access, and transaction- level fault tolerance.
  • Pig Latin is procedural and fits very naturally in the pipeline paradigm while SQL is instead declarative. In SQL users can specify that data from two tables must be joined, but not what join implementation to use. Pig Latin allows users to specify an implementation or aspects of an implementation to be used in executing a script in several ways.
  • In effect, Pig Latin programming is similar to specifying a query execution plan, making it easier for programmers to explicitly control the flow of their data processing task.
  • SQL is oriented around queries that produce a single result. SQL handles trees naturally, but has no built in mechanism for splitting a data processing stream and applying different operators to each sub-stream. Pig Latin script describes a directed acyclic graph (DAG) rather than a pipeline.
  • Pig Latin's ability to include user code at any point in the pipeline is useful for pipeline development. If SQL is used, data must first be imported into the database, and then the cleansing and transformation process can begin.
What do you mean by co-group and explain its function in Pig? 
Answer : The COGROUP command in Pig is a combination of some sort of both a GROUP and a JOIN.
COGROUP is very similar to GROUP in its function. Co-group does the joins of a data set by grouping one particular data set only. It does the grouping of the elements by their common field or key and after that it returns a set of records containing two separate bags. Now, the first bag consists of the records of the first data set with the common data set and the second bag, the second data set with the set of common data. Let us illustrate it by an example cited below.

Let us consider we have a dataset of employees and their offices:
$ cat >
employees.csv
Steve,Microsoft
Joe,Google
John,Cisco
George,Microsoft
Matt,Google
We will COGROUP the companies using Pig as cited below:
employees= LOAD 'employees.csv'
USING PigStorage(',')
AS (employee:chararray,company:chararray);
grouped = COGROUP employees BY company;
DUMP grouped;
This returns the list of companies for the employees. For each company, Pig groups the matching rows into bags. The resulting table grouped is:

groupemployees
Google{(Joe,Google),(Matt,Google)}
Microsoft{(Steve,Microsoft),(George,Microsoft)}
Cisco{(John,Cisco)}

What is the main use of a BloomMapFile in PIG?
Answer : The BloomMapFile is just a variant or the extension of a MapFile. As such, its functionality is very much similar to a MapFile. It has a fast version of the get() method, especially for sparsely populated files. BloomMapFile uses dynamic Bloom filters to provide quick membership test for the keys, because it uses in-memory. It is used in Hbase table format in PIG.

Can you please give us some good examples how Hadoop can be used in a real time environment?
Answer : Let us consider a scenario that the we have an examination which consists of 10 Multiple-choice questions (MCQs) and lets say 30 students appear for the same. Now, let us also assume that each student will attempt each question. Hence, we will have that for each question and each answer option, one particular key will be generated. As a result, we have a set of key-value pairs for all the questions and  also for all the answer options for every student. Based on the options that the students have selected, you have to analyze and find out how many students have answered correctly. This isn’t an easy task. Here Hadoop comes into picture! Hadoop helps you in solving these problems quickly and without much effort. You may also take the case of how many students have wrongly attempted a particular question.  

Does ‘ILLUSTRATE’ run MR job?
No, illustrate will not pull any MR, it will pull the internal data. On the console, illustrate will not do any job. It just shows output of each stage and not the final output.

Is the keyword ‘DEFINE’ like a function name?
Yes, the keyword ‘DEFINE’ is like a function name. Once you have registered, you have to define it. Whatever logic you have written in Java program, you have an exported jar and also a jar registered by you. Now the compiler will check the function in exported jar. When the function is not present in the library, it looks into your jar.

Is the keyword ‘FUNCTIONAL’ a User Defined Function (UDF)?
No, the keyword ‘FUNCTIONAL’ is not a User Defined Function (UDF). While using UDF, we have to override some functions. Certainly you have to do your job with the help of these functions only. But the keyword ‘FUNCTIONAL’ is a built-in function i.e a pre-defined function, therefore it does not work as a UDF.  

Explain why do we need MapReduce during Pig programming?
Pig is a high-level platform that makes many Hadoop data analysis issues easier to execute. The language we use for this platform is: Pig Latin. A program written in Pig Latin is like a query written in SQL, where we need an execution engine to execute the query. So, when a program is written in Pig Latin, Pig compiler will convert the program into MapReduce jobs. As such, MapReduce acts as the execution engine.  

What does FOREACH keyword in PIG do? 
Answer : In PigFOREACH is used to apply transformations to the data and to generate new data items based on the column data. For each element of a data bag, the respective action will be performed by FOREACH.

Cited below is the syntax of FOREACH operator.

Syntax : FOREACH bagname GENERATE expression1, expression2, …..

The meaning of the above statement is that the expressions mentioned after GENERATE will be applied to the current record of the data bag.

What warning does Pig give in case there is a type mismatch or missing field? 
Answer :  Pig does not show any warning when there is no matching field or a mismatch. Pig may sometimes record a warning bit it is difficult to find that in the log file. In case any mismatch is found, it assumes a null value in PIG.

Is COGROUP GROUP of more than 1 data set? 
Answer : COGROUP is a GROUP  of one data set. The GROUP and COGROUP operators are identical with both working with one or more relations. But there are cases which involves more than one data sets, COGROUP will group all the data sets and join them based on the common field. As such, we can say that COGROUP is a GROUP of more than one data set or relations and combination of that relations.

Related Posts


No comments :

 

Aired | The content is copyrighted and may not be reproduced on other websites. | Copyright © 2009-2016 | All Rights Reserved 2016

Contact Us | About Us | Privacy Policy and Disclaimer