In addition to its built-in functions, Hive also supports User-Defined Functions (UDFs), which enable users to extend Hive’s functionality by implementing custom functions. In this article, we will discuss the different types of UDFs supported by Hive and provide an overview of each type.
Types of UDFs in Hive
There are three main types of UDFs supported by Hive:
- Regular UDFs
- User-Defined Aggregating Functions (UDAFs)
- User-Defined Table Generating Functions (UDTFs)
-
Regular UDFs
Regular UDFs are the most common type of UDFs in Hive. They are similar to built-in functions, such as substring or concat, but are implemented by users to perform custom operations on input data. Regular UDFs take one or more input values and return a single output value. They can be used in SELECT, WHERE, and HAVING clauses of a HiveQL query.
To create a regular UDF, you need to extend the org.apache.hadoop.hive.ql.exec.UDF class and implement the evaluate() method. You can then add the JAR file containing your UDF to the Hive classpath and register the function using the CREATE TEMPORARY FUNCTION statement.
Example: A regular UDF to convert a temperature value from Celsius to Fahrenheit:
package com.example.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.DoubleWritable;
public class CelsiusToFahrenheit extends UDF {
public DoubleWritable evaluate(DoubleWritable celsius) {
if (celsius == null) {
return null;
}
double fahrenheit = (celsius.get() * 9.0 / 5.0) + 32;
return new DoubleWritable(fahrenheit);
}
}
-
User-Defined Aggregating Functions (UDAFs)
UDAFs are used to perform custom aggregation operations, such as computing the median or mode of a set of values. They are similar to built-in aggregate functions like SUM, COUNT, and AVG, but are implemented by users to perform custom aggregation logic. UDAFs can be used in GROUP BY clauses of a HiveQL query.
To create a UDAF, you need to extend the org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2 class and implement a set of required methods, including getEvaluator(), init(), iterate(), merge(), and terminate(). Like regular UDFs, you can add the JAR file containing your UDAF to the Hive classpath and register the function using the CREATE TEMPORARY FUNCTION statement.
Example: A UDAF to compute the median of a set of values:
// Implement the GenericUDAFResolver2, GenericUDAFEvaluator, and other required classes and methods
-
User-Defined Table Generating Functions (UDTFs)
UDTFs are used to generate multiple output rows from a single input row. They are similar to the built-in explode() function, which can generate multiple rows from a single array or map value. UDTFs can be used in the SELECT clause of a HiveQL query with the LATERAL VIEW syntax.
To create a UDTF, you need to extend the org.apache.hadoop.hive.ql.udf.generic.GenericUDTF class and implement the initialize(), process(), and close() methods. As with regular UDFs and UDAFs, you can add the JAR file containing your UDTF to the Hive classpath and register the function using the CREATE TEMPORARY FUNCTION statement.
Example: A UDTF to split a comma-separated list of values into multiple rows:
package com.example.hive.udtf;
import java.util.ArrayList;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector;
import org.apache.hadoop.io.Text;
public class SplitCommaSeparated extends GenericUDTF {
private StringObjectInspector stringOI;
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
if (arguments.length != 1 || !(arguments[0] instanceof StringObjectInspector)) {
throw new UDFArgumentException("SplitCommaSeparated takes exactly one string argument");
}
stringOI = (StringObjectInspector) arguments[0];
return ObjectInspectorFactory.getStandardListObjectInspector(
PrimitiveObjectInspectorFactory.writableStringObjectInspector);
}
@Override
public void process(Object[] record) throws HiveException {
if (record.length != 1 || record[0] == null) {
return;
}
String input = stringOI.getPrimitiveJavaObject(record[0]);
String[] values = input.split(",");
ArrayList<Text> result = new ArrayList<>();
for (String value : values) {
result.add(new Text(value));
}
forward(result);
}
@Override
public void close() throws HiveException {
// No cleanup required for this UDTF
}
}
User-Defined Functions (UDFs) in Hive provide a way to extend the functionality of Hive by implementing custom functions. There are three main types of UDFs supported in Hive: Regular UDFs, User-Defined Aggregating Functions (UDAFs), and User-Defined Table Generating Functions (UDTFs). By understanding the different types of UDFs and their use cases, you can create custom functions tailored to your specific data processing needs and improve the efficiency and effectiveness of your Hive queries.
Hive important pages to refer
