Installing Spark on a Linux machine can be done in a few steps. The following…
Tag: Spark_Interview
Learn how to connect Hive with Apache Spark.
HiveContext is a Spark SQL module that allows you to work with Hive data in Spark. It provides a way…
PySpark : Connecting and updating postgres table in spark SQL
Apache Spark is an open-source, distributed computing system that can process large amounts of data quickly. Spark SQL is a…
Kafka streaming with PySpark – Things you need to know – With Example
To use Kafka streaming with PySpark, you will need to have a good understanding of the following concepts: Kafka: Kafka…
How do you break a lineage in Apache Spark ? Why we need to break a lineage in Apache Spark ?
In Apache Spark, a lineage refers to the series of RDD (Resilient Distributed Dataset) operations that are performed on a…
When you should not use Apache Spark ? Explain with reason.
There are a few situations where it may not be appropriate to use Apache Spark, which is a powerful open-source…
PySpark : How to create a map from a column of structs : map_from_entries
pyspark.sql.functions.map_from_entries map_from_entries(col) is a function in PySpark that creates a map from a column of structs, where the structs have…
PySpark : Converting Unix timestamp to a string representing the timestamp in a specific format
pyspark.sql.functions.from_unixtime The “from_unixtime()” function is a PySpark function that allows you to convert a Unix timestamp (a long integer representing…
PySpark : Check if two or more arrays in a DataFrame column have any common elements
pyspark.sql.functions.arrays_overlap The arrays_overlap function is a PySpark function that allows you to check if two or more arrays in a…
PySpark : Combine the elements of two or more arrays in a DataFrame column
pyspark.sql.functions.array_union The array_union function is a PySpark function that allows you to combine the elements of two or more arrays…
PySpark : Sort an array of elements in a DataFrame column
pyspark.sql.functions.array_sort The array_sort function is a PySpark function that allows you to sort an array of elements in a DataFrame…