Tag: SparkExamples

PySpark @ Freshers.in

Convert data from the PySpark DataFrame columns to Row format or get elements in columns in row

pyspark.sql.functions.collect_list(col) This is an aggregate function and returns a list of objects with duplicates.┬áTo retrieve the data from the PySpark…

PySpark @ Freshers.in

PySpark: How to add months to a date column in Spark DataFrame (add_months)

I have a use case where I want to add months to a date column in spark DataFrame Function :…

PySpark @ Freshers.in

PySpark-How to returns the first column that is not null

pyspark.sql.functions.coalesce If you want to return the first non zero from list of column you can use coalesce function in…

PySpark @ Freshers.in

How can you convert PySpark Dataframe to JSON ?

pyspark.sql.DataFrame.toJSON There may be some situation that you need to send your dataframe to a file to a server or…

PySpark @ Freshers.in

How can I see the full column values in a Spark Dataframe ?

When we do a dataframe.show () , we can see that some of the column values got truncated. Here we…

PySpark @ Freshers.in

Converts a column containing a StructType, ArrayType or a MapType into a JSON string-PySpark(to_json)

You can convert a column containing a StructType, ArrayType or a MapType into a JSON string using to_json function. pyspark.sql.functions.to_json…

PySpark @ Freshers.in

How to replace a value with another value in a column in Pyspark Dataframe ?

In PySpark we can replace a value in one column or multiple column or multiple values in a column to…

PySpark @ Freshers.in

How to drop nulls in a dataframe : PySpark

For most of the data cleansing the first thing that you may need to do drop the nulls in the…