pyspark.sql.functions.array_max The array_max function is a built-in function in Pyspark that finds the maximum value…
Tag: PySpark
Retrieving value of a specific element in an array or map column of a DataFrame.
pyspark.sql.functions.element_at In PySpark, the element_at function is used to retrieve the value of a specific element in an array or…
In PySpark how to sort data in descending order, while putting the rows with null values at the end of the result ?
pyspark.sql.Column.desc_nulls_last In PySpark, the desc_nulls_last function is used to sort data in descending order, while putting the rows with null…
In PySpark how sort data in descending order, while putting the rows with null values at the beginning ?
pyspark.sql.Column.desc_nulls_first In PySpark, the desc_nulls_first function is used to sort data in descending order, while putting the rows with null…
Comparing PySpark with Map Reduce programming
PySpark is the Python library for Spark programming. It allows developers to interface with RDDs (Resilient Distributed Datasets) and perform…
Explain dense_rank. How to use dense_rank function in PySpark ?
In PySpark, the dense_rank function is used to assign a rank to each row within a result set, based on…
Pyspark code to read and write data from and to google Bigquery.
Here is some sample PySpark code that demonstrates how to read and write data from and to Google BigQuery: from…
How to start a serverless spark from GCP
To start a serverless Spark job on Google Cloud Platform (GCP), you can use the Cloud Dataproc service. Cloud Dataproc…
In pyspark what is the difference between Spark spark.table() and spark.read.table()
In PySpark, spark.table() is used to read a table from the Spark catalog, whereas spark.read.table() is used to read a…
PySpark : How to read date datatype from CSV ?
We specify schema = true when a CSV file is being read. Spark determines the data type of a column…
PySpark: How to accept date in a Dataframe : DateType can not accept object ‘YYYY-MM-DD’ in type
Accepting date in a Dataframe When you define a data in a a list of tuple and trying to read…