pyspark.sql.functions.ceil
In PySpark, the ceil() function is used to round a number up to the nearest integer. This function is a part of the pyspark.sql.functions module, and it can be used on both column and numeric expressions.
Here is an example of using the ceil() function in PySpark:
from pyspark.sql import SparkSession
from pyspark.sql.functions import ceil
# Create a SparkSession
spark = SparkSession.builder.appName("Ceil Example").getOrCreate()
# Create a DataFrame with some sample data
data = [(1.2,), (2.5,), (3.7,), (4.9,)]
df = spark.createDataFrame(data, ["num"])
# Use the ceil() function to round the numbers up
df = df.select(ceil(df["num"]).alias("rounded_num"))
# Show the result
df.show()
This code creates a SparkSession and a DataFrame with a single column “num” containing some sample decimal numbers. Then it uses the ceil() function to round these numbers up to the nearest integer and create a new column “rounded_num” with the result. The DataFrame is then displayed and show the rounded number.
The output of this code will be:
+-----------+
|rounded_num|
+-----------+
| 2|
| 3|
| 4|
| 5|
+-----------+The Ceil function rounds up the decimal number to nearest integer.
Spark important urls to refer
