PySpark’s degrees function plays a vital role in data transformation, especially in converting radians to degrees. This article provides a comprehensive overview of the degrees function, enriched with a practical example to enhance your understanding.
Understanding PySpark’s Degrees Function
What is the Degrees Function in PySpark?
PySpark’s degrees function is a part of the pyspark.sql.functions module. It is used to convert values in radians to degrees. This function is crucial in various fields, including physics, engineering, and geospatial analysis, where angle measurements are often needed in degrees.
Why Use the Degrees Function?
Converting radians to degrees can be necessary when dealing with trigonometric functions, geospatial coordinates, or when the data is more understandable or conventional in degrees. The degrees function in PySpark makes this conversion straightforward and efficient, especially when working with large datasets.
Practical Example with Real Data
Scenario
To illustrate the degrees function, let’s consider a dataset containing names and angle measurements in radians. We will use the following names: Sachin, Manju, Ram, Raju, David, Freshers_in, and Wilson, each associated with a random radian value.
Step-by-Step Implementation
Creating a DataFrame: We’ll create a DataFrame with names and their corresponding radian values.
from pyspark.sql import SparkSession
from pyspark.sql.functions import degrees
spark = SparkSession.builder.appName("Learning @ Degrees Example").getOrCreate()
data = [("Sachin", 1.57), ("Manju", 3.14), ("Ram", 4.71),
("Raju", 2.36), ("David", 5.89), ("Freshers_in", 0.78), ("Wilson", 6.28)]
columns = ["Name", "Radians"]
df = spark.createDataFrame(data, columns)Applying the Degrees Function: We use the degrees function to convert the radian values to degrees.
df_with_degrees = df.withColumn("Degrees", degrees(df["Radians"]))
df_with_degrees.show()
Output
+-----------+-------+------------------+
| Name|Radians| Degrees|
+-----------+-------+------------------+
| Sachin| 1.57| 89.95437383553924|
| Manju| 3.14| 179.9087476710785|
| Ram| 4.71|269.86312150661774|
| Raju| 2.36| 135.2180396508743|
| David| 5.89| 337.4721413320549|
|Freshers_in| 0.78| 44.69070802020421|
| Wilson| 6.28| 359.817495342157|
+-----------+-------+------------------+
Spark important urls to refer
