pyspark.sql.functions.monotonically_increasing_id
A column that produces 64-bit integers with a monotonic increase. The created ID is assured to be both singular and monotonically rising, but not sequential. The partition ID is currently stored in the upper 31 bits. While the record number inside each partition is currently stored in the lower 33 bits. The data frame is assumed to have fewer than 1 billion partitions, each with fewer than 8 billion records.
In PySpark, you can generate a sequence number using the monotonically_increasing_id() function. This generates a unique and increasing 64-bit integer ID for each row in a DataFrame. The ID is unique within the DataFrame, but may not be unique across different DataFrames or sessions.
Here is an example of how to use monotonically_increasing_id(), to add a sequence number column to a DataFrame:
Sample Code
from pyspark.sql.functions import monotonically_increasing_id
# Create a DataFrame
data = [("Peter Sam", 11), ("Twinkle John", 23), ("Marrie Bob", 33),("Sharone Rode", 43)]
df = spark.createDataFrame(data, ["full_name", "age"])
# Add a sequence number column
df = df.withColumn("seq_num", monotonically_increasing_id())
# Show the DataFrame
df.show()
Result
+------------+---+-----------+
| full_name|age| seq_num|
+------------+---+-----------+
| Peter Sam| 11| 0|
|Twinkle John| 23| 8589934592|
| Marrie Bob| 33|17179869184|
|Sharone Rode| 43|25769803776|
+------------+---+-----------+Spark important urls to refer
