Pandas API on Spark’s Clipboard Integration : read_clipboard

In the landscape of big data processing, the Pandas API on Spark provides a powerful bridge between Pandas simplicity and Spark scalability. One useful functionality is read_clipboard, which allows users to read text from the clipboard and pass it directly to read_csv. In this article, we’ll delve into how to leverage this feature for seamless data input operations.

Understanding read_clipboard

The read_clipboard function in the Pandas API on Spark simplifies the process of reading data from the clipboard into Spark DataFrames. This functionality is particularly useful when dealing with small to medium-sized datasets copied from various sources. Let’s explore its usage with examples.

Example Usage

Suppose you have data copied to your clipboard from a tabular source, such as a spreadsheet or a website. We can easily read this data into a Spark DataFrame using read_clipboard.

from pyspark.sql import SparkSession
import pandas as pd
# Initialize SparkSession
spark = SparkSession.builder \
    .appName("Clipboard Data to Spark : Learning @ Freshers.in ") \
    .getOrCreate()
# Read data from clipboard into Pandas DataFrame
df_pandas = pd.read_clipboard(sep='\t')
# Convert Pandas DataFrame to Spark DataFrame
df_spark = spark.createDataFrame(df_pandas)
# Show the contents of the Spark DataFrame
df_spark.show()
# Stop SparkSession
spark.stop()

+-------+---+------+
|   Name|Age|Gender|
+-------+---+------+
|  Alice| 30|Female|
|    Bob| 35|  Male|
|Charlie| 40|  Male|
|  David| 45|  Male|
+-------+---+------+