BigQuery : How to process BigQuery Data with PySpark on Dataproc ?

To process BigQuery data with PySpark on Dataproc, you will need to follow these steps:

1. Create a Google Cloud Platform (GCP) project and enable the required APIs.
2. Create a Dataproc cluster.
3. Load data into BigQuery: You can load data into BigQuery from a variety of sources including cloud storage, local disk, and streaming data.
4. Authorize Dataproc to access BigQuery: To authorize Dataproc to access BigQuery, you need to create a service account and grant it the BigQuery Data Viewer role.
5. Connect PySpark to BigQuery: You can connect PySpark to BigQuery using the ‘ pyspark.sql.SparkSession ‘ and ‘ pyspark.sql.DataFrameReader ‘ classes.
6. Read data from BigQuery into PySpark: You can read data from BigQuery into PySpark using the ‘ spark.read.format(“bigquery”).load() ‘ method.
7. Process data with PySpark: You can now use the PySpark API to process the data as you would with any other PySpark DataFrame.
8. Write data back to BigQuery: You can write the processed data back to BigQuery using the ‘ write.format(“bigquery”).mode(“append”).save() ‘  method.

Here is an example of how to connect PySpark to BigQuery, read data from BigQuery into PySpark, and write the processed data back to BigQuery:

from pyspark.sql import SparkSession

# create a Spark session
spark = SparkSession.builder.appName("BigQueryExample").getOrCreate()

# read data from BigQuery into a PySpark DataFrame
df = spark.read.format("bigquery").option("project", "freshers-project").load("freshers-dataset.freshers-table")

# process data with PySpark
result = df.groupBy("column1").agg({"column2": "sum"})

# write data back to BigQuery
result.write.format("bigquery").option("table", "freshers-dataset.freshers-table").mode("append").save()

# stop the Spark session
spark.stop()