Google Dataflow is designed to ensure data is encrypted both at rest and in transit. Here’s a brief overview of its encryption process:
Encryption at Rest: Data stored within Google Dataflow, whether temporary or persistent, is encrypted by default using either AES-256 or AES-128, depending on the hardware.
Encryption in Transit: Data transferred between Google services or over the Internet to Google Cloud is encrypted using HTTPS/TLS.
Key Management in Google Dataflow
Google Cloud provides three options for key management:
Google-managed Encryption Keys (GMEK): By default, Google manages the cryptographic keys on your behalf using its key management infrastructure.
Customer-supplied Encryption Keys (CSEK): Customers can provide their encryption keys, giving them control over the key’s creation, rotation, and destruction.
Customer-managed Encryption Keys (CMEK): Customers can generate and manage their encryption keys using Google Cloud’s Key Management Service, providing a balance between control and ease of use.
Example: Using Customer-managed Encryption Keys (CMEK) with Dataflow
For this illustration, we’ll guide you on how to set up and use CMEK with Google Dataflow:
1. Setting Up Key Management Service (KMS):
Navigate to the Google Cloud Console.
Open the side panel and go to “Security” > “Key Management.”
Click on “Create Key Ring” and provide a name and a location for the key ring.
Once the key ring is created, click on it, and then click on “Create Key.” Choose “Symmetric Encrypt/Decrypt” for the key type.
2. Granting Permissions to Dataflow:
Google Dataflow needs permissions to use the cryptographic key:
Within the created key details, click on the “IAM” tab.
Add “Cloud Dataflow Service Agent” role and grant it the “Cloud KMS CryptoKey Encrypter/Decrypter” permission.
3. Using CMEK with Dataflow Jobs:
When creating a Dataflow job, specify the Cloud KMS key:
gcloud dataflow jobs run freshers-in-rawfeeds \
--gcs-location gs://freshers-dataflow-templates/latest/feeds \
--region us-east \
--staging-location gs://freshers_data_in_gcs_bkt/staging \
--parameters inputTextFile=gs://freshers-dataflow/viewership/view-cnt-08-28-2023.txt,outputTable=freshers-in-2013:dataset.viewership \
--csek-key-name projects/freshers-in-2013/locations/us-east/keyRings/as_on_date/cryptoKeys/date
Data security is a cornerstone of any robust cloud data processing platform. Google Dataflow, with its advanced encryption capabilities, ensures that user data remains secure and confidential throughout its processing lifecycle.
