How to find array contains a given value or values using PySpark ( PySpark search in array)
array_contains You can find specific value/values in an array using spark sql function array_contains. array_contains(array, value) will return true if…
How to removes duplicate values from array in PySpark
This blog will show you , how to remove the duplicates in an column with array elements. Consider the below example….
How to extract Date, Time into two separate cell from Timestamp in Excel ?
Consider you have a date timestamp value in cell A1 A1 = 01/21/2022 11:35 PM You want to extract date…
What are the Python libraries provided by AWS Glue Version 2.0
The defaults Python libraries available in AWS Glue version 2.0 are as below boto3==1.12.4 botocore==1.15.4 certifi==2019.11.28 chardet==3.0.4 cycler==0.10.0 Cython==0.29.15 docutils==0.15.2…
How to add additional Python Libraries in a AWS Glue Development Endpoint
There are multiple scenario that you may need to use different set of python libraries in your python code or…
AWS Glue : Example on how to read a sample csv file with PySpark
Here assume that you have your CSV data in AWS S3 bucket. The next step is the crawl the data…
How to renaming Spark Dataframe having a complex schema with AWS Glue – PySpark
There can be multiple reason to rename the Spark Data frame . Even though withColumnRenamed can be used to rename…
How can you track the change metadata of a Snowflake table?
The CHANGES clause enables querying the change tracking metadata for a table within a specified interval of time without having…
How to do Pivot in Snowflake ?
PIVOT in Snowflake PIVOT rotates a table by turning the unique values from one column in the input expression into…
How can you UNPIVOT in Snowflake ?
UNPIVOT in Snowflake UNPIVOT will rotate a table by transforming columns into rows. UNPIVOT is a relational operator which accepts…