Recent Posts
How to transform columns into list of objects [arrays] on top of group by in PySpark – collect_list and collect_set
In this article we will see how to returns a set of objects in an array with or without duplicate…
How to create a table from CSV file and write SQL on top of it in Spark (Sample code)
In this article you will see how you can read a CSV file using pySpark , how to control header…
What is PPA in Ubuntu
PPA (Personal Package Archives) PPAs are software repositories that are created specifically for Ubuntu users and are simpler to install…
In AWS EC2 how to know your ubuntu version using the command line commands
lsb_release lsb stands for Linux Standard Base. Certain LSB (Linux Standard Base) and Distribution information is printed by lsb_release command….
Convert data from the PySpark DataFrame columns to Row format or get elements in columns in row
pyspark.sql.functions.collect_list(col) This is an aggregate function and returns a list of objects with duplicates. To retrieve the data from the PySpark…
How to give empty string as accepted_values in DBT
Jinja filter : as_text If we are giving ‘ ‘ [ two single quotes ] , this will consider as…
How to find if a substring exists in a string or not in Shell/Unix scripting
This will be a common use case when we need to check some substring in a log or some file. …
How to generate your project’s documentation in DBT
generate and serve generate DBT has two supported commands for generating documentation for the models. dbt docs generate is responsible…
How to delete files, directory and subdirectory which is older than N days in shell, Unix or Linux
For mostly, logs and other temp files, its a common use case to delete the files older than 30 or…
How to find the difference between two dates in DBT
datediff The datediff macro can be used to calculate the difference between two dates. {{ dbt.datediff("column_a", "column_b", "day") }} {{ dbt.datediff("column",…