The defaults Python libraries available in AWS Glue version 2.0 are as below boto3==1.12.4 botocore==1.15.4…
Author: user
In AWS EC2 how to know your ubuntu version using the command line commands
lsb_release lsb stands for Linux Standard Base. Certain LSB (Linux Standard Base) and Distribution information is printed by lsb_release command….
Convert data from the PySpark DataFrame columns to Row format or get elements in columns in row
pyspark.sql.functions.collect_list(col) This is an aggregate function and returns a list of objects with duplicates. To retrieve the data from the PySpark…
How to give empty string as accepted_values in DBT
Jinja filter : as_text If we are giving ‘ ‘ [ two single quotes ] , this will consider as…
How to find if a substring exists in a string or not in Shell/Unix scripting
This will be a common use case when we need to check some substring in a log or some file. …
How to generate your project’s documentation in DBT
generate and serve generate DBT has two supported commands for generating documentation for the models. dbt docs generate is responsible…
How to delete files, directory and subdirectory which is older than N days in shell, Unix or Linux
For mostly, logs and other temp files, its a common use case to delete the files older than 30 or…
How to find the difference between two dates in DBT
datediff The datediff macro can be used to calculate the difference between two dates. {{ dbt.datediff("column_a", "column_b", "day") }} {{ dbt.datediff("column",…
How to add instance variable available to all projects and groups in a GitLab instance and to use in .gitlab-ci.yml for pipeline execution
During building of git pipeline you may need to add some values which you dont want to expose in the…
Google to increase storage from 15GB to 1TB(1000GB)for Gmail and Drive storage
For those who are on the Workspace individual plan, Google has some excellent news. Users of the service, including independent…
Python 3.11.0 is now available
Major new features of the Python 3.11 series, are Include Fine-Grained Error Locations in Tracebacks. Exception Groups and except*. tomllib:…