Tag: Hadoop

PySpark @ Freshers.in

What is the difference between repartition() and coalesce() ?

The repartition algorithm will perform a full shuffle and creates new partitions with data that’s distributed evenly.┬áThe repartition algorithm makes…

PySpark @ Freshers.in

How to renaming Spark Dataframe having a complex schema with AWS Glue – PySpark

There can be multiple reason to rename the Spark Data frame . Even though withColumnRenamed can be used to rename…

What is the problem in having lots of small files in HDFS? What is the remediation plan?

In Hadoop ecosystem we are storing files under folders in HDFS, most of the time the folder name we are…

Explain distributed cache in Hadoop ?

Distributed cache is a facility provided by Hadoop map reduce framework to access small file needed by application during its…

What is Swappiness Value? What is the role of Swappiness Value during the cluster set up?

vm.swappiness is one of the Kernel Parameter in Linux or UNIX, vm.swappiness value is from 0-100 which controls the swapping…