How can I see the full column values in a Spark Dataframe ?

PySpark @ Freshers.in

When we do a dataframe.show () , we can see that some of the column values got truncated. Here we sill show the command to get the full value of the column 

state_data_df.show()
+-----+------+------+----------+--------------------+
|si_no|  name|salary|commission|             address|
+-----+------+------+----------+--------------------+
|    1|   Sam| 10000|      12.0|125960 W 112th Pl...|
|    2| Peter| 24000|       6.0|38500 Beverly 661...|
|    3|  John|  9000|       8.5|33215 Overland Av...|
|    4|Jaison| 12000|      11.0|321325 Overland A...|
|    5|  Mike| 15000|      22.0|523960 W 85th PL ...|
+-----+------+------+----------+--------------------+

The above we can see that the address field is truncated. If we want to see the complete values in the address we can use show command with parameter.

Syntax : show(numRows: Int, truncate: Boolean)

numRows: Number of rows that you need to see ( 20 is the default )

truncate : Its boolean, which we can say to truncate or not 

state_data_df.show(5,False)
+-----+------+------+----------+-----------------------------------------------------------+
|si_no|name  |salary|commission|address                                                    |
+-----+------+------+----------+-----------------------------------------------------------+
|1    |Sam   |10000 |12.0      |125960 W 112th PlLos Angeles, California(CA), 90045        |
|2    |Peter |24000 |6.0       |38500 Beverly 66192 Blvd Los Angeles, California 90048     |
|3    |John  |9000  |8.5       |33215 Overland Ave Cross Los Angeles, California(CA), 90034|
|4    |Jaison|12000 |11.0      |321325 Overland Ave New Los Angeles, California(CA), 90034 |
|5    |Mike  |15000 |22.0      |523960 W 85th PL Base Los Angeles, California(CA), 90045   |
+-----+------+------+----------+-----------------------------------------------------------+

 

Author: user

Leave a Reply