concat vs concat_ws
Syntax: pyspark.sql.functions.concat(*cols) pyspark.sql.functions.concat_ws(sep, *cols)
concat : concat concatenates multiple input columns together into a single column. The function works with strings, binary and compatible array columns.
concat_ws : concat_ws concatenates multiple input string columns together into a single string column, using the given separator.
concat returns None when any of the column/columns is null. All keys in the column is lost.
concat_ws returns non empty column’s after concatenating with the separator.
In short in general scenario concat_ws is recommended rather than concat.
Reference