Convert Pyspark Dataframe Column From List To String

I have this PySpark dataframe +-----------+--------------------+ |uuid | test_123 | +-----------+--------------------+ | 1 |[test, test2, test3]| |

Solution 1:

While you can use a UserDefinedFunction it is very inefficient. Instead it is better to use concat_ws function:

from pyspark.sql.functionsimport concat_ws

df.withColumn("test_123", concat_ws(",", "test_123")).show()

+----+----------------+
|uuid|        test_123|
+----+----------------+
|   1|test,test2,test3|
|   2|test4,test,test6|
|   3|test6,test9,t55o|
+----+----------------+

Solution 2:

You can create a udf that joins array/list and then apply it to the test column:

from pyspark.sql.functions import udf, col

join_udf = udf(lambda x: ",".join(x))
df.withColumn("test_123", join_udf(col("test_123"))).show()

+----+----------------+|uuid|        test_123|+----+----------------+|1|test,test2,test3||2|test4,test,test6||3|test6,test9,t55o|+----+----------------+

The initial data frame is created from:

from pyspark.sql.types import StructType, StructField
schema = StructType([StructField("uuid",IntegerType(),True),StructField("test_123",ArrayType(StringType(),True),True)])
rdd = sc.parallelize([[1, ["test","test2","test3"]], [2, ["test4","test","test6"]],[3,["test6","test9","t55o"]]])
df = spark.createDataFrame(rdd, schema)

df.show()
+----+--------------------+|uuid|            test_123|+----+--------------------+|1|[test, test2, test3]||2|[test4, test, test6]||3|[test6, test9, t55o]|+----+--------------------+

Solution 3:

As of version 2.4.0, you can use array_join.Spark docs

from pyspark.sql.functions import array_join

df.withColumn("test_123", array_join("test_123", ",")).show()