pyspark.sql.DataFrame.zipWithIndex#

DataFrame.zipWithIndex(indexColName='index')[source]#

Returns a new DataFrame by appending a column containing consecutive 0-based Long indices, similar to RDD.zipWithIndex().

The index column is appended as the last column of the resulting DataFrame.

Added in version 4.2.0.

Parameters:

indexColNamestr, default “index”: The name of the index column to append.

Returns:

DataFrame: A new DataFrame with an appended index column.

Notes

If a column with indexColName already exists in the schema, the resulting DataFrame will have duplicate column names. Selecting the duplicate column by name will throw AMBIGUOUS_REFERENCE, and writing the DataFrame will throw COLUMN_ALREADY_EXISTS.

Examples

>>> df = spark.createDataFrame(
...     [("a", 1), ("b", 2), ("c", 3)], ["letter", "number"])
>>> df.zipWithIndex().show()
+------+------+-----+
|letter|number|index|
+------+------+-----+
|     a|     1|    0|
|     b|     2|    1|
|     c|     3|    2|
+------+------+-----+

Custom index column name:

>>> df.zipWithIndex("row_id").show()
+------+------+------+
|letter|number|row_id|
+------+------+------+
|     a|     1|     0|
|     b|     2|     1|
|     c|     3|     2|
+------+------+------+