Concurrent Data Modification Conflicts

When writing data concurrently to an Iceberg table, two common types of conflicts can arise: concurrent data modification conflict and optimistic locking concurrent commit conflict. This section describes the concurrent data modification conflict.

Job failures due to data conflicts may occur under the following concurrent scenarios:

• Concurrent execution of DML operations on the same partition, such as performing Update, Delete, Insert Overwrite, or Insert operations on the same partition.
• Simultaneous execution of Compaction and DML operations.

The following describes the concurrent data conflicts that may be encountered by the Iceberg operation types currently supported by DataArts Fabric SQL.

• When the Insert Overwrite operation is performed, the following conflicts may occur:
  • Data file conflict: For partitioned tables, if during data conflict detection, new data files produced by other committed operations within the partitions targeted by the Insert Overwrite are identified, a data conflict is detected. For non-partitioned tables, if any new data files resulting from other committed operations are recognized during data conflict detection, a data conflict is flagged.
  • Delete file conflict: In the case of partitioned tables, if during data conflict detection, it is found that data files have been removed or delete files have been added within the partitions affected by the Insert Overwrite by other operations, this signifies a data conflict. Similarly, for non-partitioned tables, detecting the removal of data files or the addition of delete files by other operations during data conflict checks indicates a data conflict has occurred.
• When the Delete operation is performed, the following conflicts may occur:

  Data file conflict: During data conflict detection, if it is identified that the data file associated with the delete file has been deleted by another operation, a data conflict is detected.

The preceding concurrency conflict matrix indicates whether a conflict will occur upon committing the current operation if another operation initiates and completes its commit between the start and submission of the current operation.