Typical Delta Configurations
To set Delta parameters while submitting a DLI Spark SQL job, access the SQL Editor page and click Settings in the upper right corner. In the Parameter Settings area, set the parameters.
Parameter |
Description |
Default Value |
---|---|---|
spark.databricks.delta.retentionDurationCheck.enabled |
Whether the retention period is checked when vacuum clears files that are no longer referenced. |
true |
spark.databricks.delta.properties.defaults.deletedFileRetentionDuration or delta.deletedFileRetentionDuration |
Retention period of files that are no longer referenced by Delta. If spark.databricks.delta.retentionDurationCheck.enabled is set to true, an exception will be thrown when files that have not yet reahced their retention period are cleared. |
168 hours (1 week) |
spark.databricks.delta.properties.defaults.logRetentionDuration or delta.logRetentionDuration |
Expiration time of Delta log files. When a Delta log executes a checkpoint action, it scans for expired files that need deletion. If such files are found, they are removed to prevent the Delta log files from growing indefinitely. |
30 days |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot