Query-Based Deletion

Tableflow supports removing rows from a table that match a filter starting from agent version v833+. Common use cases include GDPR deletion requests and removing ingested credentials or other sensitive data.

Deletions apply to the Tableflow table only. Records from the source Kafka topic will not be deleted. To remove an entire table, see Table Deletion. To remove data based on its age, see Data Retention and TTL.

Danger

Deleting data is irreversible. Before creating a deletion request, review the query filter and optional Kafka timestamp range for selecting rows to delete and use the preview feature to see the sample rows to be deleted.

Write a Filter

Deletion requests use WarpStream’s query filter syntax. For example, the following filter matches rows for one user:

user_id == "abc"

You can combine conditions with and, or, and parentheses:

tenant_id == 42 and (status == "inactive" or deleted_at == NULL)

Nested fields use dot-separated paths:

payload.user_id == 123

Deletion filters support:

• ==, !=, <, <=, >, and >=
• and, or, and parentheses
• String, number, boolean, and NULL values
• Nested field paths

Deletion filters do not support LIKE, MATCH, IN, BETWEEN, arithmetic, array/map access, or type casts. A filter can contain at most 5,000 characters.

Limit by Kafka Timestamp

You can set optional Min Kafka Timestamp and Max Kafka Timestamp values to bound the timeframe for which the deletion will be applied to. These bounds are inclusive and apply to the Kafka record timestamp, not to a timestamp field in the record itself.

The minimum timestamp cannot be later than the maximum timestamp.

Resolve an Ambiguous Field

A field name is ambiguous if a column was removed and another column with the same name was later added to the table. Tableflow rejects the deletion because the columns can have different types or meanings.

The validation error lists the iceberg field IDs and types that match the name. Use those IDs to specify which columns the predicates reference.

For example say you wanted to filter on user_id:

user_id == 123

and it errors out saying that two field IDs were found for this column name, one with ID 10 and type string and the other ID 6 and type long. If your intent is to apply the deletion on both versions of the column you should rewrite your query as so:

field_id#10 == "123" or field_id#6 == 123

If you only want to delete the data matching it as a long (the one with field ID 6) you should rewrite your query as so:

field_id#6 == 123

The right-hand operand must match the data type of the target column.

Preview a Deletion

A preview applies the filter and optional Kafka timestamp range from the deletion query without deleting data. It returns a sample of up to 10 matching rows.

To preview a deletion:

1. In the WarpStream Console, open your Tableflow cluster.
2. Select the table, then open the Deletions tab.
3. Click New Deletion Request.
4. Enter the filter and optional Kafka timestamp bounds.
5. Optionally, enter a comma-separated list of column names under Preview columns to control which fields will be displayed in the preview.
6. Click Preview and review the sample rows.

Note that deletion previews are stateless, so if you leave your page it will loose any ongoing progress.

Create a Deletion Request

After reviewing the preview, click Create Deletion Request. The request will appear in the Deletion Requests list with a queued status.

Creating the same filter more than once creates independent deletion requests. Review the existing requests before submitting another one.

How Deletions Execute

Deletion requests are processed asynchronously. They move through the following statuses:

• queued: The request was accepted and is waiting to be scheduled.
• scheduled: The request was assigned for processing. Its progress appears in the Console.
• done: Tableflow finished processing all data covered by the request.
• canceled: The request was canceled and will not make further progress.

Requests normally remain queued for at least four hours. This delay allows Tableflow to process multiple requests together and reduce the amount of work required to scan a table as well as the write amplification. After a request is scheduled, its completion time depends on the table size, the number of matching records, and the available Agent capacity.

Tableflow continues to ingest data while deletions runs.

Note

A deletion request is not permanent and will not be applied to future data. Row added to the Tableflow table after the request is scheduled are not covered, even if they match the filter. Create another deletion request to remove those records.

Monitor a Deletion

Open the table’s Deletions tab and click Refresh to see the latest request status and progress:

• Processed shows how many files and records Tableflow has scanned.
• Matching rows shows how many rows matched this request’s filter. This value increases while the request is processed.
• Last updated shows when the request last made progress.
• Duration shows the total time for a completed request, or the elapsed processing time for a scheduled request.
• Lag shows how long an active request has gone without making progress.

Deletion requests can be processed together. If several requests in the same batch match one row, that row is counted once for each matching request. For this reason, adding the Matching rows values from several requests can produce a number larger than the number of rows removed from the table.

Additionally, the Agents emit the following metrics for deletions:

Metric	Type	Description
warpstream.tableflow_deletion_pending_requests	gauge	Number of query-deletion requests currently pending, grouped by status.
warpstream.tableflow_deletion_max_request_lag_seconds	gauge	Maximum query-deletion request lag in seconds by status, measured since each pending request last changed or made progress.
warpstream.tableflow_total_deleted_records	counter	Monotonically increasing count of records deleted from a Tableflow table.

Cancel a Deletion

You can cancel a request while its status is queued or scheduled:

1. Open the table’s Deletions tab.
2. Find the request and click Cancel.
3. Confirm the cancellation.

Canceling a scheduled request stops remaining work, but it does not restore data that has already been removed. Requests with a done or canceled status cannot be canceled.

When Deleted Data Stops Appearing

The done status means that Tableflow has finished applying the deletion. Tableflow must then publish a new Iceberg snapshot before query engines see the updated table state. This normally takes about five minutes.

External catalog synchronization and query-engine metadata caches can add more time. When confirming a deletion, refresh the query engine’s table metadata and query the latest snapshot.

Queries that explicitly use an older snapshot can continue to return deleted records until that snapshot expires.

When Data Is Physically Removed

Tableflow does not immediately delete replaced files from object storage. Old files are removed asynchronously after they are no longer referenced by a retained Iceberg snapshot. This cleanup can take several hours.

A done status confirms that the deletion was applied, but it does not confirm that old files have already been physically removed from object storage.

Limitations

• Query-based deletion applies only to Tableflow. It does not delete source Kafka records.
• A request applies to data added to the table before the request is scheduled. It does not continuously delete future matching data.
• A table can have at most 500 active deletion requests.
• The Console retains the 100 most recent done or canceled requests for each table.
• A filter can contain at most 5,000 characters.
• Preview returns at most 10 sample records.

Creating and canceling deletion requests are recorded in Audit Logs when Audit Logs are enabled.