> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reworkd.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplication

> Automatically generate scrapers

Reworkd automatically handles deduplicating data whenever your scrapers re-run.

## How It Works

When saving data, Reworkd uses a **unique key** (or composite key) based on the record's fields to determine if the data is new or if it is a duplicate of data that has already been saved.

| Scenario                                                   | Action Taken by Reworkd                                                     |
| ---------------------------------------------------------- | --------------------------------------------------------------------------- |
| **New row of data saved**                                  | Inserts data and marks as a `CREATE` change.                                |
| **Duplicate row of data saved**                            | Skips insertion; no duplicate is created.                                   |
| **Updating data that has been seen before (existing key)** | Updates existing record without duplication and marks as an `UPDATE` change |

## Defining your Deduplication Key

When you are creating your schema, you must also select which of the fields you want to use as part of your **primary/deduplication key**.
This deduplication key is critical to ensure you avoid duplicated data. It must:

* ✅ **Be unique** for every output row.
* ✅ **Remain stable** over time (avoid frequently changing fields).
* ✅ **Be consistent**. Regardless of what website you are on, this key must be the same for the same item.

If there is no one obvious key field, use multiple attributes to create a reliable **composite key**.

## Good vs. Poor Key Examples

#### Good key choices

* Unique ID like a **SKU** or **UPC**
* Combination of unique attributes like **Brand + Model + Color**

#### Poor key choices

* Price (frequently changes)
* Availability status (frequently fluctuating)
* Timestamp of last update