Public

Get outputs for a scraping group

Allows you to fetch all outputs for a given scraping group. The results are a materialized view of the outputs for the group; meaning the results are deduplicated. This view is updated depending on how often the group is scheduled to be re-scraped.

The results are paginated and sorted by the create_date of items in ascending order. You can fetch the next page by using the next_url or next_cursor fields in the response metadata.

Typically, you’d also want to provide a created_after filter to only fetch outputs created after a certain date. This is useful when you want to fetch new outputs since the last time you fetched outputs; thus allowing you to maintain a “real-time” view of the outputs.

GET

outputs

{group_id}

curl --request GET \
  --url https://api.reworkd.dev/v1/outputs/{group_id} \
  --header 'Authorization: Bearer <token>'

{
  "metadata": {
    "next_url": "https://api.reworkd.dev/v1/items?cursor=2612294",
    "next_cursor": 2612294
  },
  "items": [
    {
      "id": 123,
      "output_id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
      "is_approved": true,
      "create_date": "2023-02-14T00:00:00",
      "update_date": "2023-02-14T16:00:00",
      "last_scraped_date": "2023-02-14T16:00:00",
      "source_url": "http://example.com",
      "change_type": "CREATE",
      "key_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "value_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "data": {
        "age": 30,
        "email": "john@example.com",
        "images": [
          "http://example.com/profile.jpg",
          "http://example.com/photo.jpg"
        ],
        "name": "John Doe"
      },
      "tags": {
        "country": "US",
        "external_customer_id": "12345",
        "region": "US-CA"
      },
      "files": [
        {
          "create_date": "2023-02-14T00:00:00",
          "field": "image[0]",
          "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
          "file_metadata": {},
          "file_type": "jpg",
          "file_url": "http://example.com/example.jpg",
          "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
          "s3_key": "example.jpg",
          "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
          "source_url": "http://example.com",
          "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
        }
      ]
    }
  ]
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

group_id

string

required

ID of the scraping group you want to fetch outputs for. This can be found on the groups page in the individual group card.

Query Parameters

job_id

string | null

ID of the root scraping job to filter outputs by. Useful when you need to fetch results from a specific domain or data source within a group. When omitted, outputs from all jobs in the group will be returned.

Examples:

"03583f9c-6c90-4f3c-9afd-186258d6f4d6"

null

created_after

string | null

Filter outputs to only include those created or updated on or after this timestamp. Accepts ISO 8601 format (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ). Essential for incremental data syncing to avoid fetching the entire dataset on each request.

Examples:

"2025-01-15T10:00:00Z"

"2025-01-15"

null

url

string | null

Complete URL to fetch outputs for, including protocol and path. Must match exactly the URL that was processed by the scraper.

Examples:

"https://www.example.com/product/product_id_123"

limit

integer

default:10

Number of results to return per page

Required range: 1 <= x <= 1000

cursor

integer

default:0

Cursor to paginate through results

Required range: x >= 0

country

string | null

Name of the country to filter by (eg: United States)

Examples:

"United States"

region

string | null

ISO 3166-2 code of the region to filter by (eg: US-CA)

Examples:

"US-CA"

"US-TX"

Response

200

application/json

Successful Response

metadata

object

required

items

object[]

required

Represents a single output from a scraping job. This is the data that was extracted from a website by a scraper. The data field contains the extracted data normalized to the schema of the scraping group.

The files field contains the files that were extracted by the scraper. The files can be downloaded from the s3_url field.

The change type indicate if it was the first time the output was created or if it was an update or delete of an existing record. See ChangesetAction for more details.

items.id

integer

required

deprecated

items.output_id

string

required

Examples:

"206fdc72-10c0-468c-996f-ff3f8cc51592"

items.is_approved

boolean

required

A boolean describing whether or not the job this output was created from has been marked as APPROVED

Examples:

true

items.create_date

string

required

The date when this output was first created in the system

Examples:

"2023-02-14T00:00:00"

items.update_date

string

required

The date when the output data was last modified

Examples:

"2023-02-14T16:00:00"

items.last_scraped_date

string

required

The date when this output was most recently scraped, even if no changes were detected

Examples:

"2023-02-14T16:00:00"

items.source_url

string

required

Examples:

"http://example.com"

items.change_type

enum<string>

required

Denotes which type of change was made to a resource. If an output row has never been seen before, it is considered a CREATE. If an output row has been seen before, but has changed, it is considered an UPDATE. If an output row has been seen before, but is no longer present, it is considered a DELETE. If an output row has been seen before, and has not changed, it is considered a NOOP.

Available options:

CREATE,

UPDATE,

DELETE,

NOOP

items.key_hash

string

required

Examples:

"d41d8cd98f00b204e9800998ecf8427e"

items.value_hash

string

required

Examples:

"d41d8cd98f00b204e9800998ecf8427e"

items.data

object

required

The extracted data normalized to the schema of the scraping group

Examples:

{
  "age": 30,
  "email": "john@example.com",
  "images": [
    "http://example.com/profile.jpg",
    "http://example.com/photo.jpg"
  ],
  "name": "John Doe"
}

items.tags

object

required

Custom user-defined metadata associated with the source job the output was produced from. Country and region information is automatically added to the tags.

Examples:

{
  "country": "US",
  "external_customer_id": "12345",
  "region": "US-CA"
}

items.files

object[]

required

The files that were extracted by the scraper

Examples:

[
  {
    "create_date": "2023-02-14T00:00:00",
    "field": "image[0]",
    "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
    "file_metadata": {},
    "file_type": "jpg",
    "file_url": "http://example.com/example.jpg",
    "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
    "s3_key": "example.jpg",
    "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
    "source_url": "http://example.com",
    "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
  }
]

Get Cookies

curl --request GET \
  --url https://api.reworkd.dev/v1/outputs/{group_id} \
  --header 'Authorization: Bearer <token>'

{
  "metadata": {
    "next_url": "https://api.reworkd.dev/v1/items?cursor=2612294",
    "next_cursor": 2612294
  },
  "items": [
    {
      "id": 123,
      "output_id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
      "is_approved": true,
      "create_date": "2023-02-14T00:00:00",
      "update_date": "2023-02-14T16:00:00",
      "last_scraped_date": "2023-02-14T16:00:00",
      "source_url": "http://example.com",
      "change_type": "CREATE",
      "key_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "value_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "data": {
        "age": 30,
        "email": "john@example.com",
        "images": [
          "http://example.com/profile.jpg",
          "http://example.com/photo.jpg"
        ],
        "name": "John Doe"
      },
      "tags": {
        "country": "US",
        "external_customer_id": "12345",
        "region": "US-CA"
      },
      "files": [
        {
          "create_date": "2023-02-14T00:00:00",
          "field": "image[0]",
          "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
          "file_metadata": {},
          "file_type": "jpg",
          "file_url": "http://example.com/example.jpg",
          "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
          "s3_key": "example.jpg",
          "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
          "source_url": "http://example.com",
          "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
        }
      ]
    }
  ]
}