Public

Get outputs for a scraping group

Allows you to fetch all outputs for a given scraping group. The results are a materialized view of the outputs for the group; meaning the results are deduplicated. This view is updated depending on how often the group is scheduled to be re-scraped.

The results are paginated and sorted by the created_at field in ascending order. You can fetch the next page by using the next_url or next_cursor fields in the response metadata.

Typically, you’d also want to provide a created_after filter to only fetch outputs created after a certain date. This is useful when you want to fetch new outputs since the last time you fetched outputs; thus allowing you to maintain a “real-time” view of the outputs.

GET

outputs

{group_id}

curl --request GET \
  --url https://api.reworkd.dev/v1/outputs/{group_id}

{
  "metadata": {
    "next_url": "https://api.reworkd.dev/v1/items?cursor=2612294",
    "next_cursor": 2612294
  },
  "items": [
    {
      "id": 123,
      "output_id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
      "is_approved": true,
      "create_date": "2023-02-14T00:00:00",
      "source_url": "http://example.com",
      "change_type": "CREATE",
      "key_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "value_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "data": {
        "age": 30,
        "email": "john@example.com",
        "images": [
          "http://example.com/profile.jpg",
          "http://example.com/photo.jpg"
        ],
        "name": "John Doe"
      },
      "tags": {
        "country": "US",
        "external_customer_id": "12345",
        "region": "US-CA"
      },
      "files": [
        {
          "create_date": "2023-02-14T00:00:00",
          "field": "image[0]",
          "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
          "file_metadata": {},
          "file_type": "jpg",
          "file_url": "http://example.com/example.jpg",
          "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
          "s3_key": "example.jpg",
          "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
          "source_url": "http://example.com",
          "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
        }
      ]
    }
  ]
}

Path Parameters

group_id

string

required

ID of the scraping group you want to fetch outputs for. Can be found in the URL of the scraping group page.

Query Parameters

job_id

string | null

ID of the root scraping job you want to fetch outputs for. If not provided, all jobs will be used. Using this parameter allows you to fetch results for a specific domain.

Example:

"03583f9c-6c90-4f3c-9afd-186258d6f4d6"

created_after

string | null

Date (inclusive) in format YYYY-MM-DD

Example:

"2023-02-14"

limit

integer

default:10

Number of results to return per page

Required range: 1 <= x <= 1000

cursor

integer

default:0

Cursor to paginate through results

Required range: x >= 0

country

string | null

Name of the country to filter by (eg: United States)

Example:

"United States"

region

string | null

ISO 3166-2 code of the region to filter by (eg: US-CA)

Example:

"US-CA"

Response

200

application/json

Successful Response

metadata

object

required

items

object[]

required

Represents a single output from a scraping job. This is the data that was extracted from a website by a scraper. The data field contains the extracted data normalized to the schema of the scraping group.

The files field contains the files that were extracted by the scraper. The files can be downloaded from the s3_url field.

The change type indicate if it was the first time the output was created or if it was an update or delete of an existing record. See ChangesetAction for more details.

items.id

integer

requireddeprecated

items.output_id

string

required

Example:

"206fdc72-10c0-468c-996f-ff3f8cc51592"

items.is_approved

boolean

required

A boolean describing whether or not the job this output was created from has been marked as APPROVED

Example:

true

items.create_date

string

required

Example:

"2023-02-14T00:00:00"

items.source_url

string

required

Example:

"http://example.com"

items.change_type

enum<string>

required

Denotes which type of change was made to a resource. If an output row has never been seen before, it is considered a CREATE. If an output row has been seen before, but has changed, it is considered an UPDATE. If an output row has been seen before, but is no longer present, it is considered a DELETE. If an output row has been seen before, and has not changed, it is considered a NOOP.

Available options:

CREATE,

UPDATE,

DELETE,

NOOP

items.key_hash

string

required

Example:

"d41d8cd98f00b204e9800998ecf8427e"

items.value_hash

string

required

Example:

"d41d8cd98f00b204e9800998ecf8427e"

items.data

object

required

The extracted data normalized to the schema of the scraping group

Example:

{
  "age": 30,
  "email": "john@example.com",
  "images": [
    "http://example.com/profile.jpg",
    "http://example.com/photo.jpg"
  ],
  "name": "John Doe"
}

items.tags

object

required

Custom user-defined metadata associated with the source job the output was produced from. Country and region information is automatically added to the tags.

items.files

object[]

required

The files that were extracted by the scraper

Example:

[
  {
    "create_date": "2023-02-14T00:00:00",
    "field": "image[0]",
    "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
    "file_metadata": {},
    "file_type": "jpg",
    "file_url": "http://example.com/example.jpg",
    "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
    "s3_key": "example.jpg",
    "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
    "source_url": "http://example.com",
    "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
  }
]

Get Cookies

curl --request GET \
  --url https://api.reworkd.dev/v1/outputs/{group_id}

{
  "metadata": {
    "next_url": "https://api.reworkd.dev/v1/items?cursor=2612294",
    "next_cursor": 2612294
  },
  "items": [
    {
      "id": 123,
      "output_id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
      "is_approved": true,
      "create_date": "2023-02-14T00:00:00",
      "source_url": "http://example.com",
      "change_type": "CREATE",
      "key_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "value_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "data": {
        "age": 30,
        "email": "john@example.com",
        "images": [
          "http://example.com/profile.jpg",
          "http://example.com/photo.jpg"
        ],
        "name": "John Doe"
      },
      "tags": {
        "country": "US",
        "external_customer_id": "12345",
        "region": "US-CA"
      },
      "files": [
        {
          "create_date": "2023-02-14T00:00:00",
          "field": "image[0]",
          "file_checksum": "d41d8cd98f00b204e9800998ecf8427e",
          "file_metadata": {},
          "file_type": "jpg",
          "file_url": "http://example.com/example.jpg",
          "id": "206fdc72-10c0-468c-996f-ff3f8cc51592",
          "s3_key": "example.jpg",
          "s3_url": "https://files.reworkd.dev/d41d8cd98f00b204e9800998ecf8427e.jpg",
          "source_url": "http://example.com",
          "url_etag_hash": "d41d8cd98f00b204e9800998ecf8427e"
        }
      ]
    }
  ]
}