Skip to main content

ScrapeJobRead

A website-import job. Created queued, processed in the background; terminal statuses are completed, failed, and expired.

idIdrequired

Unique numeric identifier of the record.

product_idProduct Idrequired

ID of the product (workspace) that owns this record. Most endpoints scope access by product.

agent_idAgent Idrequired

ID of the agent (a configured AI assistant / bot persona) this record belongs to.

directory_id objectrequired

Knowledge directory receiving the documents; set as soon as ingestion starts.

anyOf
integer
urlUrlrequired

The URL the job was created for.

modeScrapeDiscoveryModerequired

Resolved discovery mode (site or page).

Possible values: [site, page]

context objectrequired

Client context used by page selection and content cleaning.

anyOf
string
max_pages objectrequired

Effective page cap for this job.

anyOf
integer
statusScrapeJobStatusrequired

Job lifecycle state (queued, in_progress, completed, failed, expired).

Possible values: [queued, in_progress, completed, failed, expired]

stage objectrequired

Current pipeline phase while in progress (discover, select, fetch, normalize, ingest).

anyOf
ScrapeJobStage

Possible values: [discover, select, fetch, normalize, ingest]

selection objectrequired

Audit trail of every considered URL with its score, category, one-line reason, and whether it was kept.

anyOf
  • Array [
    • property name*any
  • ]
    • stats objectrequired

    Live page counters (pages_discovered, pages_selected, pages_fetched, pages_normalized, pages_ingested, pages_failed), updated continuously while the job runs.

    anyOf
    object
    error_message objectrequired

    Failure reason for failed or expired jobs.

    anyOf
    string
    status_changed_atdate-timerequired

    Last status or liveness update time (UTC, ISO 8601).

    started_at objectrequired

    When a worker claimed the job (UTC, ISO 8601).

    anyOf
    date-time
    completed_at objectrequired

    When the job reached a terminal state (UTC, ISO 8601).

    anyOf
    date-time
    created_atdate-timerequired

    Creation time (UTC, ISO 8601).

    updated_atdate-timerequired

    Last modification time (UTC, ISO 8601).

    ScrapeJobRead
    {
      "id": 0,
      "product_id": 0,
      "agent_id": 0,
      "directory_id": 0,
      "url": "string",
      "mode": "site",
      "context": "string",
      "max_pages": 0,
      "status": "queued",
      "stage": "discover",
      "selection": [
        {}
      ],
      "stats": {},
      "error_message": "string",
      "status_changed_at": "2024-07-29T15:51:28.071Z",
      "started_at": "2024-07-29T15:51:28.071Z",
      "completed_at": "2024-07-29T15:51:28.071Z",
      "created_at": "2024-07-29T15:51:28.071Z",
      "updated_at": "2024-07-29T15:51:28.071Z"
    }