Crawl

1. Introduction

Crawl is a high-performance Web Crawling API built for developers and data teams. It packages complex web data collection capabilities into a ready-to-use data service.

Simply provide a target URL to retrieve structured web data. Crawl supports single-page scraping, batch collection, and site-wide crawling for use cases such as AI data acquisition, content analysis, and market monitoring.

Crawl uses advanced fingerprint browser technology to accurately simulate real browsing environments and navigate dynamic anti-bot controls. Browser rendering, dynamic page parsing, data cleaning, proxy orchestration, and automatic retries are built in. Results can be returned as HTML, Markdown, screenshots, and other formats. With a single API, you can collect web data reliably and efficiently without maintaining complex crawler infrastructure.

Core Benefits

Developer-Friendly API

The lightweight API is easy to integrate into an existing application.

• REST API: Complete a scrape through a single API without a complex crawler framework.
• Multi-language SDKs: Use Python, Node.js, Go, and other mainstream languages.
• Structured output: Receive consistent page metadata and Markdown content that AI and business systems can consume directly.

AI-Ready Output

The output is optimized for AI applications.

• Multiple input and output formats: Convert URLs, web pages, and documents into Markdown, PDF, or images.
• Intelligent main-content extraction: Remove noise such as advertisements and navigation while retaining the primary content.
• Lower token usage: Cleaner Markdown reduces token consumption and improves AI processing efficiency.

JavaScript Rendering

Full browser rendering makes dynamic pages easier to scrape.

• JavaScript rendering: Supports modern websites built with React, Vue, Next.js, and similar frameworks.
• Wait for page readiness: Wait for a CSS selector or for a custom duration.
• Browser automation: Click, scroll, type, and execute JavaScript.

Reliability and Retries

Built-in recovery improves scrape success rates and reduces application-side processing.

• Automatic retries: Recover automatically from network errors and page-load failures.
• Synchronous and asynchronous tasks: Support real-time requests and background batch processing.
• Task status queries: Retrieve execution status and results by Task ID.
• Batch monitoring: Track task progress, failures, and results.

High Concurrency and Queue Scheduling

Crawl is designed for large-scale data collection.

• Concurrent execution: Process multiple URLs and tasks concurrently.
• Priority queues: Schedule tasks according to priority.
• Rate limits: Control request rates by plan to reduce the risk of triggering target-site controls.

Predictable Costs

Reduce infrastructure and long-term maintenance costs.

• Flexible billing: Pay by request, successful scrape, or subscription plan.
• Free credit: Validate your use case with trial credit.
• Multiple plans: Starter, Growth, and Scale plans support different workloads.
• No infrastructure to maintain: Focus on your application instead of operating proxies, browser clusters, and task schedulers.

2. How to Use Crawl

Quick Start

Create an Nstproxy account to begin using Crawl.

You can choose one of the following billing options:

• Free trial: Receive trial credit after registration and quickly explore Crawl.
• Pay per use: Add funds to your account and pay only for actual usage, with no subscription required.
• Subscription plan: Select a plan that matches your workload for higher included credit, better pricing, and additional resource support.

Go to Dashboard → Crawl Playground, obtain an API key, and use it to call the Crawl API.

Example

The following request scrapes https://example.com and returns the page in Markdown format.

Request:

curl -X POST "https://api.nstproxy.com/api/v1/crawl/scrape/submit-sync" \
     -H "x-api-key: <API-KEY>" \
     -H "Content-Type: application/json" \
     -d '{"url": "https://example.com", "formats": ["markdown"]}'

Response:

{
  "data": {
    "code": 0,
    "data": {
      "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
      "markdownRef": "z4vHy1TDb8sRfMQturQ127OCgpu8nfInb4aTSz74dlEPiHDwaPXuPEVCvkkGVUgw2_E5lyEkFkJqss_bkAGm_3xLFHRPrq7PO-4vPhNkV4wVMEv-",
      "metadata": {
        "language": "en",
        "statusCode": 200,
        "title": "Example Domain"
      }
    },
    "status": "completed",
    "success": true
  },
  "err": false,
  "msg": "SUCCESS",
  "code": 200
}

You can also view task results on Dashboard → Crawl → Data Output. The latest 100 results are retained.

API Usage

Every endpoint requires an API token for authentication.

Header example:

x-api-key: YOUR_API_KEY
Content-Type: application/json

2.1 Scrape a Single Page

Single-page scraping retrieves one URL and returns Markdown, HTML, links, screenshots, PDF, raw data, or other requested results.

Typical use cases include:

• Extracting the main content from a web page
• Converting a page to Markdown
• Retrieving page HTML
• Extracting links
• Capturing a screenshot or exporting a PDF
• Scraping dynamically rendered content

Synchronous Scraping

A synchronous request waits for the task to complete and returns the result directly. It is suitable for real-time use cases where page processing time is predictable.

Endpoint:

POST /api/v1/crawl/scrape?async=true

Request:

{
  "url": "https://example.com",
  "formats": ["markdown"],
  "timeout": 60000,
  "onlyMainContent": true
}

Response:

{
  "code": 200,
  "err": false,
  "msg": "success",
  "data": {
    "code": 0,
    "success": true,
    "status": "completed",
    "data": {
      "markdown": "# Example Domain",
      "markdownRef": "xxx",
      "links": ["https://www.iana.org/domains/example"],
      "metadata": {
        "title": "Example Domain",
        "statusCode": 200,
        "language": "en"
      }
    }
  }
}

Asynchronous Scraping

An asynchronous request immediately returns a task ID. Use the task ID to retrieve the result later. This method is suitable for slower pages and batch workloads.

Endpoint:

POST /api/v1/crawl/scrape

Request:

{
  "url": "https://example.com",
  "formats": ["markdown"],
  "timeout": 60000,
  "onlyMainContent": true
}

Response:

{
  "code": 200,
  "err": false,
  "msg": "success",
  "data": {
    "id": "01KKKB6KPJT558N4MS5EMWF6TV",
    "status": "processing",
    "cached": false
  }
}

Retrieve the result:

GET /api/v1/crawl/scrape/result/{taskId}

2.2 Site-Wide Crawl

A site-wide crawl starts from one URL and automatically discovers and scrapes pages on the same site. Use maximum depth, maximum page count, and URL inclusion or exclusion rules to control the crawl scope.

Endpoint:

POST /api/v1/crawl

Request:

{
  "url": "https://example.com",
  "formats": ["markdown", "html"],
  "maxDepth": 3,
  "maxPages": 50,
  "includeUrls": ["*example.com/docs/*"],
  "excludeUrls": ["*example.com/admin/*"],
  "ignoreQuery": true,
  "onlyMainContent": true,
  "timeout": 60000
}

Response:

{
  "code": 200,
  "err": false,
  "msg": "success",
  "data": {
    "id": "01KKKCD3KXW3TB5R8BBWDA3VYP",
    "status": "processing"
  }
}

2.3 Query Task Status

Query a Single-Page Scrape Result

The response contains the task status, success state, and scrape result.

Endpoint:

GET /api/v1/crawl/scrape/result/{taskId}

Response:

{
  "data": {
    "code": 0,
    "data": {
      "markdownRef": "nL9gU9mOz9uHEtfXObDTo8K2wPh3F0ekLeDT_-NR-0Bl3-yl2_1kKY16Vbjy3Nj1nPpGK5o9GYnnSvAZqZxNRgrhgKuHrBS9JK4YgQHb0wYBUv20",
      "metadata": {
        "language": "en",
        "statusCode": 200,
        "title": "Example Domain"
      },
      "rawDataRef": "GTZQPXe-_oEWhBvlN1ZbOJt1unITySNEjb1QzVgv_FKTHVJrOe1h59O_hwVB2MaO98nq-V4EU6V1qdQAYz6sqT3mz-GEi85CxW7E5ptiO5MecpOsHg"
    },
    "status": "completed",
    "success": true
  },
  "err": false,
  "msg": "SUCCESS",
  "code": 200
}

Query Site-Wide Crawl Status

The response contains the crawl status, success state, and progress.

Endpoint:

GET /api/v1/crawl/status/{crawlId}

Response:

{
  "data": {
    "code": 0,
    "completed": 5,
    "failed": 0,
    "id": "01KXMX69PNHS92BA98HND5Z7T9",
    "pending": 0,
    "status": "completed",
    "total": 5
  },
  "err": false,
  "msg": "SUCCESS",
  "code": 200
}

Query Crawled Pages

Endpoint:

GET /api/v1/crawl/pages/{crawlId}

Response:

{
  "code": 200,
  "err": false,
  "msg": "success",
  "data": {
    "total": 25,
    "completed": 8,
    "nextCursor": "xxx",
    "data": [
      {
        "taskId": "01KKKB6KPJT558N4MS5EMWF6TV",
        "url": "https://example.com/docs/start",
        "success": true,
        "status": "completed",
        "data": {
          "markdownRef": "xxx",
          "htmlRef": "xxx",
          "links": ["https://example.com/docs/next"],
          "metadata": {
            "title": "Getting Started",
            "statusCode": 200
          }
        }
      }
    ]
  }
}

2.4 Read Large Results

For larger HTML, Markdown, screenshot, PDF, or raw-data results, the API may return a reference token such as:

• markdownRef
• htmlRef
• rawDataRef
• screenshotRef
• pdfRef

Use the storage endpoint to retrieve the original content:

GET /api/v1/crawl/storage/read?st={ref}

Typical use cases include:

• Downloading screenshots
• Downloading PDFs
• Reading raw page data

Output Formats

Crawl supports several output formats, including Markdown, HTML, Raw Data, PDF, Screenshot, and Links. For large results, the response may contain a reference token such as markdownRef, htmlRef, rawDataRef, pdfRef, or screenshotRef. Use the storage endpoint to retrieve the complete content.

3.1 Markdown

Markdown is the recommended output format for AI use cases. Crawl converts the page body, document content, or cleaned HTML into well-structured Markdown, reducing interference from advertisements, navigation, scripts, and other irrelevant content.

Suitable for:

• LLM context
• RAG document ingestion
• AI Agent tool results
• Knowledge-base construction
• Content summarization and question answering
• Information extraction and classification
• Document structuring

Characteristics:

• Clear text structure that models can understand easily
• Lower token usage than HTML
• Preserves headings, paragraphs, lists, links, and other basic semantic structure
• Can be combined with onlyMainContent to produce cleaner main content

3.2 Raw Data / rawHTML

Raw Data contains the original page data returned by the target site, usually the original HTML. It is closer to the network or browser response and is suitable when you want to control parsing yourself.

Suitable for:

• Custom parsing
• Preserving the original page structure
• Debugging scrape results
• Comparing content before and after cleaning
• Investigating encoding, anti-bot responses, redirects, and empty pages
• Saving original web-page snapshots

Characteristics:

• Preserves more of the original content
• Useful for technical investigation and secondary parsing
• Usually has a larger payload
• Not recommended as direct LLM input because it may contain many irrelevant tags, scripts, and styles

3.3 HTML

HTML output contains parsed and cleaned page HTML. Compared with Raw Data, it retains page structure while removing some irrelevant content. Selector parameters can further control the returned area.

Suitable for:

• Custom parsing
• Preserving page structure
• Debugging
• Displaying page content
• Additional cleaning or rule-based extraction
• Preserving tables, links, images, and other structures in the main content

Characteristics:

• Preserves HTML tags and page structure
• Use targetSelector to return a specific area
• Use removeSelectors to remove advertisements, pop-ups, and other elements
• Use includeTags and excludeTags to control retained or removed tags
• Better than Markdown for structured page parsing, but less suitable as direct LLM input

3.4 PDF

PDF output exports a page as a PDF file. It is useful when you need to preserve the page's reading layout or archive it. The file can also be used for text extraction, search indexing, or content comparison.

Suitable for:

• Basic full-text extraction
• Search indexing
• Content comparison
• Page archiving
• Report generation
• Audit and compliance retention
• Offline reading and file distribution

Characteristics:

• Preserves the page's print layout
• Suitable for archiving and downstream document processing
• Usually returned as pdfRef and retrieved through the storage endpoint
• Dynamic pages can be rendered with JavaScript before export

3.5 Screenshot

Screenshot output captures the rendered page. It is suitable for visual archiving, page-state validation, and competitor monitoring. For dynamic pages, Crawl can complete JavaScript rendering and browser actions before taking the screenshot.

Suitable for:

• Page archiving
• Visual validation
• Competitor monitoring
• Compliance retention
• Page redesign monitoring
• Preserving visual evidence of advertisements, prices, rankings, and other page elements
• Investigating failed or unexpected scrape results

Characteristics:

• Preserves the visual state of a page
• Useful when text alone is not sufficient
• Can wait for rendering with jsRender
• Can capture the entire page or a selected element after browser actions
• Usually returned as screenshotRef and retrieved through the storage endpoint

SDK Examples

Node.js, Python, and Go SDKs are available for direct integration into application services, data-collection scripts, AI Agent tools, and RAG ingestion workflows.

4.1 Node.js

Repository: github.com/nstdata-ai/crawl-node (opens in a new tab)

Installation

npm install @nstdata-ai/crawl

TypeScript Types

The Node.js SDK includes TypeScript declarations:

{
  "types": "dist/index.d.ts"
}

Synchronous Single-Page Scrape

import { NstDataClient, Format } from "@nstdata-ai/crawl";
 
const TOKEN = process.env.NSTDATA_API_TOKEN || "YOUR_API_TOKEN";
 
async function main() {
  const client = new NstDataClient(TOKEN);
 
  const res = await client.submitScrapeTaskSync({
    url: "https://example.com/",
    formats: [Format.MARKDOWN],
    timeout: 60000,
    onlyMainContent: true,
  });
 
  const markdown = await res.data?.getMarkdown();
  console.log(markdown);
}
 
main().catch(console.error);

Site-Wide Crawl

import { NstDataClient, Format } from "@nstdata-ai/crawl";
 
const client = new NstDataClient("YOUR_API_TOKEN");
 
const submit = await client.submitCrawlTask({
  url: "https://example.com/",
  formats: [Format.MARKDOWN, Format.HTML],
  maxDepth: 3,
  maxPages: 50,
  ignoreQuery: true,
  onlyMainContent: true,
});
 
console.log(submit.id);
 
const status = await client.getCrawlStatus(submit.id!);
console.log(status);

4.2 Python

Repository: github.com/nstdata-ai/crawl-py (opens in a new tab)

Installation

pip install nstdata-ai-crawl

Python Package

import nstdata_ai_crawl

Synchronous Single-Page Scrape

import os
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto, Format
 
TOKEN = os.getenv("NSTDATA_API_TOKEN", "YOUR_API_TOKEN")
 
with NstDataClient(TOKEN) as client:
    res = client.submit_scrape_task_sync(ScrapeRequestDto(
        url="https://example.com/",
        formats=[Format.MARKDOWN],
        timeout=60000,
        onlyMainContent=True,
    ))
 
    print(res.data.get_markdown())

Site-Wide Crawl

from nstdata_ai_crawl import NstDataClient, CrawlRequestDto, Format
 
with NstDataClient("YOUR_API_TOKEN") as client:
    submit = client.submit_crawl_task(CrawlRequestDto(
        url="https://example.com/",
        formats=[Format.MARKDOWN, Format.HTML],
        maxDepth=3,
        maxPages=50,
        ignoreQuery=True,
        onlyMainContent=True,
    ))
 
    print(submit.id)
 
    status = client.get_crawl_status(submit.id)
    print(status)

4.3 Go

Repository: github.com/nstdata-ai/crawl-go (opens in a new tab)

Installation

go get github.com/nstdata-ai/crawl-go

Go Module

module github.com/nstdata-ai/crawl-go

Synchronous Single-Page Scrape

package main
 
import (
    "context"
    "fmt"
    "log"
 
    crawl "github.com/nstdata-ai/crawl-go"
)
 
func main() {
    ctx := context.Background()
    client := crawl.NewClient("YOUR_API_TOKEN")
 
    res, err := client.SubmitScrapeTaskSync(ctx, &crawl.ScrapeRequestDto{
        URL:             "https://example.com/",
        Formats:         []string{crawl.FormatMarkdown},
        Timeout:         60000,
        OnlyMainContent: true,
    })
    if err != nil {
        log.Fatal(err)
    }
 
    markdown, _ := res.Data.GetMarkdown(ctx)
    fmt.Println(markdown)
}

Site-Wide Crawl

package main
 
import (
    "context"
    "fmt"
    "log"
 
    crawl "github.com/nstdata-ai/crawl-go"
)
 
func main() {
    ctx := context.Background()
    client := crawl.NewClient("YOUR_API_TOKEN")
 
    submit, err := client.SubmitCrawlTask(ctx, &crawl.CrawlRequestDto{
        URL:             "https://example.com/",
        Formats:         []string{crawl.FormatMarkdown, crawl.FormatHTML},
        MaxDepth:        3,
        MaxPages:        50,
        IgnoreQuery:     true,
        OnlyMainContent: true,
    })
    if err != nil {
        log.Fatal(err)
    }
 
    fmt.Println(submit.Id)
 
    status, err := client.GetCrawlStatus(ctx, submit.Id)
    if err != nil {
        log.Fatal(err)
    }
 
    fmt.Println(status)
}

Integration

Nstdata Crawl can serve as a general-purpose web data collection layer for AI Agents, RAG systems, data pipelines, and internal enterprise applications. It reliably retrieves, renders, and converts web content into formats suitable for downstream processing, so each application does not need to maintain its own crawlers, browser clusters, proxies, retry logic, and content-cleaning pipeline.

5.1 AI Agent Integration

AI Agents often need current information from company websites, news pages, product documentation, blogs, forums, or public resources. Nstdata Crawl can act as the Agent's web-reading tool: the Agent provides a URL, Crawl returns Markdown, HTML, or structured output, and the Agent then summarizes, answers questions, compares content, or extracts fields.

Typical workflow:

1. The Agent receives a user question or task objective.
2. The Agent determines which URLs it needs to access.
3. The Agent calls Nstdata Crawl to scrape the pages.
4. Crawl returns cleaned Markdown.
5. The Agent uses the Markdown to summarize, answer questions, extract structured data, or generate a report.

This approach is useful for applications that need real-time web content:

5.2 RAG Integration

In a RAG system, web content normally passes through collection, cleaning, chunking, embedding, and ingestion. Nstdata Crawl handles collection and content cleaning, converting complex pages into Markdown and reducing downstream text-processing work.

Typical workflow:

1. Use Nstdata Crawl to scrape target pages or an entire site.
2. Clean and normalize the returned Markdown.
3. Split the content into chunks by heading, paragraph, or token count.
4. Generate vectors with an embedding model.
5. Store the text, vectors, and metadata in a vector database.
6. When a user asks a question, retrieve relevant chunks and pass them to an LLM to generate an answer.

Common integration components include:

This architecture is suitable for enterprise knowledge bases, documentation Q&A, product-manual assistants, industry research libraries, and competitor intelligence databases.

Error Codes

6.1 HTTP Status Codes

6.2 Task Failure Codes Returned with HTTP 200

Concurrency, Rate Limits, and Performance

• Average single-page response time: Average response latency is within 6 seconds, depending on page type, scrape complexity, and proxy-node quality.
• Maximum timeout: To optimize resource scheduling, the maximum timeout for one request is 60 seconds.
• Page size limit: Crawl supports pages up to 50 MB to maintain efficient parsing and content quality.
• Proxy region selection: Custom geographic locations are supported for localized data collection.
• Queue priorities: Growth and Scale subscribers receive higher queue priority.

3. Billing

Crawl supports Pay Per Use and subscription billing.

Pay Per Use: No subscription is required. Charges are deducted from the account balance according to actual usage.

Subscription: The subscription amount is converted into Included Credits. Included Credits can pay for usage-based services, including Pay Per Use proxy traffic, Crawl, and Proxy Manager. Included Credits are consumed before other account balances.

Included Credits are valid for 30 days and follow the subscription billing cycle. They can only be used during the current billing cycle, and unused credit does not roll over to the next cycle. Visit the Subscriptions page for detailed pricing and features.

• Billing unit (request/page render):
  • A request is billable when page content is successfully retrieved. Receiving an HTTP response without a network error—including selected statuses such as 404 and 403—is considered a successful request.
  • Bandwidth is billed according to actual usage.
• JavaScript rendering and other services:
  • JavaScript rendering: Included in the base service at no additional charge.
  • Markdown extraction: Included in the base service at no additional charge.
  • PDF generation: Included in the base service at no additional charge.
  • Screenshots: Included in the base service at no additional charge.
• Proxy billing: Proxy traffic and usage are billed separately as needed.
• Failed-request policy: Requests that fail to retrieve page content because of a system error are not billed.
• Free trial credit: New users receive $1 USD in trial credit after registering and applying for a trial.
• Pay Per Use rate: The base rate is $1.2 per 1,000 requests.
• Enterprise plans: Custom service agreements and pricing are available for enterprise requirements.

4. FAQ

1. How is Crawl different from a standard HTTP request?

   A standard HTTP request normally retrieves only the static HTML returned by a server. It cannot handle JavaScript rendering, asynchronous content loading, client-side routing, or complex page behavior.

   Crawl runs in a real browser environment and supports dynamic rendering, content extraction, screenshots, PDF generation, proxy access, and task management. It is better suited to production web data collection.

2. Does Crawl support JavaScript rendering?

   Yes. For pages that load content with JavaScript, Crawl can wait for rendering to complete. You can configure a wait duration or wait for a specific element.

3. Does Crawl support authenticated cookies?

   Yes. Pass cookies in the request to access content that requires a signed-in session or other permissions.

4. Does Crawl support screenshots?

   Yes. Crawl can capture screenshots for page archiving, visual analysis, content review, and similar use cases.

5. Does Crawl support multiple URLs?

   Multiple URLs are currently supported through concurrent tasks. Submit multiple asynchronous single-page scrape tasks and use concurrency controls for large-scale collection.

6. Does Crawl support site-wide crawling?

   Yes. A site-wide crawl automatically discovers pages from an entry URL. Use maximum depth, maximum page count, inclusion rules, and exclusion rules to control the scope.

7. Does Crawl support webhooks?

   Webhooks are not currently supported. Query task status to retrieve execution progress and final results. Webhook support is planned for a future release.

8. Does Crawl support custom headers?

   Yes. Configure custom headers in the request, including language, User-Agent, and application-specific identifiers.

9. Can I select a proxy region?

   Yes. Crawl Proxy supports country or region selection to improve regional access, collection stability, and target-site availability.

10. Are failed requests billed?

    No charge is incurred when a system error prevents Crawl from retrieving page content.