Operate Scrawl from Telegram.

Step 1 of 11 · Customer and Admin

Start the bot

Open Scrawl and check that it is ready.

Do this

/start

Greets you and shows the main menu.

/status

Shows whether Scrawl and the services it depends on are healthy.

/help

Lists the commands available to you on this deployment.

You should see. Scrawl answers, and /help shows the commands you are allowed to use.

Tip. The inline menu works on a dedicated deployment. On the hosted service, type the commands instead.

Step 2 of 11 · Customer

Fetch and inspect a page

Get one page through Scrawl's checks and look at what came back.

Do this

/scrapy_fetch <url>

Fetches the URL once it passes the checks.

/scrapy_view <url>

Shows the page's text response.

/scrapy_parse <url>

Parses it with your spider; the buttons set depth, callback, metadata, and output.

/scrape <url>

On a dedicated deployment: the page as Markdown in one step; add links for its links instead.

You should see. The response, or a clear refusal when the URL is not allowed.

Tip. Private and internal addresses are always refused, whatever the deployment.

Step 3 of 11 · Customer

Run a crawl

Run a spider from the catalog as a job.

Do this

/scrapy_list

Lists the spiders you may run.

/scrapy_crawl <spider>

Starts the spider as a job and gives you its id.

/scrapy_check

Runs the project's checks first, if you want to.

You should see. A job id; the bot tells you when the job finishes.

Step 4 of 11 · Customer

Follow jobs and download results

See what is running and get the results in the format you want.

Do this

/jobs

Your active jobs.

/artifacts <job_id>

The job's results; pick one, then a format.

/export

Sets the format you get by default.

/cancel <job_id>

Stops a job of yours.

You should see. Result files in Markdown, text, JSON, raw, or parse output.

Tip. /history lists your finished jobs, so you can fetch their results later.

Step 5 of 11 · Customer

Extract files from a page

Collect images, documents, data files, or media that a page links to.

Do this

/scrapy_extract <url>

Choose what to collect; /scrapy_images and the others collect one kind.

/scrapy_asset_confirm <job_id>

Confirms the job after you read its quote.

/scrapy_asset_artifacts <job_id>

Downloads what was collected.

You should see. The files, checked and packaged as the job's results.

Tip. These commands appear only where asset extraction is switched on.

Step 6 of 11 · Customer

Quick text, link, and pattern extraction

Pull text, links, or matches from a page in one command on the hosted service.

Do this

/bs4_text <url>

The page's visible text; /bs4_links gives its links.

/bs4_select <url> <css>

What a CSS selector matches.

/regex_test

Tries a pattern on sample text before you use it.

/regex_extract <url> <pattern>

Matches the pattern on a page; /regex_save keeps it by name.

You should see. The extracted text, links, or matches in the chat.

Step 7 of 11 · Customer

Use AI

Plan extractions, extract structured data, and ask questions about your data.

Do this

/llm

Shows whether AI is set up here.

/ai_plan <objective>

Drafts a safe extraction plan; on a dedicated deployment, Confirm runs it.

/ai_provider

On a dedicated deployment, adds your own AI provider.

/scrape <url> json <what to extract>

With your provider: the fields you name, as JSON.

/ai_ask

Answers from your datasets, citing the records it used.

You should see. Plans, structured data, or cited answers, from the AI provider configured here.

Tip. AI is optional. Without it, every other command works the same.

Step 8 of 11 · Customer

Use your own proxy

Send your jobs through a proxy you provide.

Do this

/proxy

How bringing your own proxy works.

/proxy_set <endpoint>

On the hosted service, stores it encrypted. Use a private chat.

/proxy_test

Checks that it works; /proxy_clear removes it.

You should see. Your jobs leave through your proxy, still subject to Scrawl's checks.

Step 9 of 11 · Customer

Your hosted account

Accept the hosted service's terms and follow your credits.

Do this

/trust_pledge

Read the Trust Pledge, then Accept pledge.

/balance

Your credit balance.

/plans

The credit packs on offer.

You should see. An account ready to run jobs on the hosted service.

Tip. The hosted service is not publicly available yet.

Step 10 of 11 · Admin

Decide who can use the bot

Give and remove access on a dedicated deployment.

Do this

/access

The access cockpit: Refresh, Grant user, Revoke user.

/access_grant <user_id>

Lets a Telegram user use the bot.

/access_revoke <user_id>

Removes their access.

You should see. The people you list can use the bot; nobody else can.

Tip. Owners are set in the deployment's configuration and cannot be revoked from Telegram.

Step 11 of 11 · Admin

Set up projects and spiders

Create a project, generate and edit spiders, and run an uploaded spider.

Do this

/scrapy_startproject <name>

Creates a project; Set active workspace selects it.

/scrapy_genspider <name> <domain>

Pick a template to generate the spider.

/scrapy_edit <spider>

Edits its source.

/scrapy_runspider

Send a spider file; Confirm runs it once it is validated.

/scrapy_shell <url>

Inspect a response: status, headers, links, a callback, or the body.

You should see. Spiders your users can run with /scrapy_crawl.