webscraping.ai
Module webscraping.ai
API
Definitions
ballerinax/webscraping.ai Ballerina library
Overview
WebScraping.AI is a web scraping API that provides LLM-powered tools, Chromium JavaScript rendering, rotating proxies and built-in HTML parsing. This connector lets Ballerina applications fetch page HTML and Markdown text, select page areas with CSS selectors, ask questions about a page, extract structured fields, run search engine queries and read structured data from supported sites. It supports version 3.2.2 of the WebScraping.AI API.
Key features
- Retrieve page HTML or clean Markdown text, with JavaScript rendering and proxy options
- Select one or more page areas by CSS selector
- Ask an LLM questions about a page and extract structured fields from it
- Search engine results (SERP) and structured data for supported sites
- Check your remaining API credits and concurrency
Setup guide
To use the WebScraping.AI connector, you need an API key.
-
Create an account at webscraping.ai and sign in.
-
Open the dashboard and copy your API key.
-
Store the key in a
Config.tomlfile in your project, for example:apiKey = "<YOUR_API_KEY>"
Quickstart
To use the webscraping.ai connector in your Ballerina application, update the .bal file as follows:
Step 1: Import the module
import ballerinax/webscraping.ai;
Step 2: Instantiate a new connector
configurable string apiKey = ?; final ai:Client scraper = check new ({apiKey});
Step 3: Invoke the connector operation
public function main() returns error? { ai:Account _ = check scraper->getAccount(); }
Step 4: Run the Ballerina application
bal run
Examples
The WebScraping.AI connector provides practical examples illustrating usage in various scenarios. Explore these examples, covering the following use cases:
- Page content summarizer - Check the account quota, convert a web page to Markdown and ask an LLM a question about it.
- Search result extractor - Run a search query and extract a structured field from the top result.
Clients
webscraping.ai: Client
WebScraping.AI scraping API provides LLM-powered tools with Chromium JavaScript rendering, rotating proxies, and built-in HTML parsing.
Constructor
Gets invoked to initialize the connector.
init (ApiKeysConfig apiKeyConfig, ConnectionConfig config, string serviceUrl)- apiKeyConfig ApiKeysConfig - API keys for authorization
- config ConnectionConfig {} - The configurations to be used when initializing the
connector
- serviceUrl string "https://api.webscraping.ai" - URL of the target service
getAnswer
Get an answer to a question about a given web page
Parameters
- queries *GetAnswerQueries - Queries to be sent with the request
getFields
function getFields(map<string|string[]> headers1, *GetFieldsQueries queries) returns FieldsResult|errorExtract structured data fields from a web page
Parameters
- queries *GetFieldsQueries - Queries to be sent with the request
Return Type
- FieldsResult|error - Parameters validation error
getHtml
Page HTML by URL
Parameters
- queries *GetHtmlQueries - Queries to be sent with the request
getText
Page text by URL (Markdown)
Parameters
- queries *GetTextQueries - Queries to be sent with the request
getSelected
function getSelected(map<string|string[]> headers1, *GetSelectedQueries queries) returns string|errorHTML of a selected page area by URL and CSS selector
Parameters
- queries *GetSelectedQueries - Queries to be sent with the request
getSelectedMultiple
function getSelectedMultiple(map<string|string[]> headers1, *GetSelectedMultipleQueries queries) returns SelectedAreas|errorHTML of multiple page areas by URL and CSS selectors
Parameters
- queries *GetSelectedMultipleQueries - Queries to be sent with the request
Return Type
- SelectedAreas|error - Parameters validation error
getSerp
function getSerp(map<string|string[]> headers, *GetSerpQueries queries) returns SerpResult|errorSearch engine results (SERP) by query
Parameters
- queries *GetSerpQueries - Queries to be sent with the request
Return Type
- SerpResult|error - Parameters validation error
getData
function getData(map<string|string[]> headers, *GetDataQueries queries) returns DataResult|errorStructured data for a page on a supported site
Parameters
- queries *GetDataQueries - Queries to be sent with the request
Return Type
- DataResult|error - Parameters validation error
getAccount
Information about your account calls quota
Records
webscraping.ai: Account
Account quota and concurrency information
Fields
- resetsAt? int - Next billing cycle start time (UNIX timestamp)
- remainingApiCalls? int - Deprecated alias of remaining_monthly_credits, kept for backward compatibility
- remainingTotalCredits? int - Total remaining API credits (monthly + pay-as-you-go)
- remainingConcurrency? int - Remaining concurrent requests
- remainingPaygCredits? int - Remaining pay-as-you-go credits, consumed after the monthly quota is exhausted (valid for 12 months after the last top-up)
- remainingMonthlyCredits? int - Remaining monthly API credits on the subscription plan (does not include pay-as-you-go credits)
- email? string - Your account email
webscraping.ai: ApiKeysConfig
Provides API key configurations needed when communicating with a remote HTTP endpoint.
Fields
- apiKey string - WebScraping.AI API key supplied as the api_key query parameter
webscraping.ai: ConnectionConfig
Provides a set of configurations for controlling the behaviours when communicating with a remote HTTP endpoint.
Fields
- httpVersion HttpVersion(default http:HTTP_2_0) - The HTTP version understood by the client
- http1Settings ClientHttp1Settings(default {}) - Configurations related to HTTP/1.x protocol
- http2Settings ClientHttp2Settings(default {}) - Configurations related to HTTP/2 protocol
- timeout decimal(default 30) - The maximum time to wait (in seconds) for a response before closing the connection
- forwarded string(default "disable") - The choice of setting
forwarded/x-forwardedheader
- followRedirects? FollowRedirects - Configurations associated with Redirection
- poolConfig? PoolConfiguration - Configurations associated with request pooling
- cache CacheConfig(default {}) - HTTP caching related configurations
- compression Compression(default http:COMPRESSION_AUTO) - Specifies the way of handling compression (
accept-encoding) header
- circuitBreaker? CircuitBreakerConfig - Configurations associated with the behaviour of the Circuit Breaker
- retryConfig? RetryConfig - Configurations associated with retrying
- cookieConfig? CookieConfig - Configurations associated with cookies
- responseLimits ResponseLimitConfigs(default {}) - Configurations associated with inbound response size limits
- secureSocket? ClientSecureSocket - SSL/TLS-related options
- proxy? ProxyConfig - Proxy server related options
- socketConfig ClientSocketConfig(default {}) - Provides settings related to client socket configuration
- validation boolean(default true) - Enables the inbound payload validation functionality which provided by the constraint package. Enabled by default
- laxDataBinding boolean(default true) - Enables relaxed data binding on the client side. When enabled,
nilvalues are treated as optional, and absent fields are handled asnilabletypes. Enabled by default.
webscraping.ai: DataResult
Structured data extracted from a page on a supported site
Fields
- data? record {}? - The page's fields. The shape depends on
providerandtype
- parseStatus? string -
okwhen the page was parsed;parse_failedwhen the page was fetched but couldn't be parsed (datamay be null or partial);not_foundwhen the page doesn't exist. All three are successful, charged requests. Treat unknown values as an open set
- requestParameters? DataResultRequestParameters - The URL as requested and how it was classified
webscraping.ai: DataResultRequestParameters
The URL as requested and how it was classified
Fields
- provider? string - The detected site, e.g.
youtube,tiktok,twitter,linkedin,instagramorreddit. New sites are added over time, so treat this as an open set
- 'type? string - The detected page kind within the site, e.g.
video,channel,playlist,profile,post,companyorjob. Also an open set
- url? string - The URL that was requested
webscraping.ai: FieldDescriptions
Descriptions of the fields to extract, keyed by field name
Fields
- title? string - Description of the page title to extract
- string... - Rest field
webscraping.ai: FieldsResult
Fields extracted from the web page, keyed by field name
Fields
- result? record { string?... } - The extracted fields, keyed by the requested field names. A field the page doesn't contain is null
webscraping.ai: GetAnswerQueries
Represents the Queries record for the operation: getAnswer
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- question? string - Question or instructions to ask the LLM model about the target page
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- format "json"|"text" (default "text") - Format of the response (text by default). "text" returns the plain text/HTML response. "json" wraps it in a JSON object with a
resultkey, for tools such as Zapier that only accept JSON
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: GetDataQueries
Represents the Queries record for the operation: getData
Fields
- country string(default "us") - Two-letter country code of the proxy used to fetch the page (
usby default). Must be one of the API's proxy countries (see thecountryparameter of the page endpoints); other values are rejected with a 400
- transcript boolean(default false) - YouTube videos only. Also fetch the video's transcript into
data.transcript(null when no matching captions are available). If the transcript fetch itself fails, the whole request fails with a 500 and is not charged
- url string - URL of a page on a supported site
- transcriptLanguage? string - YouTube videos only, with
transcript=true. Caption language to pick, e.g.enorde. Without it, English is preferred, then the first available track. If the video has no captions in that language,data.transcriptis null
webscraping.ai: GetFieldsQueries
Represents the Queries record for the operation: getFields
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- fields FieldDescriptions - Object describing fields to extract from the page and their descriptions
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: GetHtmlQueries
Represents the Queries record for the operation: getHtml
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- format "json"|"text" (default "text") - Format of the response (text by default). "text" returns the plain text/HTML response. "json" wraps it in a JSON object with a
resultkey, for tools such as Zapier that only accept JSON
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- returnScriptResult boolean(default false) - Return result of the custom JavaScript code (js_script parameter) execution on the target page (false by default, page HTML will be returned)
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: GetSelectedMultipleQueries
Represents the Queries record for the operation: getSelectedMultiple
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- selectors? string[] - Multiple CSS selectors. Repeat the parameter once per selector (selectors=h1&selectors=p); bracketed forms such as selectors[]=h1 are not recognized and return empty results
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: GetSelectedQueries
Represents the Queries record for the operation: getSelected
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- format "json"|"text" (default "text") - Format of the response (text by default). "text" returns the plain text/HTML response. "json" wraps it in a JSON object with a
resultkey, for tools such as Zapier that only accept JSON
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- selector? string - CSS selector of the page area to return. The inner HTML of the first matching element is returned; if nothing matches, the request fails with a 400 "Element not found"
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: GetSerpQueries
Represents the Queries record for the operation: getSerp
Fields
- q string - Search query
- hl string(default "en") - Two-letter language code for the results (Google
hlparameter)
- engine "google" (default "google") - Search engine to query
- gl string(default "us") - Two-letter country code for geolocation of the search (Google
glparameter)
- page int(default 1) - Results page number (10 results per page), from 1 to 100. Any other value (0, a fraction, a non-number, or above 100) is rejected with a 400 before billing
webscraping.ai: GetTextQueries
Represents the Queries record for the operation: getText
Fields
- targetPageHeaders? TargetPageHeaders - HTTP headers to pass to the target page. Can be specified either via a nested query parameter (...&headers[One]=value1&headers[Another]=value2) or as a JSON encoded object (...&headers={"One": "value1", "Another": "value2"})
- country "us"|"gb"|"de"|"it"|"fr"|"ca"|"es"|"ru"|"jp"|"kr"|"in"|"hk"|"tr" (default "us") - Country of the proxy to use (US by default)
- errorOn404 boolean(default false) - Return error on 404 HTTP status on the target page (false by default)
- js boolean(default true) - Execute on-page JavaScript using a headless browser (true by default)
- url string - URL of the target page
- timeout int(default 10000) - Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 25000; larger values are capped at 25000)
- jsTimeout int(default 2000) - Maximum JavaScript rendering time in ms. Increase it in case if you see a loading indicator instead of data on the target page
- textFormat "plain"|"xml"|"json" (default "plain") - Format of the text response (plain by default). "plain" will return the page body content as Markdown. "json" and "xml" will return a json/xml with "title", "description" and "content" (Markdown) keys
- proxy "datacenter"|"residential"|"stealth"|"auto" (default "datacenter") - Type of proxy. Use
residentialif your site restricts traffic from datacenters, orstealthfor the most heavily protected sites with advanced anti-bot detection (datacenterby default). Residential and stealth proxy requests are more expensive than datacenter, see the pricing page for details.autowalks the tiers cheapest-first inside one request (datacenter, then residential, then stealth; anti-bot challenge pages skip straight to stealth) and bills only the tier that succeeded. Failed attempts are free. The API remembers, per domain, the cheapest tier that worked and starts there for 30 days, so repeat requests to a protected site do not re-pay the failed rungs in latency. Cannot be combined withcustom_proxy
- jsScript? string - Custom JavaScript code to execute on the target page. It is evaluated as a script whose result is the value of its last expression (a top-level
returnis a syntax error), and it is only run by /html; the other endpoints accept the parameter but ignore it
- waitFor? string - CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. If the element doesn't appear within the request timeout, the request fails with a 500 error naming the selector and including the target page's HTTP status code and a preview of the page body
- returnLinks boolean(default false) - [Works only with text_format=json] Return links from the page body text (false by default). Useful for building web crawlers
- device "desktop"|"mobile"|"tablet" (default "desktop") - Type of device emulation
- errorOnRedirect boolean(default false) - Return error on redirect on the target page (false by default)
- customProxy? string - Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (<a target="_blank" href="https://webscraping.ai/proxies/smartproxy">Decodo</a>, formerly Smartproxy, for example)
webscraping.ai: OrganicResult
A single organic search result
Fields
- snippet? string - Result description snippet, when Google shows one
- date? string - The date Google displays next to the snippet, as shown (absolute or relative), when present
- displayedLink? string - The breadcrumb-style URL Google displays under the title. Falls back to
domainwhen Google shows none
- domain? string - Hostname of
linkwithout a leadingwww.
- link? string - URL of the search result
- position? int - Rank of the result within this page, starting at 1 on every page (compute
(page - 1) * 10 + positionfor an absolute rank)
- title? string - Title of the search result
webscraping.ai: RelatedSearch
A related search suggestion
Fields
- query? string - Text of the related search suggestion
webscraping.ai: SerpResult
Parsed search engine results page
Fields
- pagination? SerpResultPagination - Pagination links for the search results
- organicResults? OrganicResult[] - Organic (non-ad) search results, in rank order
- relatedSearches? RelatedSearch[] - Google's "Related searches" suggestions for this query. Omitted when the page shows none
- searchInformation? SerpResultSearchInformation - Information the search engine reported about the search itself
- searchParameters? SerpResultSearchParameters - The normalized parameters the search was run with
webscraping.ai: SerpResultPagination
Pagination links for the search results
Fields
- next? int - The next page number. Omitted when Google offers no further page
- current? int - Current results page number
webscraping.ai: SerpResultSearchInformation
What Google reported about the search itself
Fields
- queryDisplayed? string - The query the results are for. Equals
qunless Google applied a spelling fix
- showingResultsFor? string - The auto-corrected query, present only when Google applied a spelling fix
- organicResultsState? "Results for exact spelling"|"Empty showing fixed spelling results"|"Fully empty" -
Results for exact spellingwhen organic results are present forq;Empty showing fixed spelling resultswhen Google auto-corrected the query and the results are for the corrected one (seeshowing_results_for);Fully emptywhen the page had no organic results (a real empty result page is a successful, billed search)
webscraping.ai: SerpResultSearchParameters
The normalized parameters the search was run with
Fields
- q? string - Search query
- hl? string - Language code of the results
- engine? string - Search engine the query was run on
- gl? string - Country code the search was run for
- page? int - Results page number
webscraping.ai: TargetPageHeaders
HTTP headers to send to the target page, keyed by header name
Fields
- Cookie? string - Cookie header value to send to the target page
- string... - Rest field
Array types
webscraping.ai: SelectedAreas
SelectedAreas
One array per requested selector, in request order, holding the inner HTML of every element that matches it (empty when nothing matches)
Import
import ballerinax/webscraping.ai;Metadata
Released date: 1 day ago
Version: 2.0.0
License: Apache-2.0
Security
Direct vulnerabilities: No known direct vulnerabilities
Indirect vulnerabilities:
Checked on 2026-11-21 via OSV.dev
Compatibility
Platform: any
Ballerina version: 2201.12.0
GraalVM compatible: Yes
Pull count
Total: 60
Current verison: 0
Weekly downloads
Keywords
Vendor/WebScraping.AI
Area/Developer Tools
Type/Connector
Contributors