Server.Enrichment.ScanMalware
Look up a domain, hostname, IP address or URL in the public archive of ScanMalware, a free URL scanner that analyses links in isolated environments and keeps its public scans searchable. Returns the verdict, risk score and AI classification of the newest completed scan of that host or URL, with a link to the full report. No account, API key or secret is needed, and nothing is submitted: the artifact only reads scans that already exist.
Phishing pages often redirect to the brand they imitate, so the archive
lists a lure hosted on pages.dev under facebook.com when that is where
the lure ended. On shared platforms such as github.io, a subdomain is
usually an unrelated tenant. A verdict is therefore attributed to the
indicator only when the scan was submitted for that exact host or its
www name. Scans that only redirected to the host are counted in
RedirectedHere and scans of subdomains in SubdomainScans; neither lends
its verdict.
For a URL, a scan of that exact URL (ignoring a trailing slash) is
preferred and Match is url. Otherwise the newest scan of the same host
is returned and Match is host, because a scan of a home page is not a
verdict on a login page of the same site. When no scan of the host exists
the row has an empty Verdict: unknown is not the same as clean.
AIClassification and AIScamType come from the AI review of the chosen
scan, for example CONFIRMED_SCAM and Credential Phishing. Some scans
are classified by a rule rather than reviewed by a model, for example
low-risk scans of well-ranked sites; AISkipped then names the rule.
An IP address gets no verdict, because a shared hosting or CDN address is not malicious just because one page served from it is. Its row reports in how many public scans a page contacted the address (IPScans), how many of the newest ScanWindow scans were of pages served from it (HostedScans), up to five of their URLs, and the address’s ASN, with a link to its page. Private, loopback and other non-public addresses are never sent.
Run from a server notebook, for example:
SELECT * FROM Artifact.Server.Enrichment.ScanMalware(
Indicator="example.com")
To enrich hunt results, pass the indicators as rows with an Indicator column. Each distinct indicator is looked up once, Workers at a time:
SELECT * FROM Artifact.Server.Enrichment.ScanMalware(
Indicators={SELECT Domain AS Indicator FROM source()})
Each indicator costs one request for a host or an IP address, or two for
a URL, plus two (summary and AI review) when a scan is found. The
anonymous API allows 600 requests per minute per IP address. After an
HTTP 429 a lookup waits as long as Retry-After asks, at most MaxRetryWait
seconds, and tries once more; a second 429 is reported in Error. Requests
carry the User-Agent scanmalware-velociraptor/1.1.
name: Server.Enrichment.ScanMalware
author: Jonas Lejon, ScanMalware
description: |
Look up a domain, hostname, IP address or URL in the public archive of
ScanMalware, a free URL scanner that analyses links in isolated
environments and keeps its public scans searchable. Returns the verdict,
risk score and AI classification of the newest completed scan of that
host or URL, with a link to the full report. No account, API key or
secret is needed, and nothing is submitted: the artifact only reads scans
that already exist.
Phishing pages often redirect to the brand they imitate, so the archive
lists a lure hosted on `pages.dev` under `facebook.com` when that is where
the lure ended. On shared platforms such as `github.io`, a subdomain is
usually an unrelated tenant. A verdict is therefore attributed to the
indicator only when the scan was submitted for that exact host or its
`www` name. Scans that only redirected to the host are counted in
RedirectedHere and scans of subdomains in SubdomainScans; neither lends
its verdict.
For a URL, a scan of that exact URL (ignoring a trailing slash) is
preferred and Match is `url`. Otherwise the newest scan of the same host
is returned and Match is `host`, because a scan of a home page is not a
verdict on a login page of the same site. When no scan of the host exists
the row has an empty Verdict: unknown is not the same as clean.
AIClassification and AIScamType come from the AI review of the chosen
scan, for example `CONFIRMED_SCAM` and `Credential Phishing`. Some scans
are classified by a rule rather than reviewed by a model, for example
low-risk scans of well-ranked sites; AISkipped then names the rule.
An IP address gets no verdict, because a shared hosting or CDN address is
not malicious just because one page served from it is. Its row reports in
how many public scans a page contacted the address (IPScans), how many of
the newest ScanWindow scans were of pages served from it (HostedScans), up
to five of their URLs, and the address's ASN, with a link to its page.
Private, loopback and other non-public addresses are never sent.
Run from a server notebook, for example:
SELECT * FROM Artifact.Server.Enrichment.ScanMalware(
Indicator="example.com")
To enrich hunt results, pass the indicators as rows with an Indicator
column. Each distinct indicator is looked up once, Workers at a time:
SELECT * FROM Artifact.Server.Enrichment.ScanMalware(
Indicators={SELECT Domain AS Indicator FROM source()})
Each indicator costs one request for a host or an IP address, or two for
a URL, plus two (summary and AI review) when a scan is found. The
anonymous API allows 600 requests per minute per IP address. After an
HTTP 429 a lookup waits as long as Retry-After asks, at most MaxRetryWait
seconds, and tries once more; a second 429 is reported in Error. Requests
carry the User-Agent `scanmalware-velociraptor/1.1`.
reference:
- https://scanmalware.com/api-docs
type: SERVER
parameters:
- name: Indicator
type: string
description: Domain, hostname, IP address or http(s) URL to look up.
- name: Indicators
type: csv
default: |
Indicator
description: Several indicators, one per row in an Indicator column. Each distinct one is looked up once.
- name: ScanWindow
type: int
default: 100
description: How many of the newest scans of the host or IP address to consider, 1 to 100.
- name: Workers
type: int
default: 2
description: How many lookups run at once when Indicators is set.
- name: MaxRetryWait
type: int
default: 60
description: Longest wait in seconds before the single retry after an HTTP 429. 0 reports the 429 at once.
sources:
- query: |
LET Base <= "https://scanmalware.com"
LET UA <= "scanmalware-velociraptor/1.1"
// One GET, repeated once after an HTTP 429 when MaxRetryWait allows.
LET Get(Url, Params) = SELECT Response, Headers, Content
FROM http_client(url=Url, params=Params, user_agent=UA)
LET Fetch(Url, Params) = SELECT * FROM foreach(
row=Get(Url=Url, Params=Params),
query={
SELECT * FROM if(
condition=Response = 429 AND MaxRetryWait > 0 AND sleep(time=if(
condition=int(int=Headers.`Retry-After`[0]) > 0
AND int(int=Headers.`Retry-After`[0]) < MaxRetryWait,
then=int(int=Headers.`Retry-After`[0]),
else=MaxRetryWait)),
then=Get(Url=Url, Params=Params),
else={SELECT Response, Headers, Content FROM scope()})
})
// With Indicators set, every distinct indicator is looked up by its
// own call to this artifact, and nothing below sends a request.
LET Solo <= NOT Indicators
LET Value <= strip(string=Indicator)
LET IP <= ip(parse=Value)
LET IsIP <= IP != NULL
LET PublicIP <= IsIP AND IP.IsGlobalUnicast AND NOT IP.IsPrivate
LET IsURL <= Value =~ "^(?i)https?://"
LET Host <= lowcase(string=strip(suffix=".",
string=if(condition=IsURL, then=url(parse=Value).Hostname, else=Value)))
// A bare IP address fails one of the two patterns, so it never reaches
// /domains; a URL may still have an IP address as its host.
LET ValidHost <= Host =~ "^[a-z0-9_-]+(\\.[a-z0-9_-]+)+$"
AND (IsURL OR NOT Host =~ "^[0-9.]+$")
// The url filter matches the submitted and the final URL as substrings,
// so only a row submitted as this exact URL is a scan of it.
LET ExactResponse <= SELECT Response, parse_json(data=Content) AS Body
FROM if(condition=Solo AND IsURL AND ValidHost, then={
SELECT * FROM Fetch(Url=Base + "/api/v1/search/smql",
Params=dict(q="url:" + Value, `limit`=100))
})
LET ExactScans <= SELECT scan_id, url, final_url, completed_at
FROM foreach(row=ExactResponse[0].Body.results)
WHERE status = "completed"
AND strip(string=url, suffix="/") = strip(string=Value, suffix="/")
LIMIT 1
// Lists scans submitted for the host, scans of its subdomains, and scans
// of other sites that redirected here, newest first.
LET HostResponse <= SELECT Response, parse_json_array(data=Content) AS Body
FROM if(condition=Solo AND ValidHost, then={
SELECT * FROM Fetch(Url=Base + "/api/v1/domains/" + Host + "/scans",
Params=dict(`limit`=ScanWindow, status="completed"))
})
LET HostRows <= SELECT scan_id, url, final_url, completed_at,
"url" in matched_on AS Submitted,
lowcase(string=url(parse=url).Hostname) AS ScanHost
FROM foreach(row=HostResponse[0].Body)
LET OwnScans <= SELECT * FROM HostRows
WHERE Submitted AND ScanHost in (Host, "www." + Host)
LET SubdomainRows <= SELECT * FROM HostRows
WHERE Submitted AND NOT ScanHost in (Host, "www." + Host)
LET RedirectRows <= SELECT * FROM HostRows WHERE NOT Submitted
LET Chosen <= if(condition=ExactScans, then=ExactScans[0],
else=OwnScans[0])
LET SummaryResponse <= SELECT Response, parse_json(data=Content) AS Body
FROM if(condition=Chosen.scan_id =~ "^[0-9a-f-]{36}$", then={
SELECT * FROM Fetch(Url=Base + "/api/v1/scan/" + Chosen.scan_id + "/summary",
Params=dict())
})
LET Summary <= SummaryResponse[0].Body
// A scan without an AI review answers 404, which is not an error.
LET AIResponse <= SELECT Response, parse_json(data=Content) AS Body
FROM if(condition=Chosen.scan_id =~ "^[0-9a-f-]{36}$", then={
SELECT * FROM Fetch(Url=Base + "/api/v1/ai/" + Chosen.scan_id,
Params=dict())
})
LET AI <= AIResponse[0].Body.results.analysis
// total_scans counts pages that contacted the address; primary_ip
// tells which of the sampled pages were served from it.
LET IPResponse <= SELECT Response, parse_json(data=Content) AS Body
FROM if(condition=Solo AND PublicIP, then={
SELECT * FROM Fetch(Url=Base + "/api/v1/search/ip/" + str(str=IP) + "/stats",
Params=dict(`limit`=ScanWindow))
})
LET IPStats <= IPResponse[0].Body
LET HostedRows <= SELECT url FROM foreach(row=IPStats.results)
WHERE primary_ip = str(str=IP)
LET HostedURLs <= SELECT url FROM HostedRows GROUP BY url LIMIT 5
LET Failed <= SELECT Response FROM chain(
a=ExactResponse, b=HostResponse, c=SummaryResponse, d=IPResponse,
e={SELECT * FROM AIResponse WHERE Response != 404})
WHERE Response != 200
LIMIT 1
// An empty list parses to [], an unreadable body to NULL.
LET Unreadable <= SELECT Response FROM chain(
a=ExactResponse, b=HostResponse, c=SummaryResponse, d=IPResponse,
e=AIResponse)
WHERE Response = 200 AND Body = NULL
LIMIT 1
LET Single = SELECT Indicator,
if(condition=ExactScans, then="url",
else=if(condition=OwnScans, then="host")) AS Match,
Summary.security_verdict.verdict AS Verdict,
Summary.risk_score AS RiskScore,
Summary.security_verdict.risk_level AS RiskLevel,
Summary.security_verdict.confidence AS Confidence,
Summary.security_verdict.risk_factors AS RiskFactors,
AI.classification AS AIClassification,
AI.scam_type AS AIScamType,
AI.skipped_reason AS AISkipped,
Chosen.url AS ScannedURL,
Chosen.final_url AS FinalURL,
Chosen.completed_at AS ScannedAt,
if(condition=Chosen.scan_id,
then=Base + "/result/" + Chosen.scan_id,
else=if(condition=IPStats.ip_address,
then=Base + "/ip/" + str(str=IP))) AS ReportURL,
if(condition=NOT IsIP, then=len(list=OwnScans)) AS HostScans,
if(condition=NOT IsIP, then=len(list=SubdomainRows)) AS SubdomainScans,
if(condition=NOT IsIP, then=len(list=RedirectRows)) AS RedirectedHere,
IPStats.total_scans AS IPScans,
if(condition=IPStats.ip_address, then=len(list=HostedRows)) AS HostedScans,
if(condition=HostedURLs, then=HostedURLs.url) AS HostedURLs,
if(condition=IPStats.asn,
then=format(format="%v %v", args=[IPStats.asn, IPStats.asn_org])) AS ASN,
if(condition=IsIP AND NOT PublicIP,
then="Private or reserved IP address; not sent to ScanMalware.",
else=if(condition=NOT IsIP AND NOT ValidHost,
then="Indicator is not a hostname, IP address or http(s) URL.",
else=if(condition=Failed[0].Response = 429,
then="ScanMalware rate limit reached (600 requests per minute); try again later.",
else=if(condition=Failed,
then=format(format="ScanMalware returned HTTP %v.",
args=Failed[0].Response),
else=if(condition=Unreadable,
then="ScanMalware returned an unreadable response.",
else=if(condition=Chosen.scan_id AND NOT Summary.security_verdict.verdict,
then="The newest scan has no verdict yet; run again in a few minutes."))))))
AS Error,
Summary AS _Summary
FROM scope()
WHERE Value
LET Items = SELECT Item FROM chain(
a={SELECT strip(string=Indicator) AS Item FROM Indicators},
b={SELECT strip(string=Indicator) AS Item FROM scope()})
WHERE Item
GROUP BY Item
LET Batch = SELECT * FROM foreach(
row=Items,
query={
SELECT * FROM Artifact.Server.Enrichment.ScanMalware(
Indicator=Item, ScanWindow=ScanWindow, MaxRetryWait=MaxRetryWait)
},
workers=Workers)
SELECT * FROM if(condition=Solo, then=Single, else=Batch)