Generic.File.LineScan.RegexHunter

A versatile artifact that scans the contents of text files line by line and reports any line that matches a user-supplied regular expression. This artifact is designed for flexible content inspection across any operating system— Linux, Windows. It can be used to detect indicators of compromise, identify suspicious commands in configuration files, or validate specific text patterns.

The user can supply one or more glob paths and a custom regex pattern. Matching lines are returned with their file names, and—if enabled—the corresponding files can be uploaded for further analysis.

Includes:

• SkipPathRegex → exclude specific files from scanning in the paths.
• WhitelistLineRegex → ignore specific line matches even if they match the main detection regex.

name: Generic.File.LineScan.RegexHunter
author: Kaizar Lehri
description: |
  A versatile artifact that scans the contents of text files line by line and reports 
  any line that matches a user-supplied regular expression.
  This artifact is designed for flexible content inspection across any operating system—
  Linux, Windows. It can be used to detect indicators of compromise, identify 
  suspicious commands in configuration files, or validate specific text patterns.

  The user can supply one or more glob paths and a custom regex pattern. 
  Matching lines are returned with their file names, and—if enabled—the corresponding 
  files can be uploaded for further analysis.

  Includes:
  - **SkipPathRegex** → exclude specific files from scanning in the paths.
  - **WhitelistLineRegex** → ignore specific line matches even if they match the main detection regex.

type: CLIENT

parameters:
  - name: PathGlobs
    type: csv
    description: "List of glob paths to search."
    default: |
      Glob
      /etc/crontab
      /etc/cron.d/*
      /etc/cron.daily/*
      /etc/cron.hourly/*
      /etc/cron.weekly/*
      /etc/cron.monthly/*
      /var/spool/cron/*
      /var/spool/cron/crontabs/*

  - name: SuspiciousRegex
    type: regex
    description: "Regex pattern to detect potentially suspicious commands or payloads."
    default: "(curl|wget|bash|/tmp/|/var/tmp/|/dev/shm|base64|eval|python|perl|crontab -e|@reboot)"

  - name: SkipPathRegex
    type: regex
    description: |
      Exclude specific files in the paths or just the paths from Scanning.
    default: "/etc/cron.daily/apt-compat"

  - name: WhitelistLineRegex
    type: regex
    description: |
      Regex pattern to exclude specific lines (known benign content).
      Matching lines will be ignored even if they match SuspiciousRegex.
    default: "^$"

  - name: UploadMatches
    type: bool
    description: "If true, upload files that contain regex matches."
    default: false

sources:
  - query: |
      -- Step 1: Gather all candidate files
      LET found_files = SELECT OSPath, Name, Size
      FROM glob(globs=PathGlobs.Glob, nosymlink=true)
      WHERE NOT IsDir AND NOT IsLink

      -- Step 2: Skip files whose paths match SkipPathRegex
      LET filtered_files = SELECT OSPath, Name, Size
      FROM found_files
      WHERE NOT OSPath =~ SkipPathRegex

      -- Step 3: Scan each remaining file line-by-line for suspicious regex matches
      LET regex_hits = SELECT *
      FROM foreach(
        row={ SELECT OSPath FROM filtered_files },
        query={
          SELECT
            OSPath AS FileName,
            Line AS LineMatch
          FROM parse_lines(filename=OSPath)
          WHERE Line =~ SuspiciousRegex
        }
      )

      -- Step 4: Remove benign line matches
      LET filtered_hits = SELECT *
      FROM regex_hits
      WHERE NOT LineMatch =~ WhitelistLineRegex

      -- Step 5: Optionally upload matching files
      LET upload_hits = SELECT
        FileName,
        LineMatch,
        upload(file=FileName, name=FileName) AS UploadedFile
      FROM filtered_hits

      -- Step 6: Return results with or without upload
      SELECT *
      FROM if(
        condition=UploadMatches,
        then=upload_hits,
        else=filtered_hits
      )