Elastic.Flows.Upload

This server side event monitoring artifact waits for new artifacts to be collected from endpoints and automatically uploads those to an elastic server.

We use the artifact name as the name of the index. This allows users to adjust the index size/lifetime according to the artifact it is holding.

NOTE: Elastic is a database and still must have a stable schema. This means that artifacts that produce inconsistant columns and types will NOT work as expected. What will happen is that the first row that is inserted will create the Elastic database schema (In Elastic terminology “mapping”) and then any subsequent row with a different type for these fields will be rejected by Elastic.

In particular this does not work with event logs because event logs have a varied schema (The EventData field is a free form field depending on the event log itself). Therefore forwarding event log data to Elastic with this artifact will cause Elastic to drop many events!! This artifact is not suitable for forwarding Windows Event Logs!

name: Elastic.Flows.Upload
description: |
  This server side event monitoring artifact waits for new artifacts
  to be collected from endpoints and automatically uploads those to an
  elastic server.

  We use the artifact name as the name of the index. This allows users
  to adjust the index size/lifetime according to the artifact it is
  holding.

  NOTE: Elastic is a database and still must have a stable
  schema. This means that artifacts that produce inconsistant columns
  and types will **NOT** work as expected. What will happen is that
  the first row that is inserted will create the Elastic database
  schema (In Elastic terminology "mapping") and then any subsequent
  row with a different type for these fields will be rejected by
  Elastic.

  In particular this does not work with event logs because event logs
  have a varied schema (The EventData field is a free form field
  depending on the event log itself). Therefore forwarding event log
  data to Elastic with this artifact will cause Elastic to drop many
  events!! This artifact is not suitable for forwarding Windows Event
  Logs!

type: SERVER_EVENT

parameters:
  - name: ArtifactNameRegex
    default: .
    type: regex
    description: Only upload these artifacts to elastic
  - name: elasticAddresses
    default: http://127.0.0.1:9200/
  - name: Username
  - name: Password
  - name: APIKey
  - name: DisableSSLSecurity
    type: bool
    description: Disable SSL certificate verification
  - name: Threads
    type: int
    description: Number of threads to upload with
  - name: ChunkSize
    type: int
    description: Batch this many rows for each upload.
  - name: CloudID
    description: The cloud id if needed
  - name: RootCA
    description: |
      A root CA certificate in PEM for trusting TLS protected Elastic
      servers.

sources:
  - query: |
      LET completions = SELECT * FROM watch_monitoring(
             artifact="System.Flow.Completion")
             WHERE Flow.artifacts_with_results =~ ArtifactNameRegex
      LET organization <= org().name

      LET documents = SELECT * FROM foreach(row=completions,
          query={
             SELECT * FROM foreach(
                 row=Flow.artifacts_with_results,
                 query={
                     SELECT *, _value AS Artifact,
                            client_info(client_id=ClientId).os_info.hostname AS Hostname,
                            timestamp(epoch=now()) AS timestamp,
                            ClientId, Flow.session_id AS FlowId,
                            "artifact_" + regex_replace(source=_value,
                               re='[/.]', replace='_') as _index,
                            organization as Organization
                     FROM source(
                        client_id=ClientId,
                        flow_id=Flow.session_id,
                        artifact=_value)
                 })
          })

      SELECT * FROM elastic_upload(
            query=documents,
            threads=Threads,
            chunk_size=ChunkSize,
            addresses=split(string=elasticAddresses, sep=","),
            index="velociraptor",
            password=Password,
            username=Username,
            cloud_id=CloudID,
            api_key=APIKey,
            root_ca=RootCA,
            disable_ssl_security=DisableSSLSecurity,
            type="artifact")