Debugging Velociraptor

The internal profiles are built in profiles provided by the Golang ecosystem. They are mostly useful for developers but can be collected for Velociraptor as well. Metrics Metrics are counters in the program that are used to collect high level statistics about program execution. Metrics are also exported on the server using the Metrics Server. This is controlled in the configuration file Monitoring section: Monitoring: bind_address: 127.0.0.1 bind_port: 8003 metrics_url: http://localhost:8003/metrics On the server, you can collect monitoring data using curl: Golang These show the built in Golang profiles. These profiles provide detailed instrumentation of the running process. This type of information is critical for developers to understand what the code is doing, and is therefore often required in any bug reports or discussions of suspected bug with the Velociraptor developer team. These can be accessed via the link Internal -> Show built in Go Profiles in the Debug Console. Built in Golang profiles Below we document the most interesting of those.

Datastore Profiles related to the server’s datastore. Replication Reports current replication connections between master and minion. VQL Queries See currently and recently running VQL queries. Active Queries This view shows the queries currently running in this process. For example queries will run as part of notebook evaluation, currently installed event queries, or currently collecting artifacts (in the case of the offline collector). Active Queries profile The above example shows a number of queries watching a variety of event logs on the endpoint. This is because this endpoint is running the Windows.Hayabusa.Monitoring artifact, which evaluate many Sigma rules, many referring to different log sources. Each log source relies on parsing the event logs. Plugin Monitor At their core VQL queries process rows emitted from VQL plugins. We have seen previously the Active Queries tracker which provides information on currently running queries. However it is also useful to know what plugins are currently running and what parameters are used within them. This gives us a really good idea what the VQL engine is doing exactly at the moment. Plugin tracker profile The above example show the plugins currently active. We see a few instances of watch_monitoring(). Another instance of watch_etw() plugin is seen watching the Sysmon ETW stream. Finally we see some instances of watch_evtx() watching various event logs. Recent Queries The Recent Queries profile similarly shows all recent queries (even after they completed). This helps us understand what queries had run on the endpoint previously, and how long they took to complete. In the VQL profile category there are also profiles to show all recent queries (even ones that have completed already). This helps us to understand what exactly the client was recently doing. Plugins Much of Velociraptor functionality is implemented using VQL plugins. Some plugins are very sophisticated and require their own profile tracking. ETW Glob NTFS Cache Sigma Tracker Windows Event Log Watcher Zip Process Tracker Sqlite Services Velociraptor contains many service modules that help the process perform certain tasks. These usually contain specific profiles to show how they are performing. ExportContainers The Velociraptor GUI allows exporting collections from Flows or Hunts into a Zip file. If the collection is very large this can take some time. While the GUI shows some progress information: Export Progress as shown by the GUI The profile shows a lot more information: Throttler An important criteria for Velociraptor is to ensure the impact on endpoint performance is limited. This helps in cases when we need to perform intensive tasks on the endpoint. We want to make sure the end system is still usable and reduce our impact on the end user. Velociraptor achieves this by implementing a Throttler. This mechanism is able to pause query execution when the process’s average CPU utilization exceeds some limit. User Manager Reporting information about current users registered on the system. Open-close Similarly to tempfiles, it is important to know what files we have opened that should be closed. This is tracked by the Open Close Tracker. The tracker does not only track operating system files but other abstract objects within Velociraptor that need to be closed. Open Closed profile We typically want to see all files being suitably closed unless they are used currently. Some files are held open for a short time after use, to avoid needing to re-open them if accessed soon after. tempfiles Velociraptor uses temporary files for a variety is purposes. It is important to ensure that whenever we create a temporary file, we suitably remove it. The Tempfile tracker keeps track of temporary files we used. Temporary file profile The profile indicates: Which temporary files were used. Where they were created from (gives an idea why we created these files). When the file was created and removed In the above example, we see two temporary files created from the VQL tempfile() function and one created by the VQL engine during a materialize operation (e.g. expanding a LET ). All files were suitably closed as determined by the non zero destroyed time. worker Notebooks are very useful feature of the server allowing for complex post-processing of collected data. Sometimes these queries are very large and take a long time to run. To limit the amount of resources the queries can take on the server, Velociraptor only creates a limited number of notebook workers (by default 5). Inspecting the notebook workers

Profiles for services associated with each org. Services Broadcast Track generators installed via the generator() plugin. QueueManager Report the current states of server artifact event queues. VFS The VFS service post processes results from VFS operations. Notifier Information about directly connected clients.

These profiles only exist on the client. You can see those in the debug server by adding the --debug flag (you can also add the --debug_port flag to set a different port). The client will by default serve the debug server over http://localhost:6060/. Monitoring The client monitoring profile shows current information on the client event monitoring subsystem. Clients receive a Client Event Table update from the server, instructing them on a set of CLIENT_EVENT artifacts to run. The results from these artifacts are streamed back to the server in near-realtime. Client event queries are stored in the client’s writeback file so they can start immediately as soon as the client boots, even if it is not connected to the server or offline. While offline the results are buffered locally to a file, and then synced with the server at the next opportunity. Flows The Client Flows profile shows information about currently executing collections on the endpoint. It is most interesting to collect when another flow is executing on the endpoint. Client Flow Profile The profile shows the progress of the collection on the endpoint: The flow id is used to identify the flow. The duration of the collection since it was last started How many bytes were uploaded to the server? How many rows were returned to the server? These are generally the same stats the server keeps but are from the point of view of the client.