Splunk App & Technology Add-On [Current Version]

Updated 1 month ago by Elvis Hovor

INTRODUCTION

This article provides a description of the Splunk App built for TruSTAR and a step by step guide to install, setup and troubleshoot that app.

The Splunk App allows users to use context from TruSTAR’s IOCs and incident reports within their Splunk analysis workflow. TruSTAR arms security teams with high-signal intelligence from sources such as internal historical data, open and closed intelligence feeds, and anonymized incident reports from TruSTAR’s vetted community of enterprise members.

FEATURES

  1. Dashboard displaying IOCs and reports from TruSTAR that match log and event data stored in your Splunk indexes.
  2. View TruSTAR reports in the Splunk app and launch IOC search and investigations against Splunk data.
  3. SplunkES capability to generate notable events from matched data.

WORKFLOW DIAGRAM

PRE-REQUISITES

The details below summarizes the prerequisites and requirements needed for the TruSTAR Splunk app to work. Please make sure below components are downloaded/available.

Splunk Enterprise 6.6.0 or above. 

Splunk Enterprise can be downloaded from here: https://www.splunk.com/en_us/download/splunk-enterprise.html

To install Splunk Enterprise, follow guidelines given in below link: http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/InstallSplunk

Set environment variable for Splunk Home

Environment Variable: Set environment variable SPLUNK_HOME by typing the following command at the command prompt:  

     export  SPLUNK_HOME=__________________________[insert path to Splunk folder]_____________________

In OS X, the Splunk folder path is usually:      /Applications/Splunk/
In Ubuntu Linux, the Splunk folder path will likely be:   /opt/Splunk/

DOWNLOAD FILES

For Splunk Enterprise instances:

These bundles are required to successfully install the TruSTAR app on Splunk Enterprise instances. Note: These bundles can only be installed on Splunk Enterprise instances. The TruSTAR app and TA for a Splunk Cloud instance must be downloaded from the Splunkbase website (see

#

Bundle Name

Description

1

Technology Add-on for TruSTAr

This bundle will fetch reports and IoC data from TruSTAR using modular input and indexes it, after which users can search it using the Splunk search tool. This bundle needs to be installed before installing the next bundle. Current version 1.0.7

2

TruSTAR App for Splunk

This bundle contains the dashboards that display data received from TruSTAR Station. Current version 1.0.6

For Splunk Cloud instances:

Installation bundle files to be used with a Splunk Cloud instance must be downloaded from the Splunkbase website here:

TruSTAR App: https://splunkbase.splunk.com/app/3678/

TruSTAR Technology Add-On: https://splunkbase.splunk.com/app/3679/


INSTALLATION LOCATIONS

Single-Instance Splunk Enterprise Deployment

In a single-server Splunk deployment, a single instance of Splunk Enterprise serves as the data-collection node, indexer, and search head all in one. In such scenarios, install both the TruSTAR Application and Technology Add-On on this instance.

Multi-Instance Distributed Splunk Enterprise Deployment

In a distributed deployment, Splunk Enterprise is installed on at least two instances.  One node functions as the search head the remaining nodes serve as indexers and data-collection nodes. The TruSTAR Application should only be installed on the search head node. The TruSTAR Technology Add-on needs to be installed on all indexer and data-collection nodes.

In a distributed environment, please note following points: 
If you have a separate data-collection nodes, please ensure they are running the full Splunk Enterprise version.

Managed Splunk Cloud Deployment

In a managed Splunk Cloud deployment, the data indexing will take place on the cloud instance. The data collection however can take place on an on-premisis Splunk Enterprise instance used as a heavy forwarder.

Splunk cloud users can only download the integration from splunkbase after it has been cloud certified. Because the app is not currently in splunkbase, Splunk cloud users will not be able to use the new Splunk integration from TruSTAR

INSTALLATION INSTRUCTIONS

Installation through web user-interface for both Splunk Enterprise and Splunk Cloud Instances

For Manual Downloads:

Download the TruSTAR Technology add-on and TruSTAR app bundles. Download the bundle from the "manual installation section" above.

After successfully downloading follow these steps:

  1. Select Apps -> Manage Apps from the main menu bar.
  2. Click on Install app from file button
  3. First upload the Technology Add-on for TruSTAR file.
  4. Next upload the TruSTAR App for Splunk file.
  5. After successfully uploading the two files go to the App Configuration section.

Installation through Command Prompt for Splunk Enterprise Instances only

To install from the command window, go to $SPLUNK_HOME/bin folder and execute following command:

cd $SPLUNK_HOME
./splunk install app TA-trustar.spl
./splunk install app Trustar.spl

Configure the TruSTAR Technology Add-Ons (TA)

Follow these instructions for all Technology Add-Ons in your architecture.

  1. Login to your Splunk node.   
  2. If desired, create a new index in which to store the data the TA will import from TruSTAR Station (TruSTAR recommends doing this, but it's not required). By default, the TA will store data in the "main" index.
  3. Go to Settings->Data inputs.
  4. Select "TruSTAR Configuration" on the next page.
  5. Fill in the configuration details (see Table below for more details).
  6. Select "Enable Data Collection" to begin ingesting data from the TruSTAR enclaves specified in the "Enclave IDs" box.
    No indicators or reports will be ingested into Splunk if "Enable Data Collection" is not selected.

Click on 'More Settings' if you want to customize the index in which the TA stores data imported from TruSTAR and/or how often the TA polls your enclaves for new data (see table below). Leave the default values if unfamiliar with these settings.

Configuration Parameter

Required

Description

Rest Input Name

Yes

The name of rest modular input. This can be any name you like, but each Modular Input's name must be unique.

URL to Connect

Yes

Use https://station.trustar.co This is TruSTAR station URL from where data get collected by executing API calls.

API Authentication Key

Yes

Your TruSTAR API Key. The App and TA use this for making API calls. Find this key in the TruSTAR Station web interface under Settings-> API. How to find your API Key

It will be in clear text at the time of new modular input creation.

On save of Modular input, Authentication key will get stored at /storage/password entity of Splunk in encrypted format.

On edit of modular input this field will be blank.

API Secret

Yes

Your TruSTAR API Secret. It will be used for making API calls. Available under Settings-> API on TruSTAR Station. How to find your API Secret

It will be in clear text at the time of new modular input creation.

On save of Modular input, Secret key will get stored at /storage/password entity of Splunk in encrypted format.

On edit of modular input this field will be blank.

Date (UTC in "YYYY-MM-DD hh:mm:ss" format)

Optional

Submission date/timestamp for the oldest report you want to import into Splunk. Leaving this blank will result in importing data for the past 90 days prior to the moment you successfully save these configurations.

SSL Certificate Path

Optional

Path of SSL Certificate, which will be used while executing any API request to TruSTAR station. No need to give path in case of CA signed certificate.

Enable Data Collection

Yes 

Enabling data collection will cause the TA to begin importing data from the TruSTAR enclaves specified in the Enclave IDs field.


Enclave IDs

Yes

Enter Enclave ID's (alphanumeric id next to enclave name in TruSTAR Station) from which to import data. If you want to import data from multiple enclaves, separate each enclave ID with a comma. Retrieving your Enclave IDs

HTTPS Proxy Address

Optional

Proxy address to use for communication with the TruSTAR station, e.g. http://10.10.1.10
**Note: we have seen users have trouble with ZScaler proxies.**

HTTPS Proxy Port

Optional

Proxy port to use for communication with the TruSTAR station e.g. 3128

HTTPS Proxy Username

Optional

Proxy username. Your system administrators / helpdesk should be able to give you this.

HTTPS Proxy Password

Optional

Proxy password. Your system adminstrators / helpdesk should be able to give you this.

Enclave types for fetching Priority Score

Optional

Default: INTERNAL
Possible values: INTERNAL, OPEN, CLOSED. If multiple types, use comma(,) separated values.

Interval

Optional

Polling interval in seconds. This is the amount of time (in seconds) that the TA will wait before again polling the TruSTAR enclaves whose IDs you entered in the "Enclave IDs" box for new indicators / reports to import into Splunk.

Set sourcetype

Optional

Standard field of Splunk with options: automatic, manual. Default value in our case is automatic. We have kept validation of not allowing user to change from UI, as we give SourceType from code.

Index

Optional

This parameter allows user to decide which index to be used for TruSTAR data. User needs to ensure that index is already present in the Splunk environment (should have been created as part of "Configure TruSTAR Technology Add-ons" Step 1).  If no value is provided, by default the TA will import data from TruSTAR Station to the “main” index. If the user changes this setting from the Default, the user MUST follow the "Change Macro Definition" instructions below.

Click on Next button on top, after adding each value for modular input in the form.

Change Macro Definition

Note: If you did not make any changes for destination index and want your TruSTAR data to be imported into the "main" index, skip this section.

If you customized the destination index you will need to follow these steps.

  1. Open the Splunk user-interface on the the Splunk Search Head.
  2. Go to Settings-> Advanced search-> Search macros.
  3. Select "TruStar App for Splunk" in App Context dropdown.
  4. Modify `trustar_get_index` macro definition with index=”<new index name>”.
NOTE: There is no need to modify `trustar_get_index_and_sourcetype` macro.
If index=default has been selected in modular input, then update macro definition with index=main (as default index of Splunk is main)

Usage & App Commands

Dashboard

The TruSTAR app dashboard shows count of Reports and Indicators Imported and Matched in All time and Last 4 hours.

Below are details of the panels in this dashboard:

  1. Matched Data :This panel displays 4 single values of Matched data.
    1. Count of Matched Reports in Last 4 hours and a trend arrow showing the count from the previous 4 hours.
    2. Count of Matched Indicators in Last 4 hours and a trend arrow showing the count from the previous 4 hours.
    3. Count of Matched Reports in All time.
    4. Count of Matched Indicators in All time.

  2. Imported Data: This panel displays 4 single values of Imported data.
    1. Count of Imported Reports in Last 4 hours and trend of count with previous 4 hours.
    2. Count of Imported Indicators in Last 4 hours and trend of count with previous 4 hours.
    3. Count of Imported Reports in All time.
    4. Count of Imported Indicators in All time.
Reports Tab

This screen displays Report details like name, creation time, distribution, last scan and result count of matched for specific report.

  1. Report Details: User can see this dashboard on drill-down of Report Name from TruSTAR Reports dashboard. It displays all details of specific report like name, total indicators count, report body and table of all related indicators. User can investigate the indicator in raw events and also perform actions like marking an IOC as false positive so that it is not considered an IOC in future matches.
Indicators Tab

This screen displays basic details of indicators like time of download, value, count of co-related reports, status, count of matched reports. Also user can perform actions like investigate IOC in raw events and mark an IOC as false positive so that it not considered in future matches.

Settings
  1. Match Configuration: We can configure the attributes for Matching the events like:
  2. Index : Index to consider to matching the TruStar events.
  3. Timerange(in days) : Timerange for the data to be matched.(e.g If one wants to consider last 2 days of events for matching, this property should be configured as 2)
  4. Enclaves: Enclaves to consider for matching against TruStar events.
    1. Submit Enclaves Configuration:Enclaves to which the TruStar submission should happen while using AR and workflow action.

Splunk ES Setup & Configuration

trustar_get_match_reports correlation search is part of TA_trustar. By default, its in a disabled state. User has to enable it to generate notable events from matched events.

Adaptive Response Actions

TA_trustar has Submit report adaptive response action implemented. Once AR action is executed, it will submit report to TruSTAR and index the response in Splunk. It will index AR action response in default main index only.

TruStar Match Report

trustar_ioc_report.py

Technical Details

Deployment Architecture

The TruStar App for Splunk allows users to utilize the context of the TruStar platform's IOCs and incidents within their Splunk workflow. TruStar arms security teams with the highest signal intelligence from sources such as internal historical data, open and closed intelligence feeds and anonymized incident reports from TruStar's vetted community of enterprise members

Below is the topology of data collection from TruStar station to Splunk in distributed and standalone environment.

Stand-alone Splunk Deployment

In case of deploying this App on Stand-alone Splunk Deployment, User would have to install TA-trustar and Trustar App for Splunk both on Splunk instance and then configure theta to start fetching data from TruStar Station.

Distributed Splunk Deployment

In case of deploying TruStar App for Splunk on distributed setup, Following are the changes needed on each type of node.

Splunk Heavy Forwarder: On Splunk Heavy forwarder, Install TA-TruStar and configure using TruStar credentials.

 Splunk Indexer Cluster: On Splunk Indexer cluster, User would have to define specific index in case user don’t want to use default index (main) or in case user have already defined Index on Splunk Heavy Forwarder.

Splunk Search-Head Cluster: On Splunk Search Head Cluster, User would install the App & TA of TruStar App for Splunk.

App Architecture

This section describes the overall App architecture.

Indexes

Access Path: Settings → Indexes

TruSTAR App for Splunk can populate the panels based on the index defined while indexing data into the Splunk. By default data will get populated under “main” index until it’s changed while configuring data input.

Splunk recommends using Splunk’s default index (that is “main” index) for simplicity and reusability.

Refer below URL to create custom index.

Reference URL: http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Setupmultipleindexes

Note: In case changes are done in Index name, Please follow steps mentioned under macro section.

Source-Types

Access Path:  Settings  → Source Types

Source-Type are default Splunk fields to categorize and filter the indexed data to narrow down the search results. Since TruSTAR app collects two different types of data from Trustar Station, it has been indexed in below source types.

Below is the table, which shows alerts and activities data are separated.

Source-Type

Description

trustar:reports

This contains all the reports sent from TruSTAR station to Splunk using rest API call.

trustar:indicators

This contains all the indicators sent from TruSTAR station to Splunk using rest API call.

Macros

Access Path: Settings  → Advanced Search  → Search Macros

All the visualizations in TruSTAR App for Splunk are referred by a “trustar_get_index” macro, which helps App to identify the Index in which data is getting indexed.

By default, it’s referred to “main” index and in case user is changing the Index value then same changes has to be done in the macro.

TruSTAR App for Splunk has another macro called  “trustar_get_index_and_sourcetype”, which helps App to identify the index and sourcetype in which indicators of TruSTAR app should be matched.

By default, it’s referred to  index=* and in case user has some specific index and source type to consider to find matches,then it should be updated in the macro.

KNOWN LIMITATIONS

There is a known limitation in Splunk where App Icon doesn’t get visible before restarting Splunk. Hence, it’s recommended to restart post installation of the App to load the App Icon.

In Splunk Modular input that in case of failure it doesn’t show proper raised error message, but shows generic failure message on UI in windows machine.

In the case of Splunk v7.1.x, the Whitelisted Input Dropdown of Indicators Dashboard wouldn't work for ‘All’ option. Workaround is to select either ‘Yes’ or ‘No’ and filter specific data.

In the case of users on Splunk v7.0.3, the matching count for SHA256 type of indicators wouldn’t be considered.

TROUBLESHOOTING / FAQs

Q: How long does it take to setup Integration ?

A: Splunk Integration setup can take anywhere from 20 - 60 mins. This is dependent on the splunk environment and whether it is a standalone or distributed environment

Q: How do you delete/reinstall/upgrade the TruSTAR instance ?

A: User can upgrade TruSTAR app and TA through CLI or UI.

Upgrade through CLI:

  • Download tar of App or TA from Splunk base
  • Stop Splunk server
  • $SPLUNK_HOME/bin/splunk 
  • Install app APP_NAME.tgz –update 1 –auth username:password
  • Start Splunk Server

Upgrade through UI:

  • Click on Manage Apps
  • Find Trustar app And TA entry from list
  • Click on link of newer version under version column on related entry

Manual Upgrade:

  • Click on Manage Apps
  • Click on Install App from file
  • Locate Trustar TA file from local drive
  • Select to Upgrade app
  • Click Upload 

Delete old app and add-on from backend:

  • Go to $SPLUNK_HOME/etc/apps/ and remove TA-trustar and Trustar 
  • Restart Splunk

Q: After completing installation of application, the dashboards did not start populating data what do i do next?

A: Confirm that you have modified macro `trustar_get_index` with indexes selected while creating Modular input. For example, If all modular input entries have index=default, then update macro definition with index=main and save. If any specific index has been set in modular input then add it in macro definition.

  • Check following query to verify data is getting indexed into Splunk
search `trustar_get_index` | stats count by sourcetype
  • Verify that SPLUNK_HOME is pointing to correct Splunk directory.
  • Look for errors in trustar_modinput.log file.  This file will be available under $SPLUNK_HOME/var/log/trustar folder.
  • Check the modular input In case API Key or Secret Key of TruSTAR gets modified after setup in modular input.

User can update it from Modular input UI

  • Go to Settings-> Data Inputs -> TruSTAR Configuration
  • Open specific TruStar station entry, and enter new Authentication key and Secret Key in both fields.
  • On click of save, modified key will get updated for that specific TruStar Station.

Q: Can i build the App from source code?

A: Any splunk application can be built to tar file with .spl extension.
Follow steps below to build both TA and App from any Linux distribution.

tar cv <app_name> > <app_name>.tar

gzip <app_name>.tar

mv <app_name>.tar.gz <app_name>.spl

(Replace <app_name> with the name of app. For eg. TA-trustar or Trustar)

Please check out our comprehensive Splunk FAQ's here.
For other questions reach out to support@trustar.co for any additional questions.


How Did We Do?