Splunk V2

Updated 2 weeks ago by Sachit Soni

Introduction

This article provides a description of the Splunk App built for TruSTAR and a step by step guide to install, setup and troubleshoot the TruSTAR App for Splunk.

The Splunk App allows users to utilize context of TruSTAR’s IOCs and incidents within their Splunk analysis workflow. TruSTAR arms security teams with the high signal intelligence from sources such as internal historical data, open and closed intelligence feeds and anonymized incident reports from TruSTAR’s vetted community of enterprise members.

Workflow Illustration

Prerequisites

The details below summarizes the prerequisites and requirements. Please make sure below components are downloaded/available.

  • Splunk Enterprise 6.6.0 or above. 
Environment Variable: Set environment variable SPLUNK_HOME.

App Installation

The following bundles are required for successful install of the TruSTAR app.

#

Bundle Name

Description

1

Technology Add-on for Trustar

This bundle will fetch reports and IoC data from TruSTAR using modular input and indexes it which then can be searched. This bundle will need to be installed first.

2

TruSTAR App for Splunkhttps://trustar.helpdocs.io/app/content/article/zsgux8lk9e#


This bundle will have all the dashboards to display data received from TruSTAR Station.


Installation

    The installation process for TruSTAR app will be different based on your Splunk environment.

    1. Download the TruSTAR Technology add-on and  TruSTAR app bundles 

    After successfully downloading follow these steps:

    1. Select Apps -> Manage Apps from the main menu bar.
    2. Click on Install app from file button
    3. First upload the Technology Add-on splunk file.
    4. Next upload the TruSTAR App file.
    5. After successfully uploading the two files go to the App Configuration section.

    Installation on Standalone Splunk Deployment

    In a single server deployment, single instance of Splunk Enterprise works as data collection node, indexer and search head. In such scenarios, install both TA-trustar and TruSTAR applications on this node. Complete the setup of TA-trustar for starting data collection.   

    Installation on Distributed Splunk Deployment

    In a distributed deployment, Splunk Enterprise is installed on at least two instances.  One node works as search head and other node works as indexer and data collection node. The TruSTAR Application only needs to be installed on the search head node. The TA-TruSTAR addon needs to be installed on all indexer and data collections nodes.

    In a distributed environment, please note following points: 

    • If you have a separate data collection node, please ensure it’s running the full Splunk Enterprise version.

    Complete the setup of the TA-trustar on data collection node(s) only. You do not need to do a setup on search head node.

      Installation on Splunk Cloud

      In Splunk Cloud, the data indexing will take place in cloud instance. The data collection can take place in on premise Splunk instance that will work as heavy forwarder.

      Installation from Command Prompt

      To install from the command window, go to $SPLUNK_HOME/bin folder and execute following command:

      ./splunk install app TA-trustar.spl
                ./splunk install app Trustar.spl

      App Configuration

      To complete the installation process, you must first complete setup for TA-trustar.

      Setup process for TA-trustar is as follows:

      • Login into your Splunk instance, or in case of distributed Splunk the data collection node.   
      • Click on TruSTAR app from left bar.
      • Go to Settings->Data inputs.
      • Select TruSTAR Configuration on the next page.
      • Fill the configuration details (see Table below for more details).
      • Select enable data collection to start ingestion of data from enclaves in TruSTAR
      No indicators or reports will be ingested into Splunk if enable data collection is unselected
      • List the IDs of the enclaves you want to ingest data from in the Enclave IDs field separated by commas for multiple IDs
        • You can find the enclave ids in the TruSTAR platform in your profile under settings
      https://demo.trustar.co/settings/profile

      • Click on More Settings if you want to customize index name and polling frequency. Leave it at default values if unsure.

      Input Parameter

      Required

      Description

      Rest Input Name

      Yes

      The name of rest modular input. This must be unique for each Modular Input

      URL to Connect

      Yes

      Use https://station.trustar.co This is TruSTAR station URL from where data get collected by executing API calls.

      API Authentication Key

      Yes

      Authentication Key to connect to TruSTAR station. It will be used for making API calls. Available under Settings-> API on TruSTAR Station.


      It will be in clear text at the time of new modular input creation.

      On save of Modular input, Authentication key will get stored at /storage/password entity of Splunk in encrypted format.

      On edit of modular input this field will be blank.

      API Secret

      Yes

      Secret Key to connect to TruSTAR station. It will be used for making API calls. Available under Settings-> API on TruSTAR Station.


      It will be in clear text at the time of new modular input creation.

      On save of Modular input, Secret key will get stored at /storage/password entity of Splunk in encrypted format.

      On edit of modular input this field will be blank.

      Date (UTC in "YYYY-MM-DD hh:mm:ss" format)

      Optional

      Date since when you want to fetch reports from TruSTAR Station during first polling. Default will be 90 days ago.

      SSL Certificate Path

      Optional

      Path of SSL Certificate, which will be used while executing any API request to TruSTAR station. No need to give path in case of CA signed certificate.

      Enable Data CollectionYes Enabling data collection will start the polling of data from the TruSTAR enclaves specified.

      Enclave IDs
      YesEnter Enclave ID's (alphanumeric id next to enclave name in TruSTAR- Station) to pull data from. If you want to pull data from multiple ID's, separate each ID with a comma. 

      HTTPS Proxy Address

      Optional

      Proxy address to use for communication with the TruSTAR station, e.g. http://10.10.1.10

      HTTPS Proxy PortOptionalProxy port to use for communication with the TruSTAR station e.g. 3128
      HTTPS Proxy UsernameOptionalProxy username to use for communication with the TruSTAR staiton, e.g. admin
      HTTPS Proxy PasswordOptionalProxy password to use for communication with the TruSTAR station, e.g. pass
      Enclave types for fetching Priority ScoreOptionalDefault: INTERNAL
      Possible values: INTERNAL, OPEN, CLOSED. If multiple types, use comma(,) separated values.

      Interval

      Optional

      Interval in seconds. All API calls will get executed on this TruSTAR station at specified interval.

      Set sourcetype

      Optional

      Standard field of Splunk with options: automatic, manual. Default value in our case is automatic. We have kept validation of not allowing user to change from UI, as we give SourceType from code.

      Index

      Optional

      This parameter allows user to decide which index to be used for TruSTAR data. User needs to ensure that index is already present in the Splunk environment.  If no value is provided, by default it will use “main” index.

      Click on Next button on top, after adding each value for modular input in above mentioned form. Modify `trustar_get_index` macro definition by going to Settings-> Advanced search-> Search macros. It must have query on selected index in modular input. If index=default has been selected in modular input, then update macro definition with index=main (as default index of Splunk is main)

      Change Macro Definition

      Note: If you did not make any changes for destination index in App Configuration then skip this section.

      If you change the destination index for source from “default” you will need to follow these steps.

      1. Open the Search Head UI.
      2. Go to Settings-> Advanced search-> Search macros.
      3. Select "TruStar App for Splunk" in App Context dropdown.
      4. Modify `trustar_get_index` macro definition with index=”<new index name>”.

      NOTE: There is no need to modify `trustar_get_index_and_sourcetype` macro.

      Using the App

      Dashboard

      TruSTAR app consists of following dashboards.The dashboard shows count of Reports and Indicators Imported and Matched in All time and Last 4 hours.

      Below are details of the panels in this dashboard:

      1. Matched Data :This panel displays 4 single values of Matched data.
        1. Count of Matched Reports in Last 4 hours and a trend arrow showing the count from the previous 4 hours.
        2. Count of Matched Indicators in Last 4 hours and a trend arrow showing the count from the previous 4 hours.
        3. Count of Matched Reports in All time.
        4. Count of Matched Indicators in All time.


      2. Imported Data: This panel displays 4 single values of Imported data.
        1. Count of Imported Reports in Last 4 hours and trend of count with previous 4 hours.
        2. Count of Imported Indicators in Last 4 hours and trend of count with previous 4 hours.
        3. Count of Imported Reports in All time.
        4. Count of Imported Indicators in All time.
      Reports Tab

        This screen displays Report details like name, creation time, distribution, last scan and result count of matched for specific report.

        Report Details

        User can see this dashboard on drill down of Report Name from TruSTAR Reports dashboard. It displays all details of specific report like name, total indicators count, report body and table of all related indicators. User can investigate the indicator in raw events and also perform actions like marking an IOC as false positive so that it is not considered it in future matches.

        Indicators Tab

        This screen displays basic details of indicators like time of download, value, count of co-related reports, status, count of matched reports. Also user can perform actions like investigate IOC in raw events and mark an IOC as false positive so that it not considered in future matches.

        Settings

        Match Configuration

        We can configure the attributes for Matching the events like:

        Index : Index to consider to matching the TruStar events.

        Timerange(in days) : Timerange for the data to be matched.(e.g If one wants to consider last 2 days of events for matching, this property should be configured as 2)

        Enclaves: Enclaves to consider for matching against TruStar events.

        Submit Enclaves Configuration

        Enclaves to which the TruStar submission should happen while using AR and workflow action.

        Correlation Search

            trustar_get_match_reports correlation search is part of TA_trustar. By default, its in disabled state. User has enable it to generate notable events from matched events.

        Adaptive Response Actions

        TA_trustar has Submit report adaptive response action implemented. Once AR action will be executed, it will submit report to TruStar and index the response in Splunk. It will index AR action response in default main index only.

        TruStar Match Report

        trustar_ioc_report.py

        Deployment Architecture

        The TruStar App for Splunk allows users to utilize the context of the TruStar platform's IOCs and incidents within their Splunk workflow. TruStar arms security teams with the highest signal intelligence from sources such as internal historical data, open and closed intelligence feeds and anonymized incident reports from TruStar's vetted community of enterprise members

        Below is the topology of data collection from TruStar station to Splunk in distributed and standalone environment.

        Stand-alone Splunk Deployment

        Figure 1 TruStar App Splunk Standalone Deployment

        In case of deploying this App on Stand-alone Splunk Deployment, User would have to install TA-trustar and Trustar App for Splunk both on Splunk instance and then configure theta to start fetching data from TruStar Station.

        Distributed Splunk Deployment

        Figure 2 Distributed Splunk Deployment Setup

        In case of deploying TruStar App for Splunk on distributed setup, Following are the changes needed on each type of node.

        Splunk Heavy Forwarder:

        On Splunk Heavy forwarder, Install TA-TruStar and configure using TruStar credentials.

         Splunk Indexer Cluster:

        On Splunk Indexer cluster, User would have to define specific index in case user don’t want to use default index (main) or in case user have already defined Index on Splunk Heavy Forwarder.

        Splunk Search-Head Cluster:

        On Splunk Search Head Cluster, User would install the App & TA of TruStar App for Splunk.

        Troubleshooting

        Known Limitations

        • There is a known limitation in Splunk where App Icon doesn’t get visible before restarting Splunk. Hence, it’s recommended to restart post installation of the App to load the App Icon.
        • There is a known limitation in Splunk Modular input that in case of failure it doesn’t show proper raised error message,, but shows generic failure message on UI in windows machine.
        • In case of Splunk 7.1.x, the Whitelisted Input Dropdown of Indicators Dashboard wouldnt work for ‘All’ option. Workaround is to select either ‘Yes’ or ‘No’ and filter specific data.
        • In case of Splunk 7.0.3, the matching count for SHA256 type of indicators wouldn’t be considered.

        Scenarios

        After completing installation of application, all the dashboards will start populating data.

        • 1) In case you don’t see data in the dashboards, use following steps for troubleshooting.
          • Confirm that you have modified macro `trustar_get_index` with indexes selected while creating Modular input.
            • To verify this go to Settings->Data inputs->TruSTAR Configuration.
            • Check index field for each entry in list view.
            • If index=default, then splunk will take default index, which is ‘main’
            • Now go to Settings-> Advance search -> Search macros
            • Open entry with name ` trustar_get_index ` and verify definition. It must have definition on index verified in 2nd step

        For example, If all modular input entries have index=default in 2nd step, then update macro definition with index=main and save.

        If any specific index has been set in modular input then add it in macro definition.

        • Check following query to verify data is getting indexed into Splunk
        search `trustar_get_index` | stats count by sourcetype
        • Verify that SPLUNK_HOME is pointing to correct Splunk directory.
        • Look for errors in trustar_modinput.log file.  This file will be available under $SPLUNK_HOME/var/log/trustar folder.
        • 2) In case Authentication Key or Secret key of Trustar station get modified after setup in modular input.

        User can update it from Modular input UI

        • Go to Settings-> Data Inputs -> TruSTAR Configuration
        • Open specific TruStar station entry, and enter new Authentication key and Secret Key in both fields.
        • On click of save, modified key will get updated for that specific TruStar Station.

        Build App Source Code

        Any splunk application can be built to tar file with .spl extension.
        Below are the steps to build both TA and App from any Linux distribution.

        tar cv <app_name> > <app_name>.tar
                  
                  gzip <app_name>.tar
                  
                  mv <app_name>.tar.gz <app_name>.spl

        (Replace <app_name> with the name of app. For eg. TA-trustar or Trustar)

        Refer below url to know more about package and publish any splunk app.
        http://dev.splunk.com/view/webframework-developapps/SP-CAAAEMY

        Upgrade App

        User can upgrade TruSTAR app and TA through CLI or UI.

        Upgrade through CLI:
        • Download tar of App or TA from Splunk base
        • Stop Splunk server
        • $SPLUNK_HOME/bin/splunk 
        • Install app APP_NAME.tgz –update 1 –auth username:password
        • Start Splunk Server
        Upgrade through UI:
        • Click on Manage Apps
        • Find Trustar app And TA entry from list
        • Click on link of newer version under version column on related entry
        Manual Upgrade:
        • Click on Manage Apps
        • Click on Install App from file
        • Locate Trustar TA file from local drive
        • Select to Upgrade app
        • Click Upload 

        App Architecture

        This section describes the overall App architecture.

        Indexes

        Access Path: Settings → Indexes

        TruSTAR App for Splunk can populate the panels based on the index defined while indexing data into the Splunk. By default data will get populated under “main” index until it’s changed while configuring data input.

        Splunk recommends using Splunk’s default index (that is “main” index) for simplicity and reusability.

        Refer below URL to create custom index.

        Reference URL: http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Setupmultipleindexes

        Note: In case changes are done in Index name, Please follow steps mentioned under macro section.

        Source-Types

        Access Path:  Settings  → Source Types

        Source-Type are default Splunk fields to categorize and filter the indexed data to narrow down the search results. Since TruSTAR app collects two different types of data from Trustar Station, it has been indexed in below source types.

        Below is the table, which shows alerts and activities data are separated.

        Source-Type

        Description

        trustar:reports

        This contains all the reports sent from TruSTAR station to Splunk using rest API call.

        trustar:indicators

        This contains all the indicators sent from TruSTAR station to Splunk using rest API call.

        Macros

        Access Path: Settings  → Advanced Search  → Search Macros

        All the visualizations in TruSTAR App for Splunk are referred by a “trustar_get_index” macro, which helps App to identify the Index in which data is getting indexed.

        By default, it’s referred to “main” index and in case user is changing the Index value then same changes has to be done in the macro.

        TruSTAR App for Splunk has another macro called  “trustar_get_index_and_sourcetype”, which helps App to identify the index and sourcetype in which indicators of TruSTAR app should be matched.

        By default, it’s referred to  index=* and in case user has some specific index and source type to consider to find matches,then it should be updated in the macro.

        Please check out our comprehensive Splunk FAQ's here.
        For other questions reach out to support@trustar.co for any additional questions.

        How Did We Do?