8. Troubleshooting

Updated 1 month ago by Steven Chamales

Log messages, their meaning, what to do about them.

Log Level

Message

What it means?

Does this mean my TruSTAR app is not going to work?

What should I do about it?

WARN

Input <your modinput name>, Required collections not found: {‘threat_intel_meta’}

Older versions of ES required the TruSTAR Unified app to update this collection every time the app CRUD'ed records in the threat intel kvstores. Newer versions of ES deprecated this kvstore.

no.

ignore.

WARN

No checkpoint found for enclave <one of your modinput's enclave IDs>, IOC type <a TruSTAR observable type> and input name <your modinput name>. Initializing new checkpoint.

This message always prints the first time one of your modinputs downloads observables of that type from that enclave, so the modinput has not yet stored a checkpoint for that download operation.

no.

ignore.

WARN

Input <your modinput name>, Too many IOCs with same lastSeen time <epoch timestamp milliseconds>, start paging results. This may result in loss of data
  • The modinput encountered a TruSTAR "lastSeen" timestamp for which the enclave contained >1k observables.
  • If the enclave contains > 10k observables with same "lastSeen" timestamp, some of them may not end up in the kvstore.
  • This case is very rare, and should only happen if someone performed multiple indicator submissions (to the submit-indicators 1.3 endpoint or the CSV indicator submission UI) and specified the same "lastSeen" timestamp on more than 10k.

no. But some of the observables from your enclave might not arrive in the Splunk kvstores.

ignore. This edge-case will be handled in future versions of TruSTAR Unified app.

WARN

The following enclaves could not be found: [<comma-separated list of TruSTAR enclave IDs>]“

One or more of the enclave IDs specified in the modinput configs are not valid, or the "DOWNLOAD" account does not have read-access to it. It's possible that....

  • (a) your "DOWNLOAD" account did at one time have read-access to the enclave but someone (with a Station Company Administrator account) modified the "DOWNLOAD" account's permission to that enclave.
  • (b) access to that enclave has been removed from your Station company account altogether.
  • (c) your Station Company Administrator unsubscribed your company account from a particular integration, which removes that enclave from your company's access.
  • (d) The enclave ID was incorrect / invalid.

no. The Modinput will continue to download observables from other valid enclave IDs that its "DOWNLOAD" creds have access to.

You can ignore, but recommend review / update the modinput's enclave IDs list to contain only valid enclave IDs that the "DOWNLOAD" account has appropriate access to.

ERROR

09-29-2021 15:34:46.421 +0000 ERROR sendmodalert [27651 AlertNotifierWorker-0] - action=trustar_enrich_threat_activity STDERR -  ERROR: ts_spl_unified.modalerts. enrich.notable_event_service: REST API call to add indicator summaries endpoint info to notable event 0F7E38BF- AA18-4529-9961- 8E49CB2F0E70@@ notable@@ e9ecb3b 7cee274a1 5bf671b8 66381502 failed.  Reason: b' <?xml version="1.0" encoding="UTF-8"?>\n<response>\n  <messages>\n    <msg type="WARN"> insufficient permission to access this resource</msg>\n  </messages> \n</response>\n'

The Splunk user account attempting to run the "enrich" modaction does not have "update_notable_event" permissions.

The Enrich action will not work as expected until user's permissions are updated.

Give the user "ess_admin" role.

ERROR

11-12-2021 14:30:28.227 -0500 ERROR AdminManagerExternal [101463 TcpChannelThread] - Unexpected error “<class ‘splunktaucclib.rest_handler.error.RestError’>” from python handler: “REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/splunktaucclib/rest_handler/handler.py”, line 117, in wrapper\n  for name, data, acl in meth(self, *args, **kwargs):\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/splunktaucclib/rest_handler/handler.py”, line 338, in _format_all_response\n  self._encrypt_raw_credentials(cont[“entry”])\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/splunktaucclib/rest_handler/handler.py”, line 368, in _encrypt_raw_credentials\n  change_list = rest_credentials.decrypt_all(data)\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/splunktaucclib/rest_handler/credentials.py”, line 289, in decrypt_all\n  all_passwords = credential_manager._get_all_passwords()\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/solnlib/utils.py”, line 148, in wrapper\n  return func(*args, **kwargs)\n File “/opt/splunk/etc/apps/trustar_unified/bin/trustar_unified/aob_py3/solnlib/credentials.py”, line 277, in _get_all_passwords\n  clear_password += field_clear[index]\nTypeError: can only concatenate str (not “NoneType”) to str\n”. See splunkd.log/python.log for more details.

Someone copied a passwords.conf file (for any app - not necessarily TruSTAR Unified) from another Splunk instance to the instance you're working on.

TruSTAR App will not work until you are able to successfully configure the app, which you can't do until you find the offending "passwords.conf" file.

Follow guidance / directions found in this thread:

App Fails to Decrypt Encrypted Credential

Submit / Enrich Actions not working.

Possibility: Splunk User Account Permissions.

Splunk ES requires that the Splunk user account have Admin permissions in order to execute Submit or Enrich commands. You can check the error message in Splunk ES to see if your permissions are the issue.

  1. To find the error message for an adaptive response action failure, click on the hyperlink labeled View Adaptive Response Invocations. When you click on that hyperlink, you’ll be taken to a search results page.
    SplunkES_FAQ_Figure1
  2. Read through the log entries to find the error message.  The text in red box below shows the error is due to the user’s Splunk account having insufficient role permissions.
    SplunkES_FAQ_Figure2

To check if your account has the correct permissions, see the section User Requirements in the Install: TTruSTAR Splunk APP for Enterprise & Enterprise Security document.


How Did We Do?