[Rule Tuning] ESQL Rule List Search Fix

[Aegrah]

Summary

There is a "feature" where using ES|QL to search a list using the == or in operators, will return null. For example, searching the event.action list [exec, fork, end] by searching event.action == "exec" will result in null, effectively making the query hit a FN.

This PR fixes this, by either using the KQL() function where possible or using mv_expand where the KQL() function is not possible (e.g. KQL after an EVAL is unsupported).

[github-actions]

Rule: Tuning - Guidelines

These guidelines serve as a reminder set of considerations when tuning an existing rule.

Documentation and Context

• Detailed description of the suggested changes.
• Provide example JSON data or screenshots.
• Provide evidence of reducing benign events mistakenly identified as threats (False Positives).
• Provide evidence of enhancing detection of true threats that were previously missed (False Negatives).
• Provide evidence of optimizing resource consumption and execution time of detection rules (Performance).
• Provide evidence of specific environment factors influencing customized rule tuning (Contextual Tuning).
• Provide evidence of improvements made by modifying sensitivity by changing alert triggering thresholds (Threshold Adjustments).
• Provide evidence of refining rules to better detect deviations from typical behavior (Behavioral Tuning).
• Provide evidence of improvements of adjusting rules based on time-based patterns (Temporal Tuning).
• Provide reasoning of adjusting priority or severity levels of alerts (Severity Tuning).
• Provide evidence of improving quality integrity of our data used by detection rules (Data Quality).
• Ensure the tuning includes necessary updates to the release documentation and versioning.

Rule Metadata Checks

• updated_date matches the date of tuning PR merged.
• min_stack_version should support the widest stack versions.
• name and description should be descriptive and not include typos.
• query should be inclusive, not overly exclusive. Review to ensure the original intent of the rule is maintained.

Testing and Validation

• Validate that the tuned rule's performance is satisfactory and does not negatively impact the stack.
• Ensure that the tuned rule has a low false positive rate.

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ High Number of Egress Network Connections from Unusual Executable (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Command Line Obfuscation via Whitespace Padding (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Port Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Long Base64 Encoded Command via Scripting Interpreter (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Malware-Driven SSH Brute Force Attempt (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Linux Local Account Brute Force Detected (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Subnet Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Command Execution from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ File Transfer Utility Launched from Unusual Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Process Spawned from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Password Spraying Attack via SSH (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Suspicious Python Shell Command Execution (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Base64 Encoding/Decoding Activity (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ High Number of Egress Network Connections from Unusual Executable (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Command Line Obfuscation via Whitespace Padding (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Port Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Long Base64 Encoded Command via Scripting Interpreter (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Malware-Driven SSH Brute Force Attempt (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Linux Local Account Brute Force Detected (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Subnet Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Command Execution from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ File Transfer Utility Launched from Unusual Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Process Spawned from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Password Spraying Attack via SSH (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Suspicious Python Shell Command Execution (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Base64 Encoding/Decoding Activity (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ Suspicious AWS S3 Connection via Script Interpreter (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ High Number of Egress Network Connections from Unusual Executable (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Command Line Obfuscation via Whitespace Padding (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Port Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Long Base64 Encoded Command via Scripting Interpreter (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Malware-Driven SSH Brute Force Attempt (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Linux Local Account Brute Force Detected (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Subnet Scanning Activity from Compromised Host (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Command Execution from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ File Transfer Utility Launched from Unusual Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Process Spawned from Web Server Parent (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Potential Password Spraying Attack via SSH (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Suspicious Python Shell Command Execution (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Unusual Base64 Encoding/Decoding Activity (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Suspicious TCC Access Granted for User Folders (esql)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[terrancedejesus]

Why MV_EXPAND? Can we just KQL wrap it like we did for the other rules?

[Aegrah]

KQL() is not allowed after MV_EXPAND, and because we need to use MV_EXPAND on _ignored, this seemed to be the logical approach.