[DaC] additional line break added to query after export/import

[wingiti]

Command or doc area

• kibana export-rules / kibana import-rules
• import-rules-to-repo
• export-rules-from-repo
• Other DaC CLI command (name it in the field below)
• Documentation in this repo only (link the page/section below)

Other command name or doc link

No response

Summary

If you export and import a rule, the query at the destination contains an additional line break (\n) at the end.
This might not impact the functionality but it shows up if you compare the rule at source and destination system. I also expect you will get multiple trailing newlines if you redo the procedure.

The issue for this can be found at the point where the query is saved to the toml file:

detection-rules/detection_rules/rule_formatter.py

Lines 294 to 302 in 36e6f54

	# we want to preserve the threat_query format, but want to modify it in the context of encoded dump
	if threat_query:
	formatted_threat_query = "\nthreat_query = '''\n{}\n'''{}".format(threat_query, "\n\n" if bottom else "")
	top_out = top_out.replace('threat_query = "XXxXX"', formatted_threat_query)
	# we want to preserve the query format, but want to modify it in the context of encoded dump
	if query:
	formatted_query = "\nquery = '''\n{}\n'''{}".format(query, "\n\n" if bottom else "")
	top_out = top_out.replace('query = "XXxXX"', formatted_query)

The original query is prefixed and suffixed with a line break (\n) before written to the toml file.
When the query is imported again, the toml parser removes the prefix newline but not the trailing one.

Multi-line basic strings are surrounded by three quotation marks on each side and allow newlines. A newline immediately following the opening delimiter will be trimmed. All other whitespace and newline characters remain intact.

Source: https://github.com/toml-lang/toml/blob/main/toml.md

I would suggest to not add line breaks to the original query.

UPDATE:
I tested it locally and removed the surrounding newlines which worked well. But then an additional issue occurred.

I identified that already the query part is stripped, which leads to the fact that if a rule on the source system contains an "intentional" white space or new line at the end, will loose it on the target system.

detection-rules/detection_rules/rule_formatter.py

Lines 224 to 232 in 36e6f54

	if _data == "rule":
	# - We want to avoid the encoder for the query and instead use kql-lint.
	# - Linting is done in rule.normalize() which is also called in rule.validate().
	# - Until lint has tabbing, this is going to result in all queries being flattened with no wrapping,
	# but will at least purge extraneous white space
	query = contents["rule"].pop("query", "").strip()
	# - As tags are expanding, we may want to reconsider the need to have them in alphabetical order
	threat_query = contents["rule"].pop("threat_query", "").strip()

Would be better if the query part is not touched to stay the same all the time.

Expected vs actual

No response

Environment (if relevant)

No response

Reproduction (if applicable)

Export/Import a rule with a multiline query.

Related issues (optional)

No response

[eric-forte-elastic]

@wingiti thanks for bringing this up! Can you share an example where this occurs? It seems like it may require specific circumstances for this to occur.

If this is more about the general behavior in Kibana, then I am assuming you are referring to this extra space in Kibana for a rule like the following.

If this is the case then for:

I also expect you will get multiple trailing newlines if you redo the procedure.

You can re-run it many times and it will not add additional spaces. See the following for the kibana export-rules and kibana import-rules routes for confirmation of this.

This behavior is generally an artifact of using a multi-line value in the query. If you use a single line query (shown below) this difference will not occur.

This behavior is intentional, as you discovered in your update.

UPDATE:
I tested it locally and removed the surrounding newlines which worked well. But then an additional issue occurred.

I identified that already the query part is stripped, which leads to the fact that if a rule on the source system contains an "intentional" white space or new line at the end, will loose it on the target system.

The discrepancy is only on the Kibana import-rules path where the query is cast to a multi-line string regardless of whether or not it requires it being a multi-line string. This was a design decision made based on how our rule developers use the repo.

E.g. the Rule in Kibana has destination.ip:* or host.ip:* while the exported version via kibana import-rules is:

query = '''
destination.ip:* or host.ip:*
'''

We can look into supporting single line strings on Kibana import as a cosmetic feature request, but before doing so I would prefer to make sure that this is what you are referring to instead of a potentially different issue. Thanks!

[wingiti]

Thank you for looking into that.

Maybe some words about our current workflow and why this issue matters to us.
We want to create/develop rules on a QA instance. After that transport them to a staging system for final tuning and finally deploy it to a production environment.
This means we use DAC for: QA export -> Staging import -> Staging export -> Prod import

For validation we compare the output from /api/detection_engine/rules on the source and destination system.

For example take this rule: https://github.com/elastic/detection-rules/blob/d1c9cd65b40de3189e429279973a96970785356c/rules_building_block/lateral_movement_posh_winrm_activity.toml

It has a bigger query in multiline format which is fine and totally makes sense.

If I do the API request on the source system, output looks like:

...
"index": [
    "winlogbeat-*",
    "*:logs-windows.powershell*"
  ],
  "query": """event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
    ("Invoke-WmiMethod" or "Invoke-Command" or "Enter-PSSession") and "ComputerName"
  ) and
  not user.id : "S-1-5-18" and
  not file.directory : (
    "C:\\Program Files\\LogicMonitor\\Agent\\tmp" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\icinga-powershell-framework\\cache" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\SmartCardTools\\1.2.2"
  ) and not
  powershell.file.script_block_text : (
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )""",
  "filters": [
...

The export in toml looks like:

query = '''
event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
    ("Invoke-WmiMethod" or "Invoke-Command" or "Enter-PSSession") and "ComputerName"
  ) and
  not user.id : "S-1-5-18" and
  not file.directory : (
    "C:\\Program Files\\LogicMonitor\\Agent\\tmp" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\icinga-powershell-framework\\cache" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\SmartCardTools\\1.2.2"
  ) and not
  powershell.file.script_block_text : (
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )
'''

After import to the next system, the API result of the destination looks like:

...
"index": [
    "winlogbeat-*",
    "*:logs-windows.powershell*"
  ],
  "query": """event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
    ("Invoke-WmiMethod" or "Invoke-Command" or "Enter-PSSession") and "ComputerName"
  ) and
  not user.id : "S-1-5-18" and
  not file.directory : (
    "C:\\Program Files\\LogicMonitor\\Agent\\tmp" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\icinga-powershell-framework\\cache" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\SmartCardTools\\1.2.2"
  ) and not
  powershell.file.script_block_text : (
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )
""",
  "filters": [
...

And here you can see the small difference.
Source:

    "function Invoke-Command {"
  )""",
  "filters": [

Destination:

    "function Invoke-Command {"
  )
""",
  "filters": [

This happens because the toml parser trims the prefix newline but not the trailing one for import.

That is why I think this newlines should not be added and the toml should look like this:

query = '''event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
...
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )'''

The newline is also more or less visible if you go to the UI:

Additional newlines from export/import steps, as originally feared, will not be added because of the strip I discovered in my update. But it will always be one newline because of the things above.

Now coming to my issue mentioned in the update.
Let's say we removed the additional new lines in the code. The export -> import of the above mentioned rule will work fine, no difference in API comparison.

Nevertheless the strip also makes trouble in case someone adds one or multiple new line characters (by intention, by mistake or for whatever reason) to the end of his query.

API response from the source system:

  "index": [
    "winlogbeat-*",
    "*:logs-windows.powershell*"
  ],
  "query": """event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
    ("Invoke-WmiMethod" or "Invoke-Command" or "Enter-PSSession") and "ComputerName"
  ) and
  not user.id : "S-1-5-18" and
  not file.directory : (
    "C:\\Program Files\\LogicMonitor\\Agent\\tmp" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\icinga-powershell-framework\\cache" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\SmartCardTools\\1.2.2"
  ) and not
  powershell.file.script_block_text : (
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )




""",
  "filters": [

API response on the destination system:

  "index": [
    "winlogbeat-*",
    "*:logs-windows.powershell*"
  ],
  "query": """event.category:process and host.os.type:windows and
  powershell.file.script_block_text : (
    ("Invoke-WmiMethod" or "Invoke-Command" or "Enter-PSSession") and "ComputerName"
  ) and
  not user.id : "S-1-5-18" and
  not file.directory : (
    "C:\\Program Files\\LogicMonitor\\Agent\\tmp" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\icinga-powershell-framework\\cache" or
    "C:\\Program Files\\WindowsPowerShell\\Modules\\SmartCardTools\\1.2.2"
  ) and not
  powershell.file.script_block_text : (
    "Export-ModuleMember -Function @('Invoke-Expression''Invoke-Command')" and
    "function Invoke-Command {"
  )""",
  "filters": [

Even though you can discuss about whether it makes sense to keep the useless characters or not, we have again a mismatch in our integrity check.

Hope this explains my issues in a understandable way and I am curious about your thoughts.

[eric-forte-elastic]

Thanks for the explanation! This makes the situation much more clear.

So for this to occur it appears the following must be true:

• Rules in Kibana are source of truth
• Desired TOML output for rules is to match the newline formatting from Kibana API endpoint /api/detection_engine/rules, prior to the rule being viewed in the "Edit" view in Kibana for pre-built rules on stack version < 9.4
• When writing custom rules in the Kibana UI, the extra newline which is considered standard practice for our repo, is not included

For context, from a fresh rule install from the package, if I use the API endpoint in the dev console the same behavior occurs as you are seeing (Note this is from Kibana version 9.3).

Then if I go to edit the rule, you will see the extra \n be added without making any query modification.

Then if I go back and re-run the API command, it will have the newline:

If I go back to the rule to see if the rule was edited, I can see that the rule was not edited as the install timestamp matches the updated timestamp

While yes, you can use this repo to cause this behavior, this will occur whether or not you use the repo.

This discrepancy will only increase if we implement a change to the base path as you are suggesting for pre-built rules.

This behavior, is not present in the most recent Kibana versions. E.g. in version 9.4 this does not occur and the newline is present prior to viewing the rule in edit view.

clean_rule_install_9_4.mp4

Furthermore, the rule when exported via ndjson (either via API or UI) will contain this newline.

This is all for pre-built rules which have their source of truth being the packages made from this repo, which will include the newline.

For custom rules (or modified pre-built rules) you can remove this newline in Kibana and it will stick.

modify_sticks.mp4

Given this, I think the feature request is for us to provide a bypass to the current practice of adding the newline?

[eric-forte-elastic]

@wingiti can you confirm what stack version you are on as well? Thanks!

[wingiti]

I am on stack version 9.3.3 now, not sure what it was during the time the rule was initially installed. And what happened to the rule afterwards.

I can not see the behaviour you described above:
If I have an installed rule without line break at the end and I only click the edit button (+ cancel afterwards), there will not be any change in the update timestamp (like you confirmed as well) nor any changes to the API output. -> no line break added.

If I go to edit, add a line break and click on save, for sure update time and API output changes.

Not sure if there is a difference what Kibana does if I install a prebuilt rule via the button or if I import a ndjson via UI or if I use the DAC code to import a TOML.

My target/wish would be:
Nevertheless what is the source of a rule (prebuilt, imported, initially created on the system) if I use DAC to export a rule from one system and import it to another system, it should look the same on both afterwards and if compared via API output.

This means import/export as it is, no manipulation (strip, adding/removing newlines, ...) within the procedure of the DAC import/export functionality.