Issue Adding Sigma Rules from Local Repo

[jstore-embers]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

20

RAM

64 GB

Storage for /

2 TB

Storage for /nsm

12 TB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I am trying to add a private sigma ruleset offered by the DFIR Report. I can use git to pull the repo down to /nsm/rules/detect-sigma/repos/dfir-report/ and I've configured the following line in the admin interface under rule repos according to the documentation here

{"community":true,"folder":"rules/rules/sigma","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"}

I think the above is correct (not sure what I should put for license, so I copied the option from the securityonion-resources repo. Note the full path to the rules root folder in my instance is /nsm/rules/detect-sigma/repos/dfir-report/rules/rules/sigma but there are several subfolders.

I'm wondering where I may find logs that are generated when doing a full-update on the elastalert rules from the detections interface or if there's anything obviously wrong with this config.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[defensivedepth]

When you say pull down - did you download it or did you git clone it?

That dir needs to be a git repo. You should be able to run git status on the following dir: /nsm/rules/detect-sigma/repos/dfir-report/

[jstore-embers]

I do appreciate all the help so far, but I'm at a loss.

I don't see anything when I tail the sensoroni log, nor do I see anything useful when checking the logs in the Hunt interface as you outlined. Here's an example of the last entries when the SOC crashes.

{"fields":{"contentLength":1006,"method":"GET","status":"200 OK","statusCode":200,"url":"http://onion-manager:4434/identities/7bb6b477-8f6a-4356-b523-00ee541bf202"},"level":"info","timestamp":"2024-11-22T16:03:33.054727052Z","message":"HTTP request finished"}
{"fields":{"contentLength":0,"elapsedMs":5,"impl":"/build/server/detectionhandler.go:939:github.com/security-onion-solutions/securityonion-soc/server.(*DetectionHandler).syncEngineDetections","remoteAddr":"172.17.1.1:55322","requestId":"7550aa93-a25f-4db0-bd14-656566477ab4","requestMethod":"POST","requestPath":"/api/detection/sync/ElastAlert/full","requestQuery":{},"requestor":{"id":"7bb6b477-8f6a-4356-b523-00ee541bf202","createTime":"2024-11-22T16:03:33.052052479Z","updateTime":"0001-01-01T00:00:00Z","email":"jstore@embers.org","firstName":"Justin","lastName":"Store","totpStatus":"disabled","oidcStatus":"disabled","webauthnStatus":"enabled","note":"","roles":null,"status":"","searchUsername":"","password":"","passwordChanged":false},"sourceIp":"10.13.100.111","statusCode":200},"level":"info","timestamp":"2024-11-22T16:03:33.054971463Z","message":"Handled request"}
{"fields":{"detectionEngine":"elastalert","forceSync":true,"syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.055043153Z","message":"starting sync"}
{"fields":{"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b","targetFile":"/opt/sensoroni/ai_summary_repos/securityonion-resources/detections-ai/sigma_summaries.yaml"},"level":"info","timestamp":"2024-11-22T16:03:33.476455333Z","message":"reading AI summaries"}
{"fields":{"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.678954392Z","message":"successfully unmarshalled AI summaries, parsing..."}
{"fields":{"aiSummaryCount":3051,"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.679441549Z","message":"successfully parsed AI summaries"}
{"fields":{"aiSummaryCount":3051,"detectionEngine":"elastalert"},"level":"info","timestamp":"2024-11-22T16:03:33.682130624Z","message":"loaded AI summaries"}
{"fields":{"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.682164217Z","message":"successfully loaded AI summaries"}
{"fields":{"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.682171914Z","message":"successfully refreshed AI summaries"}
{"fields":{},"level":"info","timestamp":"2024-11-22T16:03:33.683401255Z","message":"no changes to sigma processing pipeline"}
{"fields":{"detectionEngine":"elastalert","syncId":"3e4b2a18-3d73-45d7-9b62-909d12a8895b"},"level":"info","timestamp":"2024-11-22T16:03:33.683429727Z","message":"successfully checked the sigma processing pipelines"}
{"fields":{"downloadSizes":{"core++":2510461,"emerging_threats_addon":328096}},"level":"info","timestamp":"2024-11-22T16:03:35.016315906Z","message":"downloaded sigma packages"}

The folder structure of the repo is shown at the bottom of this reply. All rules are contained in the three folders within the sigma folder: linux, network, and windows.

I can manually specify linux or network without issues like this.
{"community":true,"folder":"rules/sigma/linux","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"} WORKS
{"community":true,"folder":"rules/sigma/network","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"} WORKS

I can also manually specify some windows subfolders like this:
{"community":true,"folder":"rules/sigma/windows/builtin","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"} WORKS

But at least one windows subfolder breaks the soc like the windows process creation folder:
{"community":true,"folder":"rules/sigma/windows/process_creation","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"} BREAKS SOC

I figured I could use the process of elimination to narrow down the culprit by deleting files in the local repo until I got things to work, but that's not the case. Is it possible that the files I delete locally are still retained in the SOC container even after I force restart the soc and kick off a full update? I noticed running a git pull says the files are up to date and won't re-pull them as they are non-committed local changes, so I wasn't sure if the SOC was doing something similar and retaining the original files that I deleted. I figured it would start fresh after a restart.

I can delete the entire windows sub folder, so only the linux and network folders exist locally, but the SOC still crashes, even though I can specify those manually without issue.
{"community":true,"folder":"rules/sigma","license":"Elastic-2.0","repo":"file:///nsm/rules/detect-sigma/repos/dfir-report"} BREAKS SOC, even if the only subfolders are linux and network, both of which work when specified manually

Based on this testing, I assume the files I delete locally are still being used, but I can't understand why. Maybe I'm going down the wrong path here, but I can't find any logs that point to the offending rules, and removing offending rules (assuming some in the process_creation folder) doesn't seem to fix it.

[root@onion-manager dfir-report]# pwd
/nsm/rules/detect-sigma/repos/dfir-report
[root@onion-manager dfir-report]# ls -laR | grep /
./.git:
./.git/branches:
./.git/hooks:
./.git/info:
./.git/logs:
./.git/logs/refs:
./.git/logs/refs/heads:
./.git/logs/refs/remotes:
./.git/logs/refs/remotes/origin:
./.git/objects:
./.git/objects/info:
./.git/objects/pack:
./.git/refs:
./.git/refs/heads:
./.git/refs/remotes:
./.git/refs/remotes/origin:
./.git/refs/tags:
./.github:
./.github/workflows:
./rules:
./rules/sigma:
./rules/sigma/linux:
./rules/sigma/linux/process_creation:
./rules/sigma/network:
./rules/sigma/network/zeek:
./rules/sigma/windows:
./rules/sigma/windows/builtin:
./rules/sigma/windows/builtin/security:
./rules/sigma/windows/builtin/system:
./rules/sigma/windows/builtin/system/service_control_manager:
./rules/sigma/windows/builtin/terminalservices:
./rules/sigma/windows/builtin/WindowsDefender:
./rules/sigma/windows/create_remote_thread:
./rules/sigma/windows/create_stream_hash:
./rules/sigma/windows/dns_query:
./rules/sigma/windows/file:
./rules/sigma/windows/file/file_event:
./rules/sigma/windows/file/file_event/file_event:
./rules/sigma/windows/image_load:
./rules/sigma/windows/network_connection:
./rules/sigma/windows/pipe_created:
./rules/sigma/windows/powershell_module:
./rules/sigma/windows/powershell_script:
./rules/sigma/windows/process_access:
./rules/sigma/windows/process_creation:
./rules/sigma/windows/registry:
./rules/sigma/windows/registry/registry_set:
./statistics:
./statistics/dataset:
./tests:

[jstore-embers]

For completeness, here is how I setup the repo.

# Add github PAT to /nsm/rules/detect-sigma/repos/github_pat

# Initialize
git clone https://jstore-embers:$(cat /nsm/rules/detect-sigma/repos/github_pat)@github.com/The-DFIR-Report/private-sigma-ruleset.git /nsm/rules/detect-sigma/repos/dfir-report

# Add safe directory exception
git config --global --add safe.directory /nsm/rules/detect-sigma/repos/dfir-report

And here's how I update it.

# Navigate to local repo
pushd /nsm/rules/detect-sigma/repos/dfir-report

# Update
git pull

[defensivedepth]

@jstore-embers I appreciate the time and effort you put into writing this up. Im going to do some digging internally and see if I can replicate this in any way. Will let you know what I come up with.

[jstore-embers]

Thank you! I know the team at DFIR Report are aware of this thread, so if need be I could ask if I could allow you temporary access to the private repo for testing. Not sure if that's an option and I obviously want to be sure I'm not violating any terms of use, but it could be worth asking if there's no other way to replicate.

[defensivedepth]

I actually already reached out to them via their Contact Us form earlier today to see if there was a way to get temp access so that we can troubleshoot further. My concern is that this is not an issue specific to them, but some type of edge case that we need to account for.

[defensivedepth]

Greetings @jstore-embers. Good news! I have found the issue and PR'ed a fix for our next release: Security-Onion-Solutions/securityonion-soc#687

I have also added additional tests to make sure something like this scenario will be caught earlier.

Until then, you can search for any of the Sigma rules that don't have author fields and add a placeholder author field and value. That will allow all the rules to import without crashing SOC. Because it is a private ruleset, I will not list the rules here, but there are only 4 of them and you can find them with this:

grep -iLr "author" --include="*.yml" .

Thanks again for taking the time to troubleshoot this with me.

[jstore-embers]

Yeah, I copied the author from other rules, but I think I'm just a git dummy as I was running into something similar when I was trying to troubleshoot the issue initially. The soc was still crashing even after I deleted the offending rules, so I'm not positive how local changes to the repo are carried over. It seems like local changes are not honored, though I don't understand why.

While I'm curious as it underscores some misunderstanding on my part, the DFIR Report added the author to each rule and it seems to be synchronizing now that I've pulled down the latest version, so should be good on the front.

[defensivedepth]

Ok let me know if you see any other issues. Author is supposed to be an optional field, so that was an issue on our side.

[jstore-embers]

Sure. I noticed a few parsing issues that I don't think are against the Sigma schema, but the SOC is not happy when looking at the logs. Should I open new issue for these once I've had time to look at them more closely?

For example, if you have something like this:

selection:
  Details|contains: blah1
  Details|contains: blah2

There will be a parsing error saying that the "Details" field is already defined, but referring to this rule creation guide, I don't think that's necessarily invalid (I may be wrong since they're using FieldName as a placeholder). There are other examples of sigma rules online that use multiple instances of the same field within a selection.

There could be other instances of syntax that isn't exactly against the schema, but not working, but I need to spend more time digging.

https://github.com/SigmaHQ/sigma/wiki/Rule-Creation-Guide

[defensivedepth]

Yes, create a new discussion around those (to keep it to one issue per discussion) and I will take a look, thanks!

[jstore-embers]

Thanks. Started that here: #13999