Issue with Missing Data on Storage Node

[mckenziemmack]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

251G

Storage for /

558G

Storage for /nsm

38T

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,

I'm currently facing an issue where I believe I have lost all data on my storage node. The problem appears to be related to my manager storing indices that it shouldn't, which is causing the manager to hit its watermark. This seems to lead to this situation where I can no longer access or view the data on the storage node in Kibana, even though the manager still retains a few of the incorrectly stored indices.

The storage node itself is showing only 551GB of usage out of 38TB available (about 2% utilization), which makes me think that indices may have been deleted. I'm worried that the data has been lost, but I'm unsure why the storage node would automatically delete all of its storage data like this. Would it be possible that the data is still there on my storage nodes but just inaccessible?

Has anyone experienced something similar or have any insights? Any help or guidance would be greatly appreciated.

Thanks!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What returns when you run these commands:

sudo so-elasticsearch-query _cat/allocation?v
sudo so-elasticsearch-query _cat/nodes?v

If your search node has a larger /nsm partition than your manager, then it is possible the clean up script deleted some data.

Have you done any Elasticsearch ILM tuning?

[mckenziemmack]

Thank you for your response!

This is returns when those commands are ran:

[~]$ sudo so-elasticsearch-query _cat/allocation?v
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host        ip          node        node.role
   137               24                 0.0                 4.2tb        4.2tb     4.5tb     31.8tb     36.3tb           12 storage1    storage1 ip storage1    dhit
    76                0                 0.0               421.2gb      300.2gb   377.9gb      146gb      524gb           72 manager     manager ip  manager dmrt

[~]$ sudo so-elasticsearch-query _cat/nodes?v
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
storage1 ip          32          99  89   21.48   18.16    17.71 dhit      -      storage1
storage2 ip          60          80  31   21.18   24.07    24.57 dhit      -      storage2
manager ip           56          98  81   10.90   11.20    11.84 dmrt      *      manager

As a current mitigation, I have adjusted the manager's watermark settings to a low of 45%, and a high of 50%, with the flood stage staying at 90%, and this has seemed to stop the search node reallocating the large shards to the manager as moving any of the complete 50 GB shards would make the manager node over the high watermark. I'm just worried I'm maybe going down the wrong rabbit hole and this mass deletion event will happen again.

[cm-ops]

A couple of things from the above. Looks like you have two search nodes, but _cat/allocation is only showing the manager and one of them. Storage2 has no elastic data.

I would recommend you move all the elastic data off the manager and remove the data role. If there is data on the manager above the retention percentage, there is a script that will remove data to bring you under that percentage. https://docs.securityonion.net/en/2.4/elasticsearch.html#so-elasticsearch-indices-delete

Exclude the Manager's IP from cluster routing to move shards off the Manager, in Kibana > Dev Tools (change the IP below to your manager's IP):

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

Check the allocation of shards with the sudo so-elasticsearch-query _cat/allocation?v command. If you have larger shards, for example 50GB, it will take some time to move them.

Once all shards are off, remove the data role from the manager (toggle on advanced in Options at the top) in SOC -> Administration -> Configuration -> elasticsearch -> so_roles -> manager -> config -> node -> roles

[mckenziemmack]

Firstly, actually my attempt to change the watermark failed, as I am getting these errors:

[ERROR] Command '/usr/sbin/so-elasticsearch-templates-load' failed with return code: 1
[ERROR] stdout: State file /opt/so/state/estemplates.txt not found. Running so-elasticsearch-templates-load.

I tried the command you suggested

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

However, it looks like it deleted a majority of old indices on my manager and storage nodes, and some of the new indices that have been created are still reallocating to my manager.

.ds-logs-infoblox_nios.log-default-2024.11.20-000001               0 p STARTED      6631791   1.2gb   1.2gb ip   manager
.ds-logs-zeek-so-2024.11.22-000010                                 0 p STARTED     12111195  17.5gb  17.5gb ip   manager
.ds-logs-zeek-so-2024.11.22-000010                                 1 p STARTED     12111823  22.9gb  22.9gb ip   manager
.ds-logs-zeek-so-2024.11.21-000005                                 0 p STARTED     46663731  52.8gb  52.8gb ip   manager
.ds-logs-zeek-so-2024.11.21-000005                                 1 p STARTED     46662478  52.7gb  52.7gb ip   storage2
.ds-logs-zeek-so-2024.11.22-000009                                 0 p STARTED     43716497  50.1gb  50.1gb ip   storage2
.ds-logs-zeek-so-2024.11.22-000009                                 1 p RELOCATING  43716831  50.1gb  50.1gb ip   manager -> storage1 ip WOSyKTGoRhObePi0oXFBnQ storage1

[cm-ops]

Did you change the 10.0.0.1 IP to that of your manager? The transient setting is temporary and will be gone if Elasticsearch is restarted. To make it persistent you have to use that instend of transient:

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

In the above snippit I see one shard moving from you manager to your search node storage1. Keep in mind 50gb takes about 45 minutes to move with the default ES settings.

[mckenziemmack]

Yes, I did change the IP to my manager's.

I had restarted Elasticsearch at least a week before the rerouting allocation was set to exclude the managers IP.

[cm-ops]

Okay, keep an eye on moving shards, once they are all off the manager remove the data role in SOC.