Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Leak (outputs.opensearch ??) #16295

Open
bg4erem opened this issue Dec 11, 2024 · 5 comments
Open

Memory Leak (outputs.opensearch ??) #16295

bg4erem opened this issue Dec 11, 2024 · 5 comments
Assignees
@srebhan
Labels
bug unexpected problem or unintended behavior

Comments

@bg4erem
Copy link

bg4erem commented Dec 11, 2024

Relevant telegraf.conf

[global_tags]
  snmp_interfaces_agg_period_sec = "30"

[agent]
  interval = "15s"
  round_interval = true
  precision = "0s"
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  hostname = ""
  omit_hostname = false
  # snmp_translator = "gosmi"

[inputs]

[[inputs.internal]]
  collect_memstats = true


  [[inputs.snmp]]
    agents = ["switch1", "switch2", "switch3", "switch4"]
    version = 2
    path = ["/usr/share/snmp/mibs"]
    community = "public"
    # log_level = "debug"

    [[inputs.snmp.field]]
      oid = "RFC1213-MIB::sysUpTime.0"
      name = "sysUpTime"
      conversion = "float(2)"

    [[inputs.snmp.field]]
      oid = ".1.3.6.1.4.1.11.5.7.5.7.1.1.1.1.4.9.0"
      name = "HpAvgCpuString"

    [[inputs.snmp.field]]
      oid = "RFC1213-MIB::sysName.0"
      name = "sysName"
      is_tag = true

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysObjectID.0"
    name = "sysObjectID"
    is_tag = true

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysContact.0"
    name = "sysContact"
    is_tag = true

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysLocation.0"
    name = "sysLocation"
    is_tag = true

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysDescr.0"
    name = "sysDescr"
    is_tag = true

  [[inputs.snmp.table]]
    oid = "IF-MIB::ifXTable"
    name = "interface"
    inherit_tags = ["sysName", "sysUpTime", "sysDescr", "sysContact", "sysLocation" ]

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifIndex"
      name = "ifIndex"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifDescr"
      name = "ifDescr"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifOperStatus"
      name = "ifOperStatus"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifAdminStatus"
      name = "ifAdminStatus"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifPhysAddress"
      name = "ifPhysAddress"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifPromiscuousMode"
      name = "ifPromiscuousMode"
      is_tag = true

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifInErrors"
      name = "ifInErrors"

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifOutErrors"
      name = "ifOutErrors"

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifInDiscards"
      name = "ifInDiscards"

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifOutDiscards"
      name = "ifOutDiscards"

  [[inputs.snmp.table]]
    name = "ip_address"
    oid = "IP-MIB::ipAddrTable"
    inherit_tags = ["sysName", "sysDescr", "sysObjectID", "ifName" ]

    [[inputs.snmp.table.field]]
      name = "ip"
      oid = "IP-MIB::ipAdEntAddr"

    [[inputs.snmp.table.field]]
      name = "netmask"
      oid = "IP-MIB::ipAdEntNetMask"

    [[inputs.snmp.table.field]]
      name = "ifIndex"
      oid = "IP-MIB::ipAdEntIfIndex"
      is_tag = true

[[processors.enum]]
  [[processors.enum.mapping]]
    tag = "ifPromiscuousMode"

    [processors.enum.mapping.value_mappings]
      1 = true
      2 = false

[aggregators]

  [[aggregators.basicstats]]
    stats = ["non_negative_diff", "non_negative_rate"]
    drop_original = true
    period = "30s"
    namepass = ["interface"]


  [[outputs.opensearch]]
    username = "username"
    password = "password"
    index_name = "my-network-{{.Time.Format \"2006-01-02\"}}"
    template_name = "telegraf"
    manage_template = true
    overwrite_template = false
    use_pipeline = "my-network-pipeline"
    urls = ["https://opensearch_host"]
    insecure_skip_verify = true
    tls_enable = true

Logs from Telegraf

2024-12-11T15:10:54Z I! Loading config: /etc/telegraf/telegraf.conf
2024-12-11T15:10:54Z I! Starting Telegraf 1.33.0 brought to you by InfluxData the makers of InfluxDB
2024-12-11T15:10:54Z I! Available plugins: 236 inputs, 9 aggregators, 33 processors, 26 parsers, 63 outputs, 6 secret-stores
2024-12-11T15:10:54Z I! Loaded inputs: internal snmp
2024-12-11T15:10:54Z I! Loaded aggregators: basicstats
2024-12-11T15:10:54Z I! Loaded processors: enum
2024-12-11T15:10:54Z I! Loaded secretstores:
2024-12-11T15:10:54Z I! Loaded outputs: opensearch
2024-12-11T15:10:54Z I! Tags enabled: host=dev52 snmp_interfaces_agg_period_sec=30
2024-12-11T15:10:54Z I! [agent] Config: Interval:15s, Quiet:false, Hostname:"dev52", Flush Interval:10s
2024-12-11T15:10:54Z D! [agent] Initializing plugins
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifXTable"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "IF-MIB::ifXTable.1"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptable" "-Ch" "-Cl" "-c" "public" "127.0.0.1" "IF-MIB::ifXTable"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifIndex"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHighSpeed"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifDescr"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOperStatus"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifAdminStatus"
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifPhysAddress"
2024-12-11T15:10:54Z W! DeprecationWarning: Value "hwaddr" for option "field.conversion" of plugin "inputs.snmp" deprecated since version 1.33.0 and will be removed in 2.0.0: Use 'displayhint' instead
2024-12-11T15:10:54Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifPromiscuousMode"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifInErrors"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOutErrors"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifInDiscards"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOutDiscards"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifName"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifInMulticastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifInBroadcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOutMulticastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOutBroadcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCInOctets"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCInUcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCInMulticastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCInBroadcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCOutOctets"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCOutUcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCOutMulticastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifHCOutBroadcastPkts"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifLinkUpDownTrapEnable"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifConnectorPresent"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifAlias"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifCounterDiscontinuityTime"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAddrTable"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "IP-MIB::ipAddrTable.1"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptable" "-Ch" "-Cl" "-c" "public" "127.0.0.1" "IP-MIB::ipAddrTable"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAdEntAddr"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAdEntNetMask"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAdEntIfIndex"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAdEntBcastAddr"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IP-MIB::ipAdEntReasmMaxSize"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysUpTime.0"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysName.0"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysObjectID.0"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysContact.0"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysLocation.0"
2024-12-11T15:10:55Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "RFC1213-MIB::sysDescr.0"
2024-12-11T15:10:55Z W! DeprecationWarning: Value "agent_host" for option "agent_host_tag" of plugin "inputs.snmp" deprecated since version 1.29.0 and will be removed in 2.0.0: set to "source" for consistent usage across plugins or safely ignore this message and continue to use the current value
2024-12-11T15:10:55Z D! [agent] Connecting outputs
2024-12-11T15:10:55Z D! [agent] Attempting connection to [outputs.opensearch]
2024-12-11T15:10:55Z D! [outputs.opensearch] Template telegraf created or updated
2024-12-11T15:10:55Z D! [agent] Successfully connected to outputs.opensearch
2024-12-11T15:10:55Z D! [agent] Starting service inputs
2024-12-11T15:10:55Z D! [aggregators.basicstats] Updated aggregation range [2024-12-11 15:10:30 +0000 UTC, 2024-12-11 15:11:00 +0000 UTC]
2024-12-11T15:11:00Z D! [aggregators.basicstats] Updated aggregation range [2024-12-11 15:11:00 +0000 UTC, 2024-12-11 15:11:30 +0000 UTC]
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 44xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 5IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 5YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 5oxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 54xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 6IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 6YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 6oxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 64xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 7IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 7YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 7oxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 74xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 8IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 8YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 8oxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 84xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 9IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 9YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 9oxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- 94xGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- -IxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- -YxGtpMBV_R7Wv7TEBjr 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- -oxGtpMBV_R7Wv7TEBj9 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- -4xGtpMBV_R7Wv7TEBj- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- _IxGtpMBV_R7Wv7TEBj- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- _YxGtpMBV_R7Wv7TEBj- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- _oxGtpMBV_R7Wv7TEBj- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- _4xGtpMBV_R7Wv7TEBj- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- AIxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- AYxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- AoxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- A4xGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- BIxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- BYxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- BoxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- B4xGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- CIxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- CYxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- CoxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- C4xGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- DIxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- DYxGtpMBV_R7Wv7TEBn- 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- DoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- D4xGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- EIxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- EYxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- EoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- E4xGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- FIxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- FYxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- FoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- F4xGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- GIxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- GYxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- GoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- G4xGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- HIxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- HYxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- HoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- H4xGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- IIxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- IYxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- IoxGtpMBV_R7Wv7TERkJ 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- I4xGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- JIxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- JYxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- JoxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- J4xGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- KIxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- KYxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- KoxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- K4xGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- LIxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- LYxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- LoxGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Indexed to OpenSearch with status- [201] Result- created DocumentID- L4xGtpMBV_R7Wv7TERkY 
2024-12-11T15:11:05Z D! [outputs.opensearch] Successfully indexed [77] documents
2024-12-11T15:11:05Z D! [outputs.opensearch] Successfully indexed [0] documents
2024-12-11T15:11:05Z D! [outputs.opensearch] Wrote batch of 77 metrics in 121.426753ms
2024-12-11T15:11:05Z D! [outputs.opensearch] Buffer fullness: 0 / 10000 metrics
#########
Until the OOM the logs remain the same

System info

Ubuntu 22.04 LTS, Telegraf 1.33 (earlier version of Telegraf had the same issue for me)

Docker

No response

Steps to reproduce

  1. Use Telegraf 1.33
  2. Use provided config above
  3. Start Telegraf

Expected behavior

Stable work with adequate system resources consumption

Actual behavior

After Telegraf start-up, it consumes 130 MB of memory. In my measurement, after 15 min memory usage grows up to 4 GB, eventually causing OOM later when taking all available memory.

Additional info

heap1
heap2
@bg4erem bg4erem added the bug unexpected problem or unintended behavior label Dec 11, 2024
@srebhan
Copy link
Member

srebhan commented Dec 16, 2024

@bg4erem thanks for your report and your investigations! Could you try to test your setup by replacing the opensearch output with the discard output to narrow down which plugin causes the issue?

Could you please also check the Telegraf log to see if there is any hint for an issue?!?

@bg4erem
Copy link
Author

bg4erem commented Dec 17, 2024

Hi Sven @srebhan
I can confirm that in the same config but with outputs.opensearch commented out and outputs.discard enabled, there is no memory leak. Memory consumption stays stable and is about 175 M
heap

@bg4erem bg4erem changed the title Memory Leak (inputs.snmp or outputs.opensearch ??) Memory Leak (outputs.opensearch ??) Dec 17, 2024
@srebhan
Copy link
Member

srebhan commented Dec 18, 2024

Thanks @bg4erem! After a quick glance, this seems to be related to the "bulk indexer" handling in the code. I have to read deeper into this... One thing you can try is to reduce the metric_batch_size say to 100 and see if the memory stays in bounds.

As the team is out during Christmas and only be back on Jan 13th, I will take a look as soon as I'm back...

@srebhan srebhan self-assigned this Dec 18, 2024
@bg4erem
Copy link
Author

bg4erem commented Dec 18, 2024

Hi @srebhan ! Thank you. I set metric_batch_size to 100. However, Telegraf still has the memory problem, usage grows from 130M to 4300M in 11 min

@srebhan
Copy link
Member

srebhan commented Dec 20, 2024

Thanks a lot @bg4erem! Will take a look as soon as I'm back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@srebhan srebhan

Labels
bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@srebhan @bg4erem