Test new configs logstash #77
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added the feature for size and time based flushing of buffer. Added config options for max_interval and max_size. Once either one is reached the events stored in the buffer will be flushed.
Added the feature for size and time based flushing of buffer. Added config options for max_interval and max_size. Once either one is reached the events stored in the buffer will be flushed. Updated kusto.rb
tanmaya-panda1
suggested changes
Sep 18, 2024
| end | ||
| end | ||
|
|
||
| def flush |
There was a problem hiding this comment.
Can we add error handling in the flush method. If the flush callback raises an exception, it could cause the buffer to stop working properly.
| def flush | ||
| return if @buffer.empty? | ||
|
|
||
| @flush_callback.call(@buffer) |
There was a problem hiding this comment.
Also, during gracefully shutdown of the buffer, how do we ensure all events are flushed before the application exits? may be add a graceful shutdown
| class CustomSizeBasedBuffer | ||
| def initialize(max_size, max_interval, &flush_callback) | ||
| @max_size = max_size | ||
| @max_interval = max_interval |
There was a problem hiding this comment.
Considering the flush thread sleeps for the entire @max_interval. If the interval is large, it might introduce a considerable delay the buffer flush. Can we do something like this
def start_flusher_thread
@flusher_thread = Thread.new do
loop do
sleep @max_interval
flush_if_needed
end
end
end
def flush_if_needed
@mutex.synchronize do
if Time.now - @last_flush_time >= @max_interval
flush
end
end
end
Added the feature for size and time based flushing of buffer. Added config options for max_interval and max_size. Once either one is reached the events stored in the buffer will be flushed. Updated kusto.rb
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
lib/logstash/outputs/kusto.rb
Outdated
|
|
||
| # Maximum interval (in seconds) before the buffer gets flushed, defaults to 60 | ||
| config :max_interval, validate: :number, default: 60 | ||
|
|
||
| default :codec, 'json_lines' | ||
|
|
||
| def register | ||
| require 'fileutils' # For mkdir_p |
There was a problem hiding this comment.
How does it all add up ? It is just a buffer that is added.
Currently there is events written to file and then these files are flushed. With the new approach these files should go away
Right now the code changes are
a) Buffer that is taken from LA plugin
b) We write the events to the buffer and flush. How is this message encoded ?
c) can we remove the file part ? or the file buffers based on size on file ?
d) Where are the Tests ?
e) Time based flushes ?
Added the feature for size and time based flushing of buffer. Added config options for max_interval and max_size. Once either one is reached the events stored in the buffer will be flushed. Updated kusto.rb and ingestor.rb with the implementation of memory based buffer and flushing and removed file based buffer
… into testNewConfigsLogstash
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
Testing spec files
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Error
Testing spec files
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
Testing spec files
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
Testing spec files
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
This comment has been minimized.
This comment has been minimized.
monishkadas-ms
temporarily deployed
to
build
GitHub Actions
Inactive
Testing spec files
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
** Updated the kusto_spec.rb test
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Removed the temp file buffer and added retry to prevent data loss
This comment has been minimized.
This comment has been minimized.
Added exponential backoff to retries. Removed max_retries.
This comment has been minimized.
This comment has been minimized.
Updated upload_async and upload to raise errors during network downtime to trigger rescue block in flush buffer so the data sent for flushing can be restored and the flush can be attempted later. Added file buffer to store data ONLY when flush fails due to network issues. Once the network is back online the each file in the buffer is flushed and deleted first and the regular in-memory buffer is used post that.
Persists the incoming data during network down time. Checks for network availabilty before retrying.
monishkadas-ms
had a problem deploying
to
build
GitHub Actions
Failure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.