2023-10-18

Wrong tag, abandoned.

v1.7.5

1.7.5 (2023-09-07)

Added

• Kestrel Docker container image in DockerHub
• Documentation on how to use Kestrel Docker container
• case insensitive option (?i) for Elasticserach via stix-shifter data source interface (stix-shifter v6.2.1)

v1.7.4

Kestrel v1.7.4 with improved stix-shifter-diag tool and stix-shifter v6 (v6.0.3).

Added

• New simpler default STIX patterns for stix-shifter-diag
• Connector verification/install in stix-shifter-diag #388
• Custom pattern (string and file) support for stix-shifter-diag
• Debug info output support for stix-shifter-diag
• Current time as stop time support for default pattern in stix-shifter-diag
• Query-translate-only mode for stix-shifter-diag

Changed

• Fix/change the order of LIMIT and timespan in Lark file according to Kestrel doc
• stix-shifter update to v6 (6.0.3)

v1.7.3

Introducing the command line utility stix-shifter-diag to diagnose stix-shifter data source interface issues.

Check https://kestrel.readthedocs.io/en/latest/source/kestrel_datasource_stixshifter.interface.html for more details.

v1.7.2

1.7.2 (2023-07-26)

Added

• Minimal version requirements for all dependencies
• param cool_down_after_transmission in stix-shifter interface
• Unit tests on empty input variable for commands
• lark-js support for kestrel.lark #371

Changed

• Keep stix-shifter to v5 (not v6) to avoid a dependency specification issue

Fixed

• Fast translation bug on group keyword in stix-shifter mapping #370
• typeguard old version cause exception
• Exception with empty variable #254

v1.7.1

We add LIMIT keyword to GET and FIND for a user to sample the limited number of return from a live data source in the case that the total return is too large and may take many minutes/hours to retrieve.

• This makes it possible to hunt with super big data. In the super big data case, stix-shifter does not know how many entries remain, and Kestrel cannot estimate the entire retrieval time.
• Use the LIMIT keyword to get some samples of the return, check the return, refine the pattern in GET/FIND, and rerun to better hunt down the suspicious entities at a smaller scale (and finally without LIMIT in the GET/FIND)
• Check the syntax for GET and FIND in the Kestrel documentation for more details of its usage.

Known Issues

Fast translation is broken #370 and will be fixed in the next Kestrel release. Please do not enable fast translation in stix-shifter data source interface (stixshifter.yaml)

Added

• LIMIT keyword in GET/FIND
• LIMIT support in stix-shifter interface and stix-bundle interface
• Unit tests for LIMIT
• Documentation for LIMIT
• New transform function RECORD
• Documentation for RECORD
• Unit tests for RECORD

Changed

• Use prefetch results for GET/FIND if prefetched; instead of merging results with local/main query

Fixed

• stix-shifter interface translator error msg passing bugs
• stix-shifter interface transmitter error msg passing bug
• infinite loop in stix-shifter interface transmitter
• stix-shifter connector pip uninstall hanging issue
• prefetch logic error with empty return
• dataframe index error in CSV export

v1.7.0

1.7.0 (2023-06-14)

Kestrel v1.7.0 brings multi-process support to Kestrel in its stix-shifter data source interface. In addition, let's welcome our new contributor, @leila-rashidi, who implements the ADDOBSID transformer function in this release.

Added

• Multi-process support for stix-shifter data source interface

  • Each native data source query is executed in a subprocess
  • A pool of translators are created to pick up translation tasks for each transmitted page/batch
  • Ingestion is serialized in main proccess to avoid multi-process execution for SQLite
  • Two queues between transmitter/translator and translator/ingestor are used
  • Both stix-shifter translation and firepit fast-translation are supported
  • With debug flag, the translated results (JSON or DataFrame) will be dump to disk
  • Unit tests for the translator subprocess in different modes

• Additional syntax/keywords on singular timeunits

• New variable transformer function ADDOBSID

  • Add new syntax and codegen
  • Add additional documentatoin

• Unit tests on CLI

  • Invoking with kestrel x.hf
  • Invoking with python -m kestrel x.hf

Fixed

• No dumped data in stix-shifter interface when debug is enabled
• Multiprocessing conflict with runpy
• STIX-shifter module verification failure due to pypi website update

Removed

• Deprecated functions in kestrel/codegen/relations.py

Changed

• Examples in Kestrel config YAML

v1.6.1

1.6.1 (2023-05-31)

This patch release provides a temporary fix for stix-shifter #1493.

Changed

• Kestrel variable definition syntax changed back to CNAME from ECNAME

• stix-shifter data source profile config changes

  • Replace result_limit with retrieval_batch_size
  • Replace timeout with single_batch_timeout
  • Add default values for the configs
  • Document updates
  • The new fields will be processed by Kestrel before given to stix-shifter

• Replace stix-shifter sync APIs with async APIs

Added

• Scalability end-to-end testing for large query with multiple pages
• Test cases for new stix-shfiter data source configs

Fixed

• Temporary fix of stix-shifter/issues/1493

  • Add retry-once logic if server timeout (busy CPU on the client side)
  • Nullify the pipelining; need better long-term fix to enable it

• Fixed bugs and reimplement transmission_complete() in stix-shifter data source interface

v1.6.0

Major updates in Kestrel v1.6.0

1. Moving to stix-shifter v5, which makes it possible to do async calls [contribution from @cmadam]
2. Pipelining the transmission and translation/ingestion with async support in the Kestrel stix-shifter datasource interface, also upgrading the data-flow to be fileless [contribution from @tingdai]
3. Major Elasticsearch ECS mapping upgrade [contribution from @Harmedox to stix-shifter v5.3.0]
4. End-to-end testing framework + CI/CD connected with stix-shifter (live data source) and docker analytics [contribution from @cmadam]
5. Kestrel raw string implementation and documentation

Full changelog

Changed

• Upgrade stix-shifter from v4 to v5 in the stix-shifter datasource interface
• Bump stix-shifter version to v5.3.0 to include latest Elastcisearch ECS mappings
• Restrict scopes of Github workflows to eliminate unnecessary executions

Added

• stix-shifter datasource interface query procedure pipelining: a producer-consumer model for transmission and translation/ingestion
• Integration testing with stix-shifter and the first live data source---Elasticsearch
• Raw String implemented in Kestrel
• Documentation on raw String

Fixed

• Logging module reimplemented to fix #334
• asyncio bug in tests/test_fast_translate.py

Known issues

• There seems to be a bug in fast translation #341 with firepit 2.3.19. Please turn off fast translation with Kestrel 1.6.0. We will do a patch release to fix it.

v1.5.14

1.5.14 (2023-04-19)

Fixed

• A bug in firepit v2.3.16 when fast translation is in use; fixed in firepit v2.3.17
• Improved logic on prefetch skipping; fix #322
• Fixing several unit tests with the improved prefetch skipping logic