v1.5.3

1.5.3 (2022-11-23)

Added

• Multiple test cases for escaped string parsed with main/ECGP parsers

Fixed

• Escaped string in value for both ECGP and argument
• Token prefix not handled in

Changed

• Use firepit time function for timestamp parsing
• Update Lark rule transform to vtrans to avoid Lark special function misfire

Removed

• Explicit dependency python-dateutil

v1.5.2

Added

• Relative path support for environment variable starting with KESTREL #248
• Relative path support for path in LOAD/SAVE
• Relative path support for local uri, i.e., file://xxx or file://./xxx in GET
• Unit test on relative path in environment variable
• Unit test on relative path in LOAD
• Unit test on relative path in data source in GET

v1.5.1

Added

• Type checking in kestrel.semantics.reference
• New exception MissingDataSource
• Unit test on variable reference in GET
• Unit test on last data source reuse

Fixed

• Missing data source if not specified #257
• SymbolTable type error in code generation

Removed

• Obsoleted exception UnsupportedStixSyntax

A Little Beyond STIX Pattern

To be more friendly in the WHERE clause than strict STIX pattern, we introduce Extended Centered Graph Pattern (ECGP) in v1.5.0, plus complete Kestrel parser upgrade with multiple fixes (closing all issues in the Parser Upgrade milestone).

• ECGP is STIX compatible, which means one can use STIX in WHERE clause as before.

• The example of ECGP in WHERE (note that the host/endpoint is specified in a datasource, e.g., Elastic index, to avoid unnecessary data to retrieve by user or system generated queries):

• Documentation on ECGP will come in v1.5.1

• Full changelog:

Added

• Introduce ExtendedCenteredGraphPattern (ECGP) for WHERE clause

  • Support optional SCO/entity type for centered graph (STIX compatible)
  • Support optional square brackets (STIX compatible)
  • Support Single or double quotes (STIX compatible)
  • Support nested list as value (STIX compatible)
  • Support Kestrel variable as reference
  • Support escaped characters in quoted value
  • Support ECGP to string/STIX/firepit transformation
  • Support ECGP pruning (centered or extended components)
  • Support ECGP merge/extend with another ECGP
  • Parse into STIX (now ECGP) #14
  • Normalize WHERE clause between GET and expression
  • Add WHERE clause to command FIND

• Upgrade arguments (in APPLY command)

  • Support quoted string in arguments #170
  • dereferring variables in arguments

• Upgrade path (in GET/APPLY/LOAD/SAVE command)

  • Support escaped characters in quoted datasrc/analytics/path

• Upgrade JSON parser for command NEW

• Upgrade operators in syntax to be case insensitive

• Upgrade timespan

  • absolute timespan without t and quotes
  • relative timespan for FIND

• Upgrade prefetch with WHERE clause to eliminate unnecessary query

• Multiple test cases for new syntax and features

• Add macOS (arm64) install requirement to documentation

Changed

• Limit STIXPATH to ATTRIBUTE

  • command: SORT, GROUP, JOIN
  • expression clause: sort, attr

• Use explicit list like (1,2,3) or [1,2,3] for multi-value argument

• Formalize semantics processor in parser-semantics-codegen procedure

  • variable dereferencing in semantics processor
  • variable timerange extraction in semantics processor

v1.4.2

Added

• links to Black Hat 2022 website, recording, and demo/lab
• Kestrel logo in PNG
• link to the Kestrel binder service blog post

Fixed

• consistent stix-shifter and connector versions

Changed

• lowercase grammar strings

v1.4.1

Added

• multi-user cache folder support in debug mode #236
• ppid used in process identification (post-prefetch) #238
• process identification upgraded to a two-step approach
• fine-grained process identification time offsets
• per entity type prefetch config support #241
• support for automatically converting input files to STIX in stixbundle interface

Fixed

• prefetch when parent_ref not in process table
• false positives in generic relation resolution
• second execution of a failed query should raise exception
• master runtime directory test case fix
• ~ support in config file path (env var)

1.4.0

This release adds 2 new language features: relative timespans in place of exact timestamps in STIX patterns, and the ability to "bin" (aka "bucket") grouping attributes. "Binning" is a means of aggregating multiple entities into a single aggregate using a range of values (e.g. 5 minutes instead of grouping b exact timestamps).

Fixed

• Fix NameError: name 'DataSourceError' is not defined
• Pass stix-shifter profile options into translation #230

Added

• Relative timespans instead of START/STOP #181
  • e.g. LAST 5 MINUTES
• Group by "binned" (or "bucketed") attributes
  • e.g. GROUP foo BY BIN(first_observed, 5m)

Changed

• bump min Python version to 3.7
• update OCA slack invitation link

v1.3.4

Kestrel binder service now supports dynamically adding data sources.

Fixed

• broken /tmp/kestrel symbol link will crash a new session
• double close (double release resources) with context manager and aexit
• AttributeError with timestamped grouped variable #224
• subsequent GET would return no results #228

Added

• documentation on macOS debug folder path
• interface figure updated with new planned interfaces
• dynamically load stix-shifter YAML profiles #227
• new exception: MissingEntityAttribute
• unit test: disp timestamped group by

Changed

• codecov GitHub App enabled instead of codecov-bot
• stixshifter interface module connector split from interface.

v1.3.3

Fixed

• Jupyter kernel crashing upon restart

v1.3.2

Summary

Stabilize v1.3 with many bug fixes; improve auto-completion; add code coverage.

Details

See CHANGELOG.rst for complete info.

Added

• runtime warning generation for invalid entity type #200
• auto-complete relation in FIND
• auto-complete BY and variable in FIND
• add logo to readthedocs
• upgrade auto-complete keywords to be case sensitive #213
• add testing coverage into github workflows
• add codecov badge to README
• 31 unit tests for auto-completion
• the first unit test for JOIN
• two unit tests for ASSIGN
• five unit tests for EXPRESSION
• use tmp dir for generated testing data
• auto-deref with mixed ipv4/ipv6 in network-traffic

Fixed

• missing _refs handling for 2 cases out of 4 #205
• incorrectly derefering attributes after GROUP BY
• incorrectly yielding variable when auto-completing relation in FIND
• pylint errors about undefined-variables

Changed

• update grammar to separate commands yielding (or not) a variable
• change FUNCNAME from a terminal to an inlined rule
• differentiate the terminal "by"i between FIND and SORT/GROUP