Merge pull request #255 from opencybersecurityalliance/develop

v1.5.0

CHANGELOG.rst

@@ -6,6 +6,67 @@ All notable changes to this project will be documented in this file.
 
 The format is based on `Keep a Changelog`_.
 
+Unreleased
+==========
+
+1.5.0 (2022-10-24)
+==================
+
+Added
+-----
+- Introduce ExtendedCenteredGraphPattern (ECGP) for WHERE clause
+
+    - Support optional SCO/entity type for centered graph (STIX compatible)
+    - Support optional square brackets (STIX compatible)
+    - Support Single or double quotes (STIX compatible)
+    - Support nested list as value (STIX compatible)
+    - Support Kestrel variable as reference
+    - Support escaped characters in quoted value
+    - Support ECGP to string/STIX/firepit transformation
+    - Support ECGP pruning (centered or extended components)
+    - Support ECGP merge/extend with another ECGP
+    - Parse into STIX (now ECGP) #14
+    - Normalize WHERE clause between GET and expression
+    - Add WHERE clause to command FIND
+
+- Upgrade arguments (in APPLY command)
+
+    - Support quoted string in arguments #170
+    - dereferring variables in arguments
+
+- Upgrade path (in GET/APPLY/LOAD/SAVE command)
+
+    - Support escaped characters in quoted datasrc/analytics/path
+
+- Upgrade JSON parser for command NEW
+
+- Upgrade operators in syntax to be case insensitive
+
+- Upgrade timespan
+
+    - absolute timespan without ``t`` and quotes
+    - relative timespan for FIND
+
+- Upgrade prefetch with WHERE clause to eliminate unnecessary query
+
+- Multiple test cases for new syntax and features
+
+- Add macOS (arm64) install requirement to documentation
+
+Changed
+-------
+- Limit STIXPATH to ATTRIBUTE
+
+    - command: SORT, GROUP, JOIN
+    - expression clause: sort, attr
+
+- Use explicit list like ``(1,2,3)`` or ``[1,2,3]`` for multi-value argument
+
+- Formalize *semantics processor* in parser-semantics-codegen procedure
+
+    - variable dereferencing in semantics processor
+    - variable timerange extraction in semantics processor
+
 1.4.2 (2022-09-26)
 ==================
 

docs/installation/runtime.rst

@@ -39,6 +39,13 @@ default configuration:
 
     $ sqlite3 --version
 
+macOS Specific Requirement
+--------------------------
+
+Full installation of `Xcode`_ is required, especially for arm64 architecture
+(2021-). ``xcode-select --install`` may not install Python header files, or set
+incorrect architecture argument for dependent package compilation.
+
 Runtime Installation
 ====================
 
@@ -125,6 +132,7 @@ What's to Do Next
 
 .. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
 .. _Python virtual environment: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
+.. _Xcode: https://developer.apple.com/xcode/
 .. _kestrel-lang: http://github.com/opencybersecurityalliance/kestrel-lang
 .. _kestrel-jupyter: http://github.com/opencybersecurityalliance/kestrel-jupyter
 .. _firepit: http://github.com/opencybersecurityalliance/firepit

docs/language.rst

@@ -560,9 +560,17 @@ Syntax
 ^^^^^^
 ::
 
-    APPLY analytics_identifier ON var1, var2, ... WITH x=1, y=abc
+    APPLY analytics_identifier ON var1, var2, ... WITH x=abc, y=(1,2,3), z=varx.pid
 
-- Input: The command takes in one or multiple variables.
+- Input: The command takes in one or multiple Kestrel variables such as ``var1,
+  var2, ...```.
+
+- Arguments: The ``WITH`` clause specifies arguments. Different parameters are
+  splitted by ``,``. Literal string, quoted string (with escaped characters),
+  list, and nested list are supported as values. Previous Kestrel variables
+  will be de-referenced if found, e.g., ``z=varx.pid`` will enumerate all
+  ``pid`` of variable ``varx``, which may be unfolded to ``4, 108, 8716``, and
+  the final argument is ``z=(4,108,8716)`` when passed to the analytics.
 
 - Execution: The command executes the analytics specified by
   ``analytics_identifier`` like ``docker://ip_domain_enrichment`` or
@@ -919,6 +927,9 @@ Syntax
   of the entities in ``oldvar``.
 - In the third form, ``oldvar`` will be filtered and the result assigned to ``newvar``.
 
+The ``WHERE`` condition supports *ExtendedCenterGraphPattern* and references of
+Kestrel variables can be used, which will be automatically dereferenced.
+
 Examples
 ^^^^^^^^
 ::
@@ -935,6 +946,11 @@ Examples
     # filter procs for WMIC commands with timestamps
     wmic_procs = TIMESTAMPED(procs) WHERE command_line LIKE '%wmic%'
 
+    # WHERE clause examples
+    p2 = procs WHERE pid IN (4, 198, 2874)
+    p3 = procs WHERE pid = p2.pid
+    p4 = procs WHERE pid IN (p2.pid, 8888, 10002)
+    p5 = procs WHERE pid = p2.pid AND name = "explorer.exe"
 
 MERGE
 -----

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = kestrel-lang
-version = 1.4.2
+version = 1.5.0
 description = Kestrel Threat Hunting Language
 long_description = file:README.rst
 long_description_content_type = text/x-rst
@@ -32,12 +32,13 @@ install_requires =
     python-dateutil
     pandas
     requests
-    lark-parser>=0.11.3
+    lark>=1.1.3
     pyarrow>=5.0.0
     docker>=5.0.0
-    stix-shifter>=4.2.2
-    stix-shifter-utils>=4.2.2
-    firepit>=2.3.3
+    stix-shifter>=4.4.0
+    stix-shifter-utils>=4.4.0
+    firepit>=2.3.6
+    typeguard
 tests_require =
     pytest
 

src/kestrel/analytics/interface.py

@@ -90,7 +90,7 @@ def execute(uri, argument_variables, config, session_id=None, parameters=None):
         Args:
             uri (str): the full URI including the scheme and analytics name.
 
-            argument_variables ([kestrel.symboltable.VarStruct]): the list of Kestrel variables as arguments.
+            argument_variables ([kestrel.symboltable.variable.VarStruct]): the list of Kestrel variables as arguments.
 
             config (dict): a layered list/dict that contains config for the
               interface and can be edited/updated by the interface.