Merge pull request #219 from opencybersecurityalliance/develop

v1.3.2

.github/workflows/code-style.yml

@@ -32,4 +32,4 @@ jobs:
           python -m pip install black
           python -m pip install .
       - name: Code style check (please black your code)
-        run: black --check src/
+        run: python -m black --check src/

.github/workflows/stixshifter-module-verification.yml

@@ -23,4 +23,4 @@ jobs:
           python -m pip install pytest
           python -m pip install .
       - name: Sample STIX-shifter Connector Package Verification on PyPI
-        run: pytest -vv tests/test_stixshifter.py -k test_verify_package_origin
+        run: python -m pytest -vv tests/test_stixshifter.py -k test_verify_package_origin

.github/workflows/unit-testing.yml

@@ -29,12 +29,21 @@ jobs:
         uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
-      - name: Install Kestrel package
+      - name: Install Kestrel
         run: |
           python -m pip install --upgrade pip
           python -m pip install --upgrade setuptools
-          python -m pip install pytest
           python -m pip install .
-          python -m pip install stix-shifter-modules-stix_bundle
-      - name: Unit testing
-        run: pytest -vv
+      - name: Unit testing with coverage report
+        run: |
+          python -m pip install pytest
+          python -m pip install pytest-cov
+          # pytest-cov does not support automatic sub-package recognition in src/
+          # pass in all sub-package names through multiple --cov arguments
+          # use ls, xargs, and sed carefully regarding diff between mac/BSD and linux
+          python -m pytest -vv --cov-report=xml $(ls src | grep -v '.egg-info' | xargs | sed -r 's/^| / --cov=/g')
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          fail_ci_if_error: true
+          verbose: true

.github/workflows/unused-import.yml

@@ -32,4 +32,4 @@ jobs:
           python -m pip install unimport
           python -m pip install .
       - name: Check
-        run: unimport --check --exclude __init__.py src/
+        run: python -m unimport --check --exclude __init__.py src/

CHANGELOG.rst

@@ -6,6 +6,41 @@ All notable changes to this project will be documented in this file.
 
 The format is based on `Keep a Changelog`_.
 
+1.3.2 (2022-04-22)
+==================
+
+Added
+-----
+
+- runtime warning generation for invalid entity type #200
+- auto-complete relation in FIND
+- auto-complete BY and variable in FIND
+- add logo to readthedocs
+- upgrade auto-complete keywords to be case sensitive #213
+- add testing coverage into github workflows
+- add codecov badge to README
+- 31 unit tests for auto-completion
+- the first unit test for JOIN
+- two unit tests for ASSIGN
+- five unit tests for EXPRESSION
+- use tmp dir for generated testing data
+- auto-deref with mixed ipv4/ipv6 in network-traffic
+
+Fixed
+-----
+
+- missing ``_refs`` handling for 2 cases out of 4 #205
+- incorrectly derefering attributes after GROUP BY
+- incorrectly yielding variable when auto-completing relation in FIND
+- pylint errors about undefined-variables
+
+Changed
+-------
+
+- update grammar to separate commands yielding (or not) a variable
+- change FUNCNAME from a terminal to an inlined rule
+- differentiate the terminal "by"i between FIND and SORT/GROUP
+
 1.3.1 (2022-04-16)
 ==================
 

GOVERNANCE.rst

@@ -46,43 +46,44 @@ Release Procedure
 
 A maintainer should release a new Kestrel runtime (PyPI package name: ``kestre-lang``) following the procedure:
 
-1. Update version and changelog
+#. Update version and changelog
 
-    - Sync the local git repo to the latest of the ``develop`` branch.
-    - Update the ``version`` field in ``setup.cfg``.
-    - Add changes in ``CHANGELOG.rst`` under a new version section.
-    - Add new contributors to ``AUTHORS.rst`` if any.
-    - Commit the updates with the new version number as the message.
-    - Push the local ``develop`` branch to remote.
+    #. Sync the local git repo to the latest of the ``develop`` branch.
+    #. Update the ``version`` field in ``setup.cfg``.
+    #. Add changes in ``CHANGELOG.rst`` under a new version section.
+    #. Add new contributors to ``AUTHORS.rst`` if any.
+    #. Commit the updates with the new version number as the message.
+    #. Push the local ``develop`` branch to remote.
 
-2. Graduate code to the ``release`` branch
+#. Graduate code to the ``release`` branch
 
-    - Open a PR to merge the ``develop`` branch to the ``release`` branch. Use the version number as the PR title.
+    #. Open a PR to merge the ``develop`` branch to the ``release`` branch. Use the version number as the PR title.
 
-    - Merge the PR.
+    #. Merge the PR.
 
-3. Create a new release
+#. Create a new release
 
-    - Go to the release page and click *Draft a new release*.
+    #. Go to the release page and click *Draft a new release*.
 
-    - Type the version number as the new tag to create.
+    #. Type the version number as the new tag to create.
 
-    - Choose ``release`` branch as the *Target*.
+    #. Choose ``release`` branch as the *Target*.
 
-    - Specify a release title. Use the version number for ordinary release.
+    #. Specify a release title. Use the version number for ordinary release.
 
-    - Write a summary of the release.
+    #. Write a summary of the release.
 
         - Patch number release: copy the CHANGELOG entries.
 
         - Minor number release: may have a TLDR at the beginning highlighting the most important new feature.
 
-    - Hit the *Publish release* button.
+    #. Hit the *Publish release* button.
 
-4. After release check
+#. After release check
 
     - Check `kestrel-lang on PyPI`_ after a few minutes to confirm new package built and released.
     - May activate/pin the released version of Kestrel documentation at `readthedocs version control`_.
+    - Announce the release at OCA Kestrel channel.
 
 Vulnerability Disclosure
 ------------------------

README.rst

@@ -10,6 +10,10 @@
         :target: https://github.com/psf/black
         :alt: Code Style: Black
 
+.. image:: https://codecov.io/gh/opencybersecurityalliance/kestrel-lang/branch/develop/graph/badge.svg?token=HM4ax10IW3
+        :target: https://codecov.io/gh/opencybersecurityalliance/kestrel-lang
+        :alt: Code Coverage
+
 .. image:: https://img.shields.io/pypi/v/kestrel-lang
         :target: https://pypi.python.org/pypi/kestrel-lang
         :alt: Latest Version
@@ -114,7 +118,7 @@ Kestrel Hunting Blogs
 Talks And Demos
 ===============
 
-- 2022/04 `SC eSummit on Threat Hunting & Offense Security`_ (register to watch for free)
+- 2022/04 `SC eSummit on Threat Hunting & Offense Security`_ (free to register/playback)
 - 2021/12 `Infosec Jupyterthon 2021`_ [`IJ'21 live hunt recording`_]
 - 2021/11 `BlackHat Europe 2021`_
 - 2021/10 `SANS Threat Hunting Summit 2021`_: [`SANS'21 session recording`_]

codecov.yml

@@ -0,0 +1,2 @@
+ignore:
+  - "tests"

docs/_static/css/logo.css

@@ -0,0 +1,8 @@
+/*
+`width:auto` was rendering 0px wide for .svg files
+https://stackoverflow.com/questions/59215996/how-to-add-a-logo-to-my-readthedocs-logo-rendering-at-0px-wide
+*/
+.wy-side-nav-search .wy-dropdown > a img.logo, .wy-side-nav-search > a img.logo {
+    width: 241px;
+    margin-top: 15px;
+}

docs/conf.py

@@ -35,3 +35,12 @@ def get_version():
 html_title = project
 html_theme = "sphinx_rtd_theme"
 highlight_language = "none"
+html_logo = "../logo/logo_w_text_white.svg"
+html_theme_options = {
+    'logo_only': True,
+    'display_version': False,
+}
+html_static_path = ['_static']
+html_css_files = [
+    'css/logo.css',
+]

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = kestrel-lang
-version = 1.3.1
+version = 1.3.2
 description = Kestrel Threat Hunting Language
 long_description = file:README.rst
 long_description_content_type = text/x-rst
@@ -37,7 +37,7 @@ install_requires =
     docker>=5.0.0
     stix-shifter>=3.6.0
     stix-shifter-utils>=3.6.0
-    firepit>=2.0.0
+    firepit>=2.0.1
 tests_require =
     pytest
 

src/kestrel/absinterface/manager.py

@@ -54,9 +54,9 @@ def _parse_and_complete_uri(self, uri):
     def _get_interface_with_config(self, scheme):
         scheme = scheme.lower()
         if scheme not in self.scheme_to_interface:
-            raise nonexist_interface_exception(scheme)
+            raise self.nonexist_interface_exception(scheme)
         if scheme not in self.scheme_to_interface_name:
-            raise nonexist_interface_exception(scheme)
+            raise self.nonexist_interface_exception(scheme)
         interface_name = self.scheme_to_interface_name[scheme]
         interface_config = self.config[self.config_root_key][interface_name]
         interface = self.scheme_to_interface[scheme]

src/kestrel/codegen/commands.py

@@ -23,16 +23,18 @@
 import itertools
 from collections import OrderedDict
 
-from firepit.query import Column, Limit, Offset, Order, Projection, Query
+from firepit.deref import auto_deref
+from firepit.query import Limit, Offset, Order, Projection, Query
 from firepit.stix20 import summarize_pattern
 
 from kestrel.utils import remove_empty_dicts, dedup_ordered_dicts
 from kestrel.exceptions import *
 from kestrel.semantics import get_entity_table, get_entity_type
 from kestrel.symboltable import new_var
 from kestrel.syntax.parser import get_all_input_var_names
+from kestrel.syntax.utils import get_entity_types
 from kestrel.codegen.data import load_data, load_data_file, dump_data_to_file
-from kestrel.codegen.display import DisplayDataframe, DisplayDict
+from kestrel.codegen.display import DisplayDataframe, DisplayDict, DisplayWarning
 from kestrel.codegen.pattern import build_pattern, build_pattern_from_ids
 from kestrel.codegen.queries import (
     compile_specific_relation_to_query,
@@ -226,6 +228,7 @@ def get(stmt, session):
     return_type = stmt["type"]
     start_offset = session.config["stixquery"]["timerange_start_offset"]
     end_offset = session.config["stixquery"]["timerange_stop_offset"]
+    display = None
 
     pattern = build_pattern(
         stmt["patternbody"],
@@ -237,10 +240,14 @@ def get(stmt, session):
     )
 
     if "variablesource" in stmt:
+        input_type = get_entity_table(stmt["variablesource"], session.symtable)
+        output_type = stmt["type"]
+        if input_type != output_type:
+            pass  # TODO: new exception type?
         session.store.filter(
             stmt["output"],
             stmt["type"],
-            get_entity_table(stmt["variablesource"], session.symtable),
+            input_type,
             pattern,
         )
         output = new_var(session.store, return_var_table, [], stmt, session.symtable)
@@ -325,10 +332,15 @@ def get(stmt, session):
 
         output = new_var(session.store, return_var_table, [], stmt, session.symtable)
 
+        if not len(output):
+            if not return_type.startswith("x-") and return_type not in (
+                set(session.store.types()) | set(get_entity_types())
+            ):
+                display = DisplayWarning(f'unknown entity type "{return_type}"')
     else:
         raise KestrelInternalError(f"unknown type of source in {str(stmt)}")
 
-    return output, None
+    return output, display
 
 
 @_debug_logger
@@ -602,43 +614,38 @@ def _filter_prefetched_process(
     id_pattern = build_pattern_from_ids(return_type, entity_ids)
     if id_pattern:
         session.store.extract(prefetch_filtered_var_name, return_type, None, id_pattern)
-        _logger.debug(f"filter successful.")
+        _logger.debug("filter successful.")
         return prefetch_filtered_var_name
     else:
         _logger.info("no prefetched process found after filtering.")
         return None
 
 
 def _set_projection(store, entity_table, query, paths):
-    proj = []
-    cols = store.columns(entity_table)
-    joined = set()
-    for path in paths:
-        if path == "*":
-            return
-        if "_ref" in path:  # This seems like a hack
-            joins, table, column = store.path_joins(entity_table, None, path)
-            if table not in joined:
-                query.extend(joins)
-                joined.add(table)
-            proj.append(Column(column, table, path))
-        elif path in cols:
-            # Prevent any ambiguity
-            proj.append(Column(path, entity_table))
-        else:
-            # Not sure where it came from
-            proj.append(path)
-    query.append(Projection(proj))
+    joins, proj = auto_deref(store, entity_table, paths=paths)
+    query.joins.extend(joins)
+    if query.proj:
+        # Need to merge projections?  More-specific overrides less-specific ("*")
+        new_cols = []
+        for p in query.proj.cols:
+            if not (hasattr(p, "table") and p.table == entity_table and p.name == "*"):
+                new_cols.append(p)
+        for p in proj.cols:
+            if not (hasattr(p, "table") and p.table == entity_table and p.name == "*"):
+                new_cols.append(p)
+        query.proj = Projection(new_cols)
+    else:
+        query.proj = proj
 
 
 def _build_query(store, entity_table, qry, stmt):
     where = stmt.get("where")
     if where:
+        where.set_table(entity_table)
         qry.append(where)
     attrs = stmt.get("attrs", "*")
-    if attrs != "*":
-        cols = attrs.split(",")
-        _set_projection(store, entity_table, qry, cols)
+    cols = attrs.split(",")
+    _set_projection(store, entity_table, qry, cols)
     sort_by = stmt.get("path")
     if sort_by:
         direction = "ASC" if stmt["ascending"] else "DESC"