Merge pull request #149 from opencybersecurityalliance/develop

v1.1.5

CHANGELOG.rst

@@ -6,6 +6,15 @@ All notable changes to this project will be documented in this file.
 
 The format is based on `Keep a Changelog`_.
 
+1.1.5 (2021-11-08)
+==================
+
+Changed
+-------
+
+- dependency version bump for the open hunting stack (Black Hat Europe 2021)
+- installation documentation updates
+
 1.1.4 (2021-10-27)
 ==================
 

README.rst

@@ -69,37 +69,19 @@ do not have Python. It is preferred to install Kestrel runtime using `pip`_,
 and it is preferred to install Kestrel runtime in a `Python virtual
 environment`_.
 
-0. Update Python installer.
-
 .. code-block:: console
 
     $ pip install --upgrade pip setuptools wheel
-
-1. Install Kestrel runtime.
-
-.. code-block:: console
-
-    $ pip install kestrel-lang
-
-2. Install Kestrel Jupyter kernel if you use `Jupyter Notebook`_ to hunt.
-
-.. code-block:: console
-
     $ pip install kestrel-jupyter
     $ python -m kestrel_jupyter_kernel.setup
-
-3. (Optional) download Kestrel analytics examples for the ``APPLY`` hunt steps.
-
-.. code-block:: console
-
-    $ git clone https://github.com/opencybersecurityalliance/kestrel-analytics.git
+    $ jupyter notebook
 
 Hello World Hunt
 ================
 
 1. Copy the following 3-step hunt flow into your favorite text editor:
 
-.. code-block::
+.. code-block:: elixir
 
     # create four process entities in Kestrel and store them in the variable `proclist`
     proclist = NEW process [ {"name": "cmd.exe", "pid": "123"}
@@ -155,6 +137,7 @@ Kestrel Hunting Blogs
 #. `Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks`_
 #. `Practicing Backward And Forward Tracking Hunts on A Windows Host`_
 #. `Building Your Own Kestrel Analytics and Sharing With the Community`_
+#. `Setting Up The Open Hunting Stack in Hybrid Cloud With Kestrel and SysFlow`_
 
 Learning/Sharing With the Community
 ===================================
@@ -179,15 +162,14 @@ and Share Your Huntbook With the Community`_ to facilitate huntbook
 composition, sharing, and reuse. The session started from 3 simple hunt step
 demos---TTP pattern matching, provenance tracking, and data visualization
 analytics---then went into comprehensive hunt flow composition to convey the
-idea of hunting knowledge composition and reuse. The recording is currently
-available at SANS library and will be published by SANS.
+idea of hunting knowledge composition and reuse.
 
 Kestrel will be presented as part of the open hunting stack for hybrid cloud in
 Black Hat Europe Arsenal 2021 session: `An Open Stack for Threat Hunting in
 Hybrid Cloud With Connected Observability`_. We will hunt an APT in a hybrid
 cloud that is a variant of a typical supply chain attack yet implemented in a
 more stealthy manner. The open stack consisting of Kestrel, `SysFlow`_, and
-other open-source projects will be presented. 
+other open-source projects will be presented.
 
 Connecting With The Community
 =============================
@@ -213,9 +195,10 @@ Alliance workspace`_ and join our *kestrel* channel.
 .. _Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks: https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/
 .. _Practicing Backward And Forward Tracking Hunts on A Windows Host: https://opencybersecurityalliance.org/posts/kestrel-2021-08-16/
 .. _Building Your Own Kestrel Analytics and Sharing With the Community: https://opencybersecurityalliance.org/posts/kestrel-custom-analytics/
+.. _Setting Up The Open Hunting Stack in Hybrid Cloud With Kestrel and SysFlow: https://opencybersecurityalliance.org/posts/kestrel-sysflow-bheu21-open-hunting-stack/
 .. _Kestrel huntbook repo: https://github.com/opencybersecurityalliance/kestrel-huntbook
 .. _Kestrel analytics repo: https://github.com/opencybersecurityalliance/kestrel-analytics
-.. _SANS Threat Hunting Summit 2021: https://www.sans.org/cyber-security-summit/
-.. _Compose Your Hunts With Reusable Knowledge and Share Your Huntbook With the Community: https://www.sans.org/blog/a-visual-summary-of-sans-threat-hunting-summit-2021/
+.. _SANS Threat Hunting Summit 2021: https://www.sans.org/blog/a-visual-summary-of-sans-threat-hunting-summit-2021/
+.. _Compose Your Hunts With Reusable Knowledge and Share Your Huntbook With the Community: https://www.youtube.com/watch?v=gyY5DAWLwT0
 .. _An Open Stack for Threat Hunting in Hybrid Cloud With Connected Observability: https://www.blackhat.com/eu-21/arsenal/schedule/index.html#an-open-stack-for-threat-hunting-in-hybrid-cloud-with-connected-observability-25112
 .. _SysFlow: https://github.com/sysflow-telemetry

docs/Makefile

@@ -11,7 +11,7 @@ SPHINXBUILD   ?= sphinx-build
 SOURCEDIR     = .
 BUILDDIR      = _build
 
-clean :
+clean:
 	rm -r "$(BUILDDIR)"
 
 # Put it first so that "make" without argument is like "make help".

docs/installation.rst

@@ -2,8 +2,7 @@
 Installation
 ============
 
-Install the Kestrel runtime plus additional front ends such as Kestrel Jupyter
-Notebook kernel.
+Install the Kestrel runtime, Kestrel Jupyter front-end, and STIX-shifter connector modules.
 
 Operating Systems
 =================
@@ -55,10 +54,11 @@ requirement are:
 Runtime Installation
 ====================
 
-You can install Kestrel runtime from `stable release`_ or `source code (nightly
-built version)`_. Either way installs all packages in the ``kestrel-lang``
-repository, and dependent packages, such as ``firepit`` and ``stix-shifter``.
-See the architecture section in :doc:`overview` to understand more.
+You can install Kestrel runtime from `stable release`_ or `nightly built
+version (source code)`_. Either way installs all packages in the
+``kestrel-lang`` repository, and dependent packages, such as ``firepit`` and
+``stix-shifter``.  See the architecture section in :doc:`overview` to
+understand more.
 
 It is a good practice to install Kestrel in a `Python virtual environment`_.
 You can easily setup and activate one named *huntingspace*:
@@ -67,6 +67,7 @@ You can easily setup and activate one named *huntingspace*:
 
     $ python -m venv huntingspace
     $ . huntingspace/bin/activate
+    $ pip install --upgrade pip setuptools wheel
 
 Stable Release
 --------------
@@ -77,32 +78,20 @@ Run this command in your terminal:
 
     $ pip install kestrel-lang
 
-Source Code (Nightly Built Version)
+Nightly Built Version (Source Code)
 -----------------------------------
 
-1. Install and upgrade Python building packages ``setuptools`` and ``wheel``:
-
-.. code-block:: console
-
-    $ pip install --upgrade pip setuptools wheel
-
-2. Clone the source from the `Github repo`_:
+Run this command in your terminal:
 
 .. code-block:: console
 
     $ git clone git://github.com/opencybersecurityalliance/kestrel-lang
-    $ cd kestrel-lang
+    $ cd kestrel-lang && pip install .
 
-3. Install all packages from the repo:
-
-.. code-block:: console
+Front-Ends Installation
+=======================
 
-    $ pip install .
-
-Runtime Front Ends
-==================
-
-Kestrel runtime currently supports three front ends (see architecture figure in :doc:`overview`):
+Kestrel runtime currently supports three front-ends (see architecture figure in :doc:`overview`):
 
 1. Command-line execution utility ``kestrel``: Installed with the
    package ``kestrel``. 
@@ -124,7 +113,65 @@ Kestrel runtime currently supports three front ends (see architecture figure in
 
     - Start a Kestrel session in Python directly. See more at :doc:`source/kestrel.session`.
 
-    - Use `magic command`_ in iPython environment. ``kestrel-jupyter`` required.
+    - Use `magic command`_ in iPython environment. Check `kestrel-jupyter`_ package for usage.
+
+STIX-shifter Connector Installation
+===================================
+
+Among :ref:`data-source-and-analytics-interfaces`, STIX-shifter is the main
+data source interface currently implemented by the Kestrel runtime.
+`STIX-shifter`_ provides a federated search interface against more than a dozen
+EDRs, NDRs, and SIEM systems for data retrieval.
+
+Because of the federated nature of STIX-shifter, the project releases a string
+of Python packages (called *connectors* of STIX-shifter) for each data source.
+Depending on the data source you are connecting to, e.g., Sysmon data stored in
+Elasticsearch, you need to install the corresponding connector such as
+`stix-shifter-modules-elastic-ecs`:
+
+.. code-block:: console
+
+    $ pip install stix-shifter-modules-elastic-ecs
+
+STIX-shifter Data Source Config
+===============================
+
+After installing the STIX-shifter connector, you need to tell a Kestrel
+front-end, e.g., Jupyter, details of the data source you are connecting to.
+This is done by exporting three environment variables for each data source, e.g.:
+
+.. code-block:: console
+
+    $ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
+    $ export STIXSHIFTER_HOST101_CONNECTION='{"host":"elastic.securitylog.company.com", "port":9200, "indices":"host101"}'
+    $ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"VuaCfGcBCdbkQm-e5aOx", "api_key":"ui2lp2axTNmsyakw9tvNnw"}}'
+
+(Optional) Kestrel Analytics Download
+=====================================
+
+Want to have some Kestrel analytics ready at your fingertip? Threat
+intelligence enrichments like SANS API? Domain name lookup for IP addresses?
+Finding IP geolocations and pin them on an interactive map? Invoking machine
+learning inference function? Clone the community-contributed Kestrel analytics
+repo to start:
+
+.. code-block:: console
+
+    $ git clone https://github.com/opencybersecurityalliance/kestrel-analytics.git
+
+Go to the `analytics` directory and build the analytics docker containers to
+``APPLY`` in your hunt.
+
+Run Kestrel
+===========
+
+Now the Kestrel runtime is set up and you can run a Kestrel huntflow with the
+command-line utility or launch a Jupyter service for developing a huntbook
+interactively:
+
+.. code-block:: console
+
+   $ jupyter notebook
 
 .. _pip: https://pip.pypa.io
 .. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
@@ -134,3 +181,4 @@ Kestrel runtime currently supports three front ends (see architecture figure in
 .. _Jupyter Notebook: https://jupyter.org/
 .. _magic command: https://ipython.readthedocs.io/en/stable/interactive/magics.html
 .. _firepit: https://github.com/opencybersecurityalliance/firepit
+.. _STIX-shifter: https://github.com/opencybersecurityalliance/stix-shifter

docs/language.rst

@@ -273,7 +273,7 @@ Syntax
     - The pool can be a data source, for example, a data lake where monitored logs are
       stored, an EDR, a firewall, an IDS, a proxy server, or a SIEM system. In
       this case, the user needs to know the identifier of the data source (more
-      in section `Data And Analytics Interfaces`_). For example:
+      in section `Data Source And Analytics Interfaces`_). For example:
 
         - ``stixshifter://server101``: EDR on server 101 accessible via STIX-Shifter.
         - ``https://a.com/b.json``: sealed telemetry data in a STIX bundle.
@@ -535,7 +535,7 @@ Syntax
 
   There is no limitation for what an analytics could do besides the input and
   output specified by its corresponding Kestrel analytics interface (see `Data
-  And Analytics Interfaces`_).
+  Source And Analytics Interfaces`_).
 
   An analytics could run entirely locally and then just do a table lookup. It could
   reach out to the internet like the VirusTotal servers. It could perform
@@ -916,8 +916,10 @@ Comment
 
 Comment strings in Kestrel start with ``#`` to the end of the line.
 
-Data And Analytics Interfaces
-=============================
+.. _data-source-and-analytics-interfaces:
+
+Data Source And Analytics Interfaces
+====================================
 
 Kestrel aims to keep it open and easy to add data source and analytics---not
 only adding data source through the STIX-Shifter interface and adding analytics

docs/tutorial.rst

@@ -16,7 +16,7 @@ install Kestrel is to use pip:
 
 .. code-block:: console
 
-    $ pip install --upgrade pip
+    $ pip install --upgrade pip setuptools wheel
     $ pip install kestrel-lang
 
 If you need more control, check out the following guide on :doc:`installation` for more details.
@@ -200,8 +200,8 @@ STIX-Shifter is automatically installed when installing ``kestrel``. However,
 you need to install additional STIX-Shifter connector packages for each
 specific data sources. Example connectors:
 
-- Sysmon data in Elasticsearch: ``stix-shifter-modules-elastic_ecs``.
-- Sysflow data in Elasticsearch: ``stix-shifter-modules-elastic_ecs``.
+- Sysmon data in Elasticsearch: ``stix-shifter-modules-elastic-ecs``.
+- Sysflow data in Elasticsearch: ``stix-shifter-modules-elastic-ecs``.
 - CarbonBlack Response: ``stix-shifter-modules-carbonblack``.
 - CarbonBlack Cloud: ``stix-shifter-modules-cbcloud``.
 - IBM QRadar: ``stix-shifter-modules-qradar``.
@@ -210,7 +210,7 @@ For example, to access Sysmon data in Elasticsearch, install the corresponding c
 
 .. code-block:: console
 
-    $ pip install stix-shifter-modules-elastic_ecs
+    $ pip install stix-shifter-modules-elastic-ecs
 
 Suppose you set up an Elasticsearch server at ``elastic.securitylog.company.com``
 with default port ``9200``. You would add the Sysmon monitored host to it as index

setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = kestrel-lang
-version = 1.1.4
+version = 1.1.5
 description = Kestrel Threat Hunting Language
 long_description = file:README.rst
 long_description_content_type = text/x-rst
@@ -28,14 +28,14 @@ scripts = bin/kestrel
 python_requires = >= 3.6
 install_requires =
     toml
-    lark-parser>=0.11.3
     python-dateutil
     pandas
+    requests
+    lark-parser>=0.11.3
     pyarrow>=5.0.0
     docker>=5.0.0
-    requests[socks]
-    stix-shifter>=3.5.1
-    stix-shifter-utils>=3.5.1
+    stix-shifter>=3.6.0
+    stix-shifter-utils>=3.6.0
     firepit>=1.3.0
 tests_require =
     pytest