🚧 Time Series as Dynamical Systems: Neural ODE Basics (#31)

* update checklist * add ts dataset: ECB exchange rate * update some of the dl sections * add neural ode intro * fix bibtex * add example of differential equations * add example of differential equations * add examples of first order ode * planning * update neural ode and dynamical systems * remove empty line * code samples for dataset * code samples for dataset * add neural ode results * mypy only for a module * mypy only for a module * mypy only for a module * test pandoc * test pandoc * add pandoc
emptymalei · Nov 27, 2023 · 4611d07 · 4611d07
1 parent 959ced4
commit 4611d07
Show file tree

Hide file tree

Showing 37 changed files with 1,508 additions and 115 deletions.
diff --git a/.github/workflows/publish-docs-test.yaml b/.github/workflows/publish-docs-test.yaml
@@ -28,4 +28,5 @@ jobs:
         with:
           python-version: "3.10"
       - run: pip install -r requirements.txt
+      - uses: nikeee/setup-pandoc@v1
       - run: mkdocs build
diff --git a/.github/workflows/publish-docs-wtih-pdf.yaml b/.github/workflows/publish-docs-wtih-pdf.yaml
@@ -27,6 +27,7 @@ jobs:
       - uses: actions/setup-python@v2
         with:
           python-version: "3.10"
+      - uses: nikeee/setup-pandoc@v1
       - run: google-chrome --version
       - run: echo $(which chrome)
       - run: sudo apt update && sudo apt install python3-cffi python3-brotli libpango-1.0-0 libpangoft2-1.0-0

diff --git a/.github/workflows/publish-docs.yaml b/.github/workflows/publish-docs.yaml
@@ -27,5 +27,6 @@ jobs:
       - uses: actions/setup-python@v2
         with:
           python-version: "3.10"
+      - uses: nikeee/setup-pandoc@v1
       - run: pip install -r requirements.txt
       - run: mkdocs gh-deploy --force
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -35,3 +35,22 @@ repos:
       - id: poetry-check
       - id: poetry-export
         args: ["-f", "requirements.txt", "-o", "requirements.txt"]
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.7.1
+    hooks:
+      - id: mypy
+        args: [--no-strict-optional, --ignore-missing-imports]
+        files: (dl/notebooks/ts_dl_utils/)
+  - repo: https://github.com/ambv/black
+    rev: 23.11.0
+    hooks:
+    - id: black
+      language: python
+      args:
+        - "--line-length=88"
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        name: isort (python)
+        args: ["--profile", "black"]
diff --git a/dl/assets/csl/nature-publishing-group-vancouver.csl b/dl/assets/csl/nature-publishing-group-vancouver.csl
@@ -0,0 +1,153 @@
+<?xml version="1.0" encoding="utf-8"?>
+<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-GB">
+  <info>
+    <title>Nature Publishing Group - Vancouver</title>
+    <id>http://www.zotero.org/styles/nature-publishing-group-vancouver</id>
+    <link href="http://www.zotero.org/styles/nature-publishing-group-vancouver" rel="self"/>
+    <link href="http://www.zotero.org/styles/bmj" rel="template"/>
+    <link href="http://www.nature.com/tpj/TPJ_gta.pdf" rel="documentation"/>
+    <author>
+      <name>Sebastian Karcher</name>
+    </author>
+    <category citation-format="numeric"/>
+    <category field="science"/>
+    <summary>Superscript Vancouver style for some NPG journals, with et-al-min="7" and et-al-use-first="6"</summary>
+    <updated>2014-06-22T12:00:00+00:00</updated>
+    <rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
+  </info>
+  <macro name="author">
+    <names variable="author" suffix=". ">
+      <name sort-separator=" " initialize-with="" name-as-sort-order="all" delimiter=", " delimiter-precedes-last="always" delimiter-precedes-et-al="never"/>
+      <label form="short" prefix=" (" suffix=")"/>
+      <et-al term="et-al" font-style="italic"/>
+      <substitute>
+        <names variable="editor"/>
+      </substitute>
+    </names>
+  </macro>
+  <macro name="editor">
+    <names variable="editor" suffix=". ">
+      <name sort-separator=" " initialize-with="" name-as-sort-order="all" delimiter=", " delimiter-precedes-last="always" delimiter-precedes-et-al="never"/>
+      <label form="short" prefix=" (" suffix=")" strip-periods="true"/>
+      <et-al term="et-al" font-style="italic"/>
+    </names>
+  </macro>
+  <macro name="publisher">
+    <group delimiter=", ">
+      <group delimiter=": ">
+        <text variable="publisher"/>
+        <text variable="publisher-place"/>
+      </group>
+      <group>
+        <date variable="issued">
+          <date-part name="year"/>
+        </date>
+      </group>
+    </group>
+  </macro>
+  <macro name="access">
+    <choose>
+      <if variable="page" match="none">
+        <choose>
+          <if variable="DOI">
+            <text variable="DOI" prefix=" doi:"/>
+          </if>
+          <else-if variable="URL">
+            <text variable="URL"/>
+            <group prefix=" (" suffix=")." delimiter=" ">
+              <text term="accessed"/>
+              <date variable="accessed">
+                <date-part name="day" prefix=" "/>
+                <date-part name="month" prefix=" " form="short" strip-periods="true"/>
+                <date-part name="year"/>
+              </date>
+            </group>
+          </else-if>
+        </choose>
+      </if>
+    </choose>
+  </macro>
+  <macro name="journal-title">
+    <choose>
+      <if type="article-journal article-magazine" match="any">
+        <text variable="container-title" form="short" font-style="italic" strip-periods="true"/>
+      </if>
+      <else>
+        <text variable="container-title" suffix=". " form="short"/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="title">
+    <choose>
+      <if type="book thesis" match="any">
+        <text variable="title" font-style="italic"/>
+      </if>
+      <else>
+        <text variable="title"/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="edition">
+    <choose>
+      <if is-numeric="edition">
+        <group delimiter=" ">
+          <number variable="edition" form="ordinal"/>
+          <text term="edition" form="short"/>
+        </group>
+      </if>
+      <else>
+        <text variable="edition" suffix="."/>
+      </else>
+    </choose>
+  </macro>
+  <citation collapse="citation-number">
+    <sort>
+      <key variable="citation-number"/>
+    </sort>
+    <layout delimiter="," vertical-align="sup">
+      <text variable="citation-number"/>
+    </layout>
+  </citation>
+  <bibliography et-al-min="7" et-al-use-first="6" second-field-align="flush">
+    <layout suffix=".">
+      <text variable="citation-number" suffix=" "/>
+      <text macro="author"/>
+      <text macro="title" suffix=". "/>
+      <choose>
+        <if type="bill book graphic legal_case legislation motion_picture report song" match="any">
+          <group prefix=" " delimiter=" ">
+            <text macro="edition"/>
+            <text macro="publisher"/>
+          </group>
+        </if>
+        <else-if type="chapter paper-conference" match="any">
+          <group prefix=" " suffix=". ">
+            <text term="in" suffix=": " text-case="capitalize-first"/>
+            <text macro="editor"/>
+            <text variable="container-title" font-style="italic"/>
+          </group>
+          <group prefix=" " delimiter=", ">
+            <text macro="publisher"/>
+            <group delimiter=" ">
+              <label variable="page" form="short" strip-periods="true"/>
+              <text variable="page"/>
+            </group>
+          </group>
+        </else-if>
+        <else>
+          <text macro="journal-title"/>
+          <group suffix=".">
+            <date variable="issued" prefix=" ">
+              <date-part name="year"/>
+            </date>
+            <group prefix="; ">
+              <text variable="volume" font-weight="bold"/>
+              <text variable="page" prefix=": "/>
+            </group>
+          </group>
+        </else>
+      </choose>
+      <text macro="access"/>
+    </layout>
+  </bibliography>
+</style>
diff --git a/dl/assets/references/dl-fundamentals.bib b/dl/assets/references/dl-fundamentals.bib
@@ -33,6 +33,35 @@ @online{Liu2020-yh
 }
 
 
+@online{Chen2018-mp,
+  title         = "Neural Ordinary Differential Equations",
+  author        = "Chen, Ricky T Q and Rubanova, Yulia and Bettencourt, Jesse
+                   and Duvenaud, David",
+  abstract      = "We introduce a new family of deep neural network models.
+                   Instead of specifying a discrete sequence of hidden layers,
+                   we parameterize the derivative of the hidden state using a
+                   neural network. The output of the network is computed using
+                   a black-box differential equation solver. These
+                   continuous-depth models have constant memory cost, adapt
+                   their evaluation strategy to each input, and can explicitly
+                   trade numerical precision for speed. We demonstrate these
+                   properties in continuous-depth residual networks and
+                   continuous-time latent variable models. We also construct
+                   continuous normalizing flows, a generative model that can
+                   train by maximum likelihood, without partitioning or
+                   ordering the data dimensions. For training, we show how to
+                   scalably backpropagate through any ODE solver, without
+                   access to its internal operations. This allows end-to-end
+                   training of ODEs within larger models.",
+  month         =  "19~" # jun,
+  year          =  2018,
+  url           = "http://arxiv.org/abs/1806.07366",
+  file          = "All Papers/CHEN/Chen et al. 2018 - Neural Ordinary Differential Equations.pdf",
+  archivePrefix = "arXiv",
+  eprint        = "1806.07366",
+  primaryClass  = "cs.LG",
+  arxivid       = "1806.07366"
+}
 
 @online{Le_Cun2006-ta,
   title  = "A Tutorial on {Energy-Based} Learning",
@@ -570,3 +599,29 @@ @online{Pascanu2012-qv
   primaryClass  = "cs.LG",
   eprint        = "1211.5063"
 }
+
+
+@MISC{Jaeger2013-vx,
+  title  = "A tutorial on training recurrent neural networks, covering {BPTT},
+            {RTRL}, {EKF} and the ``echo state network'' approach",
+  author = "Jaeger, Herbert and Kirkpatrick, James",
+  url    = "https://www.ai.rug.nl/minds/uploads/ESNTutorialRev.pdf",
+  file   = "All Papers/JAEGER/Jaeger and Kirkpatrick - A tutorial on training recurrent neural networks, covering BPTT, RTRL, EKF and the 'echo state network' approach.pdf"
+}
+
+
+@software{Poli_TorchDyn_Implicit_Models,
+author = {Poli, Michael and Massaroli, Stefano and Yamashita, Atsushi and Asama, Hajime and Park, Jinkyoo and Ermon, Stefano},
+title = {{TorchDyn: Implicit Models and Neural Numerical Methods in PyTorch}},
+url = {https://github.com/DiffEqML/torchdyn}
+}
+
+@software{Chen_torchdiffeq_2021,
+author = {Chen, Ricky T. Q.},
+license = {MIT},
+month = jun,
+title = {{torchdiffeq}},
+url = {https://github.com/rtqichen/torchdiffeq},
+version = {0.2.2},
+year = {2021}
+}
diff --git a/dl/assets/references/general.bib b/dl/assets/references/general.bib
@@ -130,3 +130,12 @@ @ARTICLE{Imai2021-kk
   issn      = "1047-1987,1476-4989",
   language  = "en"
 }
+
+
+@misc{enwiki:1186188179,
+author = "{Wikipedia contributors}",
+title = "Lorenz system --- {Wikipedia}{,} The Free Encyclopedia",
+year = "2023",
+url = "https://en.wikipedia.org/w/index.php?title=Lorenz_system&oldid=1186188179",
+note = "[Online; accessed 23-November-2023]"
+}
diff --git a/dl/dynamical-systems/assets/differential-equations/ode-exponential-growth.png b/dl/dynamical-systems/assets/differential-equations/ode-exponential-growth.png
diff --git a/dl/dynamical-systems/assets/differential-equations/ode_infection_model.png b/dl/dynamical-systems/assets/differential-equations/ode_infection_model.png
diff --git a/dl/dynamical-systems/assets/differential-equations/ode_receprocal.png b/dl/dynamical-systems/assets/differential-equations/ode_receprocal.png
diff --git a/dl/dynamical-systems/assets/differential-equations/ode_sin_t.png b/dl/dynamical-systems/assets/differential-equations/ode_sin_t.png
diff --git a/dl/dynamical-systems/assets/neural-ode/neural-ode-and-functions.jpg b/dl/dynamical-systems/assets/neural-ode/neural-ode-and-functions.jpg
diff --git a/dl/dynamical-systems/index.md b/dl/dynamical-systems/index.md
@@ -0,0 +1,29 @@
+# Dynamical Systems
+
+A lot of time series data are generated by dynamical systems. One of the most cited examples is the coordinates $x(t)$, $y(t)$, $z(t)$ as functions of time $t$ in a Lorenz system.
+
+!!! note "Lorenz System"
+
+    A Lorenz system is defined by the Lorenz equations[@enwiki:1186188179]
+
+    $$
+    \begin{align}
+    \frac{\mathrm{d}x}{\mathrm{d}t} &= \sigma (y - x), \\
+    \frac{\mathrm{d}y}{\mathrm{d}t} &= x (\rho - z) - y, \\
+    \frac{\mathrm{d}z}{\mathrm{d}t} &= x y - \beta z,
+    \end{align}
+    $$
+
+    where $x$, $y$, and $z$ are the coordinates of a particle.
+
+    It is a chaotic system that is very sensitive to the initial conditions.
+
+
+!!! note "Dynamical Systems"
+
+    Many real-world systems are dynamical systems. Differential equation is a handy tool to model a dynamical system. For example, the action potentials of a squid giant axon can be modeled by the famous [Hodgkin-Huxley model](https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model).
+
+
+A naive philosophy to model time series is to come up with a set of differential equations to model the time series. However, finding clean and interpretable differential equations is not easy. It has been the top game in physics for centuries.
+
+In the following sections, we will discuss a few solutions to model data as dynamical systems.