Merge Dev into Main July 10 2024

.github/workflows/bandit.yaml

@@ -0,0 +1,32 @@
+name: Bandit Security Linting
+
+on: [pull_request]
+
+jobs:
+  bandit:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install bandit
+
+    - name: Run Bandit
+      run: |
+        bandit -r middleware resources app.py
+
+    - name: Upload Bandit results
+      uses: actions/upload-artifact@v2
+      with:
+        name: bandit-report
+        path: bandit_output.txt
+

.github/workflows/pull.yaml

@@ -48,7 +48,7 @@ jobs:
     - name: Test with pytest
       run: |
         pip install pytest pytest-cov
-        pytest app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
+        pytest tests/resources/app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
 
   setup_client:
     defaults:

.github/workflows/python_checks.yml

@@ -18,4 +18,5 @@ jobs:
         uses: reviewdog/action-flake8@v3
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
+          flake8_args: --ignore E501,W291  # Does not check for max line exceed or trailing whitespace
           level: warning

.github/workflows/run_pytest.yml

@@ -0,0 +1,31 @@
+name: Test using Pytest
+
+on:
+  pull_request:
+
+jobs:
+  run_tests:
+    env:
+      SECRET_KEY: ${{ secrets.SECRET_KEY }}
+      DEV_DB_CONN_STRING: ${{secrets.DEV_DB_CONN_STRING}}
+      DO_DATABASE_URL: ${{secrets.DEV_DB_CONN_STRING}}  # TODO: Ideally, won't need both for testing, but this is for safety for now
+    name: Run Pytest
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.11'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+        python -m spacy download en_core_web_sm
+        pip install pytest pytest-cov
+    - name: Run tests
+      run: |
+        pytest tests/integration
+        pytest tests/middleware
+        pytest tests/resources
+        pytest tests/test_endpoints.py
+

README.md

@@ -52,25 +52,32 @@ pip install -r requirements.txt
 
 ### 5. Add environment secrets
 
-Either add a `.env` file to your local root directory or manually export these secrets: `DO_DATABASE_URL` and `VITE_VUE_API_BASE_URL`.  
+In both the local root directory and the `/client` directory, either:
+1. Either add a `.env` file to your local root directory with the below secrets
+2. or manually export the below secrets
 
 Reach out to contact@pdap.io or make noise in Discord if you'd like access to these keys.
 
 ```
 # .env
 
 DO_DATABASE_URL="postgres://data_sources_app:<password>@db-postgresql-nyc3-38355-do-user-8463429-0.c.db.ondigitalocean.com:25060/defaultdb"
+DEV_DB_CONN_STRING="postgresql://data_sources_app_v2:<password>@pdap-db-dev-do-user-8463429-0.c.db.ondigitalocean.com:25060/pdap_dev_db?sslmode=require"
 VITE_VUE_API_BASE_URL="http://localhost:5000"
 VITE_VUE_APP_BASE_URL="http://localhost:8888"
+SECRET_KEY="mySecretKey"
 ```
 
 ```
 # shell
 
 export DO_DATABASE_URL=postgres://data_sources_app:<password>@db-postgresql-nyc3-38355-do-user-8463429-0.c.db.ondigitalocean.com:25060/defaultdb
+export DEV_DB_CONN_STRING="postgresql://data_sources_app_v2:<password>@pdap-db-dev-do-user-8463429-0.c.db.ondigitalocean.com:25060/pdap_dev_db?sslmode=require"
 export VITE_VUE_API_BASE_URL="http://localhost:5000"
 export VITE_VUE_APP_BASE_URL="http://localhost:8888"
+export SECRET_KEY="mySecretKey"
 ```
+Additionally, if you are testing the email functionality, you will need to also provide the `MAILGUN_KEY` environment variable as well (also obtainable from the sources mentioned above).
 
 ### 6. Allow your IP address
 
@@ -104,11 +111,20 @@ npm run dev
 
 ## Testing
 
-All unit tests for the API live in the app_test.py file. It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated. Tests are currently run with pytest and can be run locally with the `pytest` command.
+### Location
+All unit and integration tests for the API live in the `tests` folder
 
-Endpoints are structured for simplified testing and debugging. Code for interacting with the database is contained in a function suffixed with "_results" and tested against a local sqlite database instance. Limited rows (stored in the DATA_SOURCES_ROWS and AGENCIES_ROWS variables in app_test_data.py) are inserted into this local instance on setup, you may need to add additional rows to test other functionality fully. 
+It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated. 
 
-Remaining API code is stored in functions suffixed with "_query" tested against static query results stored in app_test_data.py. Tests for hitting the endpoint directly should be included in regular_api_checks.py, makes sure to add the test function name in the list at the bottom so it is included in the Github actions run every 15 minutes.
+
+### How to run tests
+Some tests involve interfacing with the development database, which copies the production database's data and schema daily.
+
+To ensure such tests properly connect to the database, create or amend an `.env` file in the root direct of the project with the environment variable `DEV_DB_CONN_STRING`. Provide as a value a connection string giving you access to the `data_sources_app` user. If you do not have this connection string, DM a database administrator.
+
+Tests are currently run with pytest and can be run locally with the `pytest` command.
+
+Remaining API code is stored in functions suffixed with "_query" tested against static query results stored in app_test_data.py.
 
 ```
 pip install pytest

app.py

@@ -1,5 +1,5 @@
 from flask import Flask
-from flask_restful import Api
+from flask_restx import Api
 from flask_cors import CORS
 from resources.User import User
 from resources.Login import Login
@@ -27,7 +27,6 @@ def add_resource(api, resource, endpoint, **kwargs):
 
 def create_app() -> Flask:
     psycopg2_connection = initialize_psycopg2_connection()
-
     app = Flask(__name__)
     api = Api(app)
     CORS(app)