Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mc issue 310 refactor security #320

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
7def8a5
Refactor code using handle_exceptions decorator
maxachis Apr 27, 2024
5698ab7
Black reformatting
maxachis Apr 27, 2024
3b2c241
update readme
josh-chamberlain Apr 30, 2024
68199c9
Add middleware tests for database interactions
maxachis May 3, 2024
3092634
Add testing helpers for database interactions
maxachis May 3, 2024
85ec2d7
Refactor test methods and introduce database testing helpers
maxachis May 3, 2024
94bf05a
Create (mostly) blank python_tests.yml file
maxachis May 3, 2024
8c7392d
Refactor test file and update test functions
maxachis May 6, 2024
0b86c49
Add middleware testing fixtures
maxachis May 6, 2024
ee11b16
Add tests for data source queries
maxachis May 6, 2024
16ebc65
Refactor pytest fixtures
maxachis May 6, 2024
197d80d
Added unit tests for archives_queries middleware
maxachis May 6, 2024
4a0efc8
Added unit tests for login_queries middleware
maxachis May 6, 2024
171b35b
Refactor and enhance get_most_recent_quick_search_query_log function
maxachis May 6, 2024
9a3ecf5
Add unit tests for quick search query middleware
maxachis May 6, 2024
23bee8b
Add unit tests for user and reset token query methods
maxachis May 6, 2024
e1fff5a
Remove redundant tests for middleware functions
maxachis May 6, 2024
6aa2170
Remove now-redundant tests
maxachis May 6, 2024
41deff0
Update README with revised testing instructions
maxachis May 6, 2024
e3e1e25
Rename Python test workflow and add configuration
maxachis May 6, 2024
b66aab9
Update test data source imports in middleware tests
maxachis May 6, 2024
f057548
Refactor test files and clean up imports
maxachis May 6, 2024
9767d9e
Improve code readability and ignore max line exceed warning
maxachis May 8, 2024
66e13d3
Update flake8 args and refine code formatting
maxachis May 8, 2024
a83bfc3
Update flake8 args and refine code formatting
maxachis May 8, 2024
6820412
Added custom exception for psycopg2 connection initialization
maxachis May 11, 2024
12d00c5
Handle UserNotFoundError in middleware and implement custom exception
maxachis May 11, 2024
c126810
Refactor query methods for data source handling
maxachis May 12, 2024
38070f0
Remove archives_put_query function and refactor related code
maxachis May 12, 2024
078d045
Update SQL commands and refactor unit tests
maxachis May 14, 2024
d752468
Refactor using black
maxachis May 14, 2024
f7a77f1
Update pytest command in pull.yaml workflow
maxachis May 14, 2024
9bb0d0f
Refactor using black
maxachis May 14, 2024
10dc54e
Add test skeletons for UserNotLoggedIn exception handling
maxachis May 14, 2024
a5c3747
Merge branch 'mc_testing_to_dev' into mc_issue_276_refactor_middlewar…
maxachis May 15, 2024
caaf833
Implement test for UserNotFoundError in login queries
maxachis May 15, 2024
bbb6edc
Merge branch 'mc_testing_to_dev' into mc_issue_181_middleware_god_fun…
maxachis May 15, 2024
9f99705
Refactor data access in tests and remove unused test
maxachis May 15, 2024
bc4f9bc
Add type hints and docstring to 'convert_data_source_matches' function
maxachis May 15, 2024
2132d0a
Update CODEOWNERS
josh-chamberlain May 17, 2024
ad5b51b
Refactor DataRequestsManager, add DBRequestMapper class and tests
maxachis May 17, 2024
3c21385
Change psycopg2 to psycopg2-binary
maxachis May 17, 2024
83835da
Temporarily comment out github action to prevent execution
maxachis May 17, 2024
5acc1ff
Uncomment test_api.yml
maxachis May 19, 2024
a40f55b
Comment test_api.yml
maxachis May 19, 2024
31f6400
Merge pull request #1 from Police-Data-Accessibility-Project/mc_testi…
maxachis May 19, 2024
cd4cd74
Merge pull request #5 from Police-Data-Accessibility-Project/mc_handl…
josh-chamberlain May 21, 2024
15e7a95
Merge remote-tracking branch 'Data-Sources-App-V2/main' into mc_issue…
maxachis May 22, 2024
2e19d92
Merge remote-tracking branch 'Data-Sources-App-V2/dev' into mc_issue_…
maxachis May 22, 2024
aed0d58
Post-merge clean-up
maxachis May 22, 2024
fe883a2
Merge remote-tracking branch 'Data-Sources-App-V2/main' into dev
maxachis May 22, 2024
d407ada
Reformat with Black
maxachis May 22, 2024
1057284
Merge branch 'dev' into mc_issue_281_middleware_god_function_actual
maxachis May 22, 2024
073f60d
Remove broken import
maxachis May 22, 2024
f3bb63d
Update data source queries in SearchTokens
maxachis May 22, 2024
82274c9
Merge pull request #9 from Police-Data-Accessibility-Project/mc_issue…
maxachis May 22, 2024
409d4a8
Add conftest.py for pytest fixtures
maxachis May 22, 2024
ebd3ef4
Add User model to middleware
maxachis May 22, 2024
ba4e4a8
Reformat with black
maxachis May 22, 2024
c092c5d
Add Flask and Flask-SQLAlchemy to requirements.txt
maxachis May 22, 2024
2f30d89
Correct requirements.txt
maxachis May 22, 2024
85e6fce
Correct requirements.txt
maxachis May 22, 2024
cbe2713
Merge pull request #12 from Police-Data-Accessibility-Project/mc_sqla…
maxachis May 22, 2024
ea71511
Refactor token management and search endpoints
maxachis May 23, 2024
7765fe7
Refactor search endpoints and token management
maxachis May 23, 2024
5a9bfab
Refactor search functionalities and improve token handling
maxachis May 23, 2024
41aa4d5
Correct bug in SearchTokens get method
maxachis May 23, 2024
10337fa
Merge remote-tracking branch 'Data-Sources-App-V2/mc_issue_285_refact…
maxachis May 23, 2024
9c57626
Refactor application startup in app.py
maxachis May 23, 2024
3ac41a4
Remove GET and POST methods from PsycopgResource
maxachis May 23, 2024
ee5435b
Change connection type in quick_search_query_wrapper function
maxachis May 23, 2024
7abef15
Add access token insertion logic
maxachis May 23, 2024
793ef4d
Refactor access token insertion logic
maxachis May 23, 2024
e4797de
Add unit tests for endpoint behavior
maxachis May 23, 2024
d336701
Add unit tests for SearchTokens resource
maxachis May 23, 2024
f1d1363
Refactor return statements in data_source_queries.py
maxachis May 23, 2024
e6cd7a1
Update error responses in PsycopgResource.py
maxachis May 23, 2024
06a0297
Update to use Flask's make_response in quick_search_query.py
maxachis May 23, 2024
6a697ef
Added custom exception for unknown endpoint in SearchTokens
maxachis May 23, 2024
4aa78d4
Added tests for search tokens service error handling
maxachis May 23, 2024
67dd1a5
Reformat with Black
maxachis May 23, 2024
b104b2e
Reformat with Black
maxachis May 23, 2024
03a6db5
Reformat with Black
maxachis May 23, 2024
58431f3
Refactor error handling to use custom exceptions
maxachis May 24, 2024
0f3cb7a
Add tests for custom exceptions in authentication process
maxachis May 24, 2024
11cc622
Reformat with Black
maxachis May 24, 2024
bff0e22
Merge pull request #7 from Police-Data-Accessibility-Project/mc_issue…
josh-chamberlain May 24, 2024
a8b88e0
Refactor create_app function in app.py
maxachis May 25, 2024
ddc9d26
Remove unused methods from PsycopgResource class.
maxachis May 25, 2024
80b55e3
Add thorough tests for all application endpoints
maxachis May 25, 2024
06db561
Refactor test_app_with_mock fixture in app_test.py
maxachis May 25, 2024
2b2f3a5
Replace tuple return type with namedtuple in security middleware
maxachis May 25, 2024
f53d8f8
Add Bandit Security Linting to GitHub Actions
maxachis May 26, 2024
fae254f
Update Bandit execution command in workflow
maxachis May 26, 2024
006cec2
Removed cat command from Bandit run in GitHub Actions
maxachis May 26, 2024
b931fc1
Adjustment
maxachis May 26, 2024
12a9ba9
Adjustment
maxachis May 26, 2024
e05fbc1
Update Bandit Security Linting trigger event
maxachis May 26, 2024
2f3a61d
Expand targets for Bandit Security Linting
maxachis May 26, 2024
07318bb
Merge pull request #17 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 26, 2024
f9e4852
Refactor SQL queries for enhanced security
maxachis May 26, 2024
299975d
Merge pull request #18 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 26, 2024
2672d5e
Merge pull request #16 from Police-Data-Accessibility-Project/mc_add_…
maxachis May 26, 2024
bd86884
Merge pull request #14 from Police-Data-Accessibility-Project/mc_add_…
maxachis May 27, 2024
daf1608
Remove test parameters from query functions
maxachis May 27, 2024
6cb12c9
Refactor fixture import paths in test files
maxachis May 27, 2024
926798c
Move helper functions and refactor tests
maxachis May 27, 2024
845dd33
Merge pull request #19 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 29, 2024
86e6cd3
Refactor test suite by relocating helper functions and updating clien…
maxachis May 30, 2024
087e6a3
Add integration tests for various endpoints
maxachis May 30, 2024
3a264db
Improve readability of code via tweaks and docstrings
maxachis May 30, 2024
34cbec2
Update headers in integration tests
maxachis May 30, 2024
89343d0
Merge pull request #20 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 30, 2024
b3ac9e7
Fixed response status code for data source not found to return 200 in…
maxachis May 31, 2024
2cee3cd
Fix typo in assertion for data source URL in integration test.
maxachis May 31, 2024
42b4f0d
Add unit tests for getting and updating data sources by ID in test_Da…
maxachis May 31, 2024
cc1f61a
Merge pull request #21 from Police-Data-Accessibility-Project/mc_issu…
maxachis May 31, 2024
4008157
Refactor helper_functions.py to generate a random email for test user…
maxachis Jun 1, 2024
988a3b4
Implemented various test cases to check the validity and expiration s…
maxachis Jun 1, 2024
e05622b
Refactor security middleware to improve readability and efficiency in…
maxachis Jun 1, 2024
3e610c5
Add function to delete session token from database
maxachis Jun 1, 2024
a4b6ef9
Refactored RefreshSession.py to include delete_session_token function…
maxachis Jun 1, 2024
192088f
Fixed create_test_user so that email is set to a random email if not …
maxachis Jun 1, 2024
759c957
Merge pull request #22 from Police-Data-Accessibility-Project/mc_fix_…
maxachis Jun 1, 2024
2efa94d
Merge remote-tracking branch 'Data-Sources-App-V2/dev' into mc_issue_…
maxachis Jun 1, 2024
07f5969
✨ Add ClientWithMockDB namedtuple in fixtures.py and update test_Data…
maxachis Jun 2, 2024
583e855
✨ Add bypass for api_required decorator in tests/resources/__init__.py
maxachis Jun 2, 2024
cf8be37
✅ Add check_response_status function in tests/helper_functions.py to …
maxachis Jun 2, 2024
b5cee84
🛠️ refactor: Refactor token retrieval and deletion functions in login…
maxachis Jun 2, 2024
a92d61c
Refactor RefreshSession.py to handle custom exceptions and improve co…
maxachis Jun 2, 2024
2af5634
Refactor endpoint test fixture import in test_endpoints.py"
maxachis Jun 2, 2024
d84cbab
refactor token_results to get_session_token_user_data & update tests.
maxachis Jun 2, 2024
5f00c5a
✨ Add test_RefreshSession.py with refresh session test cases
maxachis Jun 2, 2024
d9dd825
Merge pull request #23 from Police-Data-Accessibility-Project/mc_issu…
maxachis Jun 2, 2024
04de044
🐛 fix(middleware): Update INSERT_LOG_QUERY to remove redundant dateti…
maxachis Jun 2, 2024
6a8a2cf
Update timestamp field name in query
maxachis Jun 2, 2024
1c0a8a0
Refactor code in resources/QuickSearch.py
maxachis Jun 2, 2024
24a9a57
Fix bug in test_search_tokens.py
maxachis Jun 2, 2024
dcacd66
Add tests for quick search functionality in test_QuickSearch.py"
maxachis Jun 2, 2024
3cfa69b
Merge pull request #24 from Police-Data-Accessibility-Project/mc_issu…
maxachis Jun 2, 2024
c4cfa66
Merge remote-tracking branch 'Data-Sources-App-V2/dev' into mc_issue_…
maxachis Jun 2, 2024
8f2aa2d
✨🔍 Refactor test functions to use helper function for checking respon…
maxachis Jun 2, 2024
01454af
Merge branch 'dev' into mc_issue_285_refactor_search_tokens
maxachis Jun 2, 2024
adfe6b4
Merge pull request #13 from Police-Data-Accessibility-Project/mc_issu…
maxachis Jun 2, 2024
7450935
refactor(api): Update quick_search_query.py function signatures
maxachis Jun 2, 2024
be682c7
✨ Add Response return type to data_source_by_id_wrapper function
maxachis Jun 2, 2024
ba6c4ca
✨ Add test for data_source_by_id_wrapper function
maxachis Jun 2, 2024
43d80ac
⚡️ refactor: refactor quick search endpoint response handling in inte…
maxachis Jun 2, 2024
f888606
💻 Add test cases for quick search query middleware and eliminate now…
maxachis Jun 2, 2024
d4b365d
⚙️: remove now-redundant Resource tests in test_DataSources.py
maxachis Jun 2, 2024
f1e47c6
Merge pull request #25 from Police-Data-Accessibility-Project/mc_issu…
maxachis Jun 2, 2024
4e85a2b
⚙️ Adjust status code for data source not found response
maxachis Jun 2, 2024
144d4ac
Merge pull request #26 from Police-Data-Accessibility-Project/mc_issu…
maxachis Jun 2, 2024
4b40a16
Merge remote-tracking branch 'Data-Sources-App-V2/dev' into mc_issue_…
maxachis Jun 2, 2024
6ca9cf8
⚡️ Refactor and update security middleware and tests for API key vali…
maxachis Jun 3, 2024
12fbb30
Remove redundant NoAPIKeyError handling
maxachis Jun 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
/client/ @joshuagraber
* @mbodeantor
* @josh-chamberlain
32 changes: 32 additions & 0 deletions .github/workflows/bandit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Bandit Security Linting

on: [pull_request]

jobs:
bandit:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install bandit

- name: Run Bandit
run: |
bandit -r middleware resources app.py

- name: Upload Bandit results
uses: actions/upload-artifact@v2
with:
name: bandit-report
path: bandit_output.txt

2 changes: 1 addition & 1 deletion .github/workflows/pull.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Test with pytest
run: |
pip install pytest pytest-cov
pytest app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
pytest tests/resources/app_test.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html

setup_client:
defaults:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/python_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ jobs:
uses: reviewdog/action-flake8@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
flake8_args: --ignore E501,W291 # Does not check for max line exceed or trailing whitespace
level: warning
25 changes: 25 additions & 0 deletions .github/workflows/test_api.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#name: Test API using Pytest
#
#on:
# pull_request:
#
#jobs:
# test_api:
# env:
# SECRET_KEY: ${{ secrets.SECRET_KEY }}
# DEV_DB_CONN_STRING: ${{secrets.DEV_DB_CONN_STRING}}
# name: Test API
# runs-on: ubuntu-latest
# steps:
# - uses: actions/checkout@v4
# - uses: actions/setup-python@v4
# with:
# python-version: '3.11'
# - name: Install dependencies
# run: |
# python -m pip install --upgrade pip
# pip install -r requirements.txt
# python -m spacy download en_core_web_sm
# pip install pytest pytest-cov
# - name: Run tests
# run: pytest tests --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# data-sources-app
# data-sources-app-v2

Development of the next big iteration of the data sources app according to https://github.com/Police-Data-Accessibility-Project/data-sources-app/issues/248

An API and UI for searching, using, and maintaining Data Sources.

Expand Down Expand Up @@ -102,9 +104,18 @@ npm run dev

## Testing

All unit tests for the API live in the app_test.py file. It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated. Tests are currently run with pytest and can be run locally with the `pytest` command.
### Location
All unit and integration tests for the API live in the `tests` folder

It is best practice to add tests for any new feature to ensure it is working as expected and that any future code changes do not affect its functionality. All tests will be automatically run when a PR into dev is opened in order to ensure any changes do not break current app functionality. If a test fails, it is a sign that the new code should be checked or possibly that the test needs to be updated.


### How to run tests
Some tests involve interfacing with the development database, which copies the production database's data and schema daily.

To ensure such tests properly connect to the database, create or amend an `.env` file in the root direct of the project with the environment variable `DEV_DB_CONN_STRING`. Provide as a value a connection string giving you access to the `data_sources_app` user. If you do not have this connection string, DM a database administrator.

Endpoints are structured for simplified testing and debugging. Code for interacting with the database is contained in a function suffixed with "_results" and tested against a local sqlite database instance. Limited rows (stored in the DATA_SOURCES_ROWS and AGENCIES_ROWS variables in app_test_data.py) are inserted into this local instance on setup, you may need to add additional rows to test other functionality fully.
Tests are currently run with pytest and can be run locally with the `pytest` command.

Remaining API code is stored in functions suffixed with "_query" tested against static query results stored in app_test_data.py. Tests for hitting the endpoint directly should be included in regular_api_checks.py, makes sure to add the test function name in the list at the bottom so it is included in the Github actions run every 15 minutes.

Expand Down
6 changes: 2 additions & 4 deletions app.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@
api.add_resource(resource, endpoint, resource_class_kwargs=kwargs)


def create_app() -> Flask:
psycopg2_connection = initialize_psycopg2_connection()

def create_app(psycopg2_connection) -> Flask:

Check warning on line 28 in app.py

View workflow job for this annotation

GitHub Actions / Lint

[flake8] reported by reviewdog 🐶 Missing docstring in public function Raw Output: ./app.py:28:1: D103 Missing docstring in public function
app = Flask(__name__)
api = Api(app)
CORS(app)
Expand Down Expand Up @@ -57,5 +55,5 @@


if __name__ == "__main__":
app = create_app()
app = create_app(initialize_psycopg2_connection())
app.run(debug=True, host="0.0.0.0")
Loading
Loading