Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Db maven module #174

Merged
merged 15 commits into from
Oct 20, 2023
Merged

Db maven module #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
99c01bd
Created DB maven module that contains all of the database code needed…
ctevse Oct 9, 2023
93b16e9
Updated gitignore to ignore all maven target folders
ctevse Oct 9, 2023
19f0176
Refactored Crawler to use db module for models and database access
ctevse Oct 9, 2023
dc87ad3
Migrated patchfinder models into db module
ctevse Oct 10, 2023
afccbfe
Removed database helper from crawler constructor
ctevse Oct 10, 2023
fb13c19
Migrated one models into db package
ctevse Oct 17, 2023
317a5d0
Changed raw description insert to be performed in batch
ctevse Oct 19, 2023
2cca00b
Replaced logger with @Slf4j annotation
ctevse Oct 19, 2023
7301800
Removed pom.xml from the .gitignore and added db/pom.xml
ctevse Oct 20, 2023
fc52c61
Fixed workflows to use new module building
ctevse Oct 20, 2023
ff84ba4
Fixed failing tests
ctevse Oct 20, 2023
fbbfe46
Split ReconcilerMain Tests into smaller units and applied better mocking
ctevse Oct 20, 2023
45d8f67
Added MockitoExtension to tests that require mockito
ctevse Oct 20, 2023
0dd0e8a
Sorted listFiles() return so that the return is consistent
ctevse Oct 20, 2023
3594869
Added liquibase plugin to db module
ctevse Oct 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 2 additions & 7 deletions .github/workflows/crawler-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,6 @@ jobs:
distribution: 'temurin'
cache: 'maven'

- name: Set up dependencies
run: mvn dependency:resolve

# Setup MySQL DB for tests that require connection (Might not need this)
- name: Initialize Database
env:
Expand All @@ -67,13 +64,11 @@ jobs:

- name: Run Liquibase Update
run: |
mvn liquibase:update --projects=crawler
mvn liquibase:update --projects=db

- name: Test Crawler
run: |
cd crawler
mvn dependency:go-offline
mvn test
mvn test --projects=db,crawler

- name: Publish Test Report
if: success() || failure()
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/patchfinder-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,13 +67,11 @@ jobs:
LIQUIBASE_COMMAND_PASSWORD: ${{ env.HIKARI_PASSWORD }}
run: |
mysql -e 'CREATE DATABASE nvip;' -u${{ env.HIKARI_USER }} -p${{ env.HIKARI_PASSWORD }}
mvn liquibase:update --no-transfer-progress --projects=patchfinder
mvn liquibase:update --no-transfer-progress --projects=db

- name: Test and Build PatchFinder with Maven
run: |
cd patchfinder
mvn dependency:go-offline
mvn package
mvn test --projects=db,patchfinder

- name: Publish Test Report
if: success() || failure()
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/pne-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,11 @@ jobs:
LIQUIBASE_COMMAND_PASSWORD: ${{ env.HIKARI_PASSWORD }}
run: |
mysql -e 'CREATE DATABASE nvip;' -u${{ env.HIKARI_USER }} -p${{ env.HIKARI_PASSWORD }}
mvn liquibase:update --no-transfer-progress --projects=productnameextractor
mvn liquibase:update --no-transfer-progress --projects=db

- name: Test and Build Product Extractor with Maven
run: |
cd productnameextractor
mvn dependency:go-offline
mvn package
mvn test --projects=db,productnameextractor

- name: Publish Test Report
if: success() || failure()
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/reconciler-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,7 @@ jobs:

- name: Test and Build Reconciler with Maven
run: |
cd reconciler
mvn dependency:go-offline
mvn test
mvn test --projects=db,reconciler

- name: Publish Test Report
if: success() || failure()
Expand Down
7 changes: 1 addition & 6 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ logs
*.iml

# Class outputs
/target/
/crawler/target/
/reconciler/target/
/patchfinder/target/
/productnameextractor/target/
**/target/**
/productnameextractor/nvip_data/data/test_results.csv

# Large files
Expand All @@ -30,7 +26,6 @@ nvip_data/pypa-repo/
Servers/
nvip_data/
/nvip_data/mysql-database/
pom.xml
env.list


Expand Down
84 changes: 19 additions & 65 deletions crawler/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,12 @@

<!-- List of dependencies for NVIP -->
<dependencies>
<dependency>
<groupId>edu.rit.se.nvip</groupId>
<artifactId>db</artifactId>
<version>2.0</version>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
Expand All @@ -156,11 +162,6 @@
<artifactId>HikariCP</artifactId>
</dependency>

<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
</dependency>

<dependency>
<groupId>com.mysql</groupId>
<artifactId>mysql-connector-j</artifactId>
Expand All @@ -172,18 +173,6 @@
<artifactId>org.eclipse.jgit</artifactId>
</dependency>

<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>webdrivermanager</artifactId>
<version>5.5.3</version>
</dependency>

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.13.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/xml-apis/xml-apis -->
<dependency>
<groupId>xml-apis</groupId>
Expand Down Expand Up @@ -225,30 +214,12 @@
<version>5.7.1</version>
</dependency>

<!-- edu.stanford.nlp -->
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>4.5.2</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>4.5.2</version>
<classifier>models</classifier>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.30</version>
</dependency>

<dependency>
<groupId>org.python</groupId>
<artifactId>jython-standalone</artifactId>
<version>2.7.2</version>
</dependency>

<!-- Apache FileUtils -->
<dependency>
<groupId>commons-io</groupId>
Expand All @@ -257,14 +228,6 @@
<type>jar</type>
</dependency>

<!-- Crawler4J -->
<dependency>
<groupId>de.hs-heilbronn.mi</groupId>
<artifactId>crawler4j-with-sleepycat</artifactId>
<version>5.0.1</version>
<type>pom</type>
</dependency>

<!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
<dependency>
<groupId>org.jsoup</groupId>
Expand Down Expand Up @@ -293,28 +256,6 @@
<version>3.0</version>
</dependency>

<!-- <version>1.0.0-beta5</version> -->
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>1.0.0-beta7</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-nlp</artifactId>
<version>1.0.0-beta7</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-nn</artifactId>
<version>1.0.0-beta7</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-parallel-wrapper</artifactId>
<version>1.0.0-beta7</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.nd4j/nd4j-native-platform -->
<dependency>
<groupId>org.nd4j</groupId>
Expand Down Expand Up @@ -351,6 +292,19 @@
<version>2.0.1</version>
</dependency>

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.13.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/io.github.bonigarcia/webdrivermanager -->
<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>webdrivermanager</artifactId>
<version>5.5.3</version>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
Expand Down
Loading
Loading