README.md

financeprediction.io

Finance prediction app to view historical and current market data, weekly predictions on BTC / stocks / forex, and get support from AI driven services. Developed with Spring Boot, Maven, Thymeleaf, Django, AWS, GCP, Docker, MySQL.

The structure of the codebase is as follows:

main/
├── java/
│   └── com/
│       └── ts/
│           └── finpredict/
│               ├── assets/
│               ├── FinPredict/
│               │   ├── config/
│               │   │   └── PredictorEntityConfig.java
│               │   ├── controller/
│               │   │   ├── about/
│               │   │   ├── advice/
│               │   │   │   └── AdviceWorker.java
│               │   │   ├── chatbot/
│               │   │   │   └── ChatbotWorker.java
│               │   │   ├── FinPredictController.java
│               │   │   ├── marketdata/
│               │   │   │   └── MarketDataWorker.java
│               │   │   ├── predictor/
│               │   │   │   └── PredictorWorker.java
│               │   │   └── requests/
│               │   │       ├── AdviceRequests.java
│               │   │       ├── ChatbotRequests.java
│               │   │       ├── MarketDataRequests.java
│               │   │       └── PredictorRequests.java
│               │   ├── FinPredictApplication.java
│               │   └── model/
│               │       ├── dao/
│               │       │   ├── PredictorDailyDAO.java
│               │       │   ├── PredictorDailyDAOImpl.java
│               │       │   ├── PredictorWeeklyDAO.java
│               │       │   └── PredictorWeeklyDAOImpl.java
│               │       ├── entity/
│               │       │   ├── Advice.java
│               │       │   ├── Chatbot.java
│               │       │   ├── MarketData.java
│               │       │   ├── Predictor.java
│               │       │   ├── PredictorDailyEntity.java
│               │       │   └── PredictorWeeklyEntity.java
│               │       └── service/
│               │           ├── PredictorDailyService.java
│               │           ├── PredictorDailyServiceImpl.java
│               │           ├── PredictorWeeklyService.java
│               │           └── PredictorWeeklyServiceImpl.java
│               └── util/
│                   ├── errors/
│                   │   └── shared/
│                   │       ├── PageErrorResponse.java
│                   │       ├── PageExceptionHandler.java
│                   │       └── PageNotFoundException.java
│                   ├── helpers/
│                   └── sql/
│                       ├── predictor-current-day-ddl.sql
│                       └── predictor-current-week-ddl.sql
└── resources/
    ├── application.properties
    ├── static/
    │   ├── css/
    │   │   ├── about.css
    │   │   ├── advice.css
    │   │   ├── market.css
    │   │   └── predictor.css
    │   ├── pages/
    │   │   └── index.html
    │   └── shared/
    │       └── index.css
    └── templates/
        ├── about/
        │   └── about.html
        ├── advice/
        │   └── advice.html
        ├── marketdata/
        │   └── marketdata.html
        └── predictor/
            └── predictor.html

Figure 1: High level view

The application consists of the following main components:

1. Client: Springboot frontend web application which sends requests to the Finance Predict API. Currently the frontend is deployed using Docker on GCP App Engine.
2. Finance Predict API: Finance prediction API developed using django and deployed to AWS EC2. API serves requests for finance prediction, market data queries and chatbot queries.
3. Services: The following services are used:

• OpenAI: Services chatbot requests
• polygon.io: Provides market data for requests from the Finance Predict API

5. Machine learning: ML models for BTC and stock predictions developed using TensorFlow / PyTorch. ml-job-scheduler handles the automated job runs from preprocessing, training, predictions, postprocessing, etc. The logs of the job runs are stored in S3 and the data is stored in MongoDB.
6. Databases: ML prediction data is stored within MongoDB.
7. Data engineering: Performs manual data migration using an external ETL / ELT API developed in Go, etl-elt-api
8. Security: AWS security services (AWS Inspector and GuardDuty) which monitors the security of APIs. Later, data and logs from the services are queried and viewed using an external security tool.
9. Monitoring: Monitoring service which collects data and logs from APIs using CloudWatch, then stores them in S3 to be viewed as a dashboard via an external monitoring tool.
10. Notifications: Receives various data and logs in S3 buckets and later sends emails (on issues or failures) using SQS and Lambda.

Setting up the development environment:

1. Cloning the repository: You would first need to clone this repository on the host you want to set up your development environment:

git clone https://github.com/tahmid-saj/fin-predict.git

2. Installing dependencies: The maven dependencies are provided in the pom.xml file. They wil only need to be installed using a valid IDE such as IntelliJ IDEA or NetBeans.
3. API: Client requests to go a the Finance Predict API which provides the following:

• Finance prediction
• Historic market data
• Chatbot responses

4. Services: API keys are used from the following services in the Finance Predict API:

• OpenAI: Services chatbot requests
• polygon.io: Provides market data for requests from the Finance Predict API

5. Databases: A MongoDB cluster and collections are created, and a connection is established with the Finance Predict API.
6. Data engineering: The set up for the development environment for the data engineering tool can be found here.
7. Machine learning: The set up for the development environment for the ML job scheduler can be found here.
8. AWS: Setting up the AWS services is an optional step as this is on a development environment. However, the same services could be used to create the tools mentioned in the high level view.
9. Running the client: The client can be run using an IDE or Docker using:

• To package your application with Maven run:

mvn clean package

• Test the jar file:

java -jar target/<SNAPSHOT NAME>.jar

• Build the docker image:

docker build -t <PROJECT NAME> .

• Tag the docker image:

docker tag <PROJECT NAME> <YOUR USERNAME>/<PROJECT NAME>

• Push the docker image to Docker Hub:

docker push <YOUR USERNAME>/<PROJECT NAME>:<TAG NAME>

After the image is pushed, it can be used in GCP App Engine or with a docker container in the development environment.