Skip to content

Latest commit

 

History

History
98 lines (74 loc) · 3 KB

README.md

File metadata and controls

98 lines (74 loc) · 3 KB

🚀 SyncFlow

SyncFlow Diagram

Welcome to SyncFlow, a powerful tool for synchronizing data with external services effortlessly!

Tech Stack Used 🧑‍💻

  • Python 🐍
  • Django (Django ORM for communicating with the DB, web framework for accepting webhook data) 🦄
  • RabbitMQ (as a message queue between Celery (consumer) and Django (producer)) 🐇
  • Redis (as a backend result storage)
  • Celery (to spawn workers and as a consumer for RabbitMQ) 🥬
  • Celery Beat (for API polling and periodic task scheduling, not required if no periodic task scheduling exists)
  • [Optional] Flower (web interface) to monitor Celery tasks 🌻

Features 👀

  • Integrate with one connect with all
  • Easy Framework for Data Synchronization with External Services
  • Removes repetitive tasks, so you can focus on integration only.
  • High Fault Tolerance using RabbitMQ and Redis
  • Fast Performance using Celery

Content 🗒️

Setup

Prerequisites of Setup

Step 1: Clone the repository.

Step 2: Create a virtual environment (recommended):

python3 -m venv venv
source venv/bin/activate

Step 3: Enter the SyncFlow directory, our whole application lies there.

Step 4: Install dependencies:

pip install -r requirements.txt

Step 5: Apply migrations:

python manage.py migrate

Step 6: Create a .env file and place it in the same directory as the README. Your .env should look like:

SECRET_KEY =
STRIPE_API_KEY =
STRIPE_ENDPOINT_SECRET =
CELERY_BROKER_URL =   # RabbitMQ URL in our case but can be other message queues as well
CELERY_RESULT_BACKEND = # Redis in our case but can be something else
FLOWER_BROKER = # RabbitMQ in our case
FLOWER_PORT =

Step 7: Make sure RabbitMQ and Redis are running.

Step 8: Start the development server:

python manage.py runserver

Step 9: Start Celery Worker

In a new terminal window, navigate to your project directory and start the Celery worker:

celery -A syncflow worker --loglevel=info

Step 10: Start Celery Beat

If you have periodic tasks that need to be scheduled, you can start Celery Beat:

celery -A syncflow beat --loglevel=info

Step 11: Monitor Celery Tasks (Optional)

To monitor your Celery tasks, you can use tools like Flower. Populate your .env with Flower configurations.

celery flower -A syncflow --port=5555

You can access the Flower dashboard at http://localhost:5555.

That's it! Your Django app with Celery, RabbitMQ, and Redis is up and running. You can now use Celery to perform background tasks and monitor their progress if needed.