Merge pull request #301 from StuffAnThings/develop

3.6.3

.github/pull_request_template.md

@@ -26,4 +26,4 @@ Please delete options that are not relevant.
 - [ ] I have performed a self-review of my own code
 - [ ] I have commented my code, particularly in hard-to-understand areas
 - [ ] I have added or updated the docstring for new or existing methods
-- [ ] I have added tests when applicable
+- [ ] I have modified this PR to merge to the develop branch

.github/workflows/dependabot-approve-and-auto-merge.yml

@@ -19,7 +19,7 @@ jobs:
       # will not occur.
       - name: Dependabot metadata
         id: dependabot-metadata
-        uses: dependabot/fetch-metadata@v1.4.0
+        uses: dependabot/fetch-metadata@v1.5.0
         with:
           github-token: "${{ secrets.GITHUB_TOKEN }}"
       # Here the PR gets approved.

.github/workflows/develop.yml

@@ -46,6 +46,8 @@ jobs:
         with:
           context: ./
           file: ./Dockerfile
+          build-args: |
+            "BRANCH_NAME=develop"
           platforms: linux/amd64,linux/arm64,linux/arm/v7
           push: true
           tags: ${{ secrets.DOCKER_HUB_USERNAME }}/qbit_manage:develop

.github/workflows/version.yml

@@ -14,6 +14,8 @@ jobs:
 
       - name: Check Out Repo
         uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
 
       - name: Login to Docker Hub
         uses: docker/login-action@v2

CHANGELOG

@@ -1,6 +1,20 @@
+# Requirements Updated
+- pre-commit updated to 3.3.3
+- requests updated to 2.31.0
+- ruamel.yaml updated to 0.17.26
+- Adds new dependency bencodepy to generate hash for cross-seed
+- Adds new dependency GitPython for checking git branches
+
 # Bug Fixes
-- Fixes bug in cross_seed (Fixes #270)
-- Bug causing RecycleBin not to be created when full path is defined. (Fixes #271)
-- Fixes Uncaught exception while emptying recycle bin (Fixes #272)
+- Changes HardLink Logic (Thanks to @ColinHebert for the suggestion) Fixes #291
+- Additional error checking (Fixes #282)
+- Fixes #287 (Thanks to @buthed010203 #290)
+- Fixes Remove Orphan crashing when multiprocessing (Thanks to @buthed010203 #289)
+- Speed optimization for Remove Orphan (Thanks to @buthed010203 #299)
+- Fixes Remove Orphan from crashing in Windows (Fixes #275)
+- Fixes #292
+- Fixes #201
+- Fixes #279
+- Updates Dockerfile to debloat and move to Python 3.11
 
-**Full Changelog**: https://github.com/StuffAnThings/qbit_manage/compare/v3.6.1...v3.6.2
+**Full Changelog**: https://github.com/StuffAnThings/qbit_manage/compare/v3.6.2...v3.6.3

Dockerfile

@@ -1,15 +1,29 @@
-FROM python:3.10-alpine
-
-# install packages
-RUN apk add --no-cache gcc g++ libxml2-dev libxslt-dev shadow bash curl wget jq grep sed coreutils findutils unzip p7zip ca-certificates
+FROM python:3.11-slim-buster
+ARG BRANCH_NAME=master
+ENV BRANCH_NAME ${BRANCH_NAME}
+ENV TINI_VERSION v0.19.0
+ENV QBM_DOCKER True
 
 COPY requirements.txt /
 
-RUN echo "**** install python packages ****" \
+# install packages
+RUN echo "**** install system packages ****" \
+ && apt-get update \
+ && apt-get upgrade -y --no-install-recommends \
+ && apt-get install -y tzdata --no-install-recommends \
+ && apt-get install -y gcc g++ libxml2-dev libxslt-dev libz-dev bash curl wget jq grep sed coreutils findutils unzip p7zip ca-certificates \
+ && wget -O /tini https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-"$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
+ && chmod +x /tini \
  && pip3 install --no-cache-dir --upgrade --requirement /requirements.txt \
- && rm -rf /requirements.txt /tmp/* /var/tmp/*
+ && apt-get --purge autoremove gcc g++ libxml2-dev libxslt-dev libz-dev -y \
+ && apt-get clean \
+ && apt-get update \
+ && apt-get check \
+ && apt-get -f install \
+ && apt-get autoclean \
+ && rm -rf /requirements.txt /tmp/* /var/tmp/* /var/lib/apt/lists/*
 
 COPY . /app
 WORKDIR /app
 VOLUME /config
-ENTRYPOINT ["python3", "qbit_manage.py"]
+ENTRYPOINT ["/tini", "-s", "python3", "qbit_manage.py", "--"]

README.md

@@ -16,7 +16,7 @@ This is a program used to manage your qBittorrent instance such as:
 * Automatically add [cross-seed](https://github.com/mmgoodnow/cross-seed) torrents in paused state. **\*Note: cross-seed now allows for torrent injections directly to qBit, making this feature obsolete.\***
 * Recheck paused torrents sorted by lowest size and resume if completed
 * Remove orphaned files from your root directory that are not referenced by qBittorrent
-* Tag any torrents that have no hard links and allows optional cleanup to delete these torrents and contents based on maximum ratio and/or time seeded
+* Tag any torrents that have no hard links outisde the root folder and allows optional cleanup to delete these torrents and contents based on maximum ratio and/or time seeded
 * RecycleBin function to move files into a RecycleBin folder instead of deleting the data directly when deleting a torrent
 * Built-in scheduler to run the script every x minutes. (Can use `--run` command to run without the scheduler)
 * Webhook notifications with [Notifiarr](https://notifiarr.com/) and [Apprise API](https://github.com/caronc/apprise-api) integration

VERSION

@@ -1 +1 @@
-3.6.2
+3.6.3

config/config.yml.sample

@@ -136,7 +136,7 @@ tracker:
     tag: other
 
 nohardlinks:
-  # Tag Movies/Series that are not hard linked
+  # Tag Movies/Series that are not hard linked outside the root directory
   # Mandatory to fill out directory parameter above to use this function (root_dir/remote_dir)
   # This variable should be set to your category name of your completed movies/completed series in qbit. Acceptable variable can be any category you would like to tag if there are no hardlinks found
   movies-completed:

modules/__init__.py

@@ -13,8 +13,10 @@
 with open(version_file_path) as f:
     version_str = f.read().strip()
 
+# Get only the first 3 digits
+version_str_split = version_str.rsplit("-", 1)[0]
 # Convert the version string to a tuple of integers
-__version_info__ = tuple(map(int, version_str.split(".")))
+__version_info__ = tuple(map(int, version_str_split.split(".")))
 
 # Define the version string using the version_info tuple
 __version__ = ".".join(str(i) for i in __version_info__)

modules/config.py

@@ -110,27 +110,28 @@ def __init__(self, default_dir, args):
             self.data["notifiarr"] = self.data.pop("notifiarr")
         if "webhooks" in self.data:
             temp = self.data.pop("webhooks")
-            if "function" not in temp or ("function" in temp and temp["function"] is None):
-                temp["function"] = {}
-
-            def hooks(attr):
-                if attr in temp:
-                    items = temp.pop(attr)
-                    if items:
-                        temp["function"][attr] = items
-                if attr not in temp["function"]:
-                    temp["function"][attr] = {}
-                    temp["function"][attr] = None
-
-            hooks("cross_seed")
-            hooks("recheck")
-            hooks("cat_update")
-            hooks("tag_update")
-            hooks("rem_unregistered")
-            hooks("rem_orphaned")
-            hooks("tag_nohardlinks")
-            hooks("cleanup_dirs")
-            self.data["webhooks"] = temp
+            if temp is not None:
+                if "function" not in temp or ("function" in temp and temp["function"] is None):
+                    temp["function"] = {}
+
+                def hooks(attr):
+                    if attr in temp:
+                        items = temp.pop(attr)
+                        if items:
+                            temp["function"][attr] = items
+                    if attr not in temp["function"]:
+                        temp["function"][attr] = {}
+                        temp["function"][attr] = None
+
+                hooks("cross_seed")
+                hooks("recheck")
+                hooks("cat_update")
+                hooks("tag_update")
+                hooks("rem_unregistered")
+                hooks("rem_orphaned")
+                hooks("tag_nohardlinks")
+                hooks("cleanup_dirs")
+                self.data["webhooks"] = temp
         if "bhd" in self.data:
             self.data["bhd"] = self.data.pop("bhd")
         self.dry_run = self.commands["dry_run"]

modules/core/cross_seed.py

@@ -2,6 +2,7 @@
 from collections import Counter
 
 from modules import util
+from modules.torrent_hash_generator import TorrentHashGenerator
 
 logger = util.logger
 
@@ -39,7 +40,7 @@ def cross_seed(self):
                 dest = os.path.join(self.qbt.torrentinfo[t_name]["save_path"], "")
                 src = os.path.join(dir_cs, file)
                 dir_cs_out = os.path.join(dir_cs, "qbit_manage_added", file)
-                category = self.qbt.get_category(dest)
+                category = self.qbt.torrentinfo[t_name].get("Category", self.qbt.get_category(dest))
                 # Only add cross-seed torrent if original torrent is complete
                 if self.qbt.torrentinfo[t_name]["is_complete"]:
                     categories.append(category)
@@ -67,12 +68,28 @@ def cross_seed(self):
                         self.client.torrents.add(
                             torrent_files=src, save_path=dest, category=category, tags="cross-seed", is_paused=True
                         )
-                        util.move_files(src, dir_cs_out)
+                        self.qbt.torrentinfo[t_name]["count"] += 1
+                        try:
+                            torrent_hash_generator = TorrentHashGenerator(src)
+                            torrent_hash = torrent_hash_generator.generate_torrent_hash()
+                            util.move_files(src, dir_cs_out)
+                        except Exception as e:
+                            logger.warning(f"Unable to generate torrent hash from cross-seed {t_name}: {e}")
+                        try:
+                            if torrent_hash:
+                                torrent_info = self.qbt.get_torrents({"torrent_hashes": torrent_hash})
+                        except Exception as e:
+                            logger.warning(f"Unable to find hash {torrent_hash} in qbt: {e}")
+                        if torrent_info:
+                            torrent = torrent_info[0]
+                            self.qbt.torrentvalid.append(torrent)
+                            self.qbt.torrentinfo[t_name]["torrents"].append(torrent)
+                            self.qbt.torrent_list.append(torrent)
                 else:
                     logger.print_line(f"Found {t_name} in {dir_cs} but original torrent is not complete.", self.config.loglevel)
                     logger.print_line("Not adding to qBittorrent", self.config.loglevel)
             else:
-                error = f"{t_name} not found in torrents. Cross-seed Torrent not added to qBittorrent."
+                error = f"{tr_name} not found in torrents. Cross-seed Torrent not added to qBittorrent."
                 if self.config.dry_run:
                     logger.print_line(error, self.config.loglevel)
                 else:

modules/core/remove_orphaned.py

@@ -1,13 +1,10 @@
 import os
+from concurrent.futures import ThreadPoolExecutor
 from fnmatch import fnmatch
-from itertools import repeat
-from multiprocessing import cpu_count
-from multiprocessing import Pool
 
 from modules import util
 
 logger = util.logger
-_config = None
 
 
 class RemoveOrphaned:
@@ -21,56 +18,34 @@ def __init__(self, qbit_manager):
         self.root_dir = qbit_manager.config.root_dir
         self.orphaned_dir = qbit_manager.config.orphaned_dir
 
-        global _config
-        _config = self.config
-        self.pool = Pool(processes=max(cpu_count() - 1, 1))
+        max_workers = max(os.cpu_count() - 1, 1)
+        self.executor = ThreadPoolExecutor(max_workers=max_workers)
         self.rem_orphaned()
-        self.cleanup_pool()
+        self.executor.shutdown()
 
     def rem_orphaned(self):
         """Remove orphaned files from remote directory"""
         self.stats = 0
         logger.separator("Checking for Orphaned Files", space=False, border=False)
         torrent_files = []
-        root_files = []
         orphaned_files = []
         excluded_orphan_files = []
 
-        if self.remote_dir != self.root_dir:
-            local_orphaned_dir = self.orphaned_dir.replace(self.remote_dir, self.root_dir)
-            root_files = [
-                os.path.join(path.replace(self.remote_dir, self.root_dir), name)
-                for path, subdirs, files in os.walk(self.remote_dir)
-                for name in files
-                if local_orphaned_dir not in path
-            ]
-        else:
-            root_files = [
-                os.path.join(path, name)
-                for path, subdirs, files in os.walk(self.root_dir)
-                for name in files
-                if self.orphaned_dir not in path
-            ]
+        root_files = self.executor.submit(util.get_root_files, self.remote_dir, self.root_dir, self.orphaned_dir)
 
         # Get an updated list of torrents
         logger.print_line("Locating orphan files", self.config.loglevel)
         torrent_list = self.qbt.get_torrents({"sort": "added_on"})
-        torrent_files_and_save_path = []
-        for torrent in torrent_list:
-            torrent_files = []
-            for torrent_files_dict in torrent.files:
-                torrent_files.append(torrent_files_dict.name)
-            torrent_files_and_save_path.append((torrent_files, torrent.save_path))
+
         torrent_files.extend(
             [
                 fullpath
-                for fullpathlist in self.pool.starmap(get_full_path_of_torrent_files, torrent_files_and_save_path)
+                for fullpathlist in self.executor.map(self.get_full_path_of_torrent_files, torrent_list)
                 for fullpath in fullpathlist
-                if fullpath not in torrent_files
             ]
         )
 
-        orphaned_files = set(root_files) - set(torrent_files)
+        orphaned_files = set(root_files.result()) - set(torrent_files)
 
         if self.config.orphaned["exclude_patterns"]:
             logger.print_line("Processing orphan exclude patterns")
@@ -108,30 +83,27 @@ def rem_orphaned(self):
             self.config.send_notifications(attr)
             # Delete empty directories after moving orphan files
             if not self.config.dry_run:
-                orphaned_parent_path = set(self.pool.map(move_orphan, orphaned_files))
+                orphaned_parent_path = set(self.executor.map(self.move_orphan, orphaned_files))
                 logger.print_line("Removing newly empty directories", self.config.loglevel)
-                self.pool.starmap(util.remove_empty_directories, zip(orphaned_parent_path, repeat("**/*")))
+                self.executor.map(lambda dir: util.remove_empty_directories(dir, "**/*"), orphaned_parent_path)
 
         else:
             logger.print_line("No Orphaned Files found.", self.config.loglevel)
 
-    def cleanup_pool(self):
-        self.pool.close()
-        self.pool.join()
-
-
-def get_full_path_of_torrent_files(torrent_files, save_path):
-    fullpath_torrent_files = []
-    for file in torrent_files:
-        fullpath = os.path.join(save_path, file)
-        # Replace fullpath with \\ if qbm is running in docker (linux) but qbt is on windows
-        fullpath = fullpath.replace(r"/", "\\") if ":\\" in fullpath else fullpath
-        fullpath_torrent_files.append(fullpath)
-    return fullpath_torrent_files
-
-
-def move_orphan(file):
-    src = file.replace(_config.root_dir, _config.remote_dir)  # Could be optimized to only run when root != remote
-    dest = os.path.join(_config.orphaned_dir, file.replace(_config.root_dir, ""))
-    util.move_files(src, dest, True)
-    return os.path.dirname(file).replace(_config.root_dir, _config.remote_dir)  # Another candidate for micro optimizing
+    def move_orphan(self, file):
+        src = file.replace(self.root_dir, self.remote_dir)
+        dest = os.path.join(self.orphaned_dir, file.replace(self.root_dir, ""))
+        util.move_files(src, dest, True)
+        return os.path.dirname(file).replace(self.root_dir, self.remote_dir)
+
+    def get_full_path_of_torrent_files(self, torrent):
+        torrent_files = map(lambda dict: dict.name, torrent.files)
+        save_path = torrent.save_path
+
+        fullpath_torrent_files = []
+        for file in torrent_files:
+            fullpath = os.path.join(save_path, file)
+            # Replace fullpath with \\ if qbm is running in docker (linux) but qbt is on windows
+            fullpath = fullpath.replace(r"/", "\\") if ":\\" in fullpath else fullpath
+            fullpath_torrent_files.append(fullpath)
+        return fullpath_torrent_files

modules/core/tag_nohardlinks.py

@@ -186,6 +186,7 @@ def tag_nohardlinks(self):
         """Tag torrents with no hardlinks"""
         logger.separator("Tagging Torrents with No Hardlinks", space=False, border=False)
         nohardlinks = self.nohardlinks
+        check_hardlinks = util.CheckHardLinks(self.root_dir, self.remote_dir)
         for category in nohardlinks:
             torrent_list = self.qbt.get_torrents({"category": category, "status_filter": "completed"})
             if len(torrent_list) == 0:
@@ -199,7 +200,7 @@ def tag_nohardlinks(self):
                 continue
             for torrent in torrent_list:
                 tracker = self.qbt.get_tags(torrent.trackers)
-                has_nohardlinks = util.nohardlink(
+                has_nohardlinks = check_hardlinks.nohardlink(
                     torrent["content_path"].replace(self.root_dir, self.remote_dir), self.config.notify
                 )
                 if any(tag in torrent.tags for tag in nohardlinks[category]["exclude_tags"]):