Skip to content

Latest commit

 

History

History
1392 lines (1049 loc) · 52 KB

README.md

File metadata and controls

1392 lines (1049 loc) · 52 KB
Raw

tget 5.7.3 - torrent get

tget is a suite of programs: tget and plexfix.

tget grew out of my dissatisfaction with FlexGet's behavior and configuration. Don't get me wrong, FlexGet is an amazing program in its own right, but there were some things I couldn't get it to do. And configuration is clumsy, at best. Coming from the Lisp world, YAML just sucks. Go ahead and compare tget's example configuration file to that of FlexGet. The power and flexibility of Lisp make the comparison a shutout. Of course, that assumes tget does what you want it to do.

The main limitation I wanted to overcome by writing tget, though, was to set the download quality based on time: wait 6 hours (from publication date), then download the x264 SD version, but if after 12 hours after that a version isn't available, then download the x264 720p version, and if either of those aren't available for 2 days then download the XviD SD version. The latter is really my last choice, because the quality can be pretty crappy. What I just described is not possible in FlexGet. tget makes this pretty easy.

tget isn't nearly as functional as FlexGet, though, and the feed parsing only works (currently) with two sites (TvT and BTN). I'm always looking to add more.

plexfix works around deficiencies in Plex, which doesn't recognize certain filename formats. For example, Foo-612.mp4 would not be recognized by Plex, perhaps because the naming convention is ambiguous--is it season 6 and espisode 12 or the June 12th episode of Foo? Plex punts of filenames like this. plexfix creates symbolic links so that Plex can see the correct season and episode.

Table of Contents

How it works
Installation
Getting started
Configuration
Maintenance tasks
Cleanup
plexfix
Usage
Example configuration file

How it works

tget has these basic phases:

  • parsing the configuration file,

  • fetching the RSS feeds defined by the configuration file, and

  • downloading torrent files for episodes never seen before, which meet the criteria defined by the configuration file.

To accomplish the above, tget maintains a database of series and episodes, with various data about them.

Here are some details about the above phases:

  • The configuration file defines which series to monitor. When the configuration file is read, entries are created in the database for each series. You can examine this stored information with command line arguments to tget.

  • When an RSS feed is fetched, any episodes related to series defined by the configuration file, and thus in the database, are stored in the database, but marked as not downloaded.

  • Each series in the database has a complete-to slot. This tells tget what episode number the series is completed to, and tget will never download anything at or before this episode.

    Say you've downloaded all the episodes of Breaking Bad up to S05E08. Any episodes which appear in the RSS feed at or before this episode will be ignored and never downloaded.

    There are command line arguments, which start with --catch-up, which can be used to manipulate the complete-to slot of a series.

  • Episodes are sometimes downloaded out of order. You will not miss an episode of a series if it appears in the RSS feed out of order.

    Say the complete-to for a series is S05E08, and episode S05E10 appears on one invocation of tget and S05E09 appears in a later invocation. Both episodes will be downloaded and the complete-to will not be updated to S05E10 until after episode S05E09 is downloaded.

  • When the torrent file is downloaded, the resulting file can be given to transmission-remote for processing (transmission-remote is a command line tool used to control Transmission, the cross-platform BitTorrent client) or put in a specific directory watched by your BitTorrent client. See the example configuration file for information on how to use either of these methods.

Installation

tget has been tested on Linux and Mac OS X and depends on two other software packages:

transmission-remote -- [OPTIONAL] a program that can communicate to a local or remote instance of Transmission. tget uses it to download the actual episodes . I run Transmission on Mac OS X, on the same local area network. transmission-remote makes this easy. On CentOS and Fedora, install the transmission-common rpm. On a Mac, I suggest looking to MacPorts.

Allegro Common Lisp -- For now, at least, no binary packages of tget are available and you need Allegro Common Lisp to build it.

After cloning the repo, build tget:

$ make

That should produce a directory tget/, which can be installed with:

$ make install

You can make a Makefile.local to override features of the GNU Make file.

Getting started

After installing the software, the next thing you need is a configuration file. See the next section for information on configuration, and the section at the end for an example configuration file, which you can edit for your needs.

The database used by tget is stored in $HOME/.tget.d. You can change this location with the --root command line argument.

Next, you probably want to create a new database which has everything you care about marked as already downloaded. The following command will start from scratch and, assuming the RSS feed you are using supports it, mark 6 months of episodes as already downloaded:

$ tget --reset --learn --feed-interval 180

Next, you'll want to set up cron to periodically run tget, perhaps every day at 3AM:

0 3 * * *  source $HOME/.bashrc; MAILTO=username@domain /usr/local/bin/tget --cron

Running more often is fine, but you need to be careful not to run afoul of the site rules for how often you can download their RSS feed. You can use the MAILTO environment variable to change the user that receives the output of tget. $HOME/.bashrc is sourced to pick up some environment variables used by the example configuration file. You can omit this part if you do not need any environment variables.

Configuration

At the highest level, the configuration file defines these entities:

  • transmission-remote settings - in the config example below I pull items from the environment, mainly because I don't want to share that private information in this documentation. You can store yours in the configuration file directly.

  • RSS feeds - this is where tget gets the information on what is available to download. Each feed is site specific and possibly user specific.

  • quality settings - these define the quality of the files you want to download. It's a very important aspect of the configuration, as it will directly affect your viewing pleasure.

  • download groups - if you have multiple people that are watching episodes, or multiple sites from which you are downloading, then you likely will want to use different groups with differing options for each group. Each group might, for example, have different download locations.

  • TV shows you care about - the defseries macro defines the shows you want to download. A series is associated with a group, at a minimum, and can override the quality of what you want to download. For example, you may want to download 720p versions of Top Gear, but would accept a lower resolution of other shows.

That's the big picture. There is an example configuration file below. It is fully annotated and is a good place to start.

set-torrent-handler handler

Define how .torrent files will be handled. There are two choices: have transmission-remote handle them or define a directory in which the .torrent files are placed. For the latter, the configuration file option would be:

(set-torrent-handler (pathname "~/Downloads/"))

This would cause tget to store all downloaded .torrent files in the directory ~/Downloads/.

The second choice is to use make-tranmission-remote-handler to define how to communicate with transmission-remote.

make-transmission-remote-handler ...options...

The values for each option have direct correspondence to transmission-remote command line arguments, given by the table below. Each keyword given in the table must be paired with a value, like this:

:host "download.example.com"

The :host part must come before the value.

keyword(s) transmission-remote argument
:host & :port host:port
:username & :password --auth username:password
:add-paused t --start-paused
:add-paused nil --no-start-paused
:trash-torrent-file t --trash-torrent
:ratio -sr ratio
:download-path --download-dir path
:docker-container-name See below
:docker-user See below
:ssh-user See below
:ssh-identity See below

When a subdirectory is specified by defseries, tget must ensure it exists before telling transmission to download into it. There are two methods for tget communicating with transmission: via docker or SSH.

You may specify one of the group of arguments, :docker-* or :ssh-*, but not both.

:docker-container-name and :docker-user are the container's name and user inside the container (for su) to create directories.

:ssh-user and :ssh-identity are the user and identity file to create directories.

SSH example:

(set-torrent-handler
 (make-transmission-remote-handler
  :host (sys:getenv "TRANSMISSION_HOST")
  :port (sys:getenv "TRANSMISSION_PORT")
  :username (sys:getenv "TRANSMISSION_USER")
  :password (sys:getenv "TRANSMISSION_PASS")
  :ssh-user (sys:getenv "TRANSMISSION_SSH_USER")
  :ssh-identity (sys:getenv "TRANSMISSION_SSH_IDENTITY")
  :add-paused nil
  :trash-torrent-file t
  :ratio "1.04"))

In this example, the values for :host and other options are being pulled from the environment with sys:getenv.

Docker example:

(set-torrent-handler
 (make-transmission-remote-handler
  ...
  :docker-container-name "transmission"
  :docker-user (sys:getenv "USER")
  ...))

defquality &key priority container source codec resolution

The valid values for each keyword option (the names after &key) are:

:priority -- any positive integer from 1 to 100. The higher the number the higher the priority the that has this quality will be given if there are multiple matching episodes.

:container -- the acceptable containers for the quality. A container is, essentially, the file type of the downloaded file (e.g. mp4). Valid values:

:avi, :mkv, :vob, :mpeg, :mp4, :iso, :wmv, :ts, :m4v, :m2ts

:source -- the acceptable source for the quality. The source is where the stream originated. :hdtv is a common source. Valid values:

:pdtv, :hdtv, :dsr, :dvdrip, :tvrip, :vhsrip, :bluray, :bdrip, :brrip, :dvd5, :dvd9, :hddvd, :web-dl

:codec -- the codec used to encode the original source (e.g. XviD or x264, aka h.264). Valid values:

:x264, :h.264, :xvid, :mpeg2, :divx, :dvdr, :vc-1, :wmv, :bd

:resolution -- the resolution of the encoded image (e.g. 720p). Valid values:

:sd, :720p, :1080p, :1080i

deftracker name &key url debug-feed public download-delay disabled ratio upload-limit

name -- the name of the tracker, a keyword (e.g. :eztv).

url -- the URL that points to an RSS feed for this tracker. If you require a password to access the tracker's RSS feed, it would need to be contained in the URL.

debug-feed -- a filename which is to be substituted for the actual URL, for debugging purposes.

public -- a boolean which indicates if this tracker is public or not.

download-delay -- the delay imposed on downloads for this tracker. This option allows you to give priority to different trackers, based of time.

disabled -- a boolean indicating if the tracker has been disabled. It is useful when trackers go offline for extended periods of time.

ratio -- the ratio to be applied to this specific tracker. It takes precedence over the global ratio specified for Transmission, if that is used.

upload-limit -- set the upload limit for this tracker. This is useful for when the tracker unreliably gives you upload credit for what you upload (I'm looking at you TvT!). Sometimes, if you upload too fast the tracker will not count a significant amount of your upload data.

defgroup name &key rss-url trackers delay ratio quality download-path

name -- the name of the group, a keyword (e.g. :bruce).

:trackers -- a list of trackers which apply to this group.

:rss-url -- the URL of the RSS feed. This option has been deprecated in favor of the :trackers option.

:delay -- nil or a positive integer, which represents the delay, in hours, that episodes should be delayed from download. NOTE: this is distinct from any site-specific delay that might be available in the RSS feed.

ratio -- the share ratio given to all downloads for this group. It's a string not a floating point number, so we don't have to worry about the printed representation of floats. It no ratio is given, then the default in Transmission is used. A ratio of "-1" means seed forever.

:quality -- a user-defined quality symbol. See defquality.

:download-path -- The path used by transmission-remote to store the downloaded file for this group. Because the path can be remote, no checking on the validity of the path is done.

defseries name group &key private delay quality catch-up subdir date-based aliases

Required arguments:

name -- the name of the series. Case is not significant, and single quotes are removed in parsing.

private -- a boolean indicating this series should only be downloaded from a tracker listed as private.

group -- the group to which this series belongs.

Optional arguments:

:delay -- this allows the group delay to be overriden.

:quality -- this allows the group quality to be overriden.

:catch-up -- do not download any episodes of this series at or before this season and episode. An example value is S05E08. Case differences are ignored (e.g., you can use s or S).

:subdir -- put episodes of this series into a subdirectory of the group download path. This is a workaround for limitations in Plex Media Server, which will not recognize episodes of shows with dates in their names instead of episode numbers. The Daily Show and The Colbert Report are two examples of these types of shows.

:date-based -- indicate this series is date based and turn off the accounting for complete seasons. This is useful for shows like The Daily Show and The Colbert Report, which don't have seasons and the episodes for these are ordered by date.

:aliases -- a list of aliases for name. This feature exists because many sites use different names for series, The Daily Show with Jon Stewart vs. The Daily Show. Here's how you specify it:

:aliases ("The Daily Show")

Configuration options

*repack-window-seconds* -- the number of seconds within the download of an original file that a repack will be downloaded. The default value is:

86400

This value is the number of seconds after an episode is downloaded that we will download a repack of that episode. A repack is a re-release of an episode to fix problems in the original.

Maintenance tasks

Maintenance task: making the database smaller

After some time, tget can slow down due to the database growing. It has to do with how tget operates, temporarily storing episodes while it is deciding what to download. These temporarily-stored episodes are deleted almost immediately, but they leave holes in the database. To clean them out, you can

$ tget --compact-database

This will backup the current database and then compact it to make it more efficient. It will usually dramatically decrease the size of the database, if it's been in use for a month or more.

Maintenance task: adding a new series

The easiest way to add a new series is to add it to the configuration file, like this:

(defseries "New Series" :group1)

However, sometimes you might want to add it in a caught up state. That would be:

(defseries "New Series" :group1 :catch-up "S05E08")

Which would cause episodes from S05E09 and later to be downloaded, and nothing before it.

Maintenance task: removing a series

There are two ways you can remove a series.

The first method entails removing it from the configuration file and then running tget to remove it from the database. The removing from the database part would be:

$ tget --delete-series "True Blood"

The second method entails changing the configuration file to note this series should be removed:

(defseries "Series Name" :group1 :remove t)

You can remove this entry in the configuration file after tget has been run once.

Maintenance task: removing an episode or episodes

You may remove a single episode or episodes.

Let's say someone uploaded pre-release versions of some really popular show, in SD with a really crappy frame rate. Say it was the first 4 episodes of the new Season 5 of the show. tget downloaded them and you want to delete them:

$ tget --delete-episode "Game of Bones S05E01"
$ tget --delete-episode "Game of Bones S05E02"
$ tget --delete-episode "Game of Bones S05E03"
$ tget --delete-episode "Game of Bones S05E04"
$ tget --catch-up-series "Game of Bones S04"

The last command was to make sure you get them in the future, but also make sure to upgrade your config file so that only high quality episodes will be downloaded in the future:

(defseries "Game of Bones :me :private t :delay 0 :quality :720p)

Cleanup

This is a mode of tget, when the --cleanup or -c arguments are given. It is intended to be run manually or from cron. I recommend that you run it manually for a while, since it will take a while to tune the configuration so that you do not receive hit and runs from any tracker.

With no optional arguments, nothing is removed. tget prints the status of torrents and videos:

$ tget -c 
These torrents are complete:

T name                                     %done ratio seeded      left        
F at midnight 2015.06.10                   100%  1.22  1d 15:45:19 
F the daily show 2015.06.09                100%  0.64  3d 01:45:17 
F at midnight 2015.06.08                   100%  1.02  3d 18:37:16 

These torrents are incomplete:

T name                                     %done ratio seeded      left        
F the daily show 2015.06.10                100%  0.60  2d 00:45:26     23:14:34
B the daily show 2015.06.11                100%  0.22  23:45:30     2d 00:14:30
B the nightly show with larry wilmore 2015 100%  0.22  19:43:26     2d 04:16:34
F the graham norton show S17E10            100%  1.10  14:31:43     2d 09:28:17
B the nightly show with larry wilmore 2015 100%  0.24  1d 19:37:36  1d 04:22:24
F hannibal S03E02                          100%  1.10  1d 10:41:49  1d 13:18:11
F at midnight 2015.06.09                   100%  1.01  2d 18:44:23     05:15:37
F at midnight 2015.06.11                   100%  0.88  15:42:07     2d 08:17:53
F the ultimate fighter S21E08              100%  0.49  2d 09:36:32     14:23:28
B the nightly show with larry wilmore 2015 100%  0.23  1d 19:37:09  1d 04:22:51
B the nightly show with larry wilmore 2015 100%  0.29  1d 19:37:22  1d 04:22:38
B ridiculousness S06E29                    100%  0.24  10:45:16     2d 13:14:44
F childrens hospital us S06E13             100%  0.62  09:36:38     2d 14:23:22

TV:Adrian+Kevin:
NO seeding:ridiculousness.S06E29.mp4
TV:Kevin:
NO 68h<72h:Last.Week.Tonight.With.John.Oliver.S02E16.HDTV.x264-BATV.mp4
NO seeding:the.daily.show.2015.06.09.nick.offerman.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.10.colin.quinn.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.11.mark.ruffalo.hdtv.x264-crooks.mp4
NO 68h<72h:the.daily.show.2015.06.08.nicola.sturgeon.hdtv.x264-crooks.mp4
NO seeding:at.midnight.2015.06.08.mp4
NO seeding:at.midnight.2015.06.11.mp4
NO seeding:The.Graham.Norton.Show.S17E10.720p.HDTV.x264-FTP.mkv
NO seeding:The.Ultimate.Fighter.S21E08.HDTV.x264-KOENiG.mp4

To remove the completed torrents, use --remove:

$ tget -c --remove
These torrents were removed:

T name                                     %done ratio seeded      left        
F at midnight 2015.06.10                   100%  1.22  1d 15:45:47 
F the daily show 2015.06.09                100%  0.64  3d 01:45:45 
F at midnight 2015.06.08                   100%  1.02  3d 18:37:44 

These torrents are incomplete:

T name                                     %done ratio seeded      left        
F the daily show 2015.06.10                100%  0.60  2d 00:45:54     23:14:06
B the daily show 2015.06.11                100%  0.22  23:45:58     2d 00:14:02
B the nightly show with larry wilmore 2015 100%  0.22  19:43:54     2d 04:16:06
F the graham norton show S17E10            100%  1.10  14:32:11     2d 09:27:49
B the nightly show with larry wilmore 2015 100%  0.24  1d 19:38:04  1d 04:21:56
F hannibal S03E02                          100%  1.10  1d 10:42:17  1d 13:17:43
F at midnight 2015.06.09                   100%  1.01  2d 18:44:51     05:15:09
F at midnight 2015.06.11                   100%  0.88  15:42:35     2d 08:17:25
F the ultimate fighter S21E08              100%  0.49  2d 09:37:00     14:23:00
B the nightly show with larry wilmore 2015 100%  0.23  1d 19:37:37  1d 04:22:23
B the nightly show with larry wilmore 2015 100%  0.29  1d 19:37:50  1d 04:22:10
B ridiculousness S06E29                    100%  0.24  10:45:44     2d 13:14:16
F childrens hospital us S06E13             100%  0.62  09:37:06     2d 14:22:54

TV:Adrian+Kevin:
NO seeding:ridiculousness.S06E29.mp4
TV:Kevin:
NO 68h<72h:Last.Week.Tonight.With.John.Oliver.S02E16.HDTV.x264-BATV.mp4
NO 32h<72h:the.daily.show.2015.06.09.nick.offerman.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.10.colin.quinn.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.11.mark.ruffalo.hdtv.x264-crooks.mp4
NO 68h<72h:the.daily.show.2015.06.08.nicola.sturgeon.hdtv.x264-crooks.mp4
NO 10h<72h:at.midnight.2015.06.08.mp4
NO seeding:at.midnight.2015.06.11.mp4
NO seeding:The.Graham.Norton.Show.S17E10.720p.HDTV.x264-FTP.mkv
NO seeding:The.Ultimate.Fighter.S21E08.HDTV.x264-KOENiG.mp4

Now we have:

$ tget -c

These torrents are incomplete:

T name                                     %done ratio seeded      left        
F the daily show 2015.06.10                100%  0.60  2d 00:46:03     23:13:57
B the daily show 2015.06.11                100%  0.22  23:46:07     2d 00:13:53
B the nightly show with larry wilmore 2015 100%  0.22  19:44:03     2d 04:15:57
F the graham norton show S17E10            100%  1.10  14:32:20     2d 09:27:40
B the nightly show with larry wilmore 2015 100%  0.24  1d 19:38:13  1d 04:21:47
F hannibal S03E02                          100%  1.10  1d 10:42:26  1d 13:17:34
F at midnight 2015.06.09                   100%  1.01  2d 18:45:00     05:15:00
F at midnight 2015.06.11                   100%  0.88  15:42:44     2d 08:17:16
F the ultimate fighter S21E08              100%  0.49  2d 09:37:09     14:22:51
B the nightly show with larry wilmore 2015 100%  0.23  1d 19:37:46  1d 04:22:14
B the nightly show with larry wilmore 2015 100%  0.29  1d 19:37:59  1d 04:22:01
B ridiculousness S06E29                    100%  0.24  10:45:53     2d 13:14:07
F childrens hospital us S06E13             100%  0.62  09:37:15     2d 14:22:45

TV:Adrian+Kevin:
NO seeding:ridiculousness.S06E29.mp4
TV:Kevin:
NO 68h<72h:Last.Week.Tonight.With.John.Oliver.S02E16.HDTV.x264-BATV.mp4
NO 32h<72h:the.daily.show.2015.06.09.nick.offerman.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.10.colin.quinn.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.11.mark.ruffalo.hdtv.x264-crooks.mp4
NO 68h<72h:the.daily.show.2015.06.08.nicola.sturgeon.hdtv.x264-crooks.mp4
NO 10h<72h:at.midnight.2015.06.08.mp4
NO seeding:at.midnight.2015.06.11.mp4
NO seeding:The.Graham.Norton.Show.S17E10.720p.HDTV.x264-FTP.mkv
NO seeding:The.Ultimate.Fighter.S21E08.HDTV.x264-KOENiG.mp4

To remove the watched torrents before the configured waiting period of 72h is up (set the waiting period to 0, for example):

$ tget -c --remove -h0

These torrents are incomplete:

T name                                     %done ratio seeded      left        
F the daily show 2015.06.10                100%  0.60  2d 00:46:16     23:13:44
B the daily show 2015.06.11                100%  0.22  23:46:20     2d 00:13:40
B the nightly show with larry wilmore 2015 100%  0.22  19:44:16     2d 04:15:44
F the graham norton show S17E10            100%  1.10  14:32:33     2d 09:27:27
B the nightly show with larry wilmore 2015 100%  0.24  1d 19:38:26  1d 04:21:34
F hannibal S03E02                          100%  1.10  1d 10:42:39  1d 13:17:21
F at midnight 2015.06.09                   100%  1.01  2d 18:45:13     05:14:47
F at midnight 2015.06.11                   100%  0.88  15:42:57     2d 08:17:03
F the ultimate fighter S21E08              100%  0.49  2d 09:37:22     14:22:38
B the nightly show with larry wilmore 2015 100%  0.23  1d 19:37:59  1d 04:22:01
B the nightly show with larry wilmore 2015 100%  0.29  1d 19:38:12  1d 04:21:48
B ridiculousness S06E29                    100%  0.24  10:46:06     2d 13:13:54
F childrens hospital us S06E13             100%  0.62  09:37:28     2d 14:22:32

TV:Adrian+Kevin:
NO seeding:ridiculousness.S06E29.mp4
TV:Kevin:
rm /me/tplex/content/videos/kevin/Last.Week.Tonight.With.John.Oliver/Last.Week.Tonight.With.John.Oliver.S02E16.HDTV.x264-BATV.mp4
rm /me/tplex/content/videos/kevin/The.Daily.Show/the.daily.show.2015.06.09.nick.offerman.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.10.colin.quinn.hdtv.x264-crooks.mp4
NO seeding:the.daily.show.2015.06.11.mark.ruffalo.hdtv.x264-crooks.mp4
rm /me/tplex/content/videos/kevin/The.Daily.Show/the.daily.show.2015.06.08.nicola.sturgeon.hdtv.x264-crooks.mp4
rm /me/tplex/content/videos/kevin/At.Midnight/at.midnight.150608-yestv.mp4
rm /me/tplex/content/videos/kevin/At.Midnight/at.midnight.2015.06.08.mp4
NO seeding:at.midnight.2015.06.11.mp4
NO seeding:The.Graham.Norton.Show.S17E10.720p.HDTV.x264-FTP.mkv
NO seeding:The.Ultimate.Fighter.S21E08.HDTV.x264-KOENiG.mp4
$

plexfix

plexfix is a simple program run from transmission after a torrent completes. plexfix is given no arguments. The directory and name are passed in the environment variables TR_TORRENT_DIR and TR_TORRENT_NAME. plexfix uses these to determine if Plex needs a symlink to see the newly download torrent. To enable it, use the settings.json file for transmission:

"script-torrent-done-enabled": true, 
"script-torrent-done-filename": "/usr/local/lib/plexfix/plexfix",

/usr/local/lib/plexfix is the default location for the installation of plexfix.

For debugging purposes, you can also give a complete pathname to plexfix on the command line, rather than setting the above environment variables.

plexfix also accepts these arguments:

-d :: debug mode -m :: do not send mail -n :: do not execute, just say what would be done -q :: be quiet, only print error messages -v :: be verbose

An example email:

From: layer@somewhere
To: layer@somewhere
Subject: plexfix

OLD name: /me/tplex/content/videos/adrian+kevin/ridiculousness.0629-yestv.mp4
NEW name: /me/tplex/content/videos/adrian+kevin/ridiculousness.S06E29.mp4

Usage

There are two primary modes of operation: cleanup and download. Cleanup mode is activated by using the --cleanup or -c arguments. Download mode is activated by using the other primary behavior determining arguments. See below.

Cleanup mode has two parts: removing torrents from Transmission after they have seeded enough and removing video files watched by Plex.

[download] Primary behavior determining arguments (one of these must be given):

--add file-or-directory
--catch-up   
--catch-up-series series-episode-name
--check-database
--clean-database
--compact-database
--delete-episodes series-name
--delete-episode episode-description
--delete-orphans
--delete-series series-name
--dump-all
--dump-complete-to
--dump-episodes series-name
--dump-orphans
--dump-series series-name
--dump-stats
--skip series-name

[download] Behavior modifying arguments:

--auto-backup condition
--config file
--cron
--db database-name
--debug
--force
--learn
--reset
--root data-directory
--verbose or -v

[cleanup] Behavior modifying arguments:

-h
--remove
--remove-bad
--remove-seeded
--remove-watched
--torrent-info
--torrents-only

Usage details

The tget options are below. When there is an argument naming series, these are canonicalized by removing single quotes and converting to lower case. However, the series names presented to you will be stored in their original form.

  • --help

    Print full help text and exit.

The following are arguments controlling primary behavior:

  • --cleanup or -c

    Active cleanup mode. See the cleanup options below.

  • --run

    The primary mode of operation, whereby RSS feeds are retrieved, searched for new episodes and those episode torrents downloaded.

  • --add thing

    If thing is a file, it should be a .torrent file, which is manually added. If thing is a directory, then all the .torrent files in the directory are added. This circumvents any matching and assumes the episodes should be downloaded. The series name,episode and season numbers are taken directly from the file name. If the information cannot be extracted, you can rename the files to suit tget.

    If the episode has already been downloaded, then --force is required to make tget download it again.

  • --catch-up

    Go through the database and make the newest episode of each series the oldest episode that will ever be downloaded for that series; this prevents old episodes, which are released from time to time, from being downloaded.

  • --catch-up-series series-ep

    Catch series up to the episode given in the companion argument. See examples below.

  • --check-database

    Report on items in the database which can be cleaned up.

  • --clean-database

    Remove items reported by --check-database.

  • --compact-database

    This saves and restores the database, compacting it at the same time.

  • --delete-episodes series-name

    Delete episodes with series name matching series-name. This is permanent! Using this option with --auto-backup force is recommended.

    --force can be used if there are multiple episodes that match.

  • --delete-episode episode-description

    Delete the episode matching episode-description. This is permanent! Using this option with --auto-backup force is recommended. Example:

    --delete-episode "I Love Lucy S05E01"

  • --delete-orphans

    Delete orphaned series and episodes from the database. See --dump-orphans for more information.

  • --delete-series series-name

    Delete series with series name matching series-name. This is permanent! Using this option with --auto-backup force is recommended.

  • --dump-all

    Dump all episode objects to stdout.

  • --dump-complete-to

    Dump a table of series and last downloaded information for all series in the database to stdout. See --catch-up-series.

  • --dump-episodes series-name

    Dump all episode objects matching series name series-name to stdout.

  • --dump-orphans

    Dump orphaned series and episodes. Orphaned series are those which do not appear in the config file but exist in the database. Orphaned episodes are those are in the database but have no corresponding series object.

  • --dump-series series-name

    Dump all series objects matching series name series-name to stdout.

  • --dump-stats

    Dump information about the database to stdout.

  • --skip series-name

    Skip the next episode of series-name. It does so by using the last downloaded episode and incrementing it by 1.

    Note: this has no effect on date-based series. See the :date-based option to defseries.

The following options augment the options above or have the stated side effects:

  • --auto-backup {reset|program-update|schema-update|restore|force|never}

    Perform a backup of the database under specific conditions given by the companion argument:

    • compact - when the database is being compacted
    • force - always
    • never - never
    • program-update - when the program changed since the last db update
    • reset - when the database is being reset
    • schema-update - when the schema changes

    The default is to make backups for all conditions above.

  • --config file

    Load file as the configuration file instead of one of the built-in defaults. The default list (searched in this order) is to load the first file found in the list:

    • config.cl in the same directory as the program
    • $HOME/.tget.cl
    • config.cl in the tget data directory (see the --root option)

  • --cron

    Quiet mode, for cron jobs.

  • --db database

    The name of the database. The default is db in the tget data directory (see the --root). This name will itself become a directory.

  • --debug

    Run in debug mode. In this mode, torrents are not downloaded and the debug feed defined by the configuration file is used. Also, the program is more verbose. This is for testing and is not recommended. It implies --learn.

  • --force

    The meaning of this argument depends on the other arguments and context in which it is given.

  • --learn

    Mark episodes which match the configuration as downloaded. This is useful when using tget for the first time. See the examples below.

  • --reset

    Reset database before beginning operation--this removes all data from the database. The default --auto-backup settings will cause a backup to be performed before the reset.

  • --root data-directory

    Change the data directory, the defaults is $HOME/.tget.d/

  • --verbose or -v

    Verbose mode. For now, it makes --dump-episodes print detailed information on episodes.

Cleanup options:

  • -h hours

    Set the threshold for how long after a video is watched that it can be considered for removal. The value of 0 can be used to remove any videos which have been watched, for which seeding is complete.

    The default is 72 hours, or 3 days.

  • --remove

    Remove videos which meet the criteria for removal. This means removing the torrent from Transmission and removing the file from disk.

  • --remove-bad

    A special cleanup mode, which causes only unregistered torrents to be removed from Transmission. When this argument is given, without the other cleanup mode arguments (-c and --cleanup), it is used as a pass before torrents are downloaded. Along with the torrent, the episode is deleted from the database. This allows for bad torrents to be removed and redownloaded, if they should become available again.

  • --remove-seeded

    Only remove torrents from Transmission, if they meet the seeding criteria.

  • --remove-watched

    Only remove watched files from disk, if they meet the criteria.

  • --torrent-info

    List torrents torrents currently in Transmission.

  • --torrents-only

    Clean up torrents only, not files on disk.

Examples:

Toss current database and catch up on shows released in the last 180 days marking them all as `downloaded'

$ tget --reset --learn

The usefulness of this is highly dependent on how far the feed for the site you are using goes back. Many sites do not have deep feeds, but some sites have parameters that allow you to go back in time. Sadly, this feature is rare these days, as it is expensive to support.

Same, but to a `temp' database in the current directory:

$ tget --reset --learn --root $PWD --db test.db

Let's see what the series object for "Regular Show" looks like. The series name is not case sensitive:

$ tget --dump-series "regular show"

These all refer to the same series:

$ tget --dump-series "james mays man lab"
$ tget --dump-series "James Mays Man Lab"
$ tget --dump-series "James May's Man Lab"

To see the episodes of the above, you could use any of the variations on the names given, for example:

$ tget --dump-episodes "James May's Man Lab"

Compact the database:

$ tget --compact-database

Catch up series to a specific episode:

#   note that episodes before s04e21 have been downloaded:
$ tget --catch-up-series "regular show s04e20"
#   note that all of season 4 has been downloaded:
$ tget --catch-up-series "breaking bad s04"

Example configuration file

;; Config file for tget

(in-package :user)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; General options

;; This is now reset once per run, so it's not too much data to save,
;; and it's good for debugging feed issues.
(setq *log-rss* (merge-pathnames "rss.log" *tget-data-directory*))
(setq *log-xml* (merge-pathnames "error.xml" *tget-data-directory*))

;; A good resource to see why something is or isn't downloading
(setq *log-file* (merge-pathnames "ep.log" *tget-data-directory*))

;; Use transmission-remote to tell your torrent client to download the
;; episode:
(set-torrent-handler
 (make-transmission-remote-handler
  :host (sys:getenv "TRANSMISSION_HOST")
  :port (sys:getenv "TRANSMISSION_PORT")
  :username (sys:getenv "TRANSMISSION_USER")
  :password (sys:getenv "TRANSMISSION_PASS")

;;;; Now using transmission in a container, with no ssh
  :docker-container-name "transmission"
  :docker-user (sys:getenv "USER")
;;;; 
  ;;:ssh-user (sys:getenv "TRANSMISSION_SSH_USER")
  ;;:ssh-identity (sys:getenv "TRANSMISSION_SSH_IDENTITY")

  :add-paused nil
  :trash-torrent-file t
  ;; The default when there is tracker defined ratio
  :ratio 1.1))

;; An alternate method for downloading .torrent files: put them into a
;; specific directory, where your torrent client will see them.
#+ignore
(set-torrent-handler (pathname "~/Downloads/"))

;; Same in and outside the transmission container
(setq *download-root* "/me/tplex/content/videos/")

(defvar *for-mom* "/me/tplex/content/videos/for-mom/")
(defvar *tv-archive* "/me/tplex/content/videos/archive/")

(defvar *allowed-codecs* '(:x264 :h.264))

;;;;;;;;;;;;;;;;;;;
;; For --cleanup ;;
;;;;;;;;;;;;;;;;;;;

;; Minimum seed time in seconds: 3 days, for torrents that don't have a
;; tracker-specific rule that takes precedence over this value.
(setq *minimum-seed-seconds* #.(* 3600 24 3))

;; Ignore anything watched in the last 3 days:
(setq *ignore-watched-within* #.(* 24 3))

(setq *sqlite3-binary*
  ;; CentOS 6.6 doesn't have an up-to-date enough sqlite3 binary, so we use
  ;; one from sqlite.org (version 3.8.10.2).
  "/usr/local/bin/sqlite3")

;; The directories to clean
(setq *watch-directories*
  '(("/me/tplex/content/videos/kevin" "/data/videos/kevin" . "TV Shows")
    ("/me/tplex/content/videos/movies/kevin" "/data/videos/movies/kevin"
     . "Movies")
    ("/me/tplex/content/videos/tmp" "/data/videos/tmp" . "temp")))


;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Trackers

(deftracker :freshon
    ;; The feed URL is
    ;;  https://freshon.tv/rss.php?feed=dl&passkey=<passkey>"
    ;; Get the <passkey> from your profile page.
    :url "https://freshon.tv/..."
    :debug-feed :freshon
    :download-delay 0
    ;; See if limiting the upload rate allows freshon to correctly count my
    ;; upload credits.  Sick of uploading lots and not getting credit for it.
    :upload-limit 100 ;; KB/s
    :ratio 1.3
    :re "freshon\\.tv"
    :char "F"
    :setter (lambda (torrent)
	      ;; 1.2 ratio and 3 days should do it?
	      (setf (torrent-ratio-limit torrent) 1.2)
	      (setf (torrent-seed-min-time torrent) (* 3600 24 3))))

(deftracker :btn
    ;; RSS feed configured by creating a notification filter with these
    ;; options: 
    ;; - categories: episode, season
    ;; - containers: avi, mkv, mp4
    ;; - codecs: x264, h.264
    ;; - resolutions: sd, 720p
    ;; All other checkboxs were unchecked.
    ;;
    ;; passkey is in DL links.  Very hard to figure this out!
    ;;
    :url "https://broadcasthe.net/..."
    :debug-feed :btn
    :disabled #-debug nil #+debug t
    ;; Can't make the delay too long or shows will fall off the RSS feed
    :download-delay #-debug 1 #+debug 0
    :ratio 1.5
    :re "broadcasthe\\.net|landof\\.tv"
    :char "B"
    :setter (lambda (torrent)
	      (setf (torrent-ratio-limit torrent) 1.50)
	      (if* (torrent-seasonp torrent)
		 then ;; Seed for a week + slop 
		      (setf (torrent-seed-min-time torrent)
			(* 3600 24 12))
		 else ;; The rules state you need to seed to 1:1 or 24
		      ;; hours.  Give it some slop, to make sure I don't
		      ;; get a H&R
		      ;; NOTE: the BTN tracker is notorious for not
		      ;;       counting seed time, so use 3d as the minimum
		      ;;       time here to make sure we don't get a H&R
		      (setf (torrent-seed-min-time torrent) (* 3600 24 3)))))

(deftracker :mtv
    :url "https://www.morethan.tv/..."
    :disabled nil
    :download-delay 0
    :ratio 1.5
    :re "morethantv"
    :char "T"
    :setter (lambda (torrent)
	      (setf (torrent-ratio-limit torrent) 1.5)
	      (setf (torrent-seed-min-time torrent) (* 3600 24 100))))

(deftracker :shazbat
    :url "https://www.shazbat.tv/..."
    :debug-feed :shazbat
    :disabled t
    :download-delay 0
    :ratio 1.5
    :re "shazbat|bttracker\\.tv"
    :char "S"
    :setter (lambda (torrent)
	      ;; 1.2 ratio and 3 days should do it?
	      (setf (torrent-ratio-limit torrent) 1.2)
	      (setf (torrent-seed-min-time torrent) (* 3600 24 3))))

;;;;TODO: rename this to *auto-trackers* or something
(defvar *trackers*
    ;; Give the new guy first crack:
    (list #+ignore :mtv ;; for now
	  :btn
	  #+ignore :shazbat ;; hasn't worked in forever???
	  ))

;; for --cleanup, only manually downloaded
(deftracker :720pier
    :re "720pier"
    :char "7"
    :setter (lambda (torrent)
	      ;; 1.5 ratio and 3 days should do it?
	      (setf (torrent-ratio-limit torrent) 1.5)
	      (setf (torrent-seed-min-time torrent) (* 3600 24 3))))

;; for --cleanup, only manually downloaded
(deftracker :mma-tracker
    :re "mma-tracker"
    :char "M"
    :setter (lambda (torrent)
	      ;; Hard to seed stuff here, so seed longer.
	      (setf (torrent-seed-min-time torrent) (* 3600 24 4))
	      (setf (torrent-ratio-limit torrent) 1.5)))

(deftracker :ptn
    :re "piratethenet"
    :char "x"
    :setter (lambda (torrent)
	      ;; Hard to seed stuff here, so seed longer.
	      (setf (torrent-seed-min-time torrent) (* 3600 24 4))
	      (setf (torrent-ratio-limit torrent) 1.1)))

;; Public trackers, just in case
(deftracker :public-misc
    :re "tracker\\.ccc\\.de|openbittorrent\\.com|thepiratebay"
    :char "p"
    :setter (lambda (torrent)
	      ;; Public tracker, so just seed to 1:1
	      (setf (torrent-ratio-limit torrent) 1.0)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Quality settings

(defquality :high-1080p
    :priority 10
    :codec *allowed-codecs* 
    :resolution :1080p)

(defquality :high-1080i
    :priority 1
    :codec *allowed-codecs* 
    :resolution :1080i)

(defquality :normal
    ;; The priority of a quality allows selection of episodes when more
    ;; than one quality is available at the same time, as is often the
    ;; case.  Higher numerical priority is given precedence.
    :priority 50
    :source :hdtv
    :codec *allowed-codecs* 
    :resolution :sd)

(defquality :high
    :priority 40
    :codec *allowed-codecs* 
    :resolution :720p)

(defquality :low
    :priority 30
    :source :hdtv
    :codec :xvid
    :resolution :sd)

(defquality :btn
    :priority 90
    :codec *allowed-codecs* 
    :resolution :720p)

;; The additional delay waiting to download a high quality ep while
;; waiting for a normal quality one to become available
(defvar *download-hq-delay* 0)

;; The additional delay waiting to download a low quality ep while
;; waiting for a normal or high quality one to become available
(defvar *download-lq-delay* #-debug 24 #+debug 24)

;; While I'm tuning the new delays, let's be verbose:
;;(pushnew :tget-config-debug *features*)

(defun my-quality (episode &aux (tracker (episode-tracker episode))
				temp)
  (flet
      (#+tget-config-debug
       (debug-msg (temp)
	 (when temp
	   ;; we would have downloaded this if enough time had passed, so
	   ;; let's say that
	   (format t "~
DEBUG: (tracker delay + quality delay) - hours avail = ~d hours for:
       ~a~%"
		   (- temp (hours-available episode)) episode))))
    ;; My defined quality, as a function.  This allows me to download
    ;; different qualities based on different criteria.
    ;;
    ;; We also output progress reports for episodes not downloaded,
    ;; so we can monitor progress, for debugging purposes.
    (when (and (null
		;; See if there is an episode with :normal quality.  The
		;; :transient keyword is important, since it restricts the
		;; search to episodes we have *not* downloaded yet.
		(query-episode :episode episode :quality :normal :transient t))
	       (eq :high (episode-quality episode))
	       (and tracker
		    (numberp (tracker-download-delay tracker))
		    (if* (>= (hours-available episode)
			     (setq temp
			       (+ (tracker-download-delay tracker)
				  *download-hq-delay*)))
		       thenret
		       else (@log ">>>waiting for ~a more hours for this HQ ep"
				  (- temp (hours-available episode)))
			    nil)))
      ;; :normal quality is not available and the :high quality episode
      ;; has been available for a set amount of hours, then take this
      ;; one
      (return-from my-quality :high))
  
    #+tget-config-debug (debug-msg temp)
  
    (when (=~ "broadcasthe.net" (episode-torrent-url episode))
      (return-from my-quality :btn))

    (setq temp nil)
    (when (and (null
		;; See if there is an episode with :normal or :high
		;; quality.
		(or
		 (query-episode :episode episode :quality :normal :transient t)
		 (query-episode :episode episode :quality :high :transient t)))
	       (eq :low (episode-quality episode))
	       (and tracker
		    (>= (hours-available episode)
			(setq temp
			  (+ (tracker-download-delay tracker)
			     *download-lq-delay*)))))
      (return-from my-quality :low))
    
    #+tget-config-debug (debug-msg temp)
  
    :normal))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Groups

(defgroup :kevin
    :trackers '#.*trackers*
    :quality 'my-quality
    :download-path (merge-pathnames "kevin/" *download-root*))

(defgroup :temp
    :trackers '#.*trackers*
    :quality 'my-quality
    :download-path (merge-pathnames "tmp/" *download-root*))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; TV shows

;; Use ... :catch-up "S01E02" ... to start a series after the 1st ep
;; Use ... :remove t ... to delete a series
;; Use ... :subdir "dirname" ... to put the episodes into a subdirectory of
;;          the group download path -- this is a hack to make Plex Media
;;          Server see episodes of The Daily Show and The Colbert Report.

(defseries "8 Out of 10 Cats" :kevin :archive #.*for-mom*
	   :subdir "8.out.of.10.cats")
(defseries "8 Out of 10 Cats Does Countdown" :kevin :archive #.*for-mom*
	   :subdir "8.out.of.10.cats.does.countdown")
(defseries "American Experience" :kevin :subdir "American.Experience")
(defseries "Atlanta" :kevin :subdir "Atlanta")
(defseries "Barry" :kevin :subdir "Barry" :private t)
(defseries "Better Call Saul" :kevin :archive #.*tv-archive*)

(defseries "Corporate" :kevin)

(defseries "Elementary" :kevin :subdir "Elementary")
(defseries "Fargo" :kevin :quality :high :archive #.*tv-archive*)
(defseries "Frontline (US)" :kevin)
(defseries "Future Man" :kevin :catch-up "S01" :archive #.*tv-archive*)
(defseries "Good Talk With Anthony Jeselnik" :kevin :subdir "Good.Talk")

(defseries "Jeff Ross Presents Roast Battle" :kevin)
(defseries "Luther" :kevin)
(defseries "Man Seeking Woman" :kevin :catch-up "S01")
(defseries "Not Safe with Nikki Glaser" :kevin)
(defseries "Naked and Afraid" :kevin :catch-up "S01"
	   :subdir "Naked.and.Afraid")
(defseries "Naked and Afraid XL" :kevin :catch-up "S02"
	   :subdir "Naked.and.Afraid.XL")
(defseries "Nathan for You" :kevin)
(defseries "Nature" :kevin :subdir "Nature")
(defseries "Nova" :kevin :subdir "Nova")
(defseries "Penn & Teller: Fool Us" :kevin :quality :high
	   :catch-up "S02E01"
	   :aliases ("Penn and Teller Fool Us"))
(defseries "Rick and Morty" :kevin :archive #.*tv-archive*)
(defseries "Sherlock" :kevin)
(defseries "Star Trek: Discovery" :kevin :archive #.*tv-archive*
	   :aliases ("Star Trek Discovery"))
(defseries "Star Trek: Picard" :kevin :archive #.*tv-archive*)
(defseries "Taskmaster" :kevin :catch-up "S03"
	   :aliases ("Taskmaster UK" "Taskmaster (UK)")
	   :subdir "Taskmaster")
(defseries "The Graham Norton Show" :kevin)
(defseries "The Righteous Gemstones" :kevin :subdir "The.Righteous.Gemstones"
	   :private t)
(defseries "The Terror" :kevin :catch-up "S01" :archive #.*tv-archive*)
(defseries "Tosh.0" :kevin)

(defseries "Would I Lie To You?" :kevin :catch-up "S08E01"
	   :aliases ("Would I Lie To You")
	   :archive #.*for-mom*)
(defseries "Hannibal" :kevin :delay 0 :quality :high)
(defseries "Killing Eve" :kevin :quality :high
	   :subdir "Killing.Eve")
(defseries "The Mandalorian" :kevin :quality :high :archive #.*tv-archive*
	   :subdir "The.Mandalorian")
(defseries "Avenue 5" :kevin :subdir "Avenue.5")


(defseries "Dispatches from Elsewhere" :kevin :catch-up "S01E02")

(defseries "The Last Dance" :kevin :catch-up "S01E02")
(defseries "Penny Dreadful City of Angels" :kevin
  :subdir "Penny.Dreadful.City.of.Angels")
(defseries "I Know This Much is True" :kevin)

(defseries "Resident Alien" :kevin)
(defseries "Criminal UK" :kevin)

(defseries "Loki" :kevin :catch-up "S01E01")

(defseries "Kin" :kevin)

(defseries "Squid Game" :kevin :catch-up "S01" :archive #.*tv-archive*)