Modify directory.watch.added and add directory.watch.removed command

[chros73]

One of the advantages of using "inotify" to load and remove downloads by torrent meta files is the reduced file system IO (apart from it's instant).

Refers to: #374

A. Modify directory.watch.added
Allow it to accept multiple commands (as load.* commands).
There are multiple ways to achieve this but since there's no "shortcut" (the directory (that is watched) for inotify has to be separated), the final command (that is called on an "add" event) is reconstructed as a line of config.

• 1st param: the directory that will be watched
• 2nd param: the name of the main command that will be called if an "add" event is triggered (load.* commands)
• rest of the params: extra commands that will be passed as arguments to the above specified main command
  • if an extra command include commas (, parameter separator) then it needs to be included inside quotes (")

directory.watch.added = (cat,(cfg.dir.meta_downl),unsafe/),   load.start,  "d.attribs.set=unsafe,,1", print=loadedunsafe

B. Add directory.watch.removed
Similarly to loading event, add ability to handle removing events of meta files as *_untied commands:

• *_untied commands can't deal with a download individually: they go through and check all the downloads (they can generate lot of IO, especially if they are used with schedule command)
• directory.watch.removed handles this individually

Since it doesn't need any additional commands, it accepts multiple directories as parameters.

• 1st param: the command to call when a meta file is removed, only 3 commands are supported
  • d.stop, d.close, d.erase
• rest of the params: comma separated list of the directories that will be watched

directory.watch.removed = d.erase, (cat,(cfg.dir.meta_compl),various/), (cat,(cfg.dir.meta_compl),unsafe/)

C. Rename *tied commands
Rename untied commands and add redirects for the new names to preserve compatibility:

• start_tied -> tied.start
• stop_untied -> untied.stop
• close_untied -> untied.close
• remove_untied -> untied.remove

D. Limitations

• a given directory can only be specified once with either directory.watch.added or directory.watch.removed
• torrent meta files that are "added" / "removed" into/from watched directories will be discarded if rtorrent wasn't run at the time of adding / removing
  • it's a good idea to run the normal scheduled "watch" command once after rtorrent is started
• probably it won't work well with network file systems (NFS, Samba, etc.)

E. directory.watch.* commands in action
Let's assume the following:

• we have multiple watch directories with which we want to use different properties for downloads with a parameterised command (like this d.attribs.set)
• we want to provide the ability of switching inotify support for users in config
• we also want to watch directories for removed metafiles
  • the meta files is moved by rtorrent (internally) when a download is finished

# Defining directory constants
method.insert = cfg.dir.main,       string|const|private, (cat,"/mnt/Torrents/")
method.insert = cfg.dir.sub,        string|const|private, (cat,(cfg.dir.main),".rtorrent/")
method.insert = cfg.dir.meta_downl, string|const|private, (cat,(cfg.dir.sub),".downloading/")
method.insert = cfg.dir.meta_compl, string|const|private, (cat,(cfg.dir.sub),".completed/")

# Whether to use inotify for loading/removing torrent (meta) files from watch directories
method.insert = cfg.inotify.use, value|const|private, 1

# Whatch dir definitions for loading metafiles using inotify
branch = ((cfg.inotify.use)), ((directory.watch.added, (cat,(cfg.dir.meta_downl),rotating/), load.start,  "d.attribs.set=rotating,1,1"))
branch = ((cfg.inotify.use)), ((directory.watch.added, (cat,(cfg.dir.meta_downl),fullseed/), load.start,  "d.attribs.set=fullseed,1,"))
branch = ((cfg.inotify.use)), ((directory.watch.added, (cat,(cfg.dir.meta_downl),unsafe/),   load.start,  "d.attribs.set=unsafe,,1"))
branch = ((cfg.inotify.use)), ((directory.watch.added, (cat,(cfg.dir.meta_downl),various/),  load.start,  "d.attribs.set=various,,"))
branch = ((cfg.inotify.use)), ((directory.watch.added, (cat,(cfg.dir.meta_downl),load/),     load.normal, "d.attribs.set=unsafe,,1"))

# Fire up only once (after rtorrent is started) the normal scheluded load watch dirs as well even if inotify is used: load added metafiles when rtorrent hasn't been run
branch = ((cfg.inotify.use)), ((schedule2, watch_dir_1,  85, 0, ((load.start, (cat,(cfg.dir.meta_downl),rotating/*.torrent), "d.attribs.set=rotating,1,1"))))
branch = ((cfg.inotify.use)), ((schedule2, watch_dir_2,  86, 0, ((load.start, (cat,(cfg.dir.meta_downl),fullseed/*.torrent), "d.attribs.set=fullseed,1,"))))
branch = ((cfg.inotify.use)), ((schedule2, watch_dir_3,  87, 0, ((load.start, (cat,(cfg.dir.meta_downl),unsafe/*.torrent),   "d.attribs.set=unsafe,,1"))))
branch = ((cfg.inotify.use)), ((schedule2, watch_dir_4,  88, 0, ((load.start, (cat,(cfg.dir.meta_downl),various/*.torrent),  "d.attribs.set=various,,"))))
branch = ((cfg.inotify.use)), ((schedule2, watch_dir_5,  89, 0, ((load.normal,(cat,(cfg.dir.meta_downl),load/*.torrent),     "d.attribs.set=unsafe,,1"))))

# Whatch dir definitions for removing metafiles using inotify
branch = ((cfg.inotify.use)), ((directory.watch.removed, d.erase, (cat,(cfg.dir.meta_compl),various/), (cat,(cfg.dir.meta_compl),unsafe/), (cat,(cfg.dir.meta_compl),rotating/), (cat,(cfg.dir.meta_compl),fullseed/), (cat,(cfg.dir.meta_compl),load/)))

# Whatch dir definitions for loading metafiles NOT using inotify
branch = ((not, (cfg.inotify.use))), ((schedule2, watch_dir_1,  5, 10, ((load.start, (cat,(cfg.dir.meta_downl),rotating/*.torrent), "d.attribs.set=rotating,1,1"))))
branch = ((not, (cfg.inotify.use))), ((schedule2, watch_dir_2,  6, 10, ((load.start, (cat,(cfg.dir.meta_downl),fullseed/*.torrent), "d.attribs.set=fullseed,1,"))))
branch = ((not, (cfg.inotify.use))), ((schedule2, watch_dir_3,  7, 10, ((load.start, (cat,(cfg.dir.meta_downl),unsafe/*.torrent),   "d.attribs.set=unsafe,,1"))))
branch = ((not, (cfg.inotify.use))), ((schedule2, watch_dir_4,  8, 10, ((load.start, (cat,(cfg.dir.meta_downl),various/*.torrent),  "d.attribs.set=various,,"))))
branch = ((not, (cfg.inotify.use))), ((schedule2, watch_dir_5,  9, 10, ((load.normal,(cat,(cfg.dir.meta_downl),load/*.torrent),     "d.attribs.set=unsafe,,1"))))

# Removes torrents from client when its metafile (torrent file) has been deleted manually or by a script. If inotify isn't used: Run it every 5 seconds, otherwise run only once after rtorrent is started.
branch = ((cfg.inotify.use)), ((schedule2, untied_torrents, 99, 0, ((untied.remove)))), ((schedule2, untied_torrents, 8, 10, ((untied.remove))))

[pyroscope]

The limitation of "removed" is arbitrary, and it is not very flexible, e.g. does not take care of re-ties.

The better / more consistent way would be to fire an event.download.untied, and then that multi-method can take care of things – it would be called on the item(s) tied to the removed file.

And then the next logical step is call it directory.watch.tied_to_files which only takes a directory list, and also triggers event.download.tied.

And the special path '*' would mean "all existing directory.watch.added dirs".

Finally, let the old super-special tied/untied commands as-is, deprecate them wholesale, and add d.multicall.check.tied_to_files=[‹viewname›] for non-inotify schedules, which iterates over all items (as those existing commands do), but "just" fires these events. That needs state though, the last seen mtime of the tied file (or 0 for not found) has to be stored in the session state, so that the triggers can be generated (once) on changes of that state. The d.tied_to_file.set and d.delete_tied commands would reset that state accordingly.

[pyroscope]

And while looking at that d.multicall.check.tied_to_files name, d.multicall.check.base_path would be nice too – fire event.download.orphaned when an item's data is gone. Obvious handlers would be d.erase, or d.close and maybe also d.views.push_back_unique=orphaned. Yes, events are that flexible.

[pyroscope]

See #374 (comment) about why you do not need more than the handler name as an argument.

[chros73]

A. directory.watch.removed : the idea was to provide similar method as directory.watch.added.

The limitation of "removed" is arbitrary, and it is not very flexible

I agree, although it's working fine:

• a given directory can only be specified once with either directory.watch.added or directory.watch.removed

About the events: it's an interesting idea, but somebody else has to implement it :) .
Right now similar things can be achieved with the allowed commands and their corresponding events (erased, stopped, etc.) and a bit of scripting (if it's necessary).

[chros73]

B. directory.watch.added

See #374 (comment) about why you do not need more than the handler name

Thanks, it didn't work for me for whatever reason when I tried (otherwise probably I wouldn't have modified it).
Although I still like the mod (since it's done already):

• your example still work with it
• it still makes sense for a normal user to use it this way (they don't have to take an exam of your valuable rTorrent Handbook :D )

Anyway, @rakshasa will decide it.