Folder Browsing

[JonnieCache]

This PR adds folder browsing functionality to the web interface.

The way this works is it displays a list of subdirectories of the current dir, and a list of releases in that dir. It doesn't ever display individual tracks per-directory.

When a directory contains a single release (the common case for leaf dirs) the link for that row is set to target the release directly. Additionally, as the directory table in the database contains data used by the album art and other subsystems, not just audio files, the queries are set up to ensure that directories in the list must contain at least one release. This prevents the inclusion of covers directories and so on.

As you might imagine this requires somewhat acrobatic SQL. We have two codepaths, filtered and unfiltered. In the unfiltered case, we have queries like this:

SELECT 
  d."id" as col0, 
  d."version" as col1, 
  d."absolute_path" as col2, 
  d."name" as col3, 
  d."parent_directory_id" as col4, 
  d."media_library_id" as col5, 
  COUNT(DISTINCT t.release_id) as col6, 
  CASE WHEN COUNT(DISTINCT t.release_id) = 1 
  AND NOT EXISTS (
    SELECT 1 FROM directory d_c 
      INNER JOIN track t2 ON t2.directory_id = d_c.id 
    WHERE 
      d_c.parent_directory_id = d.id
  ) THEN MIN(t.release_id) ELSE NULL END as col7 
FROM directory d 
  LEFT JOIN track t ON t.directory_id = d.id 
where (d.parent_directory_id = ?) 
group by d.id 
having 
  (
    COUNT(DISTINCT t.release_id) > 0 
    OR EXISTS (
      SELECT 1 FROM directory d_child 
      WHERE 
        d_child.parent_directory_id = d.id
    )
  ) 
order by 
  d.name COLLATE NOCASE

This might look a little ugly, with the subqueries and the HAVING and so on but I tried a lot of variations and this is the fastest I came up with, it needs to be set up like this to stop it self-joining to the whole table. With this design it lists the root of my ~60k track library in <30ms. A folder with a very large number of albums takes ~200ms.

Here's what it looks like when a filter is applied:

WITH RECURSIVE ancestor_walk(directory_id, release_id) AS (
  SELECT 
    t.directory_id as col0, 
    t.release_id as col1 
  FROM track t 
  where 
    (t.codec = ?) 
  group by 
    t.directory_id, 
    t.release_id 
  UNION ALL 
  SELECT 
    d.parent_directory_id, 
    a_w.release_id 
  FROM ancestor_walk a_w 
    INNER JOIN directory d ON d.id = a_w.directory_id 
  WHERE d.parent_directory_id IS NOT NULL
) 
SELECT 
  d."id" as col0, 
  d."version" as col1, 
  d."absolute_path" as col2, 
  d."name" as col3, 
  d."parent_directory_id" as col4, 
  d."media_library_id" as col5, 
  COUNT(DISTINCT a_w.release_id) as col6, 
  CASE WHEN COUNT(DISTINCT a_w.release_id) = 1 THEN MIN(a_w.release_id) ELSE NULL END as col7 
FROM ancestor_walk a_w 
  INNER JOIN directory d ON d.id = a_w.directory_id 
where (d.parent_directory_id = ?) 
group by d.id 
order by d.name COLLATE NOCASE

The situation here is that we still need to be counting the releases in each result, to get the direct-linking-of-single-release-dirs behavior, but we need to apply the filtering to the whole subtree, to find matching tracks deep below the current browsing level. (We get the no-empty-leaves behavior for free from the filter matching.)

The way I've handled it is that we apply the filters once, in the base case of the CTE, and then we reverse-walk up the directory tree to the root for each match. This ensures that in the typical case of a narrow filter, we only perform the expensive recursion on the actually-matching resultset. The worst case is then a very permissive filter, which means a lot of walks. In practice, when filtering my library for FLACs, which matches the majority of the files, I still couldn't get the pageload to take longer than a couple of hundred ms.

The other UX thing I wanted was that in the case of a single music library, the Folders link in the header should go directly to that library, rather than showing a list of one item. This requires an extra query when building the header, but that only happens when starting a new session, and its cheap anyway.

Stuff that I didn't add includes linking from the Release page back to the folder view, I might try and add that later. Also I didn't implement pagination, instead just trying to make it fast.

[epoupon]

Hello,
Thanks for this work.

A couple of questions/remarks:

People who browse by folders usually don't rely on tags. Does it make sense to apply global filters in this view? If someone wants to browse by tag, properly tagging tracks with all the relevant information and using the existing views would be the way to go. This would eliminate a lot of complexity since there would be no need to filter out empty or non-matching folders. Furthermore, speed is really a concern: this view should be very fast, even on limited hardware like a Raspberry Pi.

I planned to add more entries to the top-level navbar, such as an entry to list podcasts and their episodes and a search widget to restore the multi-search feature, but I am not pleased with that since:

• the global filters don't apply to podcasts (that likely should be the same for folders in my opinion)
• the bar already has too many elements. It feels like LMS should maybe switch to a left panel. But this is another story :)

Can you rebase on develop?

[JonnieCache]

I've removed the filtered path from the folders view, as well as some of the ropier stuff around the URL routing, to make it easier to review.

Note that we still have the complexity around hiding empty dirs - this is only an issue in the unfiltered path, as no filter can ever match an empty dir. The same mechanism also gives us the direct-linking of album directories for free. I don't think this should be a problem for sqlite, even on a less-powerful machine. If necessary we could add a "hide empty folders" config option.

I have a raspberry pi I can test on, as well as a synology NAS that I actually use LMS on. What's your threshold for a fast pageload, and for a large library?

I rebased to develop also.