Dev Diary 11: Koto July Progress Report

In this development diary, I provide an update on progress made on Koto in the month of July 2021! In Dev Diary 10, I listed the following goals for Koto development over development sessions in July:

Finish the implementation of the artist page in the "no albums" scenario.
Design and implement the user experience for audiobook listing, selection, and playback.
Implement our 10-band equalizer and playback speed controls.

Alongside these items, I wanted to work on the logic for fetching and reporting track progress and duration, since we were dealing with an issue where some files were misreporting their progress and the track duration wasn't always reliable. Not all of the items above were completed for the month of July, however I will not let that overshadow the sheer amount of work that went into Koto during this month, so let us dive right in!

#Album-less Artists

The first item for July was to wrap up work done in June on the album-less artists view. To summarize what this view is, there may be scenarios such as storing royalty free music for content creation, various backups from services, etc. where you may store tracks (files) as a direct descendant of a folder in your Music directory. Prior to various work done in June on the indexer and this artist page, Koto would not index this content or display it. This all changed in June, so my focus for the start of July was wrapping up this work, mostly around styling. This work was completed and I am fairly happy with the current implementation. There is still some broader work I want to do on metadata fetching for tracks to not solely rely on directory structure, and this is work I intend to do before the first alpha. So you can expect more improvements to this album-less artists view in the future!

#Audiobook User Experience

Work throughout most of the rest of July was done on the audiobook user experience. On the surface, this probably sounds pretty simple. You have a folder similar to Music, with the writers (artists), then the audiobook, then the tracks themselves. Index them and just throw them into a special view, right? Yeaaaaa, no.

When designing the main library for Audiobooks, the design decision was made to not just present you with a grid (instead of a list) of the writers, but also a banner of various genres in which the audiobooks within each writer reside. This banner (alongside a smaller "strip" that would show up if you needed to scroll, both are in the mockup for obvious reasons) would have buttons for each genre, with a fancy background image and separate text (so it can be localized in the future). But you see, there is no actual metadata for a folder to indicate what "genre" it belongs to. It is just a folder. It is the tracks inside the folder that have the possibility of containing ID3 information about its genre, composer, etc. So in order to determine the genres an audiobook belongs to, we need to get the genres of each track independently, then collate (combine) all of them into a single GList in the KotoAlbum (which is the underlying struct for the audiobook). These genres would get saved as a semi-colon (;) separated string in a new column in the albums table and loaded in during the database loading during application startup. To accomplish this, I first needed to propagate a new GList called genres in our KotoTrack struct. These aren't stored persistently in the database for the track, just used to ease hand-off of genre information when adding the track to an album. This propagation is being done through via koto_track_update_metadata, which will get the genres via the taglib_tag_genre function and hand off that semi-colon separated list to a new function koto_track_set_genres. This function will split up that string into a pointer array of gchars (strings, basically), lowercase and hyphenate them (replacing white-space with hyphens), and use a new koto_track_helpers_get_corrected_genre function that will replace some genre names like alternative/indie with just indie. This reduces duplication of genres that are, in essence, the same thing. Once this GList is propagated, it is used in koto_album_add_track, where we will iterate over the list, check if our existing KotoAlbum list has this genre, and add it if needed. So all of this was done just for the banner (initially). It doesn't describe the 3 or so hours spent messing around with multiple Gtk widgets and their respective functionality just to try to get images which aren't 1:1 ratio to scale (eventually I gave up and scaled down the vectors manually and exported them as PNG, sigh). Just all the indexing and struct logic.

In the mockup shown above, you can see the view once you click on a writer. This is what I would consider an "optimistic" end goal for the audiobook view. Unfortunately, there are no open APIs for easily getting audiobook or book information based on the name of a book (and then getting its ISBN-10 or ISBN-13 code). In order to get the description, I will need to create an open source server backend that Koto will reach out to which will need to use multiple APIs such as Google Books, Open Library API, and even some possible scraping of public book data from web pages. This will need to be cached or permanently stored in the backend, with probably a shared dataset hosted somewhere for folks that want to self-host. This will get us some coverage of description information for books. If we lack information on it, we (as in the broader community) could work on collating book information to create a more comprehensive and free API for other open source developers. I want to avoid doing this client side since we would need to possibly implement OAuth support and the client API bits, require users to jump through hoops of getting API keys, and more. This is not something I will be working on immediately, likely not a priority until sometime after the first stable release of Koto. By-and-large, if you own an audiobook, chances are you at least vaguely know what it is about. But there is at least a place for it in the design. Going back to the design, you can see that we have some badges for displaying the year as well as the list of genres associated with an audiobook. They are not user-interactable at the moment, but chances are they will be incorporated into the search once I get around to working on that. Like with genres, these "folders" have no real "year" metadata (not talking about access, creation, and modification times on the folder inodes themselves). We will pull this information from the KotoTrack during koto_album_add_track and it is temporarily stored in the KotoTrack struct when we are getting the ID3 metadata (if any) from the track itself. This uses the taglib_tag_year function and sets it via koto_track_set_year. The year, like genres, is stored in the database for a given album, and presented to the user when the year is known. It is not just used for the fancy info for audiobooks either. In fact, we have a dedicated KotoAlbumInfo widget that is used both in the AudiobookView widget and in the AlbumView for your local music! All of the work done on the audiobooks for genres, years, etc. is in the KotoAlbum, which means the view of your local music gets the benefit too.

Now that we have the year information for each album, what other ways can we take advantage of this additional metadata? This is something I thought about when working on the WritersPage view that displays each AudiobookView. You may want to sort your audiobooks by chronological order, with the latest releases being first and oldest last. Or you may want to ignore chronological order entirely and use strictly alphabetical order. This is pretty much a zero cost preference is my book (pun intended), as much as Tobias from GNOME wants to claim they are. I had to implement alphabetical sorting anyways otherwise there would be literally no consistent order to the audiobooks, so why not have options for either year or alphabetical. It is not "fixing the underlying problem" (because there no problem to begin with), just providing you more options in making Koto feel like home. This preference will be extended to music as well, so you can have consistent sorting there. To facilitate this, I rearchitected our album reference storing in the KotoArtist from being a GList to a GQueue and a GListStore. Without getting into the weeds, these data structures basically just allow us to perform sorting in an easier manner. We use these same data structures in our Playlist as well to provide you the ability to sort by track position in a playlist, name, artist, etc. The list store gets sorted based on the user's preference (or default, which is chronological followed by alphabetical when no year data exists) and the view (such as a GtkFlowbox or GtkListView) automatically update based on the model changes. Okay so, we have an audiobook being displayed in this new view. Great. But what about actually playing it? I hear you. Poor pun intended. As many of you would know if you followed work on Koto, in our local music library you are able to click on the album art for an album to play it. This would dynamically generate a "ephemeral" (temporary) KotoPlaylist so you can listen to the album, go forwards / backwards, enable shuffle / repeat, etc. like you normally would. However in order to ensure we are not filling up our playlist table with needless playlists of each KotoAlbum (whether that be for an audiobook, music, or podcast) just to make sure we could continue playback of an audiobook or a podcast, we needed to know the type of the library associated with a KotoAlbum, and ensure this sort of type checking is being performed in our KotoPlaylist associated with a KotoAlbum to avoid committing it to the database unnecessarily. Fortunately for us, we already had a function for the KotoArtist that did exactly this, koto_artist_get_lib_type. So instead of create a mountain out of a molehill, our koto_album_get_lib_type function will just return the KotoLibraryType for the KotoArtist associated with a KotoAlbum. In our koto_playlist_commit function, we will now check if we should commit (store) the playlist in our database through a couple checks:

If the playlist is not ephemeral, we can go ahead and save it.
If the playlist is ephemeral, then the KotoAlbum associated with a KotoPlaylist (if any) must be in a library of type KOTO_LIBRARY_TYPE_AUDIOBOOK or KOTO_LIBRARY_TYPE_PODCAST.

If either of those conditions pass, we commit it to the database. The second of those checks actually was slightly more complex. Prior to this work, we did not have the capability to associate a KotoAlbum with a KotoPlaylist. However to avoid duplication of album metadata and ensure a cleaner reference to both via the AudiobookView, I opted to enable KotoPlaylist to have a reference to the UUID of an Album. This prevents a cyclical reference between KotoAlbum and KotoPlaylist, as well as ensuring we do not explicitly require it (which would pose a problem for custom playlists). So this functionality is now implemented through a myriad of GObject properties, getters and setters. All of this work was done to facilitate the audiobook (and podcast) listening experience. We want to ensure that audiobooks and podcasts can have playlists associated with them and that we can store not just the current chapter, book, or episode in the album + playlist but the current position in the current track as well. This means we can pick up an audiobook or podcast where we left off. These KotoPlaylist have the same UUID as the KotoAlbum itself, so loading any playlist data from the database during startup can be kept as simple as possible. These KotoPlaylists are no longer being dynamically generated when you click the "Play" button for albums in your local music library, nor when you click to start playback of an audiobook. Instead, they are created when we initialize the KotoAlbum struct itself and whenever we add a track to the album (whether during first index or database loading), we are now actually adding those to the KotoPlaylist associated with the KotoAlbum, eliminating duplicate track listings. To eliminate some weirdness we had when dynamically generating a KotoPlaylist where we would actually reverse the track order before adding them to a playlist, we have a new sorting type for a playlist that sorts by track position, so the first track is the first one in the album, second is second, and so on. This may all sound like a lot and well, it is. However despite all of this functionality getting added (which was going to be done at some point anyways), it is actually resulting in a cleaner and simpler codebase. Finally, to wrap up this grand adventure we have been on talking about all the inner workings of the KotoPlaylist and KotoAlbum functionality, we will take a short trip into the playback engine itself. This is where we handle all the logic for gstreamer, playing a track, reporting track duration and progress, etc. To facilitate the communication of where we are in a given playlist / track during playback of an audiobook or podcast, we implemented two new functions:

koto_playback_engine_update_track_position - This function gets called whenever our timer triggers koto_playback_engine_tick_track (about every 100ms) and will call out to our second function, which we pass the current progress of the track (say we are 4 minutes and 20 seconds into a 69 minute podcast).
koto_playback_engine_set_track_playback_position - This function that will check if we have a current playing track, if repeat is not enabled (since why bother storing the state then?), and if the conditions are right then call out to koto_track_set_playback_position

Whenever we play or pause playback of a track, we will update the KotoTrack's reference to its playback position. If we go backwards or forwards in the KotoPlaylist (and thus the engine), we will reset the track playback position to zero. If we change to a different album or audiobook, our KotoCurrentPlaylist will call out to a new koto_playlist_save_current_playback_state function for the current KotoPlaylist before switching to the desired one. If you were listening to an audiobook or a podcast then closed the client, before closure we will save the state too. So even if you accidentally close the app, you do not have to worry about losing your playback position! I will also be working on a system to routinely flush the state to the database (say every minute or so) so if we end up in a scenario where Koto crashes, as least you will not be that far behind in your listening experience (and hopefully the crashes can be resolved, of course). At the moment, the playlist state is not communicated in the audiobook view, because the view is not done yet. This is something that will be addressed in August. So that wraps up all of that work. However if you can believe it, that was not the only work that was done on Koto. Here is what the writer's page implementation looks like at the moment.

#The Nice Other Things

In the quieter moments of Koto development this month, typically off stream, I worked on various logic fixes and design refinements. Here is just the firehose of changes:

Fixed inconsistent updating of KotoButton iconography that would result in our primary navigation not having correct icons associated with various sections.
Refined the spacing of the primary navigation in Koto to feel less compact while also not wasting a bunch of space.
Fixed the bottom border on the headerbar not being properly styled on our light theme and enforced our text coloring on the headerbar window controls to ensure styling remains consistent regardless of the default stylesheet.
Implemented a change to our KotoExpanders in the primary navigation so you can click the whole thing instead of just the up / down arrow to expand them, increasing the overall clickable area.
Renamed "Local Library" buttons to just "Library" to be consistent with other references to it.
Implemented some fancy button hover styling to use in our primary Koto navigation.
Fixed progress and duration reporting via GStreamer! We will also get duration from our ID3 information for a track and prefer that over what is reported by GStreamer.
Changed the transition type for our page stacks to make the transitions feel less glitchy. This is actually an upstream GTK4 bug with a specific transition type.
Improved the granularity of the playerbar volume control. It no longer does silliness of incrementing by 10%.
Implemented double click logic in our Track Table to immediately start playback of a track and respective playlist.
Rewrote various legacy KotoCurrentPlaylist logic to longer use GObject Properties but rather Signals.
Fixed some issues with the KotoPlayerbar misreporting some album data. Some more work to be done on this front.

#August Goals

So no, I did not get around to implementing our 10-band equalizer and playback speed controls. I am sure you can understand why. So the goals for August are:

Finish up our first pass on the audiobook user experience. This includes presenting playback state info, clickable genre buttons in the library view for filtering, alongside that "all" button shown the mockup.
Implement our 10-band equalizer and playback speed controls.
Switch our indexer logic from using the readdir Linux syscall to using GLib / GIO functionality, fix weird threading issues by introducing mutex locks on our Cartographer HashTable. This should improve the reliability of our indexing during initial startup and help us down the road.

As much as I would like to get more work done on this front, in the last week of August and first week of September I will on vacation in Spain (as well as for an in-law's wedding). So I think the above mentioned items are pretty reasonable!