June 25, 2016

New software release: SubWay – Batch video channel downloader



Having talked a lot about software development in general in this blog, I’ve now finally released a piece of software which may be genuinely useful for an actual end-user. I hereby present SubWay, an automated YouTube channel batch video downloader, as freeware.

Note: In this blog post, I’ll talk about the background of developing the SubWay software from a software engineering point of view. If you’re just interested in downloading and using SubWay, please head over to its download page. It’s free :-)

About

YouTube has become a great place to watch high-quality video content online. There are a few apps which allow downloading YouTube videos, which is great as well. However, I always felt there was one piece missing: A software which not only lets you download a single or many videos, but which watches for new video releases on the channels you’re interested in, and downloads them automatically. This is exactly what SubWay does: Add channels to your favorites – search for new videos – download them. Again, head over to its download page to see it in action.

So I really wanted to create a software which I personally thought to be genuinely useful. I then decided to release it as freeware such that others may use it as well.

When compared to similar software products, SubWay provides this one unique feature, and it was my goal to realize it in a very user-friendly manner. This includes an easy-to-use GUI, an alternative command line client to support scheduled automation, and config files stored in a human-readable JSON format.

Implementation

SubWay actually doesn’t realize any video download functionality itself – it doesn’t even talk to the internet in any way. Instead, it relies on the youtube-dl command line tool which does the talking to online servers. All SubWay does really is invoking youtube-dl with the appropriate command line parameters, managing files, doing the appropriate calculations, and, of course, displaying the GUI.

It was in fact after I discovered the existence of the youtube-dl tool that I decided to create a download manager on top of it – I’ve never been interested in developing the complex YouTube API consumption myself, and reusing the youtube-dl functionality seemed like a great way to save me from reinventing the wheel (apart from the obvious legal issues).

Hence, the actual implementation work of SubWay mainly consisted of these parts:
youtube-dl process invocation and parsing of its JSON output
  • Developing the appropriate model entities and match them against youtube-dl output
  • Implement the actual business logic, the heart piece of which is the logic which checks for newly-released videos per channel which haven’t been downloaded yet.
  • Building the GUI
  • Building the command line client

Technology

As a professional Java developer, using a JVM technology stack was a natural choice for me. Given that the underlying youtube-dl program is written in Python, a Python implementation would have been favorable, but I literally don’t know anything about that programming language.

In fact, I implemented about 10% of functionality in Java 8, and 90% in statically-typed Groovy. Implementing the business logic in Groovy tremendously increased ease of development, code readability and maintainability. A while back, I created a blog post about the main reasons to use Groovy in a project.

Of course, this technology choice implied that the resulting fat JAR file would increase by about 400% to > 8.5MB. Considering the application is targeting a PC with enough disk space to store hundreds of video files, this issue seems negligible. As far as performance is concerned, I haven’t observed any impact which could be attributed to Groovy.

The GUI is implemented with JavaFX 8.

JSON serialization / deserialization for youtube-dl output as well as for config files uses the Jackson XML parser. Of late, I’ve written a blog post about the Groovy / Jackson combination.

Command line parsing makes use of the Apache commons cli package.

Finally, I used ProGuard for code shrinking / obfuscation.

Where’s the source code?

So far, I’ve used this blog to discuss topics related to software engineering, and to publish open source software projects which I hope to have some educational value for fellow software developers; and I will continue to use my blog this way in the future.

However, this project is different. This is the first time I created a proper “end-user” software which I think may be considered useful by a broad audience. Also, this is the first “bigger” project I ever realized using JavaFX, and I certainly have yet a lot to learn about this technology. During development, I have really concentrated on creating actual functionality, and less on creating interesting abstractions or using interesting programming techniques. I think thus that educational value of this software currently is low, and I don’t feel like the code quality reaches the usual standards of my open source projects yet.

Hence, I decided to not publish the software open source.

However, I built a few general-purpose library functions in Groovy, and I may publish them open source soon.

Development experience

Major development effort has been demanded by the following tasks:
  • External process handling
  • JSON deserialization
  • JavaFX
  • ProGuard code obfuscation, especially in combination with Groovy / JavaFX

External process handling

Invoking and consuming the youtube-dl.exe process has been quite challenging to implement; most of all because it is naturally very uncomfortable to test as in the end, you really need that actual output from the program running against the YouTube API.

The actual functionality consists of running the external process, consuming its output in real-time or as a whole, triggering callbacks, checking the sterr for errors, allowing the caller to abort the process, and doing all these things in a background thread such that the callbacks can update the UI in real time.

This needs interplay of different standard components of Java and Groovy. I eventually abstracted this to a helper class. I will publish this open source soon.

JSON deserialization

JSON deserialization happens in two places: When consuming youtube-dl’s JSON output (e.g. video information) and when loading the application’s settings files. I really wanted to store the settings in JSON format to stay (expert) user friendly.

I tried out different JSON frameworks before finally discovering that Jackson does its job really well with a minimum amount of configuration required.

JavaFX

I’m quite new to JavaFX 8 and a lot of trouble I’ve had building the GUI clearly can be explained by lack of experience. However, I still think that quite a few aspects of the framework seem… fishy, to say the least. I’ve posted a rant about Lambdas, and another one about Java 8 Promises, and I feel like there’s much room for yet another rant about JavaFX 8.

As JavaFX is here as the successor of Swing (which clearly needs a more modern-looking alternative), I expected the programming model to be much superior in every aspect. However, I found so far that this is not the case:
  • The FXML editor (which by the way is no longer shipped with the JDK) really only assists basic GUI building. The good ol’ Swing editor in NetBeans at least supported proper Java code generation (e.g. action methods).
  • The whole “data binding” story seems to be of very limited use, and its API is as bad as that of lambdas.
  • The components have in general a quite strange API, somehow similar to Swing, but yet different, and especially proper layouting is in no way simpler than in Swing.
  • A good portion of the API just doesn’t seem to make any sense at all. They tried to build an MVC-like programming model, but it is so bad that we still fall back to procedural Swing-style coding.
Maybe I’m just spoiled from clean, well-designed (server-side) Java enterprise frameworks.

I seriously consider creating a minimalistic helper framework for dealing with JavaFX. For now, my GUI code mainly consists of a collection of hacks and workarounds.

ProGuard code obfuscation

After I decided to release this program closed-source, it became apparent that I needed a means of obfuscating the resulting JAR.

Here, the use of Groovy clearly made things more complicated, even in strict statically-compiled mode. I finally decided to really only obfuscate / shrink my own code, rather than trying to shrink the whole fat JAR (including the sources of Groovy and other 3rd party libs), which however turned out to be uncomfortably challenging to realize using Maven as the build tool.

Another setback was to find that Java FX’s FXML files apparently make it impossible to apply code obfuscation as it breaks connection between the FXML and the Java code.

I decided to use ProGuard for the actual obfuscation. It’s free, highly configurable in terms of what to keep and what to obfuscate, and it can be integrated in Maven builds. On the other hand, the actual quality of code obfuscation is more or less set, and doesn’t allow for much configuration.

Conclusion

I’ve now released the very first public beta version of SubWay, v. 0.1-beta. Feel free to head over to its download page, try it out and give me feedback in the comments. I hope to build an updated version soon.

In the meantime, I will continue to discuss all kinds of software development-related topics on this blog. Thank you for your interest.