Killermatch Tech Talk: Native vs web apps, WebSQL vs IndexedDB

May 25th, 2015, by Marius Kjeldahl

The first blog post in this series, where I explain the choices I made for Killermatch 1 and 2, and hint on future versions can be found here.

Killermatch is a match diary for racket sports like tennis, squash etc. In my previous blog post I wrote about how I first wrote Killermatch as a native Android app, why I chose a hybrid approach (Cordova) for Killermatch 2 and what technologies might power future versions of Killermatch.

When the iPhone was originally launched, the only option was to write apps that ran in the browser. Later Apple gave developers access to the native APIs on the phone. Since then developers have had to choose whether to write a native app or a browser-based one. Various technologies (Cordova/Phonegap and more) provide a third option, where most of the app is written to run inside the browser, but at the same time providing various accesses to the native APIs on the phone. This is typically called a "hybrid" approach, since it provides more or less full access to the phone SDKs (that matter), but uses the browser for the GUI (instead of the native GUI running on the phone). The hybrid options typically also support Android, so the situation is the same there.

Besides the technology, the browser-based option differ from the native and hybrid options in how an app gets distributed. Native and hybrid apps can (and typically will) be distributed through app stores. The browser-based option is distributed as a link (linked to by other web pages or search engines, emails or similar). Both distribution methods have their strengths and weaknesses, but so far if you want to distribute an app to a broad (lazy) consumer audience, you need app store distribution.

One can argue that distribution through app stores is both good (typically for the consumer, easy of use, safety etc) and bad (for app creators, expensive distribution monopoly, curation control etc). When the first real smartphones came out, I believe most of the vendors hoped the new phones would be great for running web-based applications. A fact that Apple itself marketed pretty heavily initially.

Today we know things worked out differently. If you have an app with a serious userbase, you are more or less forced to provide a native app experience. It's just miles better than anything we are able to do with web technologies on mobile phones. In addition to the whole WebSQL versus IndexedDB (which I'll get to shortly), touch events and full-screen layout (and scaling) continues to be challenging on web apps. As always, with proper understanding of the issues, there are workarounds for most or all issues, but it still makes it a lot harder to make good web apps than it needs to be.

Why do web apps still suck on mobile phones? I believe there are at least two reasons. One is because Apple and Google already have a monopoly when it comes to app distribution on their respective platforms. The other is technological; The browser vendors simply haven't managed to provide the technologies needed to compete on the mobile platforms. It probably does not help of course that Apple and Google are also one of the major providers of web technology today through their contributions to webkit. One can only speculate if they would be able to move a lot faster if it was Microsoft who dominated the mobile app stores.

Besides pointing to Apple and Google and giving not-so subtle hints that open web app platforms typically do not serve their interests on mobile, I'll add a third reason; browser vendors moving too slow and lack of access to platforms. That includes other browser vendors like Firefox and Microsoft. Yeah, Microsoft is still a major player in the tech area, although there are more than one sign that their significance in at least the consumer space is diminishing (sucky browsers, clueless hardware vendors, late to the mobile party etc). One can speculate that Microsoft is in a similar situation as Google and Apple, in that they have a huge native app market that they also benefit from, even if it is no a mobile app market. Firefox is in an unfortunate situation; In addition to their own "stubbornness" in at least the whole WebSQL versus IndexedDB debacle, they have no real access to the mobile platforms. They simply can not get their browser on the mobile platforms without the blessing of Google or Apple. No wonder they need their own mobile OS (FirefoxOS)!

For my own apps, one problem that keeps annoying me to no end is the lack of a decent client-side database storage backend for web apps. For devices that are always connected, or where users do not pay for bandwidth, running client-server like the typical old web application is fine, assuming you're happy with the roundtrip delays to the server. Most mobile users are not, partly due to sucky networks in large parts of the world. So web apps on at least mobile devices need to run locally on the device. Many apps need to store data locally, and very often this data typically replicated/synced with data on a server. Typically when the mobile device has access to free and/or high-bandwidth networks (wifi).

Many years ago, AOL open sourced an excellent embedded client-side database library named SQLite. In fact is was so good that it more or less killed the whole commercial category of such products (RaimaDB, BerkleyDB and many more). Google and Apple quickly made it available users, both in their browsers and their mobile devices. And before they did, because it was so compact, fast and with a useable API (pretty much a decent subset of SQL), people embedded it with their own native apps. Some work was done to standardize the interface to SQLite in the browser, which became knowns as WebSQL. AFAIK, it powered one of the first serious web apps supporting offline mode, namely GMail, and still does today. And it still powers a lot of web apps with serious needs for decent client-side database backend storage.

Of course SQLite had a few different versions as well, and there were various ways of supporting extensions etc. I am not going to pretend I know all the details, but the browser standardization committees were not able to agree on WebSQL. I believe the argument was that one wanted to avoid defining a standard based on an existing implmentation. A noble cause indeed. At this time I believe even Firefox supported WebSQL. But after vendors agreed to go for a new standard named IndexedDB with no implementations, Firefox decided to remove WebSQL support. At this time, Microsoft still believed they ruled the world, and until fairly recently I believe they did not support either "standard".

So this goes back to 2010/2011 I believe. Since then life has been challening for many web app developers. I've personally tried using IndexedDB in my own projects since 2011, and every time I've been forced to give up. Even though at least Chrome and Firefox have had IndexedDB support since 2011, it never really worked. For a long time, neither implementation was complete. I remember trying and failing several times. One time I learned that Chrome did not support deleting a table from the API. Many times I have been able to lock up existing databases by bashing the browser reload button. Also there were differences in implementations that needed to be accounted for, and the standard itself was a moving target. So every year since I see developers telling people they should not use WebSQL but move to the IndexedDB "standard". Based on my own experience, I doubt many of these developers have ever actually tried using either API.

Fast forward to today (2015). Is IndexedDB ready yet? No, it still is not. The biggest bump in the road right now is probably Apple who have shipped a few Mobile Safari releases with IndexedDB support that simply does not work. These days, Mobile Safari is a big deal. A lot bigger deal than Firefox probably. You know what still works great in all versions of Safari? WebSQL. I'm not saying lack of WebSQL support is the reason why Firefox has lost their market share, it's a lot more complex than that. But I can tell you immediately that lack of WebSQL support is THE REASON why Killermatch does not run in Firefox. And from various questions I get and see on Stackoverflow and similar, I can guarantee you that decisions like dropping WebSQL support does contribute positively overall to Firefox's marketshare.

So if neither WebSQL or IndexedDB works everywhere yet, aren't there any alternatives? Sure. For simple stuff, localStorage works great. It's a simple key-value store which certainly is usable for many types of applications. Any libraries that builds on top of WebSQL and IndexedDB offer a standard API that just works? Sure, take your pick! The trouble is most of these provide fairly simple/thin APIs, typically modelled on what I believe is the limited type of API that IndexedDB provides, and not the rich API that WebSQL/SQLite provides. The trouble is that many apps uses fairly rich/complex relational models server side. Replicating these client-side with WebSQL is typically fairly simple and quite doable, because WebSQL uses a relational model (SQL), similar to a lot of the databases server side. So a lot of the united libraries like localForage, treo and more might do a great job of bridging IndexedDB-style interactions, but typically do a very poor job of bridging WebSQL-style interactions (if at all).

I recently watched streams from the WebRebels conference in Oslo, where Dale Harvey from Mozilla (maker of Firefox, also working on PouchDB) presented "The Native Web", a talk where he "highlights the fact that browser still generally fall behind native experiences and look at how offline architectures can help bring the best experiences...". In one part of his presentation he tosses out the same advice I've seen over and over already; "Don't use WebSQL, use IndexedDB". During the Q&A I challenge this (based on no support on Mobile Safari), and he changes his advice to "Use neither. Use PouchDB.".

So what is PouchDB? According to the PouchDB website: "PouchDB is an open-source JavaScript database inspired by Apache CouchDB that is designed to run well within the browser." I've heard it works great for it's use-case. PouchDB is a database that stores JSON "documents" and typically use map-reduce for "indexed search". Without going too much into detail, this is a lot different than the "traditional" relational model (SQL) which WebSQL uses. If your backend is modelled "document-style" (typically using CouchDB/Couchbase, Mongo or similar server-side), PouchDB might be an excellent choice. However if you have a relational (SQL) type backend that you want to replicate (typically partially) client-side, just saying "use PouchDB" might not be the best advice (YMMV). WebSQL might still be a better choice, even if it means leaving Firefox out in the cold.

As other's have written before, IndexedDB is probably better viewed as a lower level database engine, an engine which other database libraries can be written on top of. From this point of view, PouchDB looks like an excellent choice, assuming it fits your use-case. On the other hand, WebSQL continues to be an excellent choice for quite a few other use-cases, assuming you can live without Firefox and Microsoft browser support (which is a lot easier today than five years ago).

In retrospect I'll claim that Firefox probably continues to hurt itself by not supporting WebSQL until IndexedDB offers a real alternative. The whole WebSQL/IndexedDB situation has probably hurt the web in the battle against native. Worst case it will continue to do so also in the future. A lot of developers (like me) are already using WebSQL and it's native sibling SQLite on native platforms. Most are probably willing to migrate to new/better solutions long-term, but short term IndexedDB needs to prove it's worth our time. So far it really has not (quite the opposite!), and many, like me, are pushing users to Chrome, Safari and Opera, rather than Firefox and Microsoft browsers.