Behind the Mozilla Apps Developer Preview

December 15, 2011 By Anant in FOSS, Mozilla, Programming, Technology 5 Comments

On Tuesday, we launched a developer preview of the Mozilla Apps project, something I’ve been working on for the better part of the year. We released a suite of tools and documentation aimed at helping developers write, deploy and sell apps built using modern web technologies like HTML5, CSS and JavaScript. Many others have already covered the question of “why” we are doing this: all the major app ecosystems out there are closed, tied to a single vendor, and could certainly use a healthy dose of the openness. There are many great things about Apps, and many great things about the Web, and we want to bring them together. [UPDATE: Tim Berners-Lee agrees!]

In this post, I want to cover the “how”. If you’re interested in writing apps, I would point you to the documentation we have on how to build them. On the other hand, if you’re curious to learn about how the system works as a whole, read on! A lot of different pieces of technology had to come together to get where we are today.

App Manifest

A fundamental building block of the system is the app manifest. Every app in the system is represented by this JSON file, documented here, which essentially contains a set of metadata about your app: name, icons, localized descriptions and so on. This manifest file is hosted at the same domain as your app. An app is uniquely identified by the domain it is hosted at, and this manifest must be served off the very same domain (at any path), with the Content-Type header set to application/x-web-app-manifest+json. We’ve received a lot of feedback stating that this is limiting, but unfortunately almost every security knob in browsers is tuned to the domain of a given page. Because we anticipate that apps will, at some point, be able to request access to elevated privileges (to use the computer’s web camera, for example), we must restrict ourselves to one app per domain. Note that app1.example.org and app2.example.org are different domains but example.org/app1 and example.org/app2 are not.

There are many other specifications that express ideas similar to this concept of a manifest (W3C Widgets, Chrome Web Store, etc.), and we are definitely very keen to standardize the format.

API: mozApps

The next piece we introduce are a set of new DOM APIs, present under the navigator namespace. The API offers a few different functions, but the most important one is navigator.mozApps.install. This function allows a web page to initiate the “installation” process for an app, which is identified by the URL to its manifest (explained previously). Any page is able to invoke this function, so you can self-publish your apps! Just add an “Install” button to your site and call this function with the right arguments. The API also provides a function that can tell if your app is currently installed (amInstalled).

There is another set of APIs under the navigator.mozApps.mgmt namespace. These “management” functions are only available to certain privileged domains, and are able to query the DOM for a list of all apps the user has installed, launch any given one, and uninstall an app. This API is expected to be used only by web pages that the user has explicitly authorized as being able to manage their apps on their behalf. We call such web pages “Dashboards”, and we built a default one into the system (which I shall explain shortly).

The mozApps API is fully documented, but you should note that as with any early DOM API (that we hope to standardize), it is subject to change. In fact, we’re already thinking about how we can further simplify the API and make it more standards-friendly. Join the discussion!

HTML5 App Runtime

Now, what actually happens when somebody calls a function described in the mozApps API? We’d like for users to be able to look for, install and launch apps from any standards-compliant browser without having to do anything special. So, we’ve built a fully in-content implementation of the mozApps API, provided by include.js that is served off the myapps.mozillalabs.com domain (the reason for that will become apparent when we discuss dashboards). Just include that JS file in any page that uses the mozApps API and you should be good to go! This applies to self-published apps as well as stores.

Now, whenever the install method from the mozApps API is invoked, the user is greeted with a dialog asking them to confirm if they’d like to install the app:

The Dashboard

Let’s say the user confirms the installation, what next? After a set of sanity checks against the manifest of the app, it is officially installed into the users collection of apps, which we call a repo. A Dashboard is the piece of software that is responsible for letting the user manage their repo, by allowing them to launch and uninstall their apps. Recall that the mozApps.mgmt set of APIs allow a dashboard to do this, and currently the myapps.mozillalabs.com domain is white-listed. In the future we expect people to write dashboards (which is essentially an app to manage apps!) that users can authorize. When a user visits the default Mozilla Labs dashboard, they look at something like this:

We implemented a touch friendly dashboard that works on both mobile devices and the desktop, to let you re-arrange your app icons and organize them in pages. This part of the dashboard is implemented using the wonderful icongrid library, which you are more than welcome to re-use while writing your own dashboard!

Clicking on an icon will launch that app. What does launching mean? In the HTML5 app runtime, it means it will open up the app in a new browser tab. However, we’ve also been experimenting with how we can improve this experience, which we will discuss next.

App Runtime for Firefox

For Firefox users, we have the opportunity to provide enhancements to the whole app installation and launch process while we wait for the API to get standardized. We’ve written an add-on that implements the mozApps API, which will override the include.js HTML5 runtime version (so stores are encouraged to continue including the include.js version to provide the most portable experience for their users). If you have this add-on installed and install an app from any page or store, you will be greeted with a doorhanger that asks you confirm if you really intend to install this app:

Note that there’s an extra option in there that asks if you want to install the “native” app version or not. On Windows and Mac, this means that we will automatically generate a .EXE or .APP that wraps your web application into a shell that looks and feels like a real app! For example, on the Mac, we will create a menu bar and dock icon for you:

Cool? Sounds pretty familiar to the Prism experiment, right?

In addition to this style of “native” launching, users can also use the dashboard from before as usual. Launching from the dashboard will open it in an app tab, a nifty little Firefox feature.

App Runtime for Android

An important feature of Apps written using web technologies is that they can work on a variety of different devices. We want users to be able to buy an app only once and use it not only on their desktop, but also on their tablets and phones. We’re going to start out with Android (iOS has its own set of tricky technical and policy problems to deal with), by introducing an App Runtime (codename “Soup”).

The App Runtime for Android is a native Android application that lets users install, launch and manage their apps just like on the desktop:

Installing an app on Android using Soup will create an icon in your home screen, tapping on it will launch the app using our embedded web runtime. Well built web applications can now look and feel just like native android apps!

In addition, apps that you installed on the desktop can be automatically synchronized to your phone (and all your other devices) using our Sync functionality, which we will discuss next.

Sync

Users shouldn’t have to install apps on every device they own once they’ve purchased it. We’ve developed an AppSync solution for the HTML5 runtime, Firefox runtime as well as Android. In all three environments, you should be prompted to login with your BrowserID (Mozilla’s new federated & distributed Identity system) when you visit the dashboard:

Once you’ve logged into your dashboard, your apps from all your devices should start automatically synchronizing!

App Marketplace

Search is a great tool for the web, and we expect that many users will discover apps using search engines. However, directories have their place and are an invaluable tool to create a community around apps. Mozilla has been running addons.mozilla.org (AMO) for a while now, an easy place to find, install and review add-ons for Firefox. We want to build a similar store for Apps, and as part of the developer preview, we launched apps-preview.mozilla.org. This preview of our “app store” lets developers submit an app by telling us the link to their manifest (the process is very similar to how you submit an add-on for inclusion on AMO). Once the application is accepted into the store users can find, install, review and provide ratings for apps. This store uses the same mozApps install APIs we discussed earlier, and we expect that others will build their own stores. We want competition in the app store market too!

Supporting developers who want to sell apps is also important to us. We’re testing integration with Paypal as part of the developer preview, which will allow developers to sell apps across the world at various price tiers.

Receipts

How do we balance the need for developers to be able to charge for their apps, while allowing users to use an app they’ve already paid for across all their compatible devices? We’ve devised a receipt format that helps achieve this. When a user completes the payment process on a given store, the store will generate a receipt of this format and pass it to the mozApps API as part of the install data provided to the install call. The implementation responsible for providing the API will stash the receipt along with the app itself (and all devices the app is synced to).

At launch time, the app can ask for the receipt associated with itself using the amInstalled API call, do an integrity check, and send it over the original store that issued the receipt. The store can then verify that the receipt is indeed valid and notify the app, at which point the app can decide whether to let the user run it or not. We’ve provided a utility function verifyReceipt to help the app developer do all of this.

Do note, however, that this whole scheme is merely intended to help developers who don’t want to setup their own payment systems. Developers are free to write apps that use their own (or 3rd party) payment or subscription services. You could, for example, sell your app for free on the AMO store, but ask users to login when the app is launched, or implement your own in-app purchasing system. We will do what we can to help, but in the end, you’re in full control of what your users see when they launch your apps!

What’s next?

This is just the beginning, we have a lot more work to do before we can realize a flourishing and open app ecosystem for the web. Here are just some of thing we have planned for the next few months:

Building out a “Web Runtime”, or WebRT. We’ve built an initial prototype of how such a runtime might work in the add-on for Firefox, and we want this to extend this to a more robust system with auto-updates and deeper OS integration.
Capabilities. In conjunction with the WebAPI project, we want to provide apps with more device APIs and capabilities than regular web pages, while giving the user an easy way to control and hand out permissions. This includes things like camera access, filesystem APIs and more.
Web Activities. A while ago we release a prototype of the apps extension that supported what we call web activities, a way for apps to communicate with each other safely and easily. You could use this, for example, to upload a picture to a site from your favorite photo service, or to share a link from an app to all your friends using your favorite social network. The Firefox Share add-on already relies on web activities to do the latter.
Push sync & notifications. We want users to be able to “push” apps to any of their devices directly from an app store.
Standardization. It is critical for the health of the web for all of these app related APIs to be standardized and supported by all interested parties.

Most importantly, we want you to get involved and help us build!

Show me the code

I’ll end this post with a brief description of all the code behind the various pieces in hopes of attracting contributors All of our code is hosted on Github and licensed under the MPL/GPL/LGPL tri-license.

The main repository for the Apps project can be found here. It contains the source code for the HTML5 App Runtime (include.js and trusted.js are the important pieces), the App Runtime for Firefox and the Dashboard that is currently deployed on myapps.mozillalabs.com. The Dashbaord was built using IconGrid, a JavaScript library to build touch friendly scrollable pages. The App Runtime for Firefox is written using the Add-on SDK and shares a few common files with the HTML5 runtime (repo.js, urlparse.js, manifest.js and sync.js).

Source code for the Android App Runtime (codenamed ‘Soup’) can be found here. It is a regular Android application written in Java with an embedded PhoneGap instance to support the marketplace and app launching.

On the server side of things, Zamboni is the code that powers addons.mozilla.org, and was extended to support apps-preview.mozilla.org. It is built on Django. The AppSync server is also written in Python (using Cornice) and is what powers app synchronization across all three runtimes (HTML5, Firefox and Android). The AppSync server in turn talks to Sauropod, written in node.js and backed by HBase. Sauropod is a Mozilla Labs experiment aimed at building a secure storage system for user data. Tarek Ziadé has a more comprehensive overview of how all the server side pieces fit together, which you should go read!

Don’t hesitate to participate and ask questions on our mailing list. We encourage you to play around with the system and file any bugs that you may find here. Together, we can make an open, healthy app ecosystem for the web a reality. We look forward to hearing from you!

Go: Why I ♥ Google

November 11, 2009 By Anant in FOSS, Google, Mozilla, People, Plan9, Programming, Technology Leave a comment

Christmas came early this year.

Glenda2Go

Today, Google announced their new open source systems programming language: Go. I’m super excited about this, we all have been wondering what Rob Pike has been upto since he joined the big G, and now we know. Not just that, but Ken Thomson, Robert Griesemer, Ian Taylor and Russ Cox were all involved in the project, with Ken doing what he does best, writing compilers in lightning speed If that isn’t a list of heavyweight respectable computer scientists, I don’t know what is!

I think Go is poised to be the dominant systems programming language of the future. Go has nailed almost every aspect of a systems language, though some would say I’m biased. Go has been strongly influenced by Oberon, CSP languages like Limbo, and the standard libraries have tantalizing similarities to Plan 9. We’ve had Limbo and Plan 9 for a while now (more than a decade), but this is where my real love for Google begins to bubble, they took something awesome but unpopular and gave it a push to the masses. There are very few companies in the world who would attract the talent to do this, and even fewer who would open source the results. The attention Go has been getting is just mind blowing. Pike had been doing amazing work at Bell-Labs for quite a while, but none of it even got an inkling of the publicity Go is currently getting.

Google was what Pike needed to prove Utah2000 wrong.

I know one thing for sure, I’ll definitely be using my Plan 9 virtual machine a lot less; now that I can write clean concurrent programs that don’t make my head hurt, both in Linux and OS X. And GCC, I’m not shedding any tears while I bid you goodbye.

On another note, Google also announced today that they’ll be sponsoring free WiFi at a whole bunch of US airports this holiday season. For all its faults, Google definitely seems to be doing the right thing. For how long, it remains to be seen, but so far I’d say their track record has been better than excellent.

UPDATE: John Gruber points out that “judging from the copyright statements, [Go is] not an official Google project”. Could this be a result of the famous 20% time scheme?

How does Weave use Cryptography?

October 11, 2009 By Anant in FOSS, Mozilla, Programming 13 Comments

I’m back from the EU MozCamp in Prague and we all had a great time! Check out the slides from my talks: Labs Overview and Weave in Depth.

A few people at the MozCamp were interested in Weave’s use of cryptography to protect the user’s data and privacy. Although the specs for the Weave server are available, it may take someone new a while to wrap their head around the whole scheme. I’m going to attempt explaining what crypto operations we do and why we do it in this blog post.

First, let’s get some basic definitions out of the way. Symmetric cryptography means you have one key that can perform both encryption and decryption, and they are complementary operations. For Weave, we use AES with a 256 bit key, and we use it in a mode that requires an ‘initialization vector’ for every decryption. Asymmetric cryptography means there’s a pair of keys (usually called ‘public’ and ‘private’ keys). A piece of text “encrypted” by one key can only be “decrypted” by the other key. Here, we use RSA with a 2048 bit private key.

So, when a user first signs up for Weave using the wizard on their computer, we generate a (random) pair of public and private keys. Next, we use the user’s passphrase to create a symmetric key. This is done using a pretty standard algorithm known as PBKDF2 (short for “Password Key Derivation Function”). The PBKDF2 algorithm requires a ‘salt’ value which is also stored on the server. Now that we have a symmetric key, we use it to encrypt the user’s private key and upload it along with the public key to the server. Note that the passphrase is never sent to the server, so if the user’s password ever gets compromised all the attacker can get is their encrypted private key, which really isn’t of much use (especially given that the key is 2048 bits long).

Whenever a particular “engine” is to be synchronized (an engine could be Tabs, Bookmarks, History etc.) we generate a random symmetric key for that engine. This key is then encrypted using the user’s public key (now, one can only retrieve the original symmetric key with the corresponding private key) and uploaded as being associated with a particular engine. All entries (the ‘ciphertext’ property in a “Weave Basic Object”) in that engine are encrypted with the symmetric key that was generated for it.

To make things clear, let’s enumerate the steps we would take to decrypt a single tab object for user ‘foo’:

Find the user’s cluster by making a GET request to https://services.mozilla.com/user/1/foo/node/weave. It returns https://sj-weave06.services.mozilla.com/.
Fetch the user’s encrypted private key and public key from https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/privkey and https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/pubkey respectively. The user’s password is required to access these JSON objects.
Ask the user for their passphrase and generate a 256 bit symmetric key from it using PBKDF2 and the ‘salt’ found in the privkey object.
Use the generated symmetric key and the initialization vector found in the ‘iv’ property of the privkey object to decrypt the user’s private key.
Fetch the user’s encrypted tab objects from https://sj-weave06.services.mozilla.com/0.5/foo/storage/tabs/?full=1.
Fetch the corresponding symmetric key (the URL is also listed in the “encryption” property of every WBO), in this case https://sj-weave06.services.mozilla.com/0.5/foo/storage/crypto/tabs.
Decrypt the symmetric key with the user’s private key.
Use the decrypted symmetric key to decrypt any WBO from the tabs collection with the initialization vector found in the ‘bulkIV’ property of the tabs symmetric key WBO.
Profit.

A word about the formats in which the keys are actually stored in. All values are Base64. For symmetric keys, the key is stored as-is. For asymmetric keys, I wish we used a standard format like PKCS#12, but we don’t. It’s still ASN.1 though, in some format NSS exports private keys in. You need to do a bit of ASN.1 parsing to figure out the values you’re interested in.

Fortunately, I’ve already figured out most of the details for you – check out my Javascript or PHP implementations of the crypto elements required to decrypt Weave Basic Objects.

Finally, a quick note about why we do all this. Sharing is now reasonably easy, if you want to share your bookmarks with someone, you just need to encrypt the corresponding symmetric key with their public key and they’re good to go. Also, each WBO has it’s own ‘encryption’ property so this can be as granular as needed. Secondly, the passphrase is never stored anywhere (except possibly on the user’s computer) so the server never sees anything other than encrypted blobs of Base64′ed text. Along with making HTTPS mandatory, we think this is a pretty secure way of protecting the user’s data.

If you have other encryption schemes that might fit into Weave’s use cases please let us know! (We’ve already been looking at interesting developments in this area such as Tahoe). I’d also love to hear from you if you have any questions on our current cryptography scheme. We’re constantly trying to improve the security and efficiency of our system so these details are only valid until we change our scheme

Now, go write that third-party Weave client, you have no excuse not to!

The Summer of Code is here again!

March 23, 2009 By Anant in FOSS, Gentoo, Glendix, Google, Mozilla, Plan9, Programming, SoC, Technology Leave a comment

It’s that time of the year. Google is, yet again, sponsoring students to write some awesome open source code this summer. If you’re a student, and you’d like to make some money contributing to some of the most well known and exciting open source software projects out there, you’d be missing out on a lot by not applying.

If you’re wondering about what the best way to get started is, check out this great advice page. All projects have also been tagged by programming language and field in this delicious profile. You can also search for ideas here.

I’m going to be mentoring for Mozilla, Glendix (under the Plan 9 Umbrella) and Gentoo this year. Get in touch if you’re interested in any of those ideas

The window for applications opens in a few hours. Good luck everyone!

SSH-HTTP Multiplexer

December 16, 2008 By Anant in Fun, Hacks, Programming, Python 1 Comment

One of my friends wanted to run a HTTP server on his office machine, but the network it is connected to blocks all ports except 22 (SSH). Sure, he could run Apache on port 22 but that would mean he could no longer login remotely.

I wrote a quick hack in python: muxer.py, that will multiplex incoming connections between an SSH and HTTP server. It is slow, and makes all incoming SSH connections wait for 5 seconds before responding, but it works! The 5 second timeout is required because the SSH protocol specifies that the server should be the first one to send the client it’s version string, and only then will the client respond.

I should probably rewrite it in C at some point. Anyway, here’s your hack for today. Maybe someone will find use for it, or even better, come up with a better solution (this one is certainly the worst!)

FOSS.IN/08: Summary

November 30, 2008 By Anant in Conferences, FOSS, FOSS.IN, Gentoo, Glendix, Hacks, Mozilla, People, Plan9, Programming 2 Comments

As a developer, I have to say that FOSS.IN/08 is possibly the most productive conference I’ve been to until now! In just 5 days, I’ve got more things done than I have in the last 5 months

Let’s start with the Beacon workout: Nandeep joined us via VoIP and we got started almost immediately, thanks to the dynamic nature and small size of our project – we didn’t have any infrastructural trouble as a few other C/C++ projects with huge codebases and complex build systems did. We had a list of 6 tasks in mind, and we managed to complete 3 of them. Salil Kothadia got started with writing a PDO data backend, and promptly submitted the patch to us next day. Thanks Salil, hope you continue to contribute to the development of Beacon (thereby increasing the development team size by 25%)!

I also attended Philip’s workout on porting HTML::Template to Javascript. As mentioned on the Wiki page, we mostly worked on the design during the first half or so, and then moved on to writing a skeleton for the whole framework. I think this is an extremely interesting project, and am very happy to be associated with its birth. Hope we can continue the momentum and work until it is finished.

Perhaps the biggest take-away from the conference for me was the ability to give a lightning talk about Glendix, with several kernel hackers present in the audience. Christoph then kindly offered to review some of the patches during the workout. Even the possibility of Plan 9 binary emulation being considered for inclusion into the main kernel tree is amazing, let alone the fact that I got the guidance of an experienced kernel hacker for a good 2 hours! I think the effort was largely successful – I now have a better idea of what I need to do in order to get a kernel patch in order, and also got a few hints as to how I can implement the missing bits.

My primary focus at the conference was to give a talk on Mozilla Labs and Innovation. I think I managed to stir up a decent amount of interest in the various Labs initiatives. I covered the different ways in which members of the community can contribute, specifically focussing on Weave, Ubiquity and the Concept Series. We even covered how easy it is to actually write an Ubiquity command. I now look forward to increased participation by the Indian Mozilla community in Labs projects. Don’t forget to thank Mary for all the goodies!

All this, apart from regular conference happenings like catching up with old friends, making new ones and free swag (great mugs and t-shirts this time around) makes FOSS.IN/08 one of the most successful conferences I’ve been to so far! I can’t wait for the 2009 edition

Binary Sunrise

Anant Narayanan

Category Archives: Programming