TedPage

For a while we've been using dbus-test-runner in various DBus related projects to create a clean DBus session bus for our test suites. This also makes it so that we can test on headless systems that don't have a standard DBus configuration like what is available on most developer's desktop systems. This release we even got it into the Ubuntu archives so we could run our test suites on package builds. As our testing has gotten more mature and we've increased the number of builds on the Ubuntu QA Jenkins server we need to have better reporting, something like what gtester or Google Test can provide, which is difficult with an external utility.

To handle this we've taken dbus-test-runner, turned it entirely inside out, and created libdbustest. This allows for managing the DBus service that is being used in the test setup to be managed by the test framework. Which means you can have a DBus Session per test, or share them, or what ever you need. You've got choices that can match what you're trying to test. I created a small example using gtester that is part of the dbus-test-runner test suite. I expect that we'll be able to port of more of the various ubuntu-menu-bar projects' test suites in the coming weeks.

For those faithful users of the dbus-test-runner command line utility, no worries, it exists and just uses libdbustest. I expect no regressions as it passes the original test suite, and even maintains the code coverage numbers that we worked on in an earlier post. It does have a place in some testing, I expect it's usage to remain as another way to test applications and interfaces.

At this point I've merged the basic ideas behind what we're trying to achieve, but there's still a lot of time to work on the API and make it perfect. If you've got requirements or ideas please share them and let's make this useful for everyone. Also we need documentation and GObject introspection support so this little guy can be used for Python and Javascript testing as well. There's a lot of work to do, but I'm excited where this project is going.

posted on Mon, 23 Apr 2012 at 15:36 | permanent link

Over the history of human society we can see great increases in productivity through unlocking the potential of greater and greater segments of the human population. Today, more people can contribute creatively to the betterment of the world than ever before. While more work is to be done there, an eventual maximum is possible. What happens if all humans are working together for the betterment of humanity? Where will our next gains in productivity come from? At Canonical we're thinking beyond this eventual limit, and that's one of the reasons that we introduced the HUD. The HUD makes your cat more productive than ever before.

Extensive research has been done on how cats interact with keyboards, it can be seen in chat rooms all across the Internet. From "awsd" to "jijkl" these appear nonsensical to everyone else, but it is the cat trying to use a computer effectively. Using the HUD matching algorithms the cat is able to select "Launch Nukes" from the menu using these simple key combinations.

There is certainly more work to be done. For instance, it's impossible for a cat to unlock the screen at the start of a workday. This is why we're investing in kitty facial recognition at the screen saver lock screen. We're also looking at laser pointer based mouse control, by swiping the "virtual pointer" the cat is able to control the pointer on screen. Lastly, we're working a remote control helicopter based catnip delivery system to take advantage of the eventual disposable income of working cats.

We hope that you are as excited about these potential advancements in Earth's overall productivity as we are. By thinking beyond the human race we hope to find advancements that help humans as well.

Cat pic CC-BY-NC-SA by Sophie

posted on Sun, 01 Apr 2012 at 12:37 | permanent link

In order to reduce the bus factor a bit, I'm documenting how I do releases for the indicators. This is not a complex process, but not obvious either. This post will involve lots of Bazaar, Launchpad and Ubuntu. If that's not what you want, skip to the next post in your reader.

Understanding Ubuntu Distributed Development

I think the first think to understand is what our goals are in making a set of releases. It's a set of artifacts to mark a point in the continual development of the software. But, that point is marked differently by different groups of people. Largely this is mirrored by the "trunks" of Ubuntu Distributed Development. It's important to realize that Bazaar doesn't really have the concept of "trunk" in it. We name a branch in Launchpad that to make it easier to communicate, but Bazaar doesn't treat that branch any differently than any other. So for the different audiences, they each have their own trunk, and releases there mean different things. The trunks we're using are:

• Upstream Trunk. Traditionally this is what software developers think of as trunk. The place where the artificats that are edited by the developers exist
• Upstream Import Trunk. This is a branch that is maintained by the UDD tools and represents the tarball releases. The tarballs are slightly different than upstream in that they contain additional files generated by the build.
• Upstream packaging. A packaging branch that is mostly a pure representation of upstream software, perhaps more up-to-date that distro. This allows us to do test builds and maintain updates for things like our Jenkins builds independent of having to ship the change to every Ubuntu user.
• Ubuntu packaging. A branch maintained by the Ubuntu Desktop team that represents what is sent to Ubuntu users.

As we move through the release process we'll go through these branches. Some will end up slightly obscured, but I'll try to point them out along the way.

Releasing Upstream

Building and shipping the upstream tarballs is mostly straight forward, but I want to take the time to be explicit and note a couple of tricks that are useful along the way. For our examples here I'm going to release indicator-messages version 1.2.3. First grab the trunk:

$ bzr branch lp:indicator-messages trunk

Then you can edit configure.ac to update the version and commit that. It's important to commit before distcheck since our ChangeLog is built dynamically at dist and you want the top of it to be the version number that is released.

$ vi configure.ac
$ bzr commit -m 1.2.3

Note, we're not tagging, and that's for good reason! Tags are a pain to move, and as releases go there's a reasonable chance that distcheck will fail, but let's check first:

$ ./autogen.sh
$ make
$ make distcheck

Distcheck will build the tarball, uncompress it and ensure it builds and can run the test suite. Most of the time this works so you can just:

$ bzr tag 1.2.3
$ bzr push :parent

But, sometimes it doesn't. And you'll need to fix it. I'm going to leave fixing it as an exercise for the reader, but what I want to talk about is how that should look on the branch. You don't want your fix to occur after you set the version in the history. So you can step back by doing:

$ bzr uncommit
$ bzr shelve

This saves your changes. So then when you want to come back to releasing again

$ bzr unshelve
$ bzr commit -m 1.2.3

And you're back to where we left off. We need to put the tarball into Launchpad, and fortunately there's a handy script for that:

$ lp-project-upload indicator-messages 1.2.3 indicator-messages-1.2.3.tar.gz 1.2.4

An interesting thing here is the "1.2.4", where does that come from? It's creating the next milestone in Launchpad for you so that we can start assigning bugs that are Fix Committed to it and they're already there come release time. Also you'll need to take the bugs that are Fix Committed on the 1.2.3 release. You can do that with this script if there's a bunch, but if there's not too many just do it by hand.

Building a test package

We've built and released a tarball, and it passes all the tests, and all the code's been reviewed, but until you install it on your machine you're not putting any skin in the game. Let's really believe in that release. First we need to get the upstream branch. Go ahead and put that in a directory next to your trunk branch. In this example we're assuming the Ubuntu release is precise because that's the current right now.

$ bzr branch lp:~indicator-applet-developers/ubuntu/precise/indicator-messages/upstream upstream

You should at this point make sure that this branch is in sync with the Ubuntu Desktop branch. Frequently they'll make updates to the packaging or cherry pick revisions they're interested in from the developers. This isn't a necessary step to make everything work, but it makes the merge for them easier, so you should do it.

$ bzr merge lp:~ubuntu-desktop/indicator-messages/ubuntu
$ bzr commit -m "Sync from Ubuntu Desktop" 

We're now going to use the UDD toolchain to pull in the tarball and merge it into the packaging. I'm purposely providing two parameters here that are optional. The first is the tag that should be used to pull from. Chances are that if you just edited trunk the revision there would be exactly the same, but let's not take that chance. The second is the version where I'm just being more explicit about what we're doing.

$ bzr merge-upstream indicator-messages-1.2.3.tar.gz lp:indicator-messages -r tag:1.2.3 --version 1.2.3

At this point the tarball has been imported and merged into the current branch. One innocuous warning that you'll see in this step is about criss-cross merges. This usually happens when the desktop team cherry picks a branch. I have argued that this shouldn't be a warning and largely I regard this as Bazaar saying "if there are conflicts they're your fault," rarely are there ever conflicts from this.

We're now at the point where we start switching from the Bazaar-based tools to the Debian ones. We need to talk about what's changed in the Debian ChangeLog by using

$ dch -a

There will already be a section made for you buy the import, but you should continue to describe the upstream release with what has changed, specifically which bugs have been fixed. Also, I always change the version number from "1.2.3-0ubuntu1" to "1.2.3-0ubuntu1~ppa1". This is so that when I install the packages locally they'll get overridden by the ones that are built in the Ubuntu builders. Also it creates a unique namespace if I mess up that I can increment. You can then commit with the message you put into the changelog with

$ debcommit

A couple of quick things that you can check at this point. Debian packages have a way to include patches, we should have merged those upstream and if you leave them there they won't apply when building. So you can just remove them and comment as such

$ rm -rf debian/patches
$ dch -a
$ debcommit

Also you can check to ensure that you're shipping actually the tarball that you build, plus only the Debian packaging. All of the Debian packaging should be in the the debian/ directory so we can look to see if anything is not in that directory

$ bzr diff -r tag:upstream-1.2.3 | lsdiff

The astute reader will notice that I just used a tag that I didn't build. Hmm. This is where the Upstream Import branch comes into play. When we did the merge-upstream above we actually edited two branches. The first is the import branch where the clean tarball was placed. Then it was pulled into our packaging branch from that revision. I find it handy to look at the revision history at this point

$ bzr visualize

Now let's build ourselves a package. The first thing you want to do is set up your environment if you haven't done any packages before. Usually if it builds I also tag it as a version in the revision history so that I can keep track of what I've installed. So to ensure I don't mess that up I chain the commands

$ bzr builddeb && dch -r && debcommit -r

You should now have packages in the directory above the current directory that you can install.

$ sudo dpkg -i *1.2.3-0ubuntu1~ppa1*deb

Now you can test on your machine if things are generally working.

Pushing downstream

If everything looks good you should push that branch back up and then propose a merge into the Ubuntu Desktop branch. This tells the Ubuntu Desktop team that we've got a package that we think works and they should consider for usage in Ubuntu. They'll run their own set of acceptance tests, and I'm sure they'll find you if they fail.

posted on Sat, 10 Mar 2012 at 00:05 | permanent link

It has become popular (at least in the US) to start to put bandwidth caps on customers. Technologists are upset, they've seen Star Trek and they know that everyone is going to start video conferencing constantly any time now. These uses would easily go over the paltry limits that network providers have setup. Real consumers won't care, what will actually happen is the economics of connecting to the Internet will change.

We can start to see this with AT&T's recent foray into allowing phone apps to pay for the bandwidth that their app consumes. This would mean that a user could grab the new app from their favorite sports team and not worry that the video highlights will knock them over their cap, the bandwidth is "free." The reality is that the way it's paid for has changed. Instead of being amalgamated over all the users, averaged and sold at a fixed rate for all you can use, it's being paid for out of the money you're paying the application provider. If it's $10/mo to Netflix, they're giving AT&T a kickback of 10% of that to ensure you don't worry about using it.

Consumers will love this! Why? Because what will in turn happen is that the cost of the bandwidth to them will go down. Do you want a gigabit of bandwidth to your house? Sure you do, especially if it's only $20/mo. What other restrictions would you take to get that much bandwidth so cheap? I'm guessing if most people didn't feel the pain of the restrictions, they'd all take that trade. And this is where the "gain" to consumers lies. Most of them aren't doing bit torrents, they're using Hulu and Google and Youtube, all people who can give the DSL provider money to subsidize that bandwidth whether it be from subscriptions or advertising.

Network neutrality in this scenario sadly becomes a moot point. The provider isn't providing any preference to the routing, or deciding which packets to put on which line, they're just adjusting how they bill the customer. You can rest assured that no one (at least in the US) would regulate how they the ISP bills the customer. But, the restrictions would be such that consumers would choose (no mater how manipulated that choice is) one site over another based on their cost, which is effectively a negotiated price by their ISP. Again, the ISP is manipulating and choosing which sites the user is most likely to see and use.

It seems doubtful to me that all bandwidth will go to this model, only consumer level home and mobile bandwidth. Companies and Universities are unlikely to see significant change in how they purchase their own bandwidth as there isn't as much direct billing there. But, I think it might effect how people could work from home. I imagine this will result in the ability to purchase unlimited bandwidth to a single domain for VPN usage.

There is, of course, a lot more to say about this. It's an interesting pivot in how we see the Internet structured today, but was probably inevitable. We continued to squeeze the companies that owned the last miles to our houses, asking for more bandwidth for cheaper. And then we had the gall to actually use it! They're trying to figure out how to recoup those costs, and consumers aren't willing to pay more, so they have to hide it.

Comments: Identi.ca | Twitter

posted on Mon, 05 Mar 2012 at 22:40 | permanent link

One of the prongs of our strategy to increase the quality of software throughout our developments in Product Strategy was to introduce static analysis to our code where possible. After evaluating several tools we decided that Coverity was the best tool for this, and we started figuring out how to make this work on our code bases. After a fair amount of work in pulling things together I'm happy to say we're scanning projects and processing the bugs in production now. You can see the open bugs and you can find out about what data is there.

While there were various technical issues I think one of the more interesting and difficult ones was social. How can we introduce a tool that we're restricted on distribution to a community without restrictions? It would have been very easy to create two classes of developers, those in Canonical with access to the tool and those outside that wouldn't have "all the info" about what was going on. We didn't want to do that, it's not how we work.

What we did was build a tool that would take the bugs out of the Coverity Integration Manager (CIM) and put them into Launchpad. So every time a commit happens on trunk a Coverity Scan is performed, the issues are put into CIM, and then Launchpad is updated. This includes both creating new defects in Launchpad as well as closing ones that were fixed. We also take the annotations of what branches Coverity took throughout the code and create an annotated source file and attach it to the bug.

Our sync tool is Open Source, and while it's hard to test as you'd need a license for Coverity to do so, we're happy to take patches. We want to see all the needed information to work on the bug in the Lauchpad bug. If there's something you think we need to add, come talk to us, it's a conversation we're interested in.

In the end we hope that we've created an environment that allows for Coverity to be used by everyone in our development community, on largely equal footing. Currently we're only licensed to scan Unity and the other projects it uses directly. I'm excited to see how we can use this new tool to improve the quality of PS projects as we continue to expand it to scan more projects we're licensed for, along with hopefully expanding its coverage as it shows value.

Comments: Identi.ca | Twitter

posted on Fri, 24 Feb 2012 at 18:04 | permanent link

Launchpad users know that it can send quite a bit of e-mail. Okay, a LOT of e-mail. There has been effort on the Launchpad side of things to add controls to set the amount of Launchpad e-mail you get. But for some of us, even getting the mail that you need results in a fair amount of Launchpad mail. In playing with my procmail config for Launchpad mail I stumbled on this little feature that I love, and thought I'd share, as it's cleaned up my mail a lot. The secret rule is:

:0:
* ^X-launchpad-bug:.*product=\/[a-z0-9-]+
.Launchpad.Bugs.${MATCH}/

Quite simply that rule takes the project specifier on a bug mail, and uses it for the folder name that it puts the mail into. This means each project gets it's own mail box, no matter what. So even as you add groups or subscribe to new bugs, you just get new mail boxes. Works for me. Hope it helps other folks too.

Comments: Identi.ca | Twitter

posted on Wed, 01 Feb 2012 at 11:19 | permanent link

One of the neatest parts about starting to get more application data into the open is that we can start to use it in interesting ways. I'm happy to talk about a new way that we're using the menu data: the HUD. The idea behind the HUD is that you can quickly find functionality in an application without having to know the menu structure. But how does it do it? How can you make it better?

Getting the data

We're using the same Dbusmenu data that is currently exported to the global menu, just remixing it for search. We are searching through the labels in the menu items which gives us already localized data straight from the application. This means that it should work for the language that the application is in. In the future we hope to use information like accessibility data as well as any tooltips that might be attached to a menuitem (though we don't show tooltips in the global menu).

Any application that works with the window menus today should also work with HUD out of the box.

Matching the label

To match the label we're basically using an implementation of the Levenshtein distance with a few additions. What this allows us to do is rank possible solutions in a relevancy order, and present some solutions that might occur via "fat finger" or other similar type errors. But, this also means that there is some fuzzy algorithms involved in the matching which will have to be tuned.

We expect to tune them over the next few releases, and to do that we have a set of test cases that we're using for the tuning. The problem with those test cases? They're only in the languages I speak. You probably speak in more/different/better languages than I do, please feel free to propose merges that extend this test suite so that as we continue to tune the search algorithms we don't leave any language behind.

Remembering Favorites

One of the additions that we add to the distance calculation is an offset based on which entries you've used most recently. Your favorite functionality in the application. Quite simply we're storing a list of items you've used over the last thirty days and a timestamp of when you used them. This database is simple but it can be fun to look into for the curious and I wanted to talk a bit about a couple of the tools that you can use to see the data.

$ hud-list-applications

This will list all the applications that have data on them in your HUD usage database. They are identified by the path to their desktop file as determined by BAMF. You can then look at the menu items used in a specific application:

$ hud-dump-application /usr/share/applications/inkscape.desktop 

This shows the individual items that you've used, and the number of times that you've used them. If you want to inspect the exact file tracking the data it is available at:

~/.cache/indicator-appmenu/hud-usage-log.sqlite

While talking about various tools to work with HUD I thought I'd also mention that you can also, just for fun, work with HUD from the command line using the command line tool:

$ hud-cli

Application initial bias

Application designers have always had a problem figuring out how to promote specific functionality that is commonly used to the forefront, while still making the rest of the functionality easily available. The most recent ways that they've done this is with toolbars and ribbon style. You can't adjust the positioning even when you know that the particular toolbar isn't best for the user because it will mess up the user's spacial memory. HUD sidesteps this issue by providing all the options, just promoting certain ones based on usage. They're all in the same place (the HUD) but with always improving ordering.

What happens on first usage of the application? At that point we don't have any way to know what the user wants to do, we we've provide a way for the application designer to provide the most likely items for general users. Effectively, this is the HUD's version of the default toolbar setup in an application; though it automatically decays and adjusts to the user's usage pattern.

The files that control this initial bias are very simple and there is an example in the test suite. Basically they have the various menu items along with a value that describes how to preload the usage database. A '5' there would mean that 5 entries are added to the usage database for that item on the first time that application is used; one for today and each of the four days previous. In this way, as values drop off by being too old, there isn't a step function in how the item is ranked, it just slowly drops down in priority. Application designers should start to think about how they would rank the menu items in their application, and start getting these integrated into either the releases or the packages of those applications so that users have a good first experience with their application.

Development notes

The code for the HUD lives in the indicator-appmenu repository. Currently it exists on a branch that needs to be reviewed before merging, but that shouldn't be for long. I expect it to get merged to trunk in the next couple of weeks.

After that the biggest change will be integration with indicator-appmenu. It was originally implemented as it's own service to make development more agile, but it clearly shares a large amount of data with the global menu and there's no reason to have two repositories in memory of the same data. It also needs to synchronize heavily with the application menu and BAMF, which is also already in indicator-appmenu. Thanks to the magic of DBus no one should notice the change in processes as the names and objects will migrate over to the new process.

As this is more of a first prototype there are also some missing features that need to be added. The first of those is to simply improve the matching. We also need to get better descriptions from application indicators, today we're using their accessibility description (you set those, right?) but that typically has too much information. Lastly, we need to integrate better with applications that expect the about-to-show signal for their menus. This includes XUL applications and some Qt ones, so it's an important feature for making the HUD usable for everyone.

Merges and bugs should be directed towards the indicator-appmenu project and also make sure you've signed the Canonical Contributor Agreement for any code contributed.

Comments: Identi.ca | Twitter

posted on Tue, 24 Jan 2012 at 09:15 | permanent link

I'm excited about two new team members that we have, Thomas Voß and Allan LeSage, who are working to increase our quality infrastructure. One of the first things they've been working on is getting per-commit code coverage measurements of our unit tests in to Jenkins. I think that this is great because, while we have tests, we have no real way to know if we have enough tests. Code coverage isn't a magic bullet there, but it does give us some idea of where we stand.

If you had asked me (as soon as yesterday) about dbus-test-runner's test suite I would have described it as good. It's a small amount of code, and it has quite a few tests. So it must be good. Allan submitted a merge request to add code coverage measurements. Here's what came out on the summary:

Line Coverage		Functions		Branches
79.4 %	239 / 301	88.0 %	22 / 25	62.0 %	67 / 108

Ouch! I don't think we could really describe that as good. Not awful, but it should be better, especially for a test tool!

This story does have a happy ending. I took a little bit of time to make a new branch that adds tests, but also makes the utility more testable so that the code can be hit in a reasonable test. Overall, a big win for dbus-test-runner. Here's the results after that branch:

Line Coverage		Functions		Branches
95.2 %	317 / 333	96.2 %	25 / 26	84.4 %	103 / 122

Comments: Identi.ca | Twitter

posted on Fri, 09 Dec 2011 at 23:31 | permanent link

I've got an Android phone and I really wanted to get the calendars syncing with my desktop (Ubuntu). I tried the Android recommended Google Calendar, and it didn't really work for a variety of reasons. Off to try a new solution.

What I ended up doing is using Ubuntu One (U1) to sync the iCal files that Evolution Data Server (EDS) uses as its backend to the cloud. And then I published those files using U1's publish feature. Then I brought those back into my phone using ICSSync. Let's go through that step by step so you can get it set up too.

To get your EDS calendars are in your home directory under ~/.local/share/evolution/calendar so to get U1 to start syncing them to the cloud you need to execute this command u1sdtool --create-folder=/.local/share/evolution/calendar. U1 should start sync'ing that folder, and it should show up on the U1 website here.

You should there have a variety of folders that are your individual calendars. If you can then publish the calendar.ics file that is in each of these folders by clicking on the "More" button the right. You can enter the URL of the published file into ICSSync's "Add feed" dialog.

There are some downsides to this approach, specifically the answer to the U1 question "Are published files private?" But it is working well for me. Hope it works for you too!

Comments: Identi.ca | Twitter

posted on Mon, 15 Aug 2011 at 14:14 | permanent link

Richard Fontana has written an article talking about the Harmony agreements using the idea that "input = output" for Open Source projects. I think this is a misunderstanding of Open Source that lies at the foundation of his argument. Motivations of many contributors to Open Source projects are because their contribution enables them to use a large corpus of functionality for relatively small amount of work. Perhaps a better equation would be:

e^input = output

Let's look to the example of someone getting a new laptop that would perhaps not have a keyboard driver in Linux. That person would have the choice of writing or paying someone to write that driver for their laptop. What would they get in return? They'd get a fully functioning kernel with world-class functionality across a variety of subsystems. I don't think that we could say that the size of the contribution was equivalent to the amount of benefit.

This is a very good thing. This is what makes Open Source interesting to me. It's a group of people coming together with differing interests and goals with the desire to make something greater than what they could on their own. And there's trade-offs, not everyone can get all of what they want, but they all realize working together makes all of their contributions more noteworthy. The essence of their collaboration, the project itself, is what makes the Open Source great more than any individual commit or contribution.

If our goals are to make Open Source projects better, stronger and more competitive in difficult software world we should focus on how to create better projects instead of prioritizing the desires of small contributors.

Now, because I work for a company that's considered to be pushing CLA's I think it's important to reiterate that the opinions on this blog have always been my own. They always will be. Trust me, I'll get something at canonical.com for official stuff if I ever need it.

Comments: Twitter | Identi.ca

posted on Fri, 08 Jul 2011 at 10:33 | permanent link