I’m sure you’ve sensed all the buzz about Ubuntu going multitouch. I truly think that this is some great news, being the multitouch and HCI enthusiast that I am. But what if you want to test your multitouch hardware? Or if you want to actually develop multitouch applications? Here’s something for you: PyMT has just been released in version 0.5!

We’ve been working hard to make this reality, and many a new feature has been added and quite a few bugs have been squashed. I suggest you read the full changelog and, if you already have a PyMT 0.4 application, also the migration guide.

One of the coolest new things with this release is the availability of portable binary packages for Windows and OS X. Those come bundled with everything you need (on Windows, even Python) to get started. You simply download the package for your platform and run it. We didn’t provide a portable package for Ubuntu, but it’s ridiculously easy to install PyMT there anyways. On Ubuntu 10.10, all you need is:

sudo apt-get install python-pymt

PyMT has native support for multitouch devices on Linux that are supported by the kernel, all Windows 7 multitouch devices, all of Apple’s multitouch accessories and much more. If you know basic python, PyMT is the easiest way to create multitouch applications or to just test your hardware.

In future releases we’re planning to fully use a rewrite of our current OpenGL abstraction and other performance-critical parts (that we start doing in C) that will allow for much higher application speed, less battery consumption and OpenGL ES/3.0 compatibility so that we can smoothly run on portable slate/pad devices.

Lastly, see what people have done with it (planet readers, click the images to get to the videos):

I hope that sparked your interest. We also hope you enjoy PyMT. If there are any questions, bugs, problems or feature requests, let us know. There’s a mailing list, a google code issue tracker and our IRC channel at irc.freenode.net in #pymt.

Google’s Summer of Code 2010 comes to an end for me today. It has been a great time working on awesome projects like PyMT and Movid. My task was to enhance PyMT’s text input methods. One of the joys of this task was that it allowed me to work on a relatively wide scope of things. Here’s a brief list of what I worked on:

• I added a new spelling provider to PyMT that abstracts from individual spellchecking libraries. That means you can use your favorite spellchecking library, which is important considering that PyMT is cross-platform.
• I added two actual spelling providers that implement this protocol: One enchant spelling provider (usable after installing enchant) and one provider using OS X’s native AppKit spellcheckers (so you get that out of the box on OS X).
• Mathieu once wrote a basic virtual keyboard with spelling suggestions which I adapted, cleaned and merged.
• PyMT obviously already had some text input widgets, which I improved (e.g. MTTextArea).
• I began working on a version of MTTextInput with added spellchecking (like OO.org with red lines drawn for incorrectly spelled words), but that needs some more love.
• One of the more concrete objectives of my task was a Swype-like keyboard for PyMT. I created a prototype for that, see the video below.
• Another concrete objective was a split keyboard (split into two parts, one half for the left, one for the right hand) that adjusts to your hand’s properties (e.g. size). To achieve this, a substantial amount of changes was needed to our vision tracking application (Movid):
• For the keyboard to adjust to the user’s hands, a handtracking algorithm was needed that I implemented for Movid. It detects the fingertips of the hand as well as the hand’s center. These are just seen as a certain type of ‘blobs’ internally.
• These blobs need to be tracked over a sequence of frames from the camera. Additionally, we also want to find simple touches (without all the hand information). For that, I added and integrated BlobFinder and BlobTracker modules that obey a common format so they’re easily interchangable.
• When your camera senses a blob on the touch surface, the application needs to perform a mapping to get the blob into screen coordinates. We do that using a calibration module, which I had started before SoC. I finished it and merged it back into our master branch.
• As an extra feature, I added a PyMT module that you can use to calibrate your tracker from within your client application, eliminating the need to switch applications. I also added a Flash GUI for the calibration so that you can easily do it on any remote computer via our web interface.
• To actually send the handtracking to the client application, Mathieu added a TUIO2 module to Movid. I started a PyMT input provider for TUIO2. Both of which is work in progress, but I believe we’re the first project to adapt TUIO2 (there’s not even a reference implementation yet).
• The result of that can be seen in the second video below. Also, make sure to read the vimeo description.
• Other than that we now also provide portable binary packages for PyMT 0.5 for both Windows and OSX. I created the OSX package, so it’s no longer a major pain to install. You just download and run it.
• And, of course, many more fixes!

Some of that is already in PyMT 0.5. All of the Movid stuff will be in the first release. In future releases we shall see much improved versions of these prototypes and hopefully even context aware word suggestions.

Here are the two promised videos, if you’re reading this through a planet, please go directly to my blog.

Prototype WipeToType Keyboard for PyMT from Christopher Denter on Vimeo.

Ergonomic multitouch keyboard prototype from Christopher Denter on Vimeo.

Thanks to all the people who made this possible. Thanks Google, Christian, Pawel, Mathieu and Thomas, for being (a) fantastic mentor(s). It has been a great pleasure and privilege to work with you in GSoC 2010 and I sure will continue to work on both projects.

I just had the opportunity to take a video of my multitouch table with my software in action. Both hardware and software were built for my bachelor’s thesis which I handed in in march. The software that you see at the end is written in Python with PyMT, using the VTK library.

Medical Multitouch from Christopher Denter on Vimeo.

Reading through a planet? Click here!

For more information, see the video description. PS: Although it supports all platforms, it currently runs on ubuntu. :-)

Let me know what you think!

Many things happened in the last few weeks. I just want to quickly outline them in case you’ve been wondering what I’ve been doing.

PyMT Spelling

Starting with PyMT 0.5 (to be released in August), we added support for spelling correction and word suggestions. This is based on my GSoC work. The code has been polished and integrated into the master branch, which will soon lead to the 0.5 release. I’m also currently working on a text input widget that indicates incorrectly spelled words as you type (just as OpenOffice would). This much is working. In a next step, I plan to add a feature that lets you select from a list of suggestions for the word that you just tapped.

WipeToType

One of the tasks of my GSoC proposal is the implementation of a Swype-like keyboard. What this means is that you just wipe over the keys that make up the word you want to type and it automatically determines which word you intended to enter. It is clearly far beyond the scope of a single, multitouch-oriented GSoC proposal to implement something as clever as a Swype clone (especially since this also requires A LOT of backend code for the intelligence). However, something remotely similar and usable should be doable and is what I’m looking for. A while back I started something like this and quickly sketched a modified version of the virtual keyboard.

This still needs much more love, but keeping in mind that I did this in a really short amount of time I think I can say that we’re getting somewhere.

FITG in Lille

Thomas, Mathieu and I have had the chance to meet in Lille at the FITG conference and present PyMT in a talk and several workshops. This was a great opportunity and in fact, it was the biggest real-life meeting of core developers and users so far.

The conference itself was a great success, both for the organizers and us. We had many people come to us and ask questions concerning PyMT and Movid. After our talk (which I think was well-received) we decided to give an additional workshop so that people interested could play with PyMT and get help from us. The room was pretty crowded and people were standing. The workshop presented a basic PyMT overview and first steps in a ‘hello world’ fashion (At least I think that’s what Mathieu was talking about. He spoke french and Thomas and I were answering questions or translating things in English). In the evening we gathered all the people that were still at the conference and went to a nice little restaurant to chat.

The next day we gave two more workshops. The idea to do the first one came up while we had breakfast. We decided to implement a simple version of the game at linerider.com with PyMT. When we arrived an hour later, we just picked a python 2D physics library that was easy to install and started live and from scratch, without any actual code having been written (or even thought about) beforehand. Luckily it all turned out well. After almost exactly one hour (in which Mathieu helped people in the audience, Thomas pointed to and explaining stuff at the projection and I coded and talked) we had finished what we were looking for in just 60 lines of unoptimized python code. The last workshop was about advanced OpenGL. Mathieu presented some of his insights that he had gathered while optimizing PyMT’s performance (great advances have been done here, by the way).

I stayed four days in total and it was absolutely worth it. Lille is a wonderful city and the conference was fantastic. The venue itself was just mind-blowing to begin with. We had a lot of fun together and obviously worked on PyMT as well. It’s even more fun if we’re in the same room! Sincere thanks to everyone involved in making these days as awesome as they were!

Reworked TextArea

For our talk in Lille, we used a very nice presentation tool (PreseMT) which is, obviously, written with PyMT. While using it for my own bachelor’s colloquium (I’m officially a Bachelor of Science now, by the way) I noticed that entering text suffered from severe limitations of the text input widget. Given that I had no multitouch-capable device around to enter text, I did it all with my hardware keyboard. I added to PyMT’s TextArea widget the ability to resize automatically depending on the text that was entered (which is what you want in PreseMT). Furthermore, the widget now properly reacts to several special keys like the arrow keys, home, end, del, pgup and pgdown.

Portable PyMT

It is no secret that installing PyMT on OSX is a major pain. This is not our fault, though. The problem simply is that installing almost anything in non-app format involves a non-trivial compilation process using MacPorts and the like. Unfortunately, one of our dependencies (gstreamer) is not easily installed this way.

Since we really don’t want our users to go through all of this, we decided to distribute portable versions of PyMT for OSX, Linux and Windows. I did the OSX version and hope to be able to finish it soon so that it can be reviewed. With it, you just download a zip file, unzip it and go. It contains everything that is needed to run PyMT.

In the course of this, I also fixed the compilation of our OpenGL-dependant cython modules for OSX.

Conclusion

Hopefully you will see a wonderful PyMT release next month. We’ve added many new features, improvements and fixed a lot of bugs. Some of my GSoC work will also go into it. In terms of GSoC, I will finish the aforementioned spelling-aware text input widget. I also intend to improve the quality of the results of the WipeToType keyboard and implement the things left on my GSoC proposal.

The NUIGroup Google Summer of Code students (I was lucky enough to become one of them for PyMT this year) are asked to summarize their weekly activities in blog format. Given that the first week has passed I figured I should just quickly outline what I have been working on up to now.

My proposal aims at developing more advanced text input methods for PyMT.

Work on PyMT

Some of the ideas I will realize draw heavily upon spelling correction and suggestion. It is therefore necessary that PyMT can interact with a spelling backend. Given that PyMT should be kept modular, I first implemented an abstract new core provider for spelling suggestions to become independent of a specific library. I then realized two concrete implementations of this provider:

• An enchant spelling backend. This uses the enchant spelling library which can itself be used with different kinds of dictionaries.
• A spelling backend based on OSX’s AppKit spell checker.

After the foundation was laid out I adapted a virtual keyboard with spelling support that Mathieu once developed to the new API and added it to the code base. All of this is not yet finished and needs some more love before I can merge it back into the master branch. You can check the branch I’m currently working on here.

Work on Movid

While spellchecking is important for some of my upcoming widgets, some other text input approaches make use of additional information provided by the tracking application. For example, one idea I had was to split the keyboard in half and dedicate one half to each hand. The halves would then automatically orient themselves following the respective hand’s position and orientation. Theoretically, further information such as properties of the user’s hands (length of fingers, etc.) could be taken into account to lay out the keyboards. For this I obviously need some kind of hand and fingertip tracking. Luckily I implemented that for Movid already:

However, since Movid is still not ready for end users due to a missing calibration utility and a proper (generic!) blob tracker (which means I can’t use it yet either), I continued my work on both of those. Again, both of which are not finished, but I can see the light at the end of the tunnel (or rather, the light below my fingers):

I hope that we can finish all of this and push out a first version of Movid for end users soon. And obviously, I want to test my text input widgets on my multitouch table and not in the mouse simulator.

This concludes my work for week one. If you have any questions or are interested in PyMT or Movid, feel free to join our IRC channel at #pymt and #movid on irc.freenode.net.

The awesome PyMT library has just been released in version 0.4.

This is a major release that brings a ton of cool new stuff, including a new animation framework, speed & stability improvements and much more. Take a look at the release notes to see what’s new in this release.

I’m using PyMT for my thesis (see picture above) and I love it. Make sure to check the new website, too! (There’s also a new demo video in the works. I will update this posting as soon as it’s available.

For those of you who don’t know it, if something provides Multi-Touch input methods, it means that you (and potentially an almost arbitrary number of other people) can interact with the same device using as many of your fingers as you like.

This technique is relatively new to most of us and I have been blown away when I first saw a video of someone interacting with a so-called Multi-Touch table:

(If you’re reading this posting via a planet or feed reader, please click this posting’s title to see the videos on my blog directly.)

In case you like the python programming language, you might be as excited as I was to know that there’s actually a library that allows you to write multi-touch software yourself. This library is PyMT. It’s based on top of OpenGL and allows you to deal with multi-touch input events in a nicely abstracted way. PyMT is cross-platform, open-source and actively developed. It comes with many examples, a mouse simulator (in case you don’t have such a table) and (in the development branch) support for the new touchpads found in recent macbooks as well as other types of hardware (HP touchsmart, etc.). Here’s an old demo video that shows what PyMT is capable of already:

I was so impressed by what is possible that I started diving into the matter quite some time ago. I’m even building my own table at the moment. The thesis I’m currently working on also relies on PyMT. If you got an appetite, feel free to join us in #pymt on irc.freenode.net or the mailing lists.

In order to show you how easy it can be, here’s a quick demo I just wrote.

If you are interested in building your own hardware (yes, you can), let me suggest you take a look at the excellent NUI community. They have software, forums and even a book available for free for you to learn and explore.