Tablet.
Since they were first released, Tablet PCs have gained significantly in popularity; as time goes on, they will move more and more into the mainstream. Currently, Tablet PCs run on a specialized version of Microsoft Windows XP. In the future, all Windows operating systems will support Tablet PC fundamentals, such as digital Ink.
In May 2005, for the first time, notebooks sales surpassed sales of desktop computers.
Mobile most interested in what software can do, which SDKs are available, and how they can use these components in their own applications. In Mobile PC and Tablet PC scenarios, the hardware is equally important, as it is part of what drives new scenarios and opportunities.
One of the obvious next steps is for devices to get both bigger and smaller in physical dimension to supplement the currently available devices. This means that devices will have much larger displays with higher resolutions. On the other end of the scale, devices are getting smaller, moving toward the domain that is currently solely occupied by Pocket PCs.
A development that may be less obvious and less expected is the move toward touch displays. Conventional Tablet PCs use electromagnetic digitizers to sense the tip of the pen on the display. This has advantages. For instance, it allows hardware manufacturers to mount the digitizer (the device that senses the pen) behind the LC display, providing better and brighter display quality.
There are also disadvantages. For instance, you always need a special pen that can be sensed by the EM digitizer. If lost, replacing it can be expensive. It that is meant to select an icon (or interact in another meaningful way), and the tap of a fingernail. This can be handled through vector graphics and exciting new techniques and APIs such as the Windows Presentation Foundation (WPF) formerly known as Avalon. (For more information on WPF, see below.) and its ability to make very complex tasks, such as handwriting recognition, ridiculously easy, has always been nothing short of amazing. Microsoft keeps pushing the limits with a recent Community Technology Preview (CTP), made available in September 2005. This version of the SDK targets the Windows XP-based Tablet PC operating system and it provides some amazing new features, particularly in the area of Ink recognition.
Tablet PCs will be available in new form factors, both larger and smaller than current devices.
The most exciting new feature is Ink Analysis, which is Ink recognition on steroids. Although simple Ink recognition is based on a collection of strokes, Ink Analysis takes a more sophisticated approach, applying both Ink recognition and Ink parsing at the same time. These two aspects are worthy of further explanation.
Ink recognition (or handwriting recognition) refers to the task of recognizing handwriting stored in individual strokes of Ink, and returning it as strings of text. Handwriting recognition is usually tied to a specific spoken language and culture..
Ink analysis also allows for shape recognition, a feature that was previously very difficult to implement. This feature can be used to recognize simple primitives, such as rectangles, circles, different types of triangles, trapezoids, and more (about a dozen, all in all). This ability is very useful as the basis for many non-text recognition models, such as the recognition of flow diagrams and the like. The provided default set of recognized shapes should handle many needs. Figures 1 and 2 show an example of Ink interpreted through Ink analysis and exported to MS Word. And there is an opportunity for third parties wanting to extend that default set.
For some examples of what’s possible with Ink analysis, see the “Ink Recognition and Ink Analysis” article in this issue. “Get Control and Performance with the Real Time Stylus API” article in this issue.)
New in Windows Vista
In Windows Vista, many of the components that were specific to the Tablet PC version of Windows XP have been migrated into the main operating system. Fundamental features, like the ability to recognize or render Ink, are available on all versions of Windows Vista. You can see any version of Windows Vista as a Tablet PC operating system. There are a few components that are specific to actual Tablet PCs. Those fall into the category of accessory applications, such as the Windows Tablet PC Journal, or the TIP (Tablet PC Input Panel). These simply do not make sense unless you have a real Tablet PC.
Ink Analysis is Ink Recognition on steroids.
Windows Vista also sports an area of major change when it comes to handwriting: Vista supports personalization. This means that you can train the handwriting recognizer to recognize your handwriting. In Windows XP, all handwriting was recognized the same way, no matter who used the device. Some people think that recognition in Windows XP gets more accurate over time, but this really is not the case; people tend to adjust their handwriting subconsciously. People get trained by the device. Vista supports the opposite, allowing users to run a wizard that can be used to train the recognizer to various degrees, ranging from simple training for characters or words the recognizer gets wrong regularly (Figure 3), to in-depth training by providing vast handwriting samples (Figure 4). Furthermore, users can send feedback to Microsoft about incorrect recognition results, similar to how users can submit application error information to Microsoft today.
Another area that has been greatly improved in Vista is the TIP (Tablet PC Input Panel). It is now smarter in where and at what size it appears on the screen. It can also be resized, an ability that was sorely lacking in Windows XP. It can appear either as a little icon like in Windows XP, or it can appear as a docked panel that can be customized by the user. Figure 5 shows the new TIP in action.
Writing on the TIP has been improved as well. It now supports more sophisticated scratch-out gestures as well as end-of-pen erasing. The overall input area has been optimized for space, which is especially useful in languages that tend to have longer words and sentences.
The new TIP now also supports auto-complete in applications such as Internet Explorer (Figure 6 and Figure 7). In XP versions of the TIP, when you write a word, such as Microsoft, it doesn’t get communicated to the focused control (in particular, to the address bar in Internet Explorer) until you hit the insert button. Internet Explorer’s auto-complete does not appear before the insert button is tapped. In Vista, all recognized text is sent to IE’s auto-complete immediately, showing the list of sites IE associates with the word right away, making the whole user experience much more satisfying. This works in Internet Explorer, but also in other instances, such as the WinForms 2.0 auto-complete. There is also an API developers can use for this specific feature, if it is needed for text entry in controls where Ink auto-complete is not supported natively.
The overall experience for pen users has also improved in Windows Vista through the addition of “flicks.” Flicks are simple gestures you can perform by quickly drawing a line in a certain direction. Draw a quick line to the right (flick the pen to the right) to perform a gesture that is interpreted as “navigate back”. There are different degrees of flick support users can choose, from relatively simple flicks in two or four directions, to more advanced flicks in eight directions. The action associated with each flick is user configurable (Figure 8).
Programming Vista Tablets
There are changes for Tablet PC developers in Windows Vista. Some are small, such as the addition of APIs or new features in existing APIs. Others are larger and go along with the new UI development paradigm that is inherent to WinFX and WPF (Windows Presentation Foundation) development.
Ink Layout Analysis allows for semantics recognition, such as finding annotations that are anchored to a certain part of a document.
Let’s start out with API changes. The new TIP API affords much greater control over the TIP (Tablet PC Input Panel) and its features, compared to components such as PIP on Windows XP. There is a special Text Input API that can be used to control how text is recognized and inserted by the TIP. This API can be used to suggest alternate recognition results right inside the TIP. A related API is used to control the TIP’s interaction with text controls that support auto-complete (see above).
There is a feature that was lost, corrupt, or inaccessible for some reason, none of the tagging information was communicated to the recognizer. In Windows Vista, the content tagging approach is no longer supported. Instead, this is now handled programmatically through the InputScope property on the UIElement base class. Context tagging is gone, but you can set the recognizer hints in XAML.
The Windows Presentation Foundation
With WinFX, Microsoft introduces a successor to the Win32 API. The public API surface of WinFX is completely managed (written in .NET) and includes technologies such as the Windows Presentation Foundation (WPF, formerly known as Avalon), Windows Communication Foundation (WCF, formerly known as Indigo), and the Windows Workflow Foundation (WWF), among others. Tablet PC-related functionality focuses almost exclusively on user interfaces. For this reason, WPF is most important for our purposes.
In the past, all versions of Windows (including Windows XP), created almost all graphical output using a technology known as GDI (Graphics Device Interface). Although this technology has served us well for a long time, it has its problems. GDI is completely pixel-based. Everything drawn on the screen is created by pixel-based drawing operations that take place on the PC’s main CPU. Unfortunately, pixel-based drawings do not scale well, which means that pixel-based output can not easily adjust to various display resolutions. Also, modern displays have a lot of pixels, which puts significant burden on the CPU, and advanced graphics accelerator cards that are present in almost all computers these days lie dormant.
Some of the downsides of GDI become apparent in applications that require high-performance graphics output, such as games or multi-media applications (videos). For this reason, these kinds of applications do not use GDI, but DirectX, a high-performance interface capable of accessing graphics hardware directly, including the utilization of graphics accelerators known as GPUs (Graphics Processing Units), which are secondary processors optimized towards processing graphics.
The problem with using both approaches is that they do not mix very well. The issue is most easily demonstrated with videos on Web pages. Whenever a Web page with a video is viewed in a browser and the browser window is moved, the video lags behind, although the rest of the window moves very smoothly. This is a result of the combination of two rather different technologies. For Tablet PC developers, this also becomes obvious in the support for Inking. It is possible to collect and render Ink in GDI and it is also possible to collect and render Ink in DirectX. However, it is very difficult to Ink annotate a video and then write beyond the edge of the video and continue the annotation on a GDI-rendered area of the screen.
WPF takes a completely different approach. From the ground up, it was built to be a unified, high-performance graphics API. All output in WPF is based on vector graphics instead of pixels. This has the advantage of resolution independence. All graphical output can be resized or transformed in other ways at a very high level of quality. WPF also utilizes graphics hardware if it is present, making all the CPU power available for other processing.
WPF doesn’t draw screen output in the same way GDI or even DirectX did; instead, it “composes” output. Typical WPF user interfaces are composed of windows, buttons, textboxes, lines, and other WPF controls, such as videos. These controls are themselves composed of primitives defined as vectors (such as lines). The composition engine allows for great flexibility. For instance, a button control can itself be composed of a text label and a video. The same video can be used to compose the contents of a drop-down list, a menu, or any other WPF control.
What is particularly important for Tablet PC developers is that Ink is a native component of WPF. Therefore, every WPF application can support Ink natively, without the need for any additional components. Of course, because Ink is based on vectors, it is a perfect match for WPF’s composition engine and can be drawn very quickly at a high level of quality.
Before you look at your first WPF Ink example, I should discuss one more technology introduced by WinFX: XAML.
XAML is a declarative language that supplements the conventional programmatic languages. In programmatic languages, you instantiate objects, set properties, and call methods in a language such as C# or Visual Basic .NET. In a declarative environment, you use some sort of markup language (XML in this case) to specify the desired result. For instance, the following could be a programmatic approach for creating WinForms UI (as it is found in the designer-generated code region):
Button btn1 = new Button(); btn1.Top = 100; Form.Controls.Add(btn1);
In XAML, the same idea could be expressed as follows:
<Button Name="btn1" Top="100"/>
In many cases, the declarative approach is simpler and more efficient, because you only worry about the “what” and not the “how.” In the programmatic example, the button object had to be instantiated and the developer had to know whether or not the object received constructor parameters. The developer also had to know to add the button to the form’s Controls collection for the form to become functional. In the declarative approach, all of this is left up to the XAML compiler and the developer only concerns herself with the desired outcome.
XAML is a great addition to the developer’s toolbox, with advantages so compelling that practically all WPF examples are based on it. However, it is important to point out that XAML is entirely optional in WinFX, and developers are free to use the more conventional programmatic approach if they so desire.
XAML, although entirely optional, is so convenient that practically all WPF examples are written in this language.
As a simple XAML example, a window containing a few controls can be defined in the following fashion:
<Window x: <Canvas"/> </Canvas> </Window>
The result of this is shown in Figure 9. This code may appear a bit confusing for people unfamiliar with XAML or WPF, but it is actually pretty straightforward. The root tag defines a window as well as some namespaces used by XAML. The window contains a canvas element/control. This means that you want to create an area in your window that is 500x500 pixels-in “logical” pixels, at 1/96tth of an inch- in dimension that you can place other controls on (or in). The nature of this control is such that you can position other controls (rectangles in this case, but it could be anything) within it, providing absolute positions and dimensions.
At this point, you may be wondering why you needed a Canvas, rather than just positioning other controls on the window. The reason is that WPF supports many different ways of positioning controls. The Canvas that contains controls with absolute positions and dimensions is only one example (this is also known as grid layout). Another example is flow layout. Within a flow layout control, all other elements are positioned left-to-right and top-to-bottom, similar to how it is done on Web pages.
What does it take to add Ink functionality to this example? Not a lot! All you need to do is add an InkCanvas object to your composition (or replace the regular Canvas with an InkCanvas). The following example shows a modified version that supports Ink:
<Window x: <Grid> <InkCanvas"/> </InkCanvas> </Grid> </Window>
The result can be seen in Figure 10.
It is important to note that although it is possible to write XAML by hand, most people will be using visual designers, such as the recently announced Sparkle designer that is part of Microsoft’s Expression series of products (). Because Ink controls are part of the native WPF control set, interactive designers like Sparkle support Ink controls (like Ink Canvas) natively.
Pushing Forward
Microsoft is pushing very hard on Tablet PC and Mobile PC technologies as the market for these technologies gains importance. For users, this means new and better devices and software. For developers, there are a lot of opportunities, and the new versions of the Tablet PC SDKs are more sophisticated, in many ways they are easier to use, and, in the case of WinFX and Windows Vista, they are more integrated in all versions of the platform.
To sum it all up: the future for Tablet PC and Mobile PC developers and users looks very bright.