Parascript, Inc. - Natural Handwriting
Recognition and Beyond
by James Sneeringer
WAVE0222, 6/14/02
Few companies have the longevity in pen computing, or expertise
in
handwriting recognition, that the team at Parascript has. When this team
formed the core of technology at ParaGraph International, they developed
the recognition software of the Apple Newton--the first commercially
available natural handwriting recognition engine. Today, the same
technology, named XR, forms the basis for the handwriting recognition
in
PocketPC, and for Parascript software used in the recognition of forms,
handwritten checks, and postal addresses.
The Parascript Pen&Internet division is developing the
next generation of
handwriting recognition software, called riteScript. Currently, though,
the division's main product is riteMail, an application for creating
handwritten notes and drawings on a computing device (known as electronic
ink) and sending them to any email address. riteMail is currently in beta,
and includes riteShape, a technology that recognizes and automatically
perfects common shapes like circles, rectangles, triangles, and arrows.
Users can try out riteMail, as well as the riteShape and riteScript
recognition engines, online at:
http://www.ritemail.net/
With the introduction of mobile pen devices such as Anoto,
and the
Microsoft Tablet PC initiative, handwriting recognition is once again
in
the spotlight. To better understand the history and current state of
recognition technology, the WAVE Report recently attended a Parascript
presentation at the Business Forms Management Association Symposium in
Baltimore, and interviewed Dr. Leonid Kitainik, general manager of the
Pen&Internet division.
From ParaGraph to Parascript
Parascript began life as the company ParaGraph International,
which
developed the handwriting recognition features of the Apple Newton.
Following the Newton, their next handwriting recognition application was
CalliGrapher, whose customer base eventually grew to over 1 million end
users and 35 OEMs. ParaGraph and many of its technologies were bought
by
SGI, which created a subsidiary later acquired by Vadem. Vadem
subsequently sold CalliGrapher technologies to Microsoft. Today, the
CalliGrapher technology forms a part of Transcriber, the handwriting
recognition software included in all Pocket PCs and Tablet PCs.
Apple recently announced that the next version of the Mac
OS will have
native support for pen input, known as Inkwell. While Apple has announced
that this is based on “Newton Technology,” Dr. Kitainik stated
that
Parascript has not been working with them. While Inkwell is probably
based on the ParaGraph Newton software, Apple has apparently continued
the
development in-house.
Parascript Text Recognition
At the Business Forms Management Association (BFMA) Symposium
in
Baltimore, the WAVE Report sat in on a handwriting recognition session
given by Parascript. The session, entitled "Recognizing the Truth
About
Character Recognition," highlighted the issues and technology involved
in
recognizing natural handwriting. The presentation was given by Jim Terry,
Director of Total Recognition Services (TRS) for Parascript. The TRS
division of Parascript provides recognition tools and services for
companies that process printed forms, such as credit card applications.
Computers can receive and manage written language in one
of two ways. It
can receive it as ASCII characters, which are representations of the
numeric codes understandable by computers. ASCII text can be parsed by
a
machine, which allows for editing, and features such as spell check,
searching, indexing, etc. Or, it can receive the image of hand-written
text. Because such language does not have a machine code like ASCII text,
it is not parsable by the computer. It is treated and managed as a single
picture, and no features are available for managing the writing within
the
picture.
Character or handwriting recognition software uses the computer
to
translate the image of handwriting, into ASCII text. There are three
levels of recognition software. In increasing difficulty of
accomplishment, they are:
OCR - Optical Character Recognition
The recognition of machine-printed text. An example is
the numbers printed
on the bottom of checks. OCR for the most part is character-based pattern
matching, and often the printed characters are designed to maximize
recognition rates. Some OCR engines are capable of poly-font recognition,
while other, simpler ones can only recognize certain fonts.
ICR - Intelligent Character Recognition
The recognition of constrained, hand-printed characters.
Constrained text
entry fields consist of a row of boxes or combs, one per character,
rather
than an open area or line for writing. ICR is significantly more
complicated than OCR, since humans never write the same letter the same
way twice. Still, the basis for ICR is whole-character pattern
recognition. However, because of the huge variation in how each character
can be written, ICR engines rely on neural networking rather than straight
comparison. Neural networking is a complex technique that allows the
recognition software to learn and adjust to variations over time. It
also
contains rules that pertain to each set of patterns (each character),
which can be amended over time to increase the flexibility.
Important issues with ICR include characters that touch,
such as
distinguishing between "cl" and "d", and the subtle
differences between
character pairs such as "T" and "Y", "G"
and "6", etc. Adjacent characters
that touch, in particular, can cause many problems, since ICR engines
look
at only one character at a time. If two letters are touching, ICR engines
have a very difficult time parsing them into two separate letters. The
engines typically treat it as one complicated, unrecognizable character.
Good constraints on the entry field are an important part of successful
ICR. Form design has a large effect on the success rate of ICR.
NHR - Natural Handwriting Recognition
The recognition of unconstrained, natural handwriting,
print or cursive.
This is the recognition of any type of written language, cursive or
print.
Rather than character-based pattern comparison, Parascript's original
NHR
technology relies on a multi-layer process of recognition that includes
context in the analysis.
NHR is the result of two technology innovations:
1. The ability to break down all handwriting into a
unique set of cursive
strokes that characterize all handwriting.
Parascript has designed a set of 64 handwriting "elements"
known as the XR
Elements. These 64 elements can be combined to form any character
or set
of characters in cursive handwriting.
2. The ability to execute linguistic and context rules
on the fly.
Human processing while reading is parallel, rather than linear. In
a
linear process, operations are executed one at a time in a set order,
so
mistakes in earlier processes carry through to all processes that
follow.
In parallel processing, operations are carried out simultaneously,
so that
each process can check the others in real time. Parascript NHR recognition
processes work in parallel to improve success rates.
As NHR software scans a piece of handwriting, the following
is occurring:
- Cursive lines are broken down into series of XR elements,
which are
compared against a database to determine words or phrases.
- Printed characters are matched against Graph Based
Character Classifiers
(sample character patterns) to determine characters and words.
- As recognition occurs, a dynamic process immediately
begins matching
against a vocabulary database.
- The characters and words are matched against user-set
criteria. One
would not expect a letter, for instance, in a zip code field.
- The characters and words are matched against the surrounding
context.
The processes occur continuously and simultaneously as
a piece of
handwriting is analyzed.
Pen&Internet Technologies - The Next Step
Our interview with Dr. Leonid Kitainik, General Manager
of the
Pen&Internet division of Parascript, lasted 1.25 hours and covered
a
tremendous amount of ground. Dr. Kitainik joined ParaGraph International
in 1993.
Today, the main product of Pen&Internet is riteMail,
used for creating,
storing, and exchanging electronic ink. riteMail runs on multiple
platforms, and includes riteShape’s shape recognition and correction.
In
the future, riteMail will serve as the infrastructure and interface for
what Dr. Kitainik called full notes recognition, for pen-based mobile
devices such as PDAs and convergent cell phones. riteScript and riteShape
will be part of the technology, applications and services.
riteScript Natural Handwriting Recognition Technology
Dr. Kitainik stressed the term “natural handwriting
recognition."
Parascript defines natural handwriting recognition (NHR) as the ability
for a computer to recognize and convert to text any freehand writing,
cursive or print, unconstrained by any boxes or combs. Dr. Kitainik stated
that most commercially available natural handwriting software is based
on
ParaGraph or Parascript technology.
Parascript Pen&Internet NHR does not look at characters;
it looks at words
and phrases. Dr. Kitainik pointed out that people do not read or write
characters, they read and write words and phrases. He stated that that
should be the goal for handwriting recognition software as well.
CalliGrapher (and now Transcriber), as well as other Parascript
applications such as TRS, uses the XR elements to break cursive words
down
into a linear series of elements. Each series of elements then represents
a word or phrase that can be matched against a database of expected or
common words or phrases.
riteScript, however, does not use the XR technology. For
riteScript, Dr.
Kitainik stated that the Pen&Internet Division have developed a new
recognition technology, based on a new mathematical foundation. XR, he
said, is limiting. It attempts to represent the inherently ambiguous
nature of handwriting by squeezing each word into a linear sequence of
elements. The new technology creates a much broader interpretation of
the
handwriting strokes, to leave more options available during the lexical
phase of analysis.
riteScript's new NHR is much more dependent on contextual
information than
the XR-based system. As each word is processed, the NHR engine begins
comparing it to multiple databases of known or expected words and
phrases--a process known as lexical support. Since the interpretation
of
the strokes is much broader under the new riteScript NHR engine, Dr.
Kitainik stated, it can achieve much better results with good lexical
support than the XR system. He added that the new technology has greater
much lexical support than older recognition engines, including multiple
lexical sources that operate in parallel.
Again like a human, the software attempts to use the context
in which
strokes appear to determine which word or character was intended. The
engine continuously matches its interpretation of strokes with multiple
sources of commonly used phrases or constructions. In some cases, it is
also possible for the user to provide a context--such as when filling
out
a form. Dr. Kitainik stated that for the fields of a form, the lexical
context is typically known, and he expects that the riteScript success
rate will soon be able to routinely achieve at least 90% word accuracy
when recognizing form inputs.
Context can also be utilized in free form input. He used
the example of an
e-mail address hurriedly scribbled down. Since an e-mail address consists
of a known structure, with an “@” character and at least one
“.”,
riteScript can more easily identify it as such. The engine can then parse
it and interpret it more accurately by comparing the handwritten address
against known domain names and extensions.
"Full Notes Recognition" Technology
Pen&Internet has extended the concept of recognition
beyond handwritten
text:
The riteShape recognition engine interprets drawing strokes,
and converts
them to "perfect" computer shapes. For instance one may draw
a circle by
hand, and the software will convert it to a perfect circle. It works
even
with overlapping or connected shapes.
Autoformatting detects the angle at which text is written
and can either
correct it to straight, or preserve the angle after conversion to ASCII.
In addition it can sense and generate formatting features such as indent,
bullets, charts, etc.
The goal that Dr. Kitainik expressed is a pen interface
that senses the
intent of the user, and accomplishes it automatically. He called this
"full notes recognition"--it goes beyond handwriting to interpret
the
entire page and produce the desired result. His example was a person
creating a PowerPoint presentation by hand. As they write out charts and
bullet points with the pen, it converts the writing to text, and
transforms the hand formatting into standard application formatting. As
they draw out an organizational chart by hand, the boxes and circles are
converted to perfect shapes, the text is converted and formatted, and
the
lines and arrows are converted to uniform shapes. The user is presented
with a simple interface, supported by very complex background software.
Consider the example of using a current applicaton to draw
a circle with
text inside it. The user must go to the application tool bar and select
the circle tool. They must pick where the circle begins, and how big it
will be. Then they must select the text tool. They must determine where
the text will begin and how big it will be. Then they type what they want.
With full notes recognition using Pen&Internet’s technology,
the user
would simply draw the circle where they want it, write what they want
as
big as they want, and the strokes are converted to computer shapes and
text automatically.
Dr. Kitainik referred to the second scenario as a "natural
pen interface."
The vision is an interface that begins with a blank screen and interprets
your intent as you draw and write. This is the opposite of most
applications today. Modern applications provide the user with a multitude
of tools up front, and the user must explicitly select everything that
occurs. The natural pen interface presents the user with freedom of
expression, and from that, interprets intent and enacts the appropriate
tools.
According to Dr. Kitainik, full notes recognition is a "very
heavy"
technology. The context detection is extremely challenging, and the
lexical support required--in this case databases of common usages for
symbols, graphic arrays, charts, etc--is vast. While the handwriting
recognition or shape recognition on its own is small enough to run on
current mobile platforms (newer PDAs and phones), full recognition is
not.
It must run on a server, and Dr. Kitainik sees it as a back-end service
that users or carriers would subscribe to.
We pointed out--the question of user expectations looms
over this project.
The Newton faced criticism when the handwriting recognition did not live
up to over-inflated expectations. Dr. Kitainik said that they must very
carefully weigh the current capabilities of hardware, software, and
communications against what they would like to accomplish. He stated they
are trying not to over-hype this capability. No recognition system, human
or computer, will ever be 100% accurate. He pointed to the success of
CalliGrapher as an example of success with recognition.
The challenge is in very carefully understanding what people
"mean" to
do--what they intend with their strokes. The context support must be based
on what people do most often with certain stroke patterns, but also leave
the capability for new applications and variations. Dr. Kitainik stated
that this is very sophisticated and challenging work, since it can so
often be subjective. But, he stated, he does not believe in a totally
free, user-defined interface. He believes the best answer is a well
thought-out, simple interface that does not present the user with too
many
options at once.
Is this a mass-market technology? We mentioned the poor
history of pen
products. He replied that there is a user base of PDAs of about 16 million
worldwide--all pen-based devices, most with electronic ink capabilities
and some recognition technology. Based on that and the PDA market
forecasts, he does believe that a pen interface can be mass-market.
Development Status
The Pen&Internet products are in various states of development:
- riteMail - Currently in an extended public beta, expected
to ship as a
commercial product by the fall
- riteShape - Version 1 released, nearing beta version
for the second
release
- riteScript - Just made major jump to second beta version.
Expect
something solid by end of the year.
- Autoformatting - Well into beta prototypes.
Business Model
Dr. Kitainik sees the ideal business model for riteMail
as a combination
of OEMs, mobile carriers, and Pen&Internet's own services.
- OEMs will license riteMail for bundling with products,
normally paying a
license fee to Pen&Internet.
- Carriers will include riteMail in their service offerings,
sharing
revenue with Pen &Internet and OEM’s.
- Carriers or subscribers will pay a subscription fee for
the back-end
service of full recognition.
For the time being, only riteMail client software, including
riteShape
technology, is a small enough program to run directly on PDAs. Depending
on the power of the mobile unit, or its connectivity to a back-end server,
riteScript handwriting recognition will also be available. Dr. Kitainik
sees these as the basic features of riteMail in its initial release, with
the more advanced full notes recognition available as a value-add service
in a future release. Eventually, as hardware capabilities catch up, full
notes recognition may run directly on PDAs as well, and would generate
license fees from application or OS developers.
We asked--isn't it hard to get carriers to share revenue?
He answered:
"profoundly difficult." But, he sees it as inevitable that carriers
will
be forced to open their revenue streams to 3rd party developers. He stated
that they cannot survive in the future without data services, and 3rd
parties are an integral part of providing those. "It is the only
way 3G
will happen." He cited Australia and some European companies as examples
of providers opening their billing structure to outside providers.
Parascript Vision for the Future
PDA and cell phone units continue to increase in power and
capability, and
most wireless operators are in the process of rolling out data services.
Parascript believes their electronic ink and natural pen interface
technologies are poised for breakthrough growth and mass market
acceptance. They see full notes and forms processing with advanced
recognition, as a bridge between the worlds of electronic ink, and
conventional text and multimedia documents. The goal is ease of use for
taking, storing, and exchanging freehand notes in digital ink, with
recognition and conversion accomplished in the background and always
available. The focus is on a natural pen interface--freehand entry, with
the software complexity shifted to the background.
http://www.parascript.com/
|
