A COMPREHENSIVE FRAMEWORK FOR STROKE TRAJECTORY RECOVERY FOR UNCONSTRAINED HANDWRITTEN DOCUMENTS

Abstract

For a long time, handwriting analysis, such as handwriting recognition and signature verification, has been an active research area. There are two categories of handwriting, online and offline. Online handwriting is captured in real-time on a digital device such as a tablet screen with a stylus pen. In contrast, the handwritten text scanned or captured by a camera from a physical medium such as paper is referred to as offline handwriting. For offline handwriting, the input is limited to handwritten images, making handwriting analysis much more difficult. In our work, we proposed a Stroke Trajectory Recover (STR) for offline and unconstrained handwritten documents. For this purpose, we introduce large-scale word-level annotations for the English handwriting sampled from the IAM-online dataset. The current STR architectures for English handwriting use lines of text or characters of the alphabet as input. However, a word-level STR method estimates loss for each word rather than averaging DTW loss over the entire line of text. Furthermore, to avoid the stray points/artifacts in predicted stroke points, we employ a marginal Chamfer distance that penalizes large, easily noticeable deviations and artifacts. For word detection, we propose the fusion of character region scores with bounding box estimation. Since the character level annotations are not available for handwritten text, we estimate the character region scores in a weakly supervised manner. Character region scores are estimated autonomously from the word’s bounding box estimation to learn the character level information in handwriting. We propose to fuse the character region scores and images to detect words in camera-captured handwriting images. We also propose an automated evaluation to check the quality of the predicted stroke trajectory. The existing handwriting datasets have limited availability of stroke coordinates information. Hence, although the proposed system can be applied to handwriting datasets without stroke coordinates information, it is impossible to evaluate the quality of its predicted strokes using the existing methods. Therefore, in our work, we propose two measures for evaluating the quality of recovered stroke trajectories when ground truth stroke information is not given. First, we formulated an automated evaluation measure based on image matching by finding the difference between original and rendered images. We also evaluated the preservation of readability of words for original and rendered images with a transformer-based word recognition network. Since our proposed STR system works with words, we demonstrate that our method is scalable to unconstrained handwritten documents, i.e., full-page text. Finally, we present a probabilistic diffusion model conditioned on handwriting style template for generating writing strokes. In our work, we propose to learn the localized patches for handwriting style features from multiscale attention network. The multiscale attention network captures fine details about local character style and global handwriting style. Moreover, we train our diffusion model with the Dynamic Time Warping (DTW) loss function, along with the diffusion loss, which eliminates the need to train any auxiliary networks for text or writer style recognition and adversarial networks.