painting environment for the users, and so on. Among these methods there are some, which try to produce (semi-)automatic painterly eect generators, which is also our target area.
These methods produce painting-like images in a wide variety of styles and ways, each having its pros and cons. Our motivation to work in this eld was to create fully automatic, and totally image structure and feature extraction based SBR methods. Of course other methods have also incorporated some image information to some extent during the rendering process, like edge or texture information, still, there were no methods which would base the control of the entire process on image features and remain fully automatic. This was one area which we targeted.
Another area which we tried to contribute to, was something which most of SBR techniques did not address: what should happen with the generated images, in what format and in what way should they be stored and handled ? Generally the produced images are stored as raster images, or video frames, or 3D meshes in the case of 3D NPR. This can cause a lot of problems, e.g. how should these images be scaled (scaling the raster images with interpolation methods, regenerating the painting into a dierent scale ?), how should they be compressed (since line, or stroke based renderings easily gain compression artifacts), how should the use for animations be handled, and so on. We tried to address these areas and investigated these issues.
1.2 NPR/SBR Overview
In [25] Haeberli introduced a simple interactive painterly rendering program. The technique was to follow the cursor, point sample the color and paint a brush stroke with that color. A stochastic process was responsible for putting the strokes on user-dened positions. Haeberli has also de-scribed a relaxation method for painting based on a kind of Voronoi algorithm. While the later algorithm is more automatic, it gives a quite synthetic result.
In Salisbury et al.'s [91] an interactive technique was introduced which produced high quality pen and ink type of contour-based sketches of model images. A similar silhouette painting algorithm is presented in Northrup et al.'s [79]. In [11] Curtis et al. created a watercolor paint system and demonstrated how computer-generated watercolor can be used as part of an interactive watercolor paint system, as a method for automatic image watercolorization, and as a mechanism for non-photorealistic rendering of three-dimensional scenes.
In Litwinowitcz's work [71] strokes with a given center, length, radius and orientation are generated, adding random perturbations and variation, gradient-based orientation. The author also uses stroke clipping for retaining edges and anti-aliasing. Hertzmann presented an algorithm [31]
1.2. NPR/SBR Overview 5
which painted an image with a series of B-spline modeled strokes on a grid over the canvas.
The paintings were built in a series of layers, starting with a rough sketch then rened with smaller brushes where necessary. Later he implemented a back-end for rendering brush strokes [35]
produced by painterly rendering systems like this or like Meier's one described later. Hertzmann also extended his method to video processing [32].
In Szirányi et al.'s [104,105] the so-called Paintbrush Image Transformation was introduced, which was a simple random painting method using rectangular stroke templates, up to ten dif-ferent stroke scales in a coarse-to-ne way also for segmentation and classication purposes. A multiscale image decomposition method was presented, the resulting images of which looked like good quality paintings, with well dened contours. The image was described by the parameters of the consecutive paintbrush strokes. In Szirányi et al.'s [106] developments were presented, which included the usage of dynamic Monte Carlo Markov Chain optimization at the stroke acceptance step. Faster convergence was supported by a dynamic Metropolis Hastings rule.
In [45] Kaplan et al. presented an algorithm for rendering subdivision surface models of complex 3D scenes using an interactively editable particle system. Geograftals are used (special types of triangle strips), following the works of Meier [76] and Kowalski [63]. In [76] Meier used NPR techniques to create hand-paint like animations. 3D models and particle systems are used, a technique not really suited for 2D painterly processing.
An automatic painting system was introduced by Shiraishi et al. [96] The method generates rectangular brush strokes. Properties of strokes are obtained from moments of the color dierence images between the local source images and the stroke colors. The method controls the density of strokes as well as their painting order based on their sizes. The density is controlled by a dithering method with space-lling curves.
Hertzmann's other work [33] is a relaxation technique to produce painted imagery from images and video. He uses an energy function similar to the one used by the Paintbrush algorithm [104] by Szirányi et al. and minimize it in a quite sophisticated way (Hertzmann's implementation deletes, adds and relaxes the strokes, where the relaxation means the modication of some of the stroke parameters). Modifying this energy function, he is able to follow several painting styles.
In Szirányi et al.'s [105] the relation between painterly rendering and anisotropic diusion (AD) is suggested. Nonlinear partial dierential equations can be used for enhancing the main image structure. When compressing an image, AD, based on the scale-space paradigm, can enhance the basic image features to get visually better quality as Kopilovic et al. show in [48]. Determining the used stroke color based on blurring the underlying area and selecting the most recurring color
1.2. NPR/SBR Overview 6
also shows some compatibility with AD because of its integrating nature.
Hertzmann et al. [34] introduced a method which can produce ltered images based on training images. They also use lters, e.g. they apply an anisotropic diusion or similar lter (a blurring lter) for preprocessing, in order to maintain good contours but eliminate texture. The goal of this work is to try to propagate a painting's style by painting images with stroke parameters extracted from another image.
In [29] Hays et al. present a still and motion picture painting method based on generating meshes of brush strokes with varying spatio-temporal properties. Strokes have location, color, angle and motion properties. By using textures and alpha masks they can mimic dierent styles.
Like in Kovács et al. [50], motion data is used to paint video frames, in this case for propagating the stroke objects along motion trajectories, which in some cases can lead to disturbing eects with strokes moving all over the video frames, giving a uid-like feel to the image. In Park et al.'s paper [81] a brush generation technique is presented where spline brush colors are obtained by generating a color palette from selected reference images, strokes follow image edge orientations, and stroke clipping at edges is also used.
In the work of Gooch et al. [24] a method was presented that produced a painting-like image composed of strokes. This approach extracts some image features, nds the approximate medial axis of these features, and uses the medial axis to guide brush stroke creation. In [43] Jodoin et al. present a hatching simulation stroke based rendering method, N-grams and texture synthesis tools to generate stroke patterns from a training stroke model and use the generated patterns to render 2D or 3D images and models in hatching style.
Santella and DeCarlo presented a method [15] which combines aspects from the approaches of Haeberli [25], Litwinowicz [71] and Hertzmann [31]. They extended their work with a system collecting eye-tracking data from a user in Santella et al.'s [92]. In their system they use curved strokes with single color and paint in details only the picture elements that attracted the user's gaze. We will present a method which has similar goals but does not need external devices, and will control the painting process by automatically estimated relative focus regions [58]. Mignotte in [77]
presents a sketching technique based on Bayesian inference and gradient vector eld distribution models combined with a stochastic optimization algorithm. The stroke model used is a B-spline with12control points pencil stroke.
In [110] Wang et al. presented a video transformation technique based on user performed segmentation on keyframes, interpolating of regions through frames and painterly rendering the segmented areas. Spatio-temporal surfaces are constructed from segmented areas over frames for