fo un da tio n an d sy st em at ic pr oc ed ur es
Mathematical fo un da tio n an d systematic procedures are required for model generation P graph &S graph frameworks are ef fective tools
P- graph & S- graph frame works are ef fective to ol s for modeling and optimization in PNS and scheduling respectively scheduling ,respectively
These frameworks extends optimization to system atic model ge ne ra tio n system atic model ge ne ra tio n Fu rth er de ve lo pm en ti s re qu ire d fo rm od el in g an d Further de ve lo pm en tis required fo rmodeling an d optimization AAcknowledgement cknowledgement L.T . Fan (83) University Distinguished Professor tK St t U i it at K ansas St at e U ni vers ity an d ad The Mark H. and Mar garet H. H li C h i i Ei i H uli ngs C ha ir in Eng ineer ing
SS--graph framework graph framework S-graph t i f ib l l ti ( I 4)
neve rgenera tes in fe as ib le so lu tio n (vs. Issue 4)
does not need any information not included in the pr ob le m (vs Issue 5) pr ob le m (vs .Issue 5)
is ef fective in terms of CPU time t th ti lit
guaran tees th e op tima lit y 71 AAcknowledgement cknowledgement R Adonyi L K al ot ai R .Adonyi B. Bertók G. Biros
L. K al ot ai J. Klemes Z. Kovács Zs. Ercsey L.T . Fan G F
Optimal Scheduling: S-graph Framework 70 Future work (recommendations) Future work (recommendations)
S tru ct ur al ex am in at io n an d de ep un de rs ta nd in g
Structur al ex am in at io n an d de ep understanding
E xp lic it s tru ct ur e r ep re se nt at io n
P ro bl em sp ec ifi c so lv er th at ex pl oi ts th e un iq ue
Problem specific solver th at ex pl oi ts th e un iq ue features of the problem
Ve rif ic at io n of th e m et ho d an d de te rm in e its do m ai n
Ve rif ic at io n of th e method an d de te rm in e its domain O pt im iz ati on oror Search for the optimal solution? Collaboration between the development of computers an d mathematical mode ling co m pu te rs and m at he m at ic al modeling Thank you for your attention Thank you for your attention
3
2. PDE based or variational computing based images processing The image processing based on variational computing or PDE have significant benefits both theoretically, precision, flexibility in modelling and in terms of numerical implementation facility. Consider a grey-scale image: (p,q), where : R2R, and ={(p,q): p[1,M], q[1,N], M and N R+}. t,q,ptFΦ, (1) where an t is an artificial parameter and F is the operator which characterizes the desired processing algorithm (F: R2R). Function F depends on the initial image, and its first and second order spatial derivatives. 6 The elements deduced from the processed image content should be dominant. The effective implementation of complex image processing is not easy due to determining of the scalar parameters. If the number of the energy terms increases too much, the image processing becomes difficult to check, so that there is the possibility of occurrence of large errors. The effectiveness of these methods depends on: the nature of the images, the model on which the processing itself is done and the scalar parameters. The major limiting factor of the PDE or variational computing based images processing algorithms is the huge computing power requirements. The cellular wave computing, as parallel computational devices, offer a great solution to this issue.
2
Contents 1. Introduction 2. PDE based or variational computing based images processing 3. Variational computing based CNN templates design 4. Example for variational computing based CNN templates design 5. CNN image denoising methods 6. Reconstruction of damaged images by using CNN interpolation methods 7. Variational computing based CNN image segmentation method 8. Conclusions 5 A complex image processing is resulted by the combination of these equations as follows: 21FFt, (7) where and R+. The differential equation (7) can be obtained from the minimization of energy E1+E2(8) if F1 and F2result from the minimization of energies E1 and E2. With scalar parameters and ( and R+), lets us balance the complex processing between the limits described by the initial results. The image processing method should be based on a smaller number of imposed parameters at the beginning of the algorithm.
1
Variational Computing Based Image Processing Methods by using Cellular Neural Networks
Alexandru Gacsádi University of Oradea, Romania Department of Electronics and Telecommunications Universităţii Str. 1, 410087, Oradea, e-mail: agacsadi@uoradea.ro 4EMinarg, (2) where E is a given energy function, and F the first order derivative of E. Through minimizing E, results from condition: F()=0, which is a steady state solution of the: Ft, (3) where t is also an artificially introduced parameter. The equivalence of the variational method with the PDE method: dpp2 E(4) pt,pt(5) t,q,pFt1 and t,q,pFt2, (6) 9 - In template design the condition to obtain the final stable state ijx is taken into consideration for every cell where the steady state solution of the CNNs state equation is: ijNCklklklkl,ijNCklkl,ijNCijklkl,ijNCklkl,ijx)y,x,u(DxCzuByA rklrklrklrkl, (13) where ijijtx)t(xlim(14) and 0dtdx ijijxxij(15) - Cost or energy functions are used for complete analytic template design. -The way how energy functions are associated to templates A, B, C or D. In the image pre-processing by using nonlinear feedback template local and also regional properties will be taken into consideration due to the propagation of the effect between the neighbors. - The weighting of cost functions with scalar parameters between 0 and 1. 8
rjl,ikmaxl,kCj,iNr .(10) In the state equation, Aij,kl is the feedback template, the control template is Bij,kl and zij is the bias value. In relation (9) without the last two terms we call it “standard” CNN dynamics. The PDE or variational based template design is possible if the following design constrains are respected. - For ensuring network stability and maintaining it in any situation, the values of state image xij and the values of the output image yij, in the linear domain [-1,1], the necessary and sufficient condition are: (11) 1)y,x,u(DxCzuByA rklrklrklrklNCklklklkl,ijNCklkl,ijNCijklkl,ijNCklkl,ij - In this case, the output is identical with the network’s state: ijijxy(12) 73. Variational computing based CNN templates design The templates design based on variational calculus allows extension of the CNN processing methods for images and extends the areas of applicability of them. It results the possibility of implementation and experimental verification, in real time, of various mathematical models for image processing. The normalized first order CNN equation: (9)
rklrklrklrklNCklklklkl,ijNCklkl,ijNCijklkl,ijNCklkl,ijijl)y,x,u(DxCzuByAxx Nj1;Mi1for; where, i and j represent spatial coordinates; M and N are the dimensions of the network; ukl is the input of this cell, xij represents the state value of the CNN cell and ykl is the output. The cells having the notation Ckl are the neighbor cells for the Cij, namely CklNr, where Nr is the neighborhood r radius:
12
0)(E3; ie (21) 020)(66661j,1i1j,1i1j,1i1j,1i1j,ij,ij,1ij,1ij,i 020)yyyy(y6y6y6y6y1j,1i1j,1i1j,1i1j,1i1j,ij,ij,1ij,1ij,i 0uB rklNCklkl,ij0zij0xC rklNCklkl,ij0)y,x,u(D rklNCklklklkl,ij
1,10,11,1
1,00,01,0
1,10,11,1 aaaaaaaaa A 0yAyyAx rklrklNCklkl,ijijNCklkl,ijij(27) 0yayayayaya yayayayayyAy j,i0,01j,i1,01j,i1,0j,1i0,1j,1i0,1 1j,1i1,11j,1i1,11j,1i1,11j,1i1,1j,iNCklkl,ijijrkl
11
4. Example for variational computing based CNN templates design Let’s consider the cost function care assures cubic spline interpolation, as: dpdqqqp2p))q,p((E q,p 2 22222 2
Ω, (16) (p,q) is a gray-scale image, where : R2R, and ={(p,q): p[1,M], q[1,N], M and NR+}. (i,j), resulting after spatial meshing with a sampling step h, identical on lines and columns:where, : R2R, Nj1;Mi1for, hiand hj(17) M 1i
N 1j
2 2j,i2j,i22 2j,i2 3jji2i))j,i((E(19) 10
- For the CNN templates designing the zero-flux condition should be use, because in this way it is not necessary to know initial boundary conditions. - In some applications other conditionings are included that can help design templates and solving those applications. - Also, the hardware implementation takes into consideration that the obtained templates will be used on digital emulator type cellular wave computing structures implemented on FPGA It is characterized by high degree of generality in relation to certain types of dedicated structures for signal processing. This mode of implementation allows not only the use of linear templates, including type D, but also the development of multi layer structures. 15 The energy function introduced by Rudin-Osher-Fatemi: dpdqdpdq2q,p0q,p
20 ΩΩRUOSFAE(3) tem.osrufa0000d0000 Dand 0a0a1a0a0 A (4) where
1,0;xxsgnaklij and
1,0;uxdklij, with (B=0, z=0). Using the energy function proposed by Chan and Esedoglu: dpdqdpdq q,p0q,p0 ΩΩCHESE(5) ches.tem0000d0000 Dand 0a0a1a0a0 A (6) where:
1,0;xxsgnaklij and
1,0;uxsgndklij, with (B=0, z=0). 145. CNN image denoising methods In fact denoising is the recovery of an original image 0, from the noisy input . Homogeneous regions separated by edges must compose the recovered image. The linear degradation model: 0(1) where 0 is the original image, the observed and damaged image, represents the random additive noise. In image denoising the minimized energy has to contain two terms: SmoothenesfidelityData210ΦEαE)(ΦE(2) The two terms are weighted by using scalar parameters and ( and R+). 13
05.0aaaa1,11,11,11,1; 3.0aaaa0,11,01,00,1, 0a0,0 emaintpol3.t0.05-3.00.05-3.003.00.05-3.00.05- A 18
(a) (b) (c) (d) Figure 1. Denoising of an ideal image, how well the edges are preserved: a) input image without noise; b) error image ER, by using osrufa.tem; c) error image ER, by using ches.tem; d) error image ER, by using vsgn.tem. 17
The error ER (precision criterion) which occurs between the restored image, , and the original image 0, can be quantitatively measured, based on relation: Nj1;Mi1,)j,i()j,i(ER20 N*MΩ(9) In order to evaluate the behavior of the method in edge preservation, the error images ER, will be defined, based on relation: Nj1;Mi1),j,i()j,i(0ER(10) Table I. The Error of the CNN image denoising methods Errors (ER)TemplateInput image without noiseInput image with noise osrufa.tem2.1713.01 ches.tem0.518.25 vsgn.tem0.204.84 16
Using the proposed energy function: dpdqdpdq q,p0q,p0 ΩΩVSGNE(7) vsgn.tem0d0d0d0d0 Dand 0a0a1a0a0 A (8) where:
1,0;xxsgnaklij and
1,0;uxsgndklij, with (B=0, z=0). The weighting of cost functions, with scalar parameters, and , between 0 and 1. In the experiments presented below and were set to 0.01. The performances of image denoising techniques are difficult to evaluate. They can be compared using some experimental quantitative measures, but the best evaluation method seems to be the visualization of the effects on natural images. To estimate the efficiency of image denoising methods it has to be evaluated how the method works on a noisy image and what is the result on a noiseless image.21
6. Reconstruction of damaged images by using CNN interpolation methods Image inpainting is an interpolation problem where an image with missing or damaged parts is restored. The most often used image inpainting applications are for pictures or films known or damaged partially. In the recontruction of damaged images by using CNN methods it is necessary to use a mask image that does not change the elements of the image that are known at the beginning, but allow the computing of unknown elements. The existence of a mask image presumes that the user knows the positions of the elements that need to be computed. The portions that correspond to the damaged regions have zero as the initial value. 20
(a) (b) (c) (d) Figure 3. Denoising of an CT image: a) input image with noise; b) output image by using osrufa.tem; c) output image by using ches.tem; d) output image by using vsgn.tem. 19
(a) (b) (c) (d) Figure 2. Denoising of an image: a) input image with Gaussian white noise, zero mean and 0.04 variance; b) output image by using osrufa.tem; c) output image by using ches.tem; d) output image by using vsgn.tem. 24 The above is the evaluation basis of the methods proposed for the restoration of the damaged image. Let us denote the error ER (precision criterion) which occurs between the restored image, , and the original image 0. This is the reason why in the following examples, the damaged images IN were obtained from original real images where the values of some pixels are zeroes. This way, the errors that result from the use of different methods can be quantitatively measured as: Nj1;Mi1,)j,i()j,i(ER20 N*MΩ(6) Propagation distance (radius) can be quantitatively analyzed with error images obtained by using the following relation: Nj1;Mi1),j,i()j,i(0ER.(7) 23
Based on cost functions minimization (1-4), templates containing only A feedback term resulted, the others being zero (B=0, z=0, C=0, D=0): (5) tem.1polinta125.0125.0125.0125.00125.0125.0125.0125.0 A1 ; tem.2polinta025.0025.0025.0025.00 A2 ; tem.3polinta0.05-3.00.05-3.003.00.05-3.00.05- A3 ; tem.osrufaTV0a0a0a0a0 A TV where
1,0;xxsgnaklij Whatever the chosen image restoring method is, the precision of the restoration should be as good as possible, but at the same time, it is desirable that the dimensions of the image’s holes that are restored should be as large as possible, that is the interpolation propagation distance should be great. 22For two dimensional signal interpolation the following cost functions can be used:
dpdqΦΦE Ωqp, 2N1r
(1) where r represents the r radius image from every pixel’s neighborhood; dxdyqp))q,p((E q,p22 2
Ω(3) The smoothing energy can also be the total variation integral from the Rudin-Osher-Fatemi image denoising model: TVdpdq q,p0Ω4E(4) 27
The resulting template is as follows, named nel_aintpol3.tem: 3nel_ainpol0d0d0d0d0 Dand 0.05-3.00.05-3.003.00.05-3.00.05- A (8) where
1,0;xxsgndklij. bac Figure 6. a) Image to be restored IN; b) mask image; c) output image OUT using nel_ainpol3.tem. 26d e
c g
ba f Figure 4. a) Original image 0; b) image to be restored IN; c) error image ER1 for ainpol1.tem; d) error image ER2 for ainpol2.tem; e) error image ER3 for ainpol3.tem; f) error image ER4 for osrufaTV; g) error image ER5 for nel_ainpol3.tem. abcb Figure 5. a) Image to be restored IN b) mask image; c) original image. 25
In order to analyze and illustrate the efficiency of the proposed templates for restoring images with large holes, a synthetic image was used. The two holes are placed in uniform noiseless regions, one with positive value (0.70) and the other one with negative value (-0.70), the image’s pixels having standard CNN domain values [-1, +1]. Table II. The Error of the CNN image inpainting methods Errors (ER) TemplateImage with large holesImage where the unknown regions are small aintpol10.02190.0230 aintpol20.05580.0204 aintpol30.08230.0198 osrufaTV0.00920.0304 nel_ainpol30.00410.0200
30
The process used to perform image segmentation varies greatly depending on specific application, imaging modality and other factors. Currently there isn’t a specific general method of segmentation to produce acceptable results for all types of images. Each of these methods have their advantages and disadvantages, as some algorithms optimized for a particular hardware structure can no longer work as well on another structure. However, some methods available for relatively large areas, which are optimized for specific applications, can often produce better results by taking into account previous knowledge. The energy function introduced by Mumford and Shah:
dxdydxdyE RRMS202, \(1) where R is a connected, bounded, open subset of R2, 297. Variational computing based image segmentation The problem of images segmentation is an important stage in an automatic diagnosis system. For grey scale images, one may classify the segmentation methods into edge-based methods and region-based techniques. Region-growing methods can be made less sensitive to noise than simple edge-based or morphological methods, but they may become extremely computationally complex for even simple rules. Curve evolution, active surfaces, statistical approaches, and variational energy methods have become popular approaches in this field. The majority of these methods prove remarkable performances when the processed image corresponds to the model of the algorithm but fails or gives significant artifacts otherwise. 28
a c
b Figure 7. a) Original image; b) image to be restored IN; c) output image OUT using nel_ainpol3.tem. 33
where R denotes the boundary of R and n denotes the direction normal to R. In the case of CNN processing a multi-layered structure is necessary. The two main layers are necessary to obtain and K. The other layers are needed to calculate some components of the main layers. Each energy function contains weighted smoothing terms (2from (3) or 2Kfrom (4)) and weighted fidelity terms, or terms for edge conservation, (20 from (3) respectively edge calculation 2 1K
from (4)). Solving this system of partial differential equations includes an number of operations that can be effectively solved by parallel processing structures, including CNN methods. Even in the case of strict implementation of this model by numerical methods, 32
Keeping K fixed, the first equation minimizes:
RdxdyE2022KK1(3) Keeping fixed, the second equation minimizes:
0 is the original image (the feature intensity), is a curve segmenting R, is the smoothed image R2\, is the length of and and are the weights, scalar parameters, ( and R+). Minimizing this classical functional requires estimating two processes, the continuous segmented field, , and a binary edge process, . The deterministic edge detection based, region based, active contour based and stochastic methods are subsets of the more general problem of variational functional minimization. Since it is difficult to apply gradient descent with respect to , Ambrosio and Tortorelli replace by a continuous variable K and obtain:
RATE2022K1,dxdy 2K222(2) a segmented image estimate and edge process estimate K. 36
This function of energy (VSGN), is proposed to improve edge conservation behavior of CNN image filtering. To determine image edges would justify an energy function dependent on 2: RdxdyE2K(12) 0b0b0b0b0 B(13) where
1,0;uubklij with (A=0, z=0). in the segmentation methods analyzed in this paper avegrad.tem template was used for edge detection. bbbb0bbbb B where
1,0;kluijub with (A=0, z=0). (14) 35To determine the noise filtered image, the following energy functions will be used:
RLdxdyE202(8) 0d0d0d0d0 D 0a0aa41a0a0 A(9) where 1,0;25.0a and
1,02;kluijx2d, with (B=0, z=0). RVSGNdxdyE0)((10) 0d0d0d0d0 D 0a0a1a0a0 A(11) where
1,0;xxsgnaklij and
1,0;uxsgndklij, with (B=0, z=0). 34resultant accuracy is modest. This is mainly a consequence of smoothing imperfections with a function of the form 2. In addition, by implementing the image processing CNN on 8-bit digital structures, solving these equations introduces approximations and additional errors. Based on the above, as compromise solution, currently is justified the evaluation of some algorithms that eliminate the interaction between the two main layers, so basically the two partial differential equation can be solved successively. For variational computing based template design, using the standard CNN types for gray-scale image processing, all design constrains mentioned are respected. For variational computing based CNN image segmentation, in the following it will be examined the behavior of energy functions to determine the two images, the filtered image and the edged image K. The estimate segmented image, , will result from the fusion of these two images.
39
8. Conclusions Complex PDE based mathematical models are available now for image denoising, image segmentation, damaged images reconstruction. Comparing with other approaches image processing based on variational computing or PDE have significant benefits both theoretically, precision, flexibility in modelling and in terms of numerical implementation facility. These image processing methods are sometimes difficult to implement in real-time even if a large serial processing computing power is available. The cellular wave based parallel processing ensures computing-time reduction. The proposed method based on variational computing design methods of the templates offers a better efficiency in terms of image denoising and edge preservation, comparing to other previous variational computing based CNN methods. 38
(a) (b) (c) (d)Figure 9. Variational computing based CNN segmentation of real CT image: a) input image with noise; b) filtered output image, , by using vsgn.tem; c) output image, K , after edge detection by using avegrad.tem; d) segmented output image.37
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j)Figure 8. Variational computing cased CNN segmentation: a) ideal image without noise; b) output image after edge detection; c) result of segmentation of the noiseless image; d) input image with Gaussian white noise, zero mean and 0.04 variance; e) filtered output image, , by using dn_Laplace.tem; f) output image, K, after edge detection, by using avegrad.tem; g) result of image segmentation on the noisy image by using dn_Laplace.tem; h) filtered output image, , by using vsgn.tem.tem; i) output image, K, after edge detection by using avegrad.tem; j) result of image segmentation on the noisy image by using vsgn.tem. 41
Thank you for your attention! 40
8. Conclusions The proposed variational computing based CNN methods for image processing, use nonlinear templates. The hardware implementation takes into consideration that the resulted templates will be used on digital emulator type cellular wave computing structures implemented on FPGA characterized by high degree of generality in relation to certain types of dedicated structures for signal processing. The template design based on variational calculus allows extension of the CNN processing methods for images and extends the areas of applicability of them. It eases the possibility of implementation and experimental verification, in real time, of various mathematical models for image processing, if there is a large elementary templates library, which can be easily used on an accessible infrastructure. 11/28/2012
Q -E ye chip
•176 x 144 sensor-processor array •Analog processor in each cell •3x3 morphologic processor in each cell •Diffusion network •8 bits accuracy •50 GOPS •60mm2die area in 0.18um CMOS technology •<300mW typical power consumption AnaFocus, Seville-SpainQ-Eye chip architecture
Eye-RIS system