A Homogeneous Algorithm for Motion Estimation and Compensation by Using Cellular Neural Networks

A Homogeneous Algorithm for Motion Estimation and Compensation by Using

Cellular Neural Networks

Alexandru Gacsádi University of Oradea, Romania Department of Electronics and Telecommunications Universităţii Str. 1, 410087, Oradea, e-mail: agacsadi@uoradea.ro

Content

1. Introduction in motion estimation and compensation

2. PDE based or variational computing based images processing 3. Overview on motion estimation methods

4. The CNN implementation of the HS motion estimation method 5. The implementation of the CNN motion compensation

6. Experimental results

7. Conclusions

1. Introduction in motion estimation and compensation

Motion estimation = determine the displacement field

Projection center Pt+1

Pt

y

x z O

y x pt

pt+1

Image plan

3D object at t instant Object 3D at

t+1instant

Vector of

real motion Vector of

projected motion

Motion estimation

Estimated motion field

Motion compensation

Ideal:

difference = 0

Predicted image (compensated)

t t+1

2D Image

Perspective projection of a thorax

Respiring patient Thorax Real 3D object

CT RX image

Tomographic slice of a thorax

Optical case (video camera) Cas RX

Radiographic Projection (superposition of many plans)

Estimated motion ≠ Real motion

2. PDE based or variational computing based images processing

The image processing based on variational computing or PDE have significant benefits both theoretically, precision, flexibility in modelling and in terms of numerical implementation facility.

Consider a grey-scale image: (p,q),

where : R²R, and ={(p,q): p[1,M], q[1,N], M and N  R⁺}.

 





t  



Φ F

,  (1)

where an t is an artificial parameter and F is the operator which characterizes the desired processing algorithm (F: R²R).

Function F depends on the initial image, and its first and second order spatial derivatives.

 





arg ,  (2)

where E is a given energy function, and F the first order derivative of E.

Through minimizing E,  results from condition: F()=0, which is a steady state solution of the:

 

 



 F

t ,  (3)

where t is also an artificially introduced parameter.

The equivalence of the variational method with the PDE method:

 



 

E (4)

 

 

t 





 (5)

 





t  F¹ 





 and F

 

 

t  ² 





 , (6)

A complex image processing is resulted by the combination of these equations as follows:

2

1 F

t F 





 ,  (7)

where  and  R⁺.

The differential equation (7) can be obtained from the minimization of energy

E1+E2  (8)

if F1 and F2 result from the minimization of energies E1 and E2.

With scalar parameters  and  ( and  R⁺), lets us balance the complex processing between the limits described by the initial results.

The image processing method should be based on a smaller number of imposed parameters at the beginning of the algorithm.

The elements deduced from the processed image content should be dominant.

The effective implementation of complex image processing is not easy due to determining of the scalar parameters.

If the number of the energy terms increases too much, the image processing becomes difficult to check, so that there is the possibility of occurrence of large errors.

The effectiveness of these methods depends on: the nature of the images, the model on which the processing itself is done and the scalar parameters.

The major limiting factor of the PDE or variational computing based images processing algorithms is the huge computing power requirements.

The cellular wave computing, as parallel computational devices, offer a great solution to this issue.

3. Overview on motion estimation methods

Block-based methods (correlative methods):

Phase-correlation methods - high noise-sensitivity

Block-Matching - great computational time and bloc effects;

Current image at t instant Target image at t+1 instant

Displacement vector

The bloc in the searching window that

best matches the reference bloc By

Bx

dx_max

dy_max

(x,y)

General hypothesis = illumination preservation Displaced Frame Difference

Φ(x,y,t) Δt)

dy,t dx,y

Φ(x

DFD(x,y)     (9)

where (x,y,t) denote the brightness distribution of the image at the moment t d=[dx,dy]^T is the displacement vector between the moments t and t'=t-Δt

(dx and dy being the displacement vectors on x and y direction, respectively)

Differential motion estimation methods are based on spatial and temporal gradients of a sequence of images. If the brightness intensity of a pixel is not varying in time, then:

dt 0

d t 

(10)

The first order Taylor development of this last relation has as result the “equation of movement constraint” EMC (or “optical flow equation” OFE) that links the spatial and temporal gradients of brightness intensity:

t 0 dt dy y dt dx

x 





 





 





 (11)

0 v

v_x ^y _y ^t

x     

 (12)

where ^x and ^y are the spatial gradients, ^t is the temporal gradient of the brightness intensity: ^{x i}

x   





 ; ^{y j}

y  





 ; ^{t t}

t  







and dt

v_x  dx ,

dt

v_y  dy are the velocities on x and y directions

- unable to estimate great displacements (>3 pixels);

- unable to take into account the motion discontinuities.

Horn and Schunck motion estimation method

Horn & Schunck motion estimation method assumes that all the neighbor pixels, has similar movement (we say that the velocity field is uniform or smooth).

The energy to be minimized:

E E

E²  ²_flow  ²_{uniformity (13)}

2 v 2

v 2

x 2

2 x

smoothness y

v x

v y

v x

E v 

 







 



 







 



 







 



 







  (14)





E²_flow  ^x  _x ^y _y ^{t 2} (15)

4. The CNN implementation of the HS motion estimation method

i x

x  





 ; ^{y j}

y  





 ; ^{t t}

t   







i

x v

dt v

dx   ; ^x v^x_x vⁱ_i x

v  



 ; ^x v^y_x v^j_i y

v  



 ;

j

y v

dt v

dy   ; ^y v^x_y vⁱ_j x

v  



 ; ^y v^y_y v^j_j y

v  





The motion estimation results using the two images, ₁(x,y,t) and ₂(x,y,t+Δt), a two-layer CNN structure:

1 21 11 i 1

i vi A v D D Z

dt

dv      (16)

2 12 22 j 2

j vj A v D D Z

dt

dv      (17)

A² D22

Z²

D11

Z1 A1

D²¹ D¹²

Output 1= vi

Output 2= vj

Layer 1 Layer 2

- polarization images Z1 and Z2:

t 1 x

Z  

t 2 y

Z   (18)

- nonlinear templatesA1 and A2:

0 a 0 0 0 0

A= a 1-4·a a D = 0 d 0 B = 0 z = 0

0 a 0 0 0 0

(19) where the parameter a also includes the  parameter

- NonlinearD-type template D is:

0 a 0 0 0 0

A = a 1 a D_kl= 0 d_kl 0 B = 0 z= 0

0 a 0 0 0 0

(20) - forD¹¹ corresponds d₁₁(v_i)

 

- forD²¹ corresponds i j j j

21(v ) v

d     ; - forD22 corresponds: d (v )

 

Experimental results

Motion to the left

Motion to the right

(b) Motion to down

(c) Motion to up

Initial image (a)

Experimental results

5. The implementation of the CNN motion compensation

Final image Φ^f(x,y,tf)) Initial image

Φⁱ(x,y,tⁱ)

Intermediary images Φ(x,y,t)

- for each pixel it results an estimation of its movement in the both two directions (x,y) of the system of co-ordinates that is attached to image plane;

- the purpose of motion compensation is that based on estimated motion information and starting from a reference image to obtain an estimation of the real comparison image.

As a result of the motion estimation process, a pixel could be stationary or can change its position in one of the eight elementary directions:

N, N-E, E, S-E, S, S-W, W, N-W

After the application of a motion estimation technique, the pixels of the intermediary image frame Φ(x,y,t), that corresponds to any given moment t(ti,tf), could be classified in the following categories:

- pixels of “a” type that has an identical position in the two consecutive images.

These pixels does not change, at a given moment t(ti,tf), neither their positions nor their values;

- pixels of “b” type, that will moves as a result of the fact that the corresponding pixels in the two images that contains the motion information has a value greater than a current elementary value (that could be view as a quantum or a threshold).

- pixels of “c” type are those pixels that will change their values because will be covered by the pixels that will arrive in that position, overlapping the initial pixel:

c(t)=shift(b(t)) (21)

- pixels of “d” type, with unknown values, that are the result of the displacement of “b” type pixels, that liberates a location but there is no pixel arriving in that location.

In each step of the movement, the value of these pixels could be determined through spatial CNN spline-cubic interpolation:





t (

d        (22)

- pixels of “e” type that at the current time during the processing will have the same value as in the initial image Φi, due to the movement of the pixels (arrivals and departures of the pixels). The values of these pixels will be restored from the initial image:

) t ( d ) t ( c ) t ( b ) 1 t ( c ) t (

e      (23)

No

Detection of the pixels that is displacing one step (quantum) in one

of the 8 possible directions

There is still displacing pixels ? Yes

The realization of current and final mask-images in order to determine the pixels of “b”, “d” and “e” type

The increment of the threshold in the image containing motion information

Stop

Start

Shift in the intermediary image of the “b” type pixels with one step in one of the 8 possible directions, according to the motion information

Shift of the pixels in the images that contains motion information with one step in the 8 directions and decrement Determination of the values of “d”

type pixels, through CNN interpolation

Experimental results

(a) (b) (c) (d)

(a) = initial image;

(b) and (c) = intermediary images;

(d) = final image.

Experimental results

Difference between real image and motion

CNN Motion estimation

CNN Motion compensation

Motion Estimation Field

₁ 3

2

ˆ2

 Motion 2ˆ2

Compensated image

8. Conclusions

In the case of serial implementation of a motion compensation algorithm, the processing time depends on the image dimensions.

If the image processing is completely parallel, the total processing time doesn’t depend on image dimensions, being dependent only on the number of displacement steps that has to be performed and thus we can obtain real-time applications with applications in artificial vision and medical imaging.

The CNN motion estimation and compensation algorithm is based on the cost functions minimization resulting nonlinear templates.

The possible implementation of the homogeneous algorithm for motion estimation and compensation is the FPGA based solution.

Thank you for your attention!