Ground state solutions for nonlinearly coupled systems of Choquard type with lower critical exponent

Ground state solutions for nonlinearly coupled systems of Choquard type with lower critical exponent

Anran Li

, Peiting Wang and Chongqing Wei

School of Mathematical Sciences, Shanxi University, Taiyuan, People’s Republic of China Received 13 February 2020, appeared 20 September 2020

Communicated by Petru Jebelean

Abstract. In this paper, we study the existence of ground state solutions for the fol- lowing nonlinearly coupled systems of Choquard type with lower critical exponent by variational methods

(−_∆u+V(x)u= (I_α∗ |u|^Nα+1)|u|^Nα−1u+p|u|^p−2u|υ|^q, inR^N,

−∆υ+V(x)υ= (Iα∗ |υ|^Nα+1)|υ|^Nα−1υ+q|υ|^q−2υ|u|^p, inR^N. Where N ≥ 3, α ∈ (0,N), I_α is the Riesz potential, p,q ∈ 1,q

N−2N

and N p+ (N+2)q < 2N+4, ^N+α_N is the lower critical exponent in the sense of Hardy–

Littlewood–Sobolev inequality andV∈C(_R^N_,(_0,∞))is a bounded potential function.

As far as we have known, little research has been done on this type of coupled systems up to now. Our research is a promotion and supplement to previous research.

Keywords: nonlinearly coupled systems, lower critical exponent, Choquard type equa- tion, ground state solutions, variational methods.

2020 Mathematics Subject Classification: 35J10, 35J60, 35J65.

1 Introduction and main result

We are interested in the following nonlinearly coupled systems of Choquard type with lower critical exponent

(−_∆u+V(x)u= (I_α∗ |u|^Nα+1)|u|^Nα−1u+p|u|^p−2u|υ|^q, inR^N,

−_∆υ+V(x)υ= (I_α∗ |υ|^Nα+1)|υ|^Nα−1υ+q|υ|^q−2υ|u|^p, inR^N. (1.1) Where the dimensionN≥3 ofR^Nis given and functionIα :R^N\ {0} →_Ris a Riesz potential of orderα∈ (0,N)defined for eachx∈ _R^N\ {0},

I_α(x) = ^Γ(^N−₂^α) Γ(^α₂)π^N22^α|x|^N−α,

BCorresponding author. Email: anran0200@163.com

Γ denotes the classical Gamma function, ∗represents the convolution product onR^N, p,q∈ 1,q

N N−2

and N p+ (N+2)q< 2N+4,V ∈ C(_R^N,(0,∞))is a bounded potential function.

More precisely, we make the following assumptions onV, (V₁) V₀:= inf

x∈_R^NV(x)>0;

(V₂) V(x)< lim

|y|→_∞V(y) =V_∞ <_∞.

For the following Choquard equation

−_∆u+V(x)u= (I_α∗ |u|^p)|u|^p−2u, inR^N, (1.2) whenN=3,α=2, p=2 andVis a positive constant, this equation appears in several physi- cal contexts, such as standing waves for the Hartree equation, the description of the quantum physics of a polaron at rest by S. I. Pekar in [13] and the modeling of an electron trapped in its own hole in 1976 in the work of Choquard, as a certain approximation to Hartree–Fock theory of one–component plasma (see [4]). In some particular cases, this equation is also known as the Schrödinger–Newton equation, which was introduced by R. Penrose [14] in his discussion on the selfgravitational collapse of a quantum mechanical wave function. The existence and uniqueness of positive solutions for equation (1.2) withN=3,V(x)≡1,α=2 andp=2 was firstly obtained by E. H. Lieb in [4]. Later, P. L. Lions [6,7] got the existence and multiplicity results of normalized solution on the same topic. Since then, the existence and qualitative properties of solutions for equation (1.2) have been widely studied by variational methods in the recent decades. For related topics, we refer the reader to the recent survey paper [12].

To study equation (1.2) variationally, the well-known Hardy–Littlewood–Sobolev inequal- ity is the starting point. Particularly, V. Moroz and J. Van Schaftingen [9] established the exis- tence, qualitative properties and decay estimates of ground state solutions for the autonomous case of equation (1.2) with ^N_N^+α < p < ^N_N⁺₋^α₂ andV(x)≡ 1. In view of the Pohožaev identity [9–11], Choquard equation (1.2) withV is a positive constant has no nontrivial smoothH¹ so- lution when eitherp ≤ ^N_N^+α _or p≥ ^N_N⁺₋^α_{2. Usually,} ^N_N^+α is called the lower critical exponent and

N+α

N−2 is the upper critical exponent for Choquard equation in the sense of Hardy–Littlewood–

Sobolev inequality. The upper critical exponent plays a similar role as the Sobolev critical exponent in the local semilinear equations. C. O. Alves, S. Gao, M. Squassina and M. Yang[1]

established the existence of ground states for a type of critical Choquard equation with con- stant coefficients and also studied the existence and multiplicity of semi–classical solutions and characterized the concentration behavior by variational methods. G. Li and C. Tang [8]

obtained a positive ground state solution for Choquard equation with upper critical exponent when the nonlinear perturbation satisfies the general subcritical growth conditions. The lower critical exponent seems to be a new feature for Choquard equation, which is related to a new phenomenon of “bubbling at infinity” (for more details see [10]).

J. Van Schaftingen and J. Xia [15] studied the ground state solutions of the following Choquard equation with lower critical exponent and coercive potentialV,

−_∆u+V(x)u= (I_α∗ |u|^Nα+1)|u|^Nα−1u, inR^N. (1.3) Later, J. Van Schaftingen and J. Xia [16] also obtained a ground state solution for the following Choquard equation with lower critical exponent and a local nonlinear perturbation

−_∆u+u= (I_α∗ |u|^Nα+1)|u|^Nα−1u+ f(x,u)_{, inR}^N_{. (1.4)}

For the autonomous case f(x,u) = f(u)satisfies some superlinear assumptions, the existence and symmetry of ground state for equation (1.4) were also got. Furthermore, they derived a ground state solution of equation (1.4) for the nonautonomous case f(x,u) = K(x)|u|^q−2u with q∈(2, 2+ _N⁴)andK∈ L^∞(_R^N)satisfying inf_x∈RNK(x) =K_∞ =lim_|x|→∞K(x)>0.

As we mentioned above, all the results in the literature are concerned with a single equa- tion. More recently, P. Chen and X. Liu [2] obtained the existence of ground state solu- tions for the following linearly coupled systems of Choquard type with subcritical exponent p ∈(^N_N^+α,^N_N⁺₋^α₂),

(−_∆u+u= (I_α∗ |u|^p)|u|^p−2u+λυ, inR^N,

−_∆υ+υ= (Iα∗ |υ|^p)|υ|^p−2υ+λu, inR^N.

Later, M. Yang, J. de Albuquerque, E. Silva and M. Silva [19] obtained the existence of positive ground state solutions for the following linearly coupled systems of Choquard type

(−_∆u+u= (Iα∗ |u|^p)|u|^p−2u+λυ, inR^N,

−_∆υ+υ= (I_α∗ |υ|^q)|υ|^q−2υ+λu, inR^N. (1.5) when the exponents satisfy one of case 1, case 2 and case 3, and also obtained that there is no nontrivial solution for system (1.5) in case 4, where

case 1, ^N_N^+α < p< ^N_N⁺₋^α_{2 and} q= ^N_N⁺₋^α_2,

case 2, p = ^N_N^+α and ^N_N^+α <q< ^N_N⁺₋^α₂, case 3, p = ^N_N^+α and q= ^N_N⁺₋^α₂,

case 4, p,q≤ ^N_N^+α or p,q≥ ^N_N⁺₋^α₂.

Motivated by [2,15,16,19], in this paper, we will study the existence of ground state solu- tions for system (1.1). Our main result reads as followed.

Theorem 1.1. Let N ≥3,α∈ (0,N), p,q∈ 1, q N

N−2

, N p+ (N+2)q<2N+4and V satisfies (V₁),(V₂), then system(1.1)admits at least one ground state solution.

Remark 1.2. The assumption N p+ (N+2)q < 2N+4 is mainly used to get the energy estimate ofc₀in Lemma2.6. In particular,p,q∈ _1, ^N_N⁺₊²₁ satisfy our assumptions on p,q.

The method used to prove Theorem1.1 is as follows. Firstly, we establish the variational framework for system (1.1). LetH¹(_R^N)denote the normal Sobolev space equipped with the norm

kuk:=

Z

R^N(|∇u|²+|u|²)dx¹₂ . DefineX= H¹(_R^N)×H¹(_R^N)equipped with norm

k(u,v)k= (kuk²+kvk²)¹². Similar to H¹(_R^N), Xis a Hilbert space and satisfies

X,→ L^p(_R^N)×L^p(_R^N), p∈[2, 2^∗], where 2^∗ = ^2N N−2.

By Hardy–Littlewood–Sobolev inequality and Sobolev embedding theorem, the energy func- tional associated to system (1.1)

J_V(u,v) = ¹ 2

Z

R^N(|∇u|²+V(x)|u|²)dx+¹ 2

Z

R^N(|∇v|²+V(x)|v|²)dx

− ^N

2(N+α)

Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα+1dx− ^N 2(N+α)

Z

R^N(I_α∗ |v|^Nα+1)|v|^Nα+1dx

−

Z

R^N|u|^p|v|^qdx isC¹(X,R)and

hJ_V⁰ (u,v),(φ,ϕ)i=

Z

R^N(∇u∇φ+V(x)uφ)dx+

Z

R^N(∇v∇ϕ+V(x)vϕ)dx

−

Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα−1uφ)dx−

Z

R^N(I_α∗ |v|^Nα+1|v|^Nα−1vϕ)dx

−p Z

R^N|v|^q|u|^p−2uφdx−q Z

R^N|u|^p|v|^q−2vϕdx, for(φ,ϕ)∈X.

Thus, any critical point ofJ_Vis a weak solution of system (1.1). As usual, a nontrivial solution (u,v)∈ Xof system (1.1) is called a ground state solution if

J_V(u,v) =c^V_g :=_inf{J_V(u,v)_:(u,v)∈ X\ {(_{0, 0})} _and J_V⁰ (u,v) =₀}_.

Secondly, in the process of finding ground state solutions for system (1.1), the following limiting problem plays a significant role

(−_∆u+V_∞u= (Iα∗ |u|^Nα+1)|u|^Nα−1u+p|u|^p−2u|υ|^q, inR^N,

−_∆v+V_∞υ= (I_α∗ |υ|^Nα+1)|υ|^Nα−1υ+q|υ|^q−2υ|u|^p_{, inR}^N_. ^(1.6) Compared with the autonomous system (1.6), the potential V in system (1.1) breaks down the invariance under translations in R^N, then we cannot use the translation-invariant concentration–compactness argument. The strategy to prove Theorem1.1 is a comparison of the energy of the functional J_V with the functional J_V∞ associated to system (1.6). On the one hand, we construct a Palais–Smale sequence {(un,vn)}of J_V∞ at the level c₀ defined in (2.4), that is, a sequence {(u_n,v_n)}in Xsuch that J_V∞(u_n,v_n)→ c₀ and J_V⁰_∞(u_n,v_n) →0 asn → _∞.

On the other hand, we prove that up to translations the sequence {(u_n,v_n)} converges to a nontrivial solution(u,v)of system (1.6). Then, in the same way we obtain a (PS)c_V sequence of J_V. Furthermore, by the equivalent characterization ofc₀, we can show thatc_V <c₀under the assumptions on the potentialV. Based on c_V < c₀, the compactness maintains and a ground state solution for system (1.1) is obtained.

The rest of the paper is organized as follows. We give some preliminaries in Section 2. We obtain a ground state solution for system (1.6) in Section 3. Theorem1.1is proved in Section 4.

2 Preliminary

In this section, we first provide some preliminary results.

The following well-known Hardy–Littlewood–Sobolev inequality will be frequently used in this paper.

Lemma 2.1(Hardy–Littlewood–Sobolev inequality, [5]). Let p,q>1,α∈(0,N),1≤r <s<_∞ and s∈(1,^N_α)such that

1 p+ ¹

q =1+ ^α

N, 1

r − ¹ s = ^α

N. (i) Let f ∈ L^p(_R^N)and g∈ L^q(_R^N), we have

Z

R^N

Z

R^N

f(x)g(y)

|x−y|^N−αdxdy

≤C(N,α,p)kfk_Lp(_R^N)kgk_Lq(_R^N). (ii) For any f ∈ L^r(_R^N), Iα∗f ∈ L^s(_R^N)and

kI_α∗fk_Ls(_R^N) ≤C(N,α,r)kfk_Lr(_R^N).

By Hardy–Littlewood–Sobolev inequality mentioned above and the classical Sobolev em- bedding theorem, we obtain

Z

R^N(Iα∗ |u|^Nα+1)|u|^Nα+1dx ≤C(N,α)

Z

R^N|u|²dx_N^α+1

. (2.1)

This inequality can be restated as the following minimization problem S=_inf

_Z

R^N|u|²dx:u∈ H¹(_R^N)_and

Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα+1dx=₁

.

By Theorem 4.3 in [5], the infimumS is achieved by a functionu∈ H¹(_R^N)if and only if u(x) = A

ε

ε²+|x−a|^{2 N}₂

, x ∈_R^N, (2.2)

for some given constants A∈ _{R, and}a ∈ _R^N,ε ∈ (0,∞). The form of the minimizers in (2.2) suggests that a loss of compactness in equation (1.3) withV is a positive constant may occur by both of translations and dilations.

First, we recall that pointwise convergence of a bounded sequence implies weak conver- gence.

Lemma 2.2 ([18, Proposition 5.4.7]). Let N ≥ 3, q ∈ (1,∞)and {u_n}be a bounded sequence in L^q(_R^N). If u_n(x)→u(x)almost everywhere inR^N as n→_∞, then u_n *u weakly in L^q(_R^N).

Similarly as in [3], we can get the following lemma.

Lemma 2.3. Assume that {u_n} ⊂ H¹(_R^N)is a sequence satisfying that u_n * u in H¹(_R^N), then for any ϕ∈ H¹(_R^N),

nlim→_∞ Z

R^N(Iα∗ |u_n|^Nα+1)|u_n|^Nα−1u_nϕdx=

Z

R^N(Iα∗ |u|^Nα+1)|u|^Nα−1uϕdx.

Proof. For the reader’s convenience, we give a complete proof here. Up to a subsequence,{u_n} is bounded in H¹(_R^N), un * u in H¹(_R^N) and un(x) → u(x)a.e. in R^N. By Sobolev’s em- bedding theorem,{un}is bounded in L²(_R^N)∩L^2∗(_R^N), the sequence{|un|^NN+α}is bounded in L^N2N+α(_R^N). Then by Lemma2.2

|u_n|^Nα+1*|u|^Nα+1_{, in} L^N2N+α(_R^N)_.

|u_n|^Nα−1u_nϕ→ |u|^Nα−1uϕ, inL^N2N+α(_R^N)_{, for any}ϕ∈ H¹(_R^N)_.

By Lemma 2.1, the Riesz potential defines a linear continuous map from L^N2N+α(_R^N) to L^N2N−α(_R^N). We know that,

I_α∗(|u_n|^Nα−1u_nϕ)→ I_α∗(|u|^Nα−1uϕ), in L^N2N−α(_R^N). Thus,

Z

R^N(Iα∗ |un|^Nα+1)|un|^Nα−1unϕdx−

Z

R^N(Iα∗ |u|^Nα+1)|u|^Nα−1uϕdx

=

Z

R^N|u_n|^Nα+1(I_α∗(|u_n|^Nα−1u_nϕ)dx−

Z

R^N|u|^Nα+1(I_α∗(|u|^Nα−1uϕ)dx

=

Z

R^N|u_n|^Nα+1(I_α∗(|u_n|^Nα−1u_nϕ)−I_α∗(|u|^Nα−1uϕ))dx +

Z

R^N(|un|^Nα+1− |u|^Nα+1)(Iα∗(|u|^Nα−1uϕ))dx

→0, asn→_∞.

(2.3)

The proof is complete.

Lemma 2.4([17, Lemma 1.21]). Let r₀>0and s∈[2, 2^∗). If{u_n}is bounded in H¹(_R^N)and sup

y∈_R^N Z

B(y,r₀)

|u_n|^s→0, as n→_∞,

then u_n→₀in L^t(_R^N)for t∈ (_{2, 2}^∗).

Lemma 2.5. The functional JV_∞ satisfies the following properties:

(1) there existsρ>0such thatinf_{(u,v)∈X,k(u,v)k=ρ}J_V∞(u,v)>0;

(2) for any(u,v)∈ X\{(0, 0)}, it holdslim_t→_∞J_V∞(tu,tv) =−_∞.

Proof. (1) By (2.1) and the classical Sobolev inequality, we can deduce that J_V∞(u,v)≥ ¹

2min{1,V_∞}(kuk²+kvk²)− ^N 2(N+α)

Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα+1dx

− ^N

2(N+α)

Z

R^N(I_α∗ |v|^Nα+1)|v|^Nα+1dx−

Z

R^N|u|^p|v|^qdx

≥ ¹

2min{1,V_∞}k(u,v)k²−C₁(kuk^2αN+2+||v||^2αN+2)−

Z

R^N(|u|^2p+|v|^2q)dx

≥ ¹

2min{1,V_∞}k(u,v)k²−C₁k(u,v)k^2αN+2−C2k(u,v)k^2p−C3k(u,v)k^2q, whereC₁,C2are positive constants. Since p,q>1 andα>0, we have that

(u,v)∈X,infk(u,v)k=ρ

JV_∞(_u,v)>_0, provided thatρ>0 is sufficiently small.

(2) For any(u,v)∈ X\{(0, 0)}, we have J_V∞(tu,tv)≤ ^t

2

2 max{1,V_∞}(kuk²+kvk²)− ^Nt

2α N+2

2(N+α)

Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα+1dx

− ^Nt

2α N+2

2(N+α)

Z

R^N(I_α∗ |v|^Nα+1)|v|^Nα+1dx−t^p+q Z

R^N|u|^p|v|^qdx

≤ ^t

2

2 max{1,V_∞}k(u,v)k²− ^Nt

2α N+2

2(N+α)(

Z

R^N(Iα∗ |u|^Nα+1)|u|^Nα+1dx +

Z

R^N(Iα∗ |v|^Nα+1)|v|^Nα+1dx). Then the conclusion (2) follows.

By the classical Mountain Pass theorem [17], we have a minimax description at the energy levelc₀defined by

c₀ = inf

γ∈_Γmax

t∈[0,1]J_V∞(γ(t)), (2.4) where

Γ={γ∈C([_{0, 1}]_,X)_:γ(₀) = (_{0, 0})_,J_V∞(γ(₁))<₀}_. Lemma 2.6. Let N ≥ 3, α ∈ (_0,N)_{, p,q} ∈ 1,

q N N−2

and N p+ (_N+₂)_q < _2N+_{4, then} c₀ <c∗ := ^α

2(N+α)(V_∞S)^Nα+1.

Proof. We first show thatc₀ ≤c₁, where c₁ = inf

(u,v)∈X\{(0,0)}max

t≥0 JV_∞(tu,tv).

Indeed, for any(u,v)∈X\{(0, 0)}, by Lemma2.5(2), there existstu,v >0 such that J_V∞(t_u,vu,t_u,vv)<0.

Hence, by (2.4), we have

c0 ≤ max

τ∈[0,1]JV_∞(_{τtu,vu,τtu,vv})≤max

t≥0 JV_∞(_tu,tv)_{. (2.5)} It leads to c₀ ≤c₁.

By the representation formula (2.2) for the optimal functions of Hardy–Littlewood–Sobolev inequality, for eachε>0, we set

U(x) =A(1+|x|²)^−N2, x∈_R^N, U_ε(x) =ε^N2U(εx)_andV_ε(x) = ε

N+β

2 U(εx)_{, where} β ∈ ^N(p₂⁺₋^q_q⁻²⁾_,^4−N(p_q^+q−2)_{. For each}ε > ₀ the functionUε satisfies

Z

R^N|U_ε|²dx=S and Z

R^N(I_α∗ |U_ε|^Nα+1)|U_ε|^Nα+1dx=1.

Through direct computations, we have that Z

R^N|Vε|²dx=ε^β Z

R^N|U|²dx, Z

R^N(Iα∗ |Vε|^Nα+1)|Vε|^Nα+1dx=ε

β(N+α) N ,

Z

R^N|∇U_ε|²dx =ε² Z

R^N|∇U|²dx, Z

R^N|∇V_ε|²dx=ε^β+2 Z

R^N|∇U|²dx.

For everyε>0, we now consider the functionξ_ε :[_0,∞)→_{Rdefined by} ξ_ε(t)_:= J_V∞(tU_ε,tV_ε) =g(t) +h_ε(t) + f_ε(t)_, t ∈[_0,∞)_, where the functionsg,hε, fε :[0,∞)→_Rare defined by

g(t) = ¹

2V_∞St²− ^N 2(N+α)^t

2α N+2, h_ε(t) = ^t

2

2 Z

R^N|∇V_ε|²dx+ ^t

2

2V_∞ Z

R^N|V_ε|²dx− ^Nt

2(N+α) N

2(N+α)

Z

R^N(I_α∗ |V_ε|^Nα+1)|V_ε|^Nα+1dx, f_ε(t) = ^t

2

2 Z

R^N|∇U_ε|²dx−t^p+q Z

R^N|U_ε|^p|V_ε|^qdx.

Sinceξ_ε(t)> 0 whenevert >0 is small enough, lim_t→0ξ_ε(t) =0 and lim_t→_∞ξ_ε(t) =−_{∞, for} eachε>0 there existst_ε >0 such that

ξ_ε(t_ε) =_max

t≥0 ξ_ε(t)_. By the definition of the functiong, we have

c₁≤max

t≥0 ξ_ε(t) =ξ_ε(t_ε) =g(t_ε) +h_ε(t_ε) + f_ε(t_ε)≤g(t∗) +h_ε(t_ε) + f_ε(t_ε), (2.6) wheret∗= (V_∞S)^2αN satisfies that

g(t∗) =max

t≥0 g(t) = ^α 2(N+α)^V

N α+1

∞ S^Nα+1 =c∗. Sinceξ⁰_ε(t_ε) =0, we have

ε² Z

R^N|∇U|²dx+ε^β+2 Z

R^N|∇U|²dx+ε^β Z

R^N|U|²dx+V_∞S

= t_ε^2αN +t_ε^2αN Z

R^N(Iα∗ |Vε|^Nα+1)|Vε|^Nα+1dx+ (p+q)t_ε^p+q−2 Z

R^N|Uε|^p|Vε|^qdx

≥ t_ε^2αN.

(2.7)

Hence, we have lim sup_ε→0t_ε^2αN ≤ V_∞S, which is equivalent to lim sup_ε→0tε ≤ V_∞^2αNS^2αN. Notice that

t

2α N

ε

Z

R^N(Iα∗ |Vε|^Nα+1)|Vε|^Nα+1dx+ (p+q)t^pε^+q−2

Z

R^N|Uε|^p|Vε|^qdx

=ε

β(N+α)

N t_ε^2αN + (p+q)ε

N(p+q−2)+βq 2 t^p_ε^+q−2

Z

R^N|U|^p+qdx, we can obtain that

lim

ε→0

t_ε^2αN

Z

R^N(Iα∗ |Vε|^Nα+1)|Vε|^Nα+1dx+ (p+q)t^p_ε^+q−2 Z

R^N|Uε|^p|Vε|^qdx

=0. (2.8)

Then (2.7) and (2.8) imply lim inf_ε→0t_ε^2αN ≥ V_∞S. Therefore, lim_ε→0t_ε^2αN = V_∞S. It leads to limε→0t_ε =t∗.

We now observe that fε(tε) +hε(tε)≤ ¹

2ε^β+2t²_ε Z

R^N|∇U|²dx+ ¹ 2ε²t²_ε

Z

R^N|∇U|²dx +¹

2ε^βt²_εV_∞ Z

R^N|U|²dx−ε

N(p+q−2)+βq 2 t_ε^p+q−2

Z

R^N|U|^p+qdx.

Since p,q ∈ 1,q

N N−2

, N p+ (N+2)q < 2N+4 and β ∈ ^N(p₂⁺₋^q_q⁻²⁾,^4−N(p_q^+q−2)

, through direct computations, we can get that ^N(p+q₂^−2)+βq <_min{_{β, 2}}_{. Thus}

f_ε(t_ε) +h_ε(t_ε)<0, whenε>0 is small enough.

Then it follows from (2.6) thatc₁<c∗and thusc₀ <c∗ in view of (2.5).

3 Existence of ground state solutions for the limiting problem (1.6)

In this section, we will prove that the limiting problem (1.6) admits at least one ground state solution.

Before giving a complete proof, we state the following lemmas, which will be frequently used in the sequel proofs. Set

k(u,v)k_V∞ = _Z

R^N(|∇u|²+V_∞u²)dx+

Z

R^N(|∇v|²+V_∞v²)dx ¹₂

. Define

c^V_g^∞ :=inf{J_V∞(u,v):(u,v)∈ X\ {(0, 0)} and J_V⁰_∞(u,v) =0}. Lemma 3.1. If{(u_n,v_n)}is a sequence in X such that

lim inf

n→_∞ k(un,vn)k_V∞ >0, and lim

n→_∞h_Φ⁰(un,vn),(un,vn)i=0, where the functional Φ: X→_Ris defined by

Φ(u,v) = ¹

2k(u,v)k²_V∞− ^N 2(N+α)

_Z

R^N(I_α∗ |u|^Nα+1)|u|^Nα+1dx+

Z

R^N(I_α∗ |v|^Nα+1)|v|^Nα+1dx

, thenlim infn→_∞Φ(u_n,v_n)≥ c∗.

Proof. From lim

n→_∞h_Φ⁰(_un,vn)_,(_un,vn)i=0, we observe that k(un,vn)k²_V∞ =

Z

R^N(Iα∗ |un|^Nα+1)|un|^Nα+1dx+

Z

R^N(Iα∗ |vn|^Nα+1)|vn|^Nα+1dx+on(1). By the assumption lim inf_n→_∞k(u_n,v_n)k_V∞ >0 and (2.1), we can deduce that

lim inf

n→∞

Z

R^N(|u_n|²+|v_n|²)dx >0.

It follows from the definition of Sthat Z

R^N(I_α∗ |u_n|^Nα+1)|u_n|^Nα+1dx+

Z

R^N(I_α∗ |v_n|^Nα+1)|v_n|^Nα+1dx+o_n(₁)

≥

Z

R^N(V_∞|un|²+V_∞|vn|²)dx

≥V_∞S

"

Z

R^N(Iα∗ |un|^Nα+1)|un|^Nα+1dx _N^N_+α

+ Z

R^N(Iα∗ |vn|^Nα+1)|vn|^Nα+1dx _N^N_+α#

≥V_∞S _Z

R^N(I_α∗ |u_n|^Nα+1)|u_n|^Nα+1dx+

Z

R^N(I_α∗ |v_n|^Nα+1)|v_n|^Nα+1dx _N^N₊

α

,

which leads to lim inf

n→_∞ k(un,vn)k²_V∞

=lim inf

n→_∞

_Z

R^N(I_α∗ |u_n|^Nα+1)|u_n|^Nα+1dx+

Z

R^N(I_α∗ |v_n|^Nα+1)|v_n|^Nα+1dx

≥(V_∞S)^Nα+1_.

(3.1)

Therefore,

Φ(u_n,v_n) =_Φ(u_n,v_n)− ^N

2(N+α)h_Φ⁰(u_n,v_n),(u_n,v_n)i+o_n(1)

= ^α

2(N+α)ku_n,v_nk²_V∞+o_n(₁)_.

(3.2)

Then combine (3.1) with (3.2), lim inf

n→_∞ Φ(un,vn) =lim inf

n→_∞

α

2(N+α)kun,vnk²_V∞ ≥ ^α

2(N+α)(V_∞S)^1+Nα =c∗. The proof is complete.

Lemma 3.2. Let{(u_n,v_n)}be a bounded (PS)_csequence with c ∈(0,c∗)for functional J_V∞, then up to a subsequence and translations, the sequence{(un,vn)}converges weakly to some(u,v)∈ X\{(0, 0)}

such that

J_V⁰_∞(u,v) =₀ and J_V∞(u,v)∈ (_0,c]_. Proof. First we show that

lim sup

n→_∞

1 2

Z

R^N(|u_n|^2p+|v_n|^2q)dx>0. (3.3) Otherwise, up to a subsequence, we have

lim sup

n→_∞ Z

R^N|u_n|^p|v_n|^qdx≤lim sup

n→_∞ Z

R^N(|u_n|^2p+|v_n|^2q)dx =0. (3.4) Since lim

n→_∞hJ_V⁰_∞(u_n,v_n),(u_n,v_n)i=0, we have k(un,vn)k²_V∞ =

Z

R^N(Iα∗ |un|^Nα+1)|un|^Nα+1dx+

Z

R^N(Iα∗ |vn|^Nα+1)|vn|^Nα+1dx+on(1). While, J_V∞(un,vn)→c>0, n→∞, together with (3.4) and (2.1), imply that

lim inf

n→_∞ k(u_n,v_n)k_V∞ >0.

Then we deduce from Lemma3.1that c=_{lim inf}

n→_∞ JV_∞(_un,vn)

=lim inf

n→_∞ Φ(un,vn)−lim sup

n→∞

Z

R^N|un|^p|vn|^qdx

=lim inf

n→_∞ Φ(un,vn)

≥c∗,