Existence of solutions to a class of quasilinear Schrödinger systems involving the fractional p-Laplacian

Existence of solutions to a class of quasilinear Schrödinger systems involving the fractional p-Laplacian

Mingqi Xiang

, Binlin Zhang

and Zhe Wei

1College of Science, Civil Aviation University of China, Tianjin, 300300, P.R. China

2Department of Mathematics, Heilongjiang Institute of Technology, Harbin, 150050, P.R. China

Received 19 July 2016, appeared 23 November 2016 Communicated by Patrizia Pucci

Abstract. The purpose of this paper is to investigate the existence of solutions to the following quasilinear Schrödinger type system driven by the fractional p-Laplacian

(−_∆)^s_pu+a(x)|u|^p−2u=Hu(x,u,v) inR^N, (−∆)^s_qv+b(x)|v|^q−2v=Hv(x,u,v) inR^N,

where 1 <q≤ p,sp <N,(−∆)^s_mis the fractionalm-Laplacian, the coefficientsa,bare two continuous and positive functions, and Hu,Hvdenote the partial derivatives of H with respect to the second variable and the third variable. By using the mountain pass theorem, we obtain the existence of nontrivial and nonnegative solutions for the above system. The main feature of this paper is that the nonlinearities do not necessarily satisfy the Ambrosetti–Rabinowitz condition.

Keywords: Schrödinger system, fractionalp-Laplacian, mountain pass theorem.

2010 Mathematics Subject Classification: 35R11, 35A15, 35J60, 47G20.

1 Introduction

In this paper we are concerned with the following fractional Schrödinger system (−_∆)^s_pu+a(x)|u|^p−2u= H_u(x,u,v) inR^N,

(−_∆)^s_qv+b(x)|v|^q−2v= Hv(x,u,v) inR^N, (1.1) where N > ps with s ∈ (0, 1) and (−_∆)^s_m is the fractional m-Laplace operator which (up to normalization factors) may be defined along any ϕ∈C₀^∞(_R^N)as

(−_∆)^s_mϕ(x) =2 lim

ε→0⁺

Z

R^N\B_ε(x)

|ϕ(x)−ϕ(y)|^m−2(ϕ(x)−ϕ(y))

|x−y|^{N+ms dy}

BCorresponding author. Email: xiangmingqi_hit@163.com (M. Xiang), zhangbinlin2012@163.com (B. Zhang), weizhe_hit@163.com (Z. Wei)

forx∈_R^N, see [14] and the references therein for further details on the fractionalm-Laplacian.

The nonlinearities Hu and Hv denote the partial derivative of H with respect to the second variable and the third variable, respectively, and H satisfies some hypotheses to be stated in the sequence.

In particular,(−_∆)^s_m becomes to the fractional Laplacian(−_∆)^s asm=2, and it is known that(−_∆)^s_m reduces to the standardm-Laplacian ass↑1, see [14].

In fact, nonlocal and fractional operators arise in a quite natural way in many different ap- plications, such as, continuum mechanics, phase transition phenomena, population dynamics and game theory, as they are the typical outcome of stochastically stabilization of Lévypro- cesses, see for instance [3,10]. The literature on nonlocal operators and on their applications is interesting and quite large, we refer the reader to [11,23,27,30–33] and the references therein.

For the basic properties of fractional Sobolev spaces, we refer the reader to [14].

On the one hand, in the context of fractional quantum mechanics, nonlinear fractional Schrödinger equation was first proposed by Laskin in [18,19] as a result of expanding the Feynman path integral, from the Brownian-like to the Lévy-like quantum mechanical paths.

In the last years, there has been a great interest in the study of the fractional Schrödinger equation

(−_∆)^su+V(x)u= f(x,u) _inR^N_,

where the nonlinearity f satisfies some general conditions. For standing wave solutions of fractional Schrödinger equations in R^N, see, for instance, [12,15,24,29] and the reference therein. It is worthy mentioning that Autuori andPucci in [5] studied the following elliptic equations involving fractional Laplacian

(−_∆)^su+a(x)u=λω(x)|u|^p−2u−h(x)|u|^r−2u inR^N, (1.2) where λ ∈ _{R, 0} < s < 1, 2s < N, 2 < q < min{r, 2^∗_s}, 2^∗_s = 2N/(N−2s), and (−_∆)^s is the fraction Laplacian operator. The authors obtained the existence and multiplicity of entire solutions of (1.2) by using the direct method in variational methods and the mountain pass theorem. The same nonlinearities were recently considered by Pucci and Saldi in [25], where the authors established existence and multiplicity of nontrivial nonnegative entire weak solutions of a stationary Kirchhoff eigenvalue problem, involving a general nonlocal integro- differential operator. More precisely, they considered the problem

M([u]²_s,K)L_Ku=λω(x)|u|^q−2u−h(x)|u|^r−2u inR^N, [u]_s,K =

Z Z

R^2N|u(x)−u(y)|²K(x−y)dxdy,

whereλ∈_R, 0<s<1, 2s< N, andL_Kis an integro-differential nonlocal operator. The case with variable exponents was treated byPucciandZhangin [28], where the authors studied the one parameter elliptic equation

−divA(x,∇u) +a(x)|u|^p(x)−2u =λω(x)|u|^q(x)−2u−h(x)|u|^r(x)−2u inR^N,

where λ ∈ _R and A : R^N×_R^N → _R^N admits a potential A , with respect to its second variableξ and satisfies some assumptions listed in the paper.

On the other hand,ServadeiandValdinociin [30] studied the following fractional Laplacian problem

((−_∆)^su= f(x,u) inΩ,

u=0 inR^N\_Ω. ^(1.3)

whereΩis a bounded domain. Since the fractional Laplacian operator is nonlocal, some tech- nical difficulties arise when applying usual variational methods. In this paper, the authors used the mountain pass theorem to obtain the existence of solutions for problem (1.3). After that, many authors devoted to studying the fractional problems by using variational meth- ods and critical point theory. Indeed, a lot of works in the literature involve the following Ambrosetti–Rabinowitz ((AR) in short) condition on the nonlinearity

0< F(x,u):=

Z _u

0 f(x,t)dt≤µf(x,u)u, x∈_R^N, 06=u∈_R, (1.4) for some constant µ > p, see [2] for more details. In general, the (AR) condition not only ensures that the Euler–Lagrange functional associated with a problem of the form (1.1) has a mountain pass geometry, but also guarantees boundedness of the Palais–Smale sequences corresponding the functional. Although the (AR) condition is very important and widely used to get the existence of solutions for elliptic problems via variational methods, it is not fulfilled by some simple nonlinearities, such as

f₁(t) =|t|^p−2t(ln|t|+1) and f₂(t) =











|t|^p−2t− ^p−_p¹|t|^r−2t, |t| ≤₁

|t|^p−2t(ln|t|+ ¹_p), |t|>1 p< r< p^∗_s := _N^pN_−ps. In fact, these functions do not satisfy

F(x,t)≥d₁|t|^µ−d₂, x∈_R^N, t ∈_R, whered₁, d₂>0 andµ> p

which is a consequence of the (AR) condition. Here we would like to sketch some advances about this aspect. For the Laplacian case as p = q = 2, the existence of nontrivial weak solutions for nonlinear elliptic problems without assuming (AR) condition was obtained in [21]. For the single p-Laplacian case with p > 1, we refer the reader to [20]. For the (p,q)- Laplacian with 1 < q ≤ 2 ≤ p and a = b = 0, we refer the reader to [22]. For the fractional Laplacian case, the existence of infinitely many weak solutions for nonlinear elliptic problems without requiring (AR) condition was investigated in [8].

Motivated by the above works, especially [12,13], we are interested in the study of solutions for system (1.1) without the (AR) condition involving the fractional p-Laplacian. For this, let V(_R^N)be a subset ofC(_R^N)and for anyV ∈ V(_R^N),

(A₁) V is bounded from below by a positive constant;

(A₂) there existsκ>0 such that lim_|y|→∞meas({x∈ Bκ(y):V(x)≤c}) =0 for any c>0, where B_κ(y) denotes any open ball of R^N centered at y and of radius κ > 0. Note that condition (A₂), which is weaker than the coercivity assumption: V(x)→_∞ as|x| → _∞, was originally discussed byBartschandWangin [7] to overcome the lack of compactness.

Without further mentioning, we always assume a,b ∈ V(_R^N). Now we introduce some notation. Assume 1 < m < _{∞. Let} D^s,m(_R^N)denote the closure of C₀^∞(_R^N)with respect to the Gagliardo seminorm

[u]_s,m := Z Z

R^2N

|u(x)−u(y)|^m

|x−y|^{N+ms dxdy} 1/m

.

Letω∈ V(_R^N)and define E_m,ω :=

u∈ D^s,m(_R^N): Z

R^Nω(x)|u(x)|^mdx< _∞

, endowed with the norm

kuk_Em,ω := [u]^m_s,m+kuk^m_m,ω^1/m, wherekuk_m,ω= R

R^Nω|u|^mdx1/m

. Define

W:= E_p,a×E_q,b, endowed with the norm

k(u,v)k:= kuk_Ep,a+kvk_Eq,b. By the embeddingE_p,a ,→L^p(_R^N)(see [26]), we can define

λ^∗ =inf

kuk_E^p

p,a+kvk_E^p

q,b : Z

R^N|(u,v)|^pdx=1

, and deduce thatλ^∗ >_0.

Finally, the conditions we impose on the nonlinearity are:

(H₁) H ∈ C¹(_R^N ×_R²,R) such that H(x,u,v) > 0 if (u,v) 6= (0, 0), H(x, 0, 0) = 0, and Hu(x,u,v) =0 ifu≤0; Hv(x,u,v) =0 ifv ≤0;

(H₂) there existr ∈(p,q^∗_s)_andC>0 such that

|H_z(x,z)| ≤C(1+|z|^r−1), for all(x,z)∈_R^N×_R², whereHz(x,z) = (Hu(x,z),Hv(x,z));

(H3) lim

|z|→_∞

Hz(x,z)·z

|z|^p = ∞, uniformly inx ∈_R^N; (H₄) there existsg∈ L¹(_R^N)+ such that

F(x,t₁z)≤ F(x,t₂z) +g(x), for all(x,z)∈_R^N×_R²and 0<t₁≤t₂or 0< t₂ ≤t₁, whereF(x,z):= H_z(x,z)·z−pH(x,z);

(H5) There exists 0≤λ(x)≤ kλk_∞ <λ^∗for all x∈_R^N such that lim sup

|z|→0

|H_z(x,z)|

|z|^p−1 ≤λ(x)_, uniformly inx ∈_R^N.

Remark 1.1. Obviously, the functions f₁(t) and f₂(t) mentioned before satisfy (H₁)–(H₅). For the single equation, it is easy to see that (1.4) implies the more weaker condition (H₅)_. Condition (H₄) together with (H₅), introduced by Jeanjean in [16], was often used to study the existence of nontrivial solutions for superlinear problems without (AR) condition in recent years, see for example [20,22].

Now, we give the definition of weak solutions of problem (1.1).

Definition 1.2. We say that(u,v)∈Wis a (weak)solutionof problem (1.1), if hu,ϕi_s,p+hv,ψi_s,q+

Z

R^Na(x)|u(x)|^p−2u(x)ϕ(x)dx+

Z

R^Nb(x)|v(x)|^q−2v(x)ψ(x)dx

=

Z

R^N Hu(x,u,v)ϕ(x)dx+

Z

R^NHv(x,u,v)ψ(x)dx, hu,ϕi_s,m :=

Z Z

R^2N

|u(_x)−u(_y)|^m−2(_u(_x)−u(_y))(ϕ(_x)−ϕ(_y))

|x−y|^{N+ms dxdy} for any (ϕ,ψ)∈W.

The main result of this paper is the following.

Theorem 1.3. Let0< _s < ₁ <_q ≤ p < _Nq/(_N−sq)_,sq < _{N, and let} (_H1)_–(_H5)hold. Assume a,b ∈ V(_R^N). Then system(1.1) has at least one pair of nontrivial and nonnegative weak solution (u,v)∈W.

This article is organized as follows. In Section2, we give some necessary definitions and properties of the fractional Sobolev spaceW. In Section3, using the mountain pass theorem, we obtain the existence of solutions for system (1.1).

2 Preliminaries

In this section, we give some basic results of fractional Sobolev spaceW that will be used in the next section.

By Lemma 10 of [26], one has Ep,a = (Ep,a,k · k_Ep,a) and E_q,b = (E_q,b,k · k_Eq,b) are two separable, reflexive Banach spaces. Hence, by Theorem 1.12 of [1], we have the following.

Lemma 2.1. W= (_W,k · k_W)is a separable and reflexive Banach space.

Lemma 2.2. The embeddingW ,→L^ν(_R^N)×L^ν(_R^N)is continuous ifν∈[p,p^∗_s], and

k(u,v)k_ν ≤C_νk(u,v)k for all(u,v)∈W. (2.1) Proof. By Lemma 1 of [26], there existsCν >0 such that

kuk_ν ≤C_νkuk_Ep,a andkvk_ν ≤C_νkvk_Eq,b for all(u,v)∈W.

Hence,

k(u,v)k_ν =^pu²+v²

L^ν(_R^N) ≤ ku+vk_Lν(_R^N)

≤ kuk_Lν(_R^N)+kvk_Lν(_R^N)

≤Cν(kuk_Ep,a+kvk_Eq,b)

=C_νk(u,v)k. Hence the lemma is proved.

Similar to Proposition A.10 of [4], we have the following lemma.

Lemma 2.3. Let{(u_n,v_n)}_n ⊂Wbe such that(u_n,v_n)*(u,v)weakly inWas n→∞. Then, up to a subsequence,(un,vn)→(u,v)a.e. inR^N as n →_∞.

By using the same discussion as in [26, Theorem 2.1], we have the following compact embedding.

Lemma 2.4. Suppose that a,b∈ V(_R^N)and1 < q ≤ p < q^∗_s. Let ν ∈ [p,q^∗_s)be a fixed exponent.

Then the embeddings E_p,a ,→ L^ν(_R^N)and E_q,b ,→ L^ν(_R^N) are compact. Moreover, the embedding W,→L^ν(_R)×L^ν(_R)is compact.

3 Proof of Theorem 1.3

To prove Theorem1.3, we need the following mountain pass theorem under condition(C). Theorem 3.1(see [17, Theorem 6]). Let E be a real Banach space with its dual space E^∗, and suppose that J∈C¹(E,R)satisfies

max{J(0),J(e)} ≤α<β≤ inf

kuk=ρ

J(u), for someα< β,ρ>0and e∈ E withkek>ρ. Let c≥ βbe characterized by

c= _inf

γ∈_Γmax

t∈[0,1]J(γ(t))

whereΓ = {γ ∈ C([0, 1],E) : γ(0) = 0,γ(1) = e}is the set of continuous paths joining 0 and e, then there exists a sequence{un}_n⊂E such that

J(u_n)→c≥ β and kJ⁰(u_n)k_E^∗(1+ku_nk)→0, asn→_∞. This kind of sequence is usually called a Cerami sequence.

Definition 3.2. We say that a functional J : E →_R of classC¹ satisfies the Cerami condition ((C) in short) if any Cerami sequence associated with J has a strongerly convergent subse- quence inE.

The Euler–Lagrange functional associated with system (1.1) is I(u,v) = ¹

p Z Z

R^2N

|u(x)−u(y)|^p

|x−y|^{N+ps dxdy}+ ¹ p

Z

R^Na(x)|u|^pdx + ¹

q Z Z

R^2N

|v(x)−v(y)|^q

|x−y|^{N+qs dxdy}+ ¹ q

Z

R^Nb(x)|v|^qdx−

Z

R^NH(x,u,v)dx.

Clearly, the functional I is well-defined in W. Under conditions (H₁)_–(H₅), it is easy to see that the functionalI is of classC¹, and for(u,v)∈W

hI⁰(u,v)_,(_ϕ,ψ)i=hu,ϕi_s,p+

Z

R^Na|u|^p−2uϕdx+hv,ψi_s,q+

Z

R^Nb|v|^q−2vψdx

−

Z

R^NHu(x,u,v)ϕdx−

Z

R^N Hv(x,u,v)ψdx, for all(_ϕ,ψ)∈_W.

Lemma 3.3. Any Cerami sequence associated with the functional I is bounded inW.

Proof. Let{(u_n,v_n)}_n ⊂ _W be a Cerami sequence associated with I. Then there existsC > 0 independent ofnsuch that|I(un,vn)| ≤Cand(1+k(un,vn)k)I⁰(un,vn)→0 asn→_{∞. In the} sequel, we will use C to denote various positive constant that does not depend on n. Hence there existsε_n >0, withε_n →0, such that

|hI⁰(un,vn),(ϕ,ψ)i| ≤ ^εnk(_ϕ,ψ)k

1+k(un,vn)k^{, for all}(ϕ,ψ)∈Wandn∈_N. (3.1)

Choosing(ϕ,ψ) = (u_n,v_n)in (3.1), we deduce

hu_n,u_ni_s,p+

Z

R^Na|u_n|^pdx+hv_n,v_ni_s,q+

Z

R^Nb|v_n|^qdx

−

Z

R^N[H_u(x,u_n,v_n)u_n+H_v(x,u_n,v_n)v_n]dx

=|hI⁰(u_n,v_n),(u_n,v_n)i|

≤ ^εnk(u_n,v_n)k

1+k(un,vn)k ≤ ε_n≤C. (3.2)

Hence we have

− ku_nk_E^p

p,a− kv_nk^q_E

q,b+

Z

R^N[H_u(x,u_n,v_n)u_n+H_v(x,u_n,v_n)v_n]dx ≤C (3.3) Next we show that {(u_n,v_n)}_n is bounded in W. Arguing by contradiction, we assume that k(u_n,v_n)k →∞. Without loss of generality, we assume that k(u_n,v_n)k ≥1 for alln∈ _N.

Set (X_n,Y_n) := _k(^(u_u^n,vn)

n,vn)k. Clearly, k(X_n,Y_n)k = 1. Then there exists(X,Y) ∈ W such that, up to a subsequence,

(Xn,Yn)*(X,Y) inW (X_n,Y_n)→(X,Y) a.e. inR^N. Moreover, by Lemma2.4, we can assume that, up to a subsequence,

(X_n,Y_n)→(X,Y) in L^ν(_R^N)×L^ν(_R^N) for anyν∈[p,p^∗_s).

Let X_n⁻ = min{0,Xn} and Y_n⁻ = min{0,Yn}. Clearly, {(X_n⁻,Y_n⁻)} is also bounded in W.

Choosing (ϕ,ψ) = (X_n⁻,Y_n⁻)in (3.1), we obtain byk(u_n,v_n)k →_∞

o(1) = hI⁰(_un,vn)_,(_X⁻_n,Yn⁻)i k(u_n,v_n)k^{p−1 ,}

that is,

o(₁) = ¹ k(u_n,v_n)k^p−1

hu_n,X_n⁻i_s,p+

Z

R^Na|u_n|^p−2u_nX_n⁻dx+hv_n,Y_n⁻i_s,q+

Z

R^Nb|v_n|^q−2v_nY_n⁻dx

−

Z

R^N

H_u(x,u⁺_n,v⁺_n)X_n⁻+H_v(x,u⁺_n,v⁺_n)Y_n⁻ k(un,vn)k^{p−1 dx}

= ¹

k(un,vn)k^p

hu_n,u⁻_ni_s,p+

Z

R^Na|u_n|^p−2u_nu⁻_ndx+hv_n,v⁻_ni_s,q+

Z

R^Nb|v_n|^q−2v_nv⁻_ndx

−

Z

R^N

H_u(x,u⁺_n,v⁺_n)u⁻_n +H_v(x,u⁺_n,v⁺_n)v⁻_n k(u_n,v_n)k^{p dx}

= ¹

k(u_n,v_n)k^p

hu_n,u⁻_ni_s,p+

Z

R^Na|u_n|^p−2u_nu⁻_ndx+hv_n,v⁻_ni_s,q+

Z

R^Nb|v_n|^q−2v_nv⁻_ndx

≥ ¹

k(u_n,v_n)k^p

ku⁻_nk^p_E

p,a+kv⁻_nk^q_E

q,b

, (3.4)

where the last inequality follows from the following elementary inequality

|ξ⁻−η⁻|^m ≤ |ξ−η|^m−2(ξ−η)(ξ⁻−η⁻), forξ,η∈_Randm>1.

Hence, we deduce from (3.4) that as n→_∞

kX⁻_nk_Ep,a →0.

Similarly, by

o(1) = hI⁰(u_n,v_n),(X⁻_n,Y_n⁻)i k(u_n,v_n)k^{q−1 ,} we obtain

kY_n⁻k_Eq,b →0,

asn → ∞. Therefore, we get (X⁻_n,Y_n⁻)→ (0, 0)in W, this implies that (X⁻,Y⁻) = (0, 0)a.e.

inR^N. Hence,X ≥0 andY≥0 a.e. inR^N.

Set Ω⁺ = {x ∈ _R^N : X> 0 orY> 0}andΩ⁰ = {x ∈ _R^N :(X,Y) = (0, 0)}. AssumeΩ⁺ has a positive Lebesgue measure. Recall thatk(un,vn)k →∞. Hence we get

|(u_n,v_n)|=(u_n,v_n)|(X_n,Y_n)| →_∞ a.e. inΩ⁺. Thus, by(H₃)

nlim→_∞

H(_x,un,vn)

k(un,vn)k^p = lim

n→_∞

H(_x,un,vn)|(Xn,Yn)|^p

|(un,vn)|^p =_∞ a.e. inΩ⁺. Then we deduce from Fatou’s lemma that

nlim→_∞ Z

R^N

H(x,u_n,v_n)

k(un,vn)k^pdx= _lim

n→_∞ Z

R^N

H(x,u_n,v_n)|(X_n,Y_n)|^p

|(un,vn)|^{p dx}= _{∞. (3.5)} On the other hand, by|I(un,vn)| ≤C, one has

1 pkunk^p_E

p,a+ ¹ qkvnk^q_E

q,b−

Z

R^NH(x,un,vn)dx≤C.

Thus

Z

R^N

H(x,un,vn)

k(u_n,v_n)k^pdx≤ ¹ p

kunk_E^p

p,a

k(u_n,v_n)k^p +¹ q

kv_nk^q_E

q,b

k(u_n,v_n)k^p + ^C k(u_n,v_n)k^p, this together with assumption k(u_n,v_n)k ≥1 implies that

Z

R^N

H(x,un,vn)

k(u_n,v_n)k^pdx ≤ ¹ p +¹

q+ ^C

k(u_n,v_n)k^p. Therefore, we deduce from k(u_n,v_n)k →_∞that

lim sup

n→_∞ Z

R^N

H(x,u_n,v_n)

k(un,vn)k^pdx≤ ¹ p +¹

q,

which contradicts with (3.5). HenceΩ⁺has zero measure, that is,(X,Y) = (0, 0)a.e. inR^N. By the continuity of the functiont ∈ [_{0, 1}] → I(tu_n,tv_n), there exists a sequence {t_n}_n ⊂ [0, 1]such that

I(tnun,tnvn) = max

0≤t≤1I(tun,tvn). Set

(U_n,V_n):= (2θ)^1/q(X_n,Y_n) = (2θ)^{1/q 1}

k(u_n,v_n)k(u_n,v_n)∈W, whereθ >1/2. By(H₅), for anyε>0, there existsδ>0 such that

|Hz(x,z)| ≤(λ(x) +ε)|z|^p−1 for all x∈_R^N and|z| ≤δ.

For all x∈_R^N and|z|>δ, we have by(H2)

|H_z(x,z)| ≤C(1+|z|^r−1)

≤C

|^z δ

|+|z|^r−1≤C 1

δ^r−1 +₁

|z|^r−1_. Hence, we obtain

|Hz(x,z)| ≤(λ(x) +ε)|z|^p−1+Cε|z|^r−1 for all(x,z)∈_R^N×_R², (3.6) whereC_ε = C ¹

δ^r−1 +1

. Observing that H(x,z) =R1

0 H_z(x,tz)·zdt, we get

|H(x,z)| ≤ ¹

p(λ(x) +ε)|z|^p+C_ε|z|^r, for all (x,z)∈_R^N×_R². (3.7) Since (U_n,V_n) → (_{0, 0}) _in L^ν(_R^N)×L^ν(_R^N)_{, with} ν ∈ [p,q^∗_s), by using (3.7) withε = _{1, we} have

Z

R^N H(x,Un,Vn)dx≤

Z

R^N[(λ(x) +1)|(Un,Vn)|^p+C₁|(Un,Vn)|^r]dx

≤(λ^∗+1)k(U_n,V_n)k^p

L^p(_R^N)+C₁k(U_n,V_n)k^r_Lr(RN)

→0,

asn→∞, thanks tor ∈(p,q^∗_s)andλ(x)<λ^∗. So we get

nlim→_∞ Z

R^NH(x,U_n,V_n)dx=0. (3.8) Since k(u_n,v_n)k → _∞, there exists n₀ large enough such that (2θ)^1/q/k(u_n,v_n)k ∈ (0, 1) for all n ≥ n₀. Hence, for all n ≥ n₀, we obtain by q ≤ p, θ > ¹₂ and kYnk_Eq,b ≤ kXnk_Ep,a+ kY_nk_Eq,b =1

I(t_nu_n,t_nv_n)≥ I

(2θ)^1/qu_n/k(u_n,v_n)k, (2θ)^1/qv_n/k(u_n,v_n)k

= (_2θ)^p/q p kX_nk^p_E

p,a+^2θ q kY_nk^q_E

q,b−

Z

R^N H(x,U_n,V_n)dx

≥ ^2θ

p (kX_nk^p_E

p,a+kY_nk^q_E

q,b)−

Z

R^N H(x,U_n,V_n)dx

≥ ^2θ

p (kX_nk^p_E

p,a+kY_nk^p_E

q,b)−

Z

R^N H(x,U_n,V_n)dx

≥ ^2θ

p2^p−1(kXnk_Ep,a+kYnk_Eq,b)^p−

Z

R^NH(x,Un,Vn)dx

= ^2θ p2^p−1 −

Z

R^N H(x,Un,Vn)dx.

By (3.8), there exists n₁≥ n₀such that Z

R^N H(x,Un,Vn)dx≤ ^θ

p2^p−1, for all n≥n₁. Then

I(tnun,tnvn)> ^θ

p2^p−1, for all n≥n₁, this together with the arbitrariness ofθ>1/2 yields

nlim→_∞I(tnun,tnvn) =_∞. (3.9) Following the same discussion as Lemma 7.3 of [6] and using the facts thatI(0, 0) =0 and I(u_n,v_n)≤C, we can assume that t_n∈(0, 1)for alln≥ n₂≥ n₁. Thus,

ktnunk^p_E

p,a+ktnvnk^q_E

q,b−

Z

R^N[Hu(x,tnun,tnvn)tnun+Hv(x,tnun,tnvn)tnvn]dx

= hI⁰(t_nu_n,t_nv_n),(t_nu_n,t_nv_n)i=t_nd

dthI⁰(tu_n,tv_n),(tu_n,tv_n)i t=tn

=0, (3.10) for alln≥n2.

From 0≤t_n≤1 and(H₄), one has Z

R^N F(_x,tnun,tnvn)_dx ≤

Z

R^N F(_x,un,vn)_dx+

Z

R^Ng(_x)_{dx. (3.11)} Combining (3.10) with (3.11), we have

kt_nu_nk_E^p

p,a+kt_nv_nk^q_E

q,b =

Z

R^N[H_u(x,t_nu_n,t_nv_n)t_nu_n+H_v(x,t_nu_n,t_nv_n)t_nv_n]dx

=

Z

R^NpH(x,t_nu_n,t_nv_n)dx+

Z

R^NF(x,t_nu_n,t_nv_n)dx

≤

Z

R^NpH(x,tnun,tnvn)dx+

Z

R^NF(x,un,vn)dx+

Z

R^Ng(x)dx,

for all n≥n₂. Then, pI(t_nu_n,t_nv_n) =kt_nu_nk^p_E

p,a+ ^p

qkt_nv_nk^q_E

q,b−

Z

R^NpH(x,t_nu_n,t_nv_n)dx+kt_nv_nk^q_E

q,b− kt_nv_nk^q_E

q,b

= p

q −1

ktnvnk^q_E

q,b+ktnunk_E^p

p,a+ktnvnk^q_E

q,b−

Z

R^N pH(x,tnun,tnvn)dx

≤ p

q −1

kv_nk^q_E

q,b+

Z

R^N F(x,u_n,v_n)dx+

Z

R^Ng(x)dx, this together with (3.9) yields

p q−1

kv_nk^q_E

q,b+

Z

R^NF(x,u_n,v_n)dx→_∞. (3.12) On the other hand, by|I(u_n,v_n)| ≤C, we have

ku_nk^p_E

p,a+ ^p qkv_nk^q_E

q,b−p Z

R^NH(x,u_n,v_n)dx= pI(u_n,v_n)≤C.

Adding this inequality to (3.3) and using(H₄), we obtain p

q −1

kv_nk^q_E

q,b+

Z

R^N F(x,u_n,v_n)dx≤C, which contradicts with (3.12).

Therefore, we conclude that{(un,vn)}_nis bounded inW.

Lemma 3.4. The functional I satisfies condition(C).

Proof. Assume {(un,vn)}_n is a Cerami sequence. Then there exists C > 0 independent of n such that

|I(un,vn)| ≤C and (1+k(un,vn)k)I⁰(un,vn)→0.

By Lemma3.3,{(u_n,v_n)}_nis bounded inW. Hence, we can assume that

(un,vn)*(u,v) inW and (un,vn)→(u,v) inL^ν(_R^N)×L^ν(_R^N),

for any ν∈[p,q^∗_s). By using (3.6) correspondingε=1 and applying Lemma2.4, we deduce Z

R^N|H_u(x,u_n,v_n)(u_n−u) +H_v(x,u_n,v_n)(v_n−v)|dx

≤

Z

R^N(|H_u(x,u_n,v_n)|+|H_v(x,u_n,v_n)|)|(u_n−u,v_n−v)|dx

≤ √ 2

Z

R^N

h

(λ^∗+1)|(un,vn)|^p−1|(un−u,vn−v)|+C|(un,vn)|^r−1|(un−u,vn−v)|ⁱdx

≤C k(u_n−u,v_n−v)k_Lp(_R^N)+k(u_n−u,v_n−v)k_Lr(_R^N)

→0,

asn→_{∞, where}Cdenotes various positive constants. Thus we get

nlim→_∞ Z

R^N|Hu(x,un,vn)(un−u) +Hv(x,un,vn)(vn−v)|dx=0.

Taking into account that(1+k(u_n,v_n)k)I⁰(u_n,v_n)→0 and the boundedness of{(u_n,v_n)}, we have

nlim→_∞

hun,un−ui_s,p+

Z

R^Na|un|^p−2un(un−u)dx +hvn,vn−vi_s,q+

Z

R^Nb|vn|^q−2vn(vn−v)dx

=0. (3.13)

Observe that the linear functionalL(u,v):W→_Rdefined by hL(u,v),(ϕ,ψ)i=hu,ϕi_s,p+

Z

R^Na|u|^p−2uϕdx+hv,ψi_s,q+

Z

R^Nb|v|^q−2vψdx

for all(ϕ,ψ)∈ W, is bounded inWby using the Hölder inequality, see [26]. Since(un,vn)* (u,v)inW, it follows that

nlim→_∞hL(u,v),(u_n−u,v_n−v)i=0. (3.14) Combining (3.13) with (3.14), we arrive at

nlim→_∞

hL(u_n,v_n)_,(u_n−u,v_n−v)i − hL(u,v)_,(u_n−u,v_n−v)i

=₀ A similar discussion as that in [26] yields that(u_n,v_n)→(u,v)inW.

Now we are in position to prove Theorem1.3.

Proof of Theorem1.3. We first show that functional I satisfies a mountain pass geometry.

For all(u,v)∈ W, withk(u,v)k ≤1, we have by (3.7) and the definition ofλ^∗ I(u,v)≥ ¹

p(kuk_E^p

p,a+kvk^q_E

q,b)− ¹ p

Z

R^N(λ(x) +ε)|(u,v)|^pdx−Cε

Z

R^N|(u,v)|^rdx

≥ ¹ p(kuk_E^p

p,a+kvk^q_E

q,b)− ¹

p(kλk_∞+ε)

Z

R^N|(u,v)|^pdx−CεC_r^rk(u,v)k^r

≥ ¹ p(kuk_E^p

p,a+kvk^p_E

q,b)− ¹ p

(kλk_∞+ε) λ^∗

(kuk^p_E

p,a+kvk_E^p

q,b)−C_εC_r^rk(u,v)k^r. Here we use the fact that kvk^q_E

q,b ≥ kvk^p_E

q,b, since kvk_Eq,b ≤ k(u,v)k ≤ 1. Taking ε =

1

2(λ^∗− kλk_∞), we obtain I(u,v)≥ ¹

2p

1− kλk_∞ λ^∗

(kuk_E^p

p,a+kvk^p_E

q,b)−Ck(u,v)k^r

≥ ¹ 2^pp

1−kλk_∞ λ^∗

(kuk_Ep,a+kvk_Eq,b)^p−Ck(u,v)k^r

= ¹ 2^pp

1−kλk_∞

λ^∗ −Ck(u,v)k^r−p

k(u,v)k^p.

Now we takek(u,v)k = ρ ∈ (0, 1) small enough such that 1− ^kλ_λ^k∗^∞ −Cρ^r−p > 0, thanks to kλk_∞ <λ^∗. Then

I(u,v)≥ ¹ 2^ppρ^p

1− kλk_∞

λ^∗ −Cρ^r−p

=:α>0

for all (u,v)∈W, withk(u,v)k=ρ.

In view of(H2)and(H3), for a positive constant A>0, there existsCA> 0 such that H(x,z)≥ A|z|^p−C_A for all (x,z)∈_R^N×_R².

Let B₁be the unit ball inR^N and letu^∗,v^∗ ∈ C₀^∞(B₁)two positive functions. Denote byu₀,v₀ the extension ofu^∗,v^∗ to zero out of B₁. Then, fort>1

I(tu0,tv0) = ^t

p

pku0k_E^p

p,a+^t

q

qkv0k^q_E

q,b−

Z

B1

H(x,tu0,tv0)dx

≤ ^t

p

p(ku₀k_E^p

p,a−Ak(u₀,v₀)k^p

L^p(B1)) + ^t

q

qkv₀k^q_E

q,b+C_A|B₁|. Choosing Alarge enough such thatku₀k^p_E

p,a < Ak(u₀,v₀)k^p

L^p(B1), one has I(tu₀,tv₀)→ −_∞as t → _∞. Thus, there exists (e₁,e₂) = (T₀u₀,T₀v₀) such that k(e₁,e₂)k > ρ and I(e₁,e₂) < 0.

Therefore, we have proved that I satisfies a mountain pass geometry. Combining this fact with Lemma3.4, there exists(0, 0)6= (u,v)∈Wsatisfying

hu,ϕi_s,p+hv,ψi_s,q+

Z

R^Na|u|^p−2uϕdx+

Z

R^Nb|v|^q−2vψdx

=

Z

R^N[Hu(x,u,v)u+Hv(x,u,v)v]dx, for all (ϕ,ψ) ∈ W. Taking (ϕ,ψ) = (u⁻,v⁻), we have (u⁻,v⁻) = (0, 0) a.e. in R^N, that is, u≥0 andv≥0 a.e. inR^N. This ends the proof.

Acknowledgements

The authors would like to thank the referee for his/her careful review and useful suggestions.

Mingqi Xiang was supported by Natural Science Foundation of China (No. 11601515) and Tianjin Key Lab for Advanced Signal Processing (No. 2016ASP-TJ02). Binlin Zhang was sup- ported by Natural Science Foundation of Heilongjiang Province of China (No. A201306) and Research Foundation of Heilongjiang Educational Committee (No. 12541667) and Doctoral Research Foundation of Heilongjiang Institute of Technology (No. 2013BJ15).

References

[1] R. A. Adams, J. J. F. Fournier, Sobolev spaces, Pure and Applied Mathematics (Amster- dam), Vol. 140, Elsevier/Academic Press, Amsterdam, 2003.MR2424078

[2] A. Ambrosetti, P. H. Rabinowitz, Dual variational methods in critical point theory and applications,J. Funct. Anal.14(1973), 349–381.MR0370183

[3] D. Applebaum, Lévy processes—from probability to finance and quantum groups,Notices Amer. Math. Soc.51(2004), 1336–1347.MR2105239

[4] G. Autuori, P. Pucci, Existence of entire solutions for a class of quasilinear elliptic equa- tions,NoDEA Nonlinear Differential Equations Appl.20(2013), 977–1009.MR3057162 [5] G. Autuori, P. Pucci, Elliptic problems involving the fractional Laplacian inR^N, J. Dif-

ferential Equations255(2013), 2340–2362.MR3082464