Stability of extremal domains for the first eigenvalue of the Laplacian operator

Abstract.

In this paper, we compute the second variation of the first Dirichlet eigenvalue on extremal domains and establish criteria for stability. We classify the stable extremal domains in the 2-sphere and in higher-dimensional spheres when the boundary is minimal. We also prove topological bounds for stable domains with nonnegative total Gaussian curvature in a general compact Riemannian surface.

1. Introduction

Let $(M^{n},g)$ be a complete Riemannian manifold of dimension $n\geq 2$. For each compact domain $\Omega\subset M$ with smooth boundary, let $\lambda_{1}(\Omega)>0$ denote the first eigenvalue of the Laplacian operator on $\Omega$ with Dirichlet boundary condition. Given $0<v<\operatorname{vol}(M,g)$, a very interesting isoperimetric probem in spectral geometry and shape optimization is that of investigating the minimizers or, more generally, the critical points of the functional $\Omega\subset M\mapsto\lambda_{1}(\Omega)$ defined on the class of domains $\Omega\subset M$ such that $\operatorname{vol}(\Omega)=v$.

Regarding to this problem, Faber [10] and Krahn [18] proved that, in $\mathbb{R}^{n}$, geodesic balls are the only global minimizers of $\Omega\subset\mathbb{R}^{n}\mapsto\lambda_{1}(\Omega)$ subject to a volume constraint. More precisely, they proved that if $\Omega\subset\mathbb{R}^{n}$ is a compact domain, then $\lambda_{1}(\Omega)\geq\lambda_{1}(B)$, where $B\subset\mathbb{R}^{n}$ is a geodesic ball such that $\operatorname{vol}(\Omega)=\operatorname{vol}(B)$, and equality holds if and only if $\Omega$ is a geodesic ball. We note that the analogous result is true in $\mathbb{S}^{n}$, the round sphere, and $\mathbb{H}^{n}$, the hyperbolic space (see, for example, Chapter IV, Section 2, of [6]).

In turn, it is known that a compact domain $\Omega_{0}\subset(M,g)$ with smooth boundary is a critical point of the functional $\Omega\subset M\mapsto\lambda_{1}(\Omega)$ with respect to volume preserving deformations of $\Omega_{0}$ in $M$ if and only if the first eigenfunctions of the Laplacian operator of $\Omega_{0}$ with Dirichlet boundary condition solve the following overdetermined problem:

$$\left\{\begin{array}[]{rl}\Delta\varphi+\lambda_{1}(\Omega_{0})\varphi=0&\text{in }\Omega,\\ \varphi=0&\text{on }\partial\Omega,\\ \dfrac{\partial\varphi}{\partial\nu}=c&\text{on }\partial\Omega,\end{array}\right.$$

where $c\neq 0$ is a constant and $\nu$ denotes the outward unit normal vector along $\partial\Omega_{0}$. This follows from the first variation formula for the first Dirichlet eigenvalue of the Laplacian operator, known as Hadamard formula, which was originally proved by Garabedian and Schiffer [12] in Euclidean space and later by El Soulfi and Ilias [8] to general Riemannian manifolds. From now on, following [22], we say that a compact domain $\Omega_{0}\subset(M,g)$ with smooth boundary is an extremal domain if it is a critical point of $\Omega\subset M\mapsto\lambda_{1}(\Omega)$ with respect to volume preserving deformations. We refer the reader to Section 2 for details and precise definitions.

In Euclidean space $\mathbb{R}^{n}$, in hyperbolic space $\mathbb{H}^{n}$, and in the round hemisphere $\mathbb{S}^{n}_{+}$, only geodesic balls are extremal domains. This result follows from the classical work of Serrin [27], and its extension to space forms by Kumaresan and Prajapat [19], on elliptic overdetermined problems, which was proved using the moving plane method introduced by Alexandrov in [2].

Extremal domains share many similarities with embedded hypersurfaces of constant mean curvature, and significant progress has been made in recent years regarding their existence, regularity, and classification. For example, in [22], Pacard and Sicbaldi demonstrated the existence of extremal domains with small volume in compact Riemannian manifolds $(M,g)$ by assuming that the scalar curvature of $M$ has a nondegenerate critical point. In [7], Delay and Sicbaldi improved this result by removing the condition on the scalar curvature of $M$.

Recently, Lamboley and Sicbaldi [20] established that for any connected compact Riemannian manifold $(M,g)$ and given $0<v<\operatorname{vol}(M,g)$, there exists an open set $\Omega^{\star}\subset M$ that is smooth except for a singular set of codimension less than $n-5$, satisfying

$$\lambda_{1}(\Omega^{\star})=\inf\{\lambda_{1}(\Omega):\Omega\subset M\text{ is an open subset and }\operatorname{vol}(\Omega)=v\}.$$

In particular, $\Omega^{\star}$ is a smooth extremal domain if $n=2,3$ or $4$. On the other hand, Espinar and Mazet [9] proved that simply connected extremal domains (and, in fact, $f$-extremal disks) in $\mathbb{S}^{2}$ are geodesic disks.

It is worth mentioning that all the above results have parallels in the context of hypersurfaces of constant mean curvature (see, for example, [31], [21], [3], [15], [13], and [17]). For more results on the existence of extremal domains, we refer the reader to [28] and [29].

Furthermore, numerous examples of extremal domains can be constructed on Riemannian manifolds endowed with an isometric action by a compact Lie group (see Section 2.1). In particular, this construction provides many examples of extremal domains on the round sphere $\mathbb{S}^{n}$ that are not geodesic balls. This shows that the moving plane method used by Serrin in [27] fails in this case.

In this paper, we initiate the investigation of the stability properties of extremal domains. We say that an extremal domain $\Omega\subset(M,g)$ is stable if it minimizes the first Dirichlet eigenvalue functional up to the second order with respect to volume preserving deformations.

In Section 3, we derive the second variation formula and establish a criterion for stability, obtaining important insights into the geometry of extremal domains. Using this criterion, we give a complete characterization of stable extremal domains in the round sphere $\mathbb{S}^{2}$. We prove the following theorem:

Theorem 1.1.

Let $\Omega\subset(\mathbb{S}^{2},g_{can})$ be an extremal domain. If $\Omega$ is stable, then it is necessarily a geodesic disk.

This theorem can be considered as the parallel of the Barbosa-do Carmo-Eschenburg theorem [4]. It is interesting to observe that Theorem 1.1, combined with Lamboley-Sicbaldi’s theorem proved in [20] and cited above, provides an alternative proof of the Faber-Krahn inequality in $\mathbb{S}^{2}$, which states that for any $0<v<4\pi$, if $\Omega\subset\mathbb{S}^{2}$ is a compact domain with area $v$, then:

$$\lambda_{1}(\Omega)\geq\lambda_{1}(D),$$

where $D\subset\mathbb{S}^{2}$ is a geodesic disk with area $v$ and equality holds if and only if $\Omega$ is a geodesic disk.

In the case that $n=2$ and $M^{2}=(\mathbb{S}^{2},g_{can})$, Theorems 1.1 and 1.2 in [20] imply that there exists a smooth domain $\Omega^{\star}$ such that $\lambda_{1}(\Omega)\geq\lambda_{1}(\Omega^{\star})$. In particular, $\Omega^{\star}$ is a stable extremal domain and, in fact, a global minimizer of $\lambda_{1}$ with an area constraint. By Theorem 1.1 above, we have that $\Omega^{\star}$ is a geodesic disk $D$. Moreover, if $\lambda_{1}(\Omega)=\lambda_{1}(D)$, then $\Omega$ is a stable extremal domain. Applying Theorem 1.1 again we have that $\Omega$ is also a geodesic disk, thus proving the Faber-Krahn inequality in $\mathbb{S}^{2}$.

In general, for stable extremal domains $\Omega$ in a given Riemannian surface $(M^{2},g)$, we prove the following restriction on the topology of $\Omega$.

Theorem 1.2.

Let $(M^{2},g)$ be an orientable compact Riemannian surface. If $\Omega\subset M$ is a stable extremal domain with nonnegative total Gauss curvature, that is, $\int_{\Omega}K_{g}\,da\geq 0$, then the only possible values for the genus $g_{\Omega}$ of $\Omega$ and the number $r$ of connected components of $\partial\Omega$ are:

1. $g_{\Omega}=0$ and $r\in\{1,2,3,4,5\}$;

2. $g_{\Omega}=1$ and $r\in\{1,2,3,4,5,6,7\}$.

In particular, it follows that stable extremal domains in a sphere $(\mathbb{S}^{2},g)$ with nonnegative curvature have at most $5$ boundary components.

Since the Faber-Krahn inequality is true on $\mathbb{S}^{n}$ for any $n\geq 2$, it is a very natural question to ask if the characterization proved in Theorem 1.1 can be extended to stable extremal domains in $\mathbb{S}^{n}$, for $n\geq 3$. By using a result due to Reilly (see [23, Theorem 4]), we are able to give the following description of stable extremal domains $\Omega\subset\mathbb{S}^{n}$, $n\geq 3$, in the case $\partial\Omega$ is a minimal hypersurface.

Theorem 1.3.

Let $\Omega\subset(\mathbb{S}^{n},g_{can})$ be an extremal domain such that $\partial\Omega$ is a compact minimal hypersurface. If $\Omega$ is stable, then $\Omega$ is a hemisphere.

Finally, we would like to mention that our methods are inspired by those used by Ros and Vergasta [25] to study stable constant mean curvature hypersurfaces with free boundary in convex domains of $\mathbb{R}^{n}$.

2. Basic definitions and first variation formula for $\lambda_{1}$

Let $(M,g)$ be a smooth Riemannian manifold. For each smooth compact domain $\Omega\subset M$, let $\lambda_{1}(\Omega)$ denote its first eigenvalue of the Laplacian operator with Dirichlet boundary condition. We will start this section by defining what we mean by a local deformation of $\Omega$ in $M$.

A smooth local deformation of $\Omega$ in $M$ is a smooth one-parameter family of domains $\Omega_{t}\subset M$, $t\in(-\varepsilon,\varepsilon)$, given by $\Omega_{t}=f_{t}(\Omega)$, where $f_{t}:M\to M$ is the smooth flow of some smooth vector field $V\in\mathfrak{X}(M)$. If, in addition, $\operatorname{vol}(\Omega_{t})=\operatorname{vol}(\Omega)$ for all $t\in(-\varepsilon,\varepsilon)$, then we say that $\Omega_{t}=f_{t}(\Omega)$ is a volume preserving local deformation of $\Omega$.

Let $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\varepsilon)$, be a smooth local deformation of a smooth compact domain $\Omega\subset M$. As the first Dirichlet eigenvalue of the Laplacian operator is simple, it follows from the implicit function theorem that the function $t\in(-\varepsilon,\varepsilon)\mapsto\lambda_{1}(\Omega_{t})$ is smooth.

Definition 2.1 (Extremal domains).

We say that a smooth compact domain $\Omega\subset(M,g)$ is extremal if

$$\left.\dfrac{d}{dt}\right|_{t=0}\lambda_{1}(\Omega_{t})=0,$$

for any volume preserving local deformation $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\epsilon)$, of $\Omega$ in $M$.

Next, we will state the Hadarmad formula proved by El Soufi and Ilias [8] which gives a first variation formula for the functional $\Omega\subset M\mapsto\lambda_{1}(\Omega)$.

Proposition 2.2 (El Soufi and Ilias, [8]).

Let $(M,g)$ be a smooth Riemannian manifold and let $\Omega\subset M$ be a smooth compact domain. Let $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\varepsilon)$, be a smooth local deformation of $\Omega$. We have

(2.1) $$\left.\dfrac{d}{dt}\right|_{t=0}\lambda_{1}(\Omega_{t})=-\int_{\partial\Omega}v\left(\dfrac{\partial\varphi}{\partial\nu}\right)^{2}d\ell,$$

where $\nu$ is the outward unit normal vector along $\partial\Omega$, $v=\left\langle V,\nu\right\rangle$ is the normal displacement of $\partial\Omega$ induced by the deformation,and $\varphi\in C^{\infty}(\Omega)$ is the first Dirichlet positive eigenfunction of the Laplacian operator on $\Omega$ such that $\|\varphi\|_{L^{2}(\Omega)}=1$

We notice that if $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\varepsilon)$, is volume preserving, that is, $\operatorname{vol}(\Omega_{t})=\operatorname{vol}(\Omega)$ for all $t\in(-\varepsilon,\varepsilon)$, then $\int_{\partial\Omega}v\,d\ell=0$. Conversely, if $v\in C^{\infty}(\partial\Omega)$ is such that $\int_{\partial\Omega}v\,d\ell=0$, then there exists a smooth local deformation $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\varepsilon)$, such that $df_{t}/dt|_{t=0}=v\nu$ on $\partial\Omega$.

Therefore, it follows as a consequence of (2.1), that $\Omega$ is an extremal domain if and only if its first eigenfunctions of Laplacian operator with Dirichlet boundary condition solve the following overdetermined elliptic problem:

$$\left\{\begin{array}[]{rl}\Delta\varphi+\lambda_{1}(\Omega)\varphi=0&\textrm{in }\Omega,\\ \varphi=0&\textrm{on }\partial\Omega,\\ \dfrac{\partial\varphi}{\partial\nu}=c&\textrm{on }\partial\Omega,\end{array}\right.$$

where $c\neq 0$.

2.1. Examples

Assume that there exists an isometric action of a compact Lie group $\mathsf{G}$ on a Riemannian manifold $M$, and let $\Omega\subset M$ be a domain invariant under the action of $\mathsf{G}$ with a single connected boundary component. If $\varphi$ is a first eigenfunction, then $\varphi\circ\theta$ also solves the Dirichlet eigenvalue problem in $\Omega=\theta(\Omega)$ for all $\theta\in\mathsf{G}$. Since $\lambda_{1}(\Omega)$ is simple, it follows that $\varphi\circ\theta=c_{\theta}\varphi$, defining a group homomorphism $\theta\mapsto c_{\theta}$. Due to the compactness of $\mathsf{G}$, we have $c_{\theta}=1$, proving that $\varphi$ is $\mathsf{G}$-invariant. This invariance implies that $\frac{\partial\varphi}{\partial\nu}$ is constant along $\partial\Omega$.

A basic non-trivial example using this construction is given by the action $\mathsf{G}=\mathsf{SO}(k+1)\times\mathsf{SO}(n-k)$ on the unit round sphere $\mathbb{S}^{n}\subset\mathbb{R}^{n+1}$ when we consider $\mathbb{R}^{n+1}=\mathbb{R}^{k+1}\oplus\mathbb{R}^{n-k}$. In this case, the invariant domains are the solid tori $\Omega_{r}$, $r\in(0,\pi/2)$, whose boundaries are given by the principal orbits of $\mathsf{G}$, $\Sigma(r)=\mathbb{S}^{k}(\cos r)\times\mathbb{S}^{n-k-1}(\sin r)$. In particular, $\partial\Omega(\pi/4)$ is a minimal hypersurface. Note that, by choosing suitable values for $r\in(0,\pi/2)$, we can also construct extremal $\mathsf{G}$-domains from this family with two boundary components.

Another family of examples of extremal domains we can construct using this method are the solid tori $D^{n-1}_{r}\times\mathbb{S}^{1}\subset T^{n}$, $r\in(0,1)$, where $T^{n}$ stands for the flat $n$-torus. From this particular family, Sicbaldi [28] and Schlenk-Sicbaldi [26] constructed new extremal domains using bifurcation techniques.

3. Second variation formula for $\lambda_{1}$ and stability of extremal domains

Let $(M,g)$ and $\Omega_{0}\subset M$ be a smooth Riemannian manifold and a smooth compact domain, respectively. Suppose that $\Omega_{0}$ is extremal. Since $\Omega_{0}$ is a critical point of the functional $\Omega\subset M\mapsto\lambda_{1}(\Omega)$, it is natural to investigate the second variation of this functional at $\Omega_{0}$. In this section, we will present a second variation formula for this functional, in a very general setting. As a consequence, we will give a criterion for a given extremal domain $\Omega_{0}\subset M$ to be stable.

Definition 3.1 (Stable extremal domains).

We say that an extremal domain $\Omega_{0}\subset(M,g)$ is stable if

$$\left.\dfrac{d^{2}}{dt^{2}}\right|_{t=0}\lambda_{1}(\Omega_{t})\geq 0,$$

for any volume preserving local deformation $\Omega_{t}=f_{t}(\Omega)$, $t\in(-\varepsilon,\epsilon)$, of $\Omega_{0}$ in $M$.

In the following proposition, we present a formula for the second variation for the functional $\Omega\subset M\mapsto\lambda_{1}(\Omega)$ at an extremal domain $\Omega_{0}$, with respect to volume preserving local deformations.

Proposition 3.2 (Second variation formula for $\lambda_{1}$).

Let $\Omega_{0}\subset(M,g)$ be an extremal domain and let $\varphi_{0}\in C^{\infty}(\Omega_{0})$ be the first Dirichlet positive eigenfunction of the Laplacian operator on $\Omega_{0}$ with $\|\varphi_{0}\|_{L^{2}(\Omega_{0})}=1$. For any volume preserving local deformation $\Omega_{t}=f_{t}(\Omega_{0})$, $t\in(-\varepsilon,\varepsilon)$, of $\Omega_{0}$ in $M$, we have

(3.1) $$\left.\dfrac{d^{2}}{dt^{2}}\right|_{t=0}\lambda_{1}(\Omega_{t})=2c^{2}\int_{\partial\Omega_{0}}\Big{(}v\frac{\partial\widehat{v}}{\partial\nu}+Hv^{2}\Big{)}\,d\ell,$$

where $H$ is the mean curvature of $\partial\Omega_{0}$ with respect to the outward unit normal vector $\nu$, $v=\left\langle df_{t}/dt|_{t=0},\nu\right\rangle$ is the normal displacement of $\partial\Omega_{0}$ induced by the deformation, $c=\partial\varphi_{0}/\partial\nu$ and $\widehat{v}$ is a $(\Delta+\lambda_{1}(\Omega_{0}))$-extension of $v$, that is, $\widehat{v}$ is a solution to the following problem:

$$\left\{\begin{array}[]{rl}\Delta\widehat{v}+\lambda_{1}\widehat{v}=0&\textrm{in }\Omega_{0},\\ \widehat{v}=v&\textrm{on }\partial\Omega_{0}.\end{array}\right.$$

Proof.

See the Appendix A. ∎

Remark 3.3.

Let $\Omega\subset(M,g)$ be a smooth compact domain and let $\varphi$ be a first Dirichlet eigenfunction of the Laplacian operator on $\Omega$. Given $v\in C^{\infty}(\Sigma)$, as a consequence of the Fredholm alternative, there exists a $(\Delta+\lambda_{1}(\Omega))$-extension $\widehat{v}$ of $v$ if and only if

(3.2) $$\int_{\partial\Omega}v\dfrac{\partial\varphi}{\partial\nu}\,d\ell=0.$$

Moreover, $\widehat{v}$ is unique up to an addition of a multiple of $\varphi$ (see, for example, Lemma 2.5 of [30]). In the case $\Omega_{0}\subset(M,g)$ is an extremal domain, we know that $\partial\varphi_{0}/\partial\nu$ is constant along $\partial\Omega$. Thus, (3.2) becomes

$$\int_{\partial\Omega_{0}}v\,d\ell=0,$$

which is equivalent to say that the local deformation of $\Omega_{0}$ given by $v$ is volume preserving.

Remark 3.4.

The second variation formula given by (3.1) is presented in [16, Theorem 2.5.6] in the case $(M,g)$ is the standard Euclidean space $\mathbb{R}^{n}$. To the best of the authors knowledge, it is the first time this formula for extremal domains in general Riemmanian manifolds appears in the literature.

We recall that volume preserving local deformations of $\Omega_{0}$ are in bijection with functions in the space $\mathcal{F}(\partial\Omega_{0})=\{v\in C^{\infty}(\partial\Omega_{0}):\int_{\partial\Omega_{0}}v\,d\ell=0\}$ of smooth functions on the boundary of $\Omega_{0}$ with zero average. Motivated by Proposition 3.1, we consider the quadratic form

$$Q:\mathcal{F}(\partial\Omega_{0})\times\mathcal{F}(\partial\Omega_{0})\to\mathbb{R}$$

given by

$$Q(v,w)=\int_{\partial\Omega_{0}}\left(v\frac{\partial\widehat{w}}{\partial\nu}+Hvw\right)\,d\ell,$$

where $\widehat{w}$ is a $(\Delta+\lambda_{1}(\Omega_{0}))$-extension of $w$ to $\Omega_{0}$. This setup allows us to define the index of an extremal domain.

Definition 3.5 (Index of Extremal Domains).

If $\Omega_{0}\subset(M,g)$ is an extremal domain, then the Morse index of $\Omega_{0}$ is given by

$$\operatorname{Ind}(\Omega_{0})=\max\{\dim V:V\subset\mathcal{F}(\partial\Omega_{0})\text{ and }Q\text{ is negative definite on }V\}.$$

By this definition, the index represents the number of linearly independent, volume-preserving variations that decrease the first Dirichlet eigenvalue. In particular, $\Omega_{0}$ is stable if and only if $\operatorname{Ind}(\Omega_{0})=0$.

Now, we have the following useful criterion for the stability of extremal domains.

Proposition 3.6.

An extremal domain $\Omega_{0}\subset(M,g)$ is stable if and only if

$$\int_{\Omega_{0}}|\nabla v|^{2}\,da-\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}v^{2}\,da+\int_{\partial\Omega_{0}}Hv^{2}\,d\ell\geq 0,$$

for all $v\in C^{\infty}(\Omega_{0})$ such that $\int_{\partial\Omega_{0}}v\,d\ell_{g}=0$.

Proof.

Given $v\in C^{\infty}(\Omega_{0})$, define

$$S(v,v)=S_{0}(v,v)+\int_{\partial\Omega_{0}}Hv^{2}\,d\ell,$$

where $S_{0}(u,v)=\int_{\Omega_{0}}\langle\nabla u,\nabla v\rangle\,da-\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}uv\,da$.

Integrating by parts, we obtain that

(3.3) $$S(\widehat{v|_{\partial\Omega_{0}}},\widehat{v|_{\partial\Omega_{0}}})=Q(v|_{\partial\Omega_{0}},v|_{\partial\Omega_{0}}),$$

where $\widehat{v|_{\partial\Omega_{0}}}$ is a $(\Delta+\lambda_{1}(\Omega_{0}))$-extension of $v|_{\partial\Omega_{0}}$.

Now, since $S_{0}(u,u)\geq 0$ for all $u\in C^{\infty}(\Omega_{0})$ such that $u=0$ on $\partial\Omega_{0}$, we have that

(3.4) $$S(\widehat{v|_{\partial\Omega_{0}}},\widehat{v|_{\partial\Omega_{0}}})\leq S(v,v),$$

for all $v\in C^{\infty}(\Omega_{0})$. Therefore, from (3.3) and (3.4) we conclude the proof. ∎

We conclude this section with some properties of Jacobi functions associated to the first Dirichlet eigenvalue.

We say that a function $v\in\mathcal{F}(\partial\Omega)$ is a Jacobi function of $\lambda_{1}$ if

$$\int_{\partial\Omega}\left(\dfrac{\partial\widehat{v}}{\partial\nu}+Hv\right)w\,d\ell=0,$$

for all $w\in\mathcal{F}(\partial\Omega)$.

The next lemma presents basic facts about Jacobi functions of $\lambda_{1}$ that will be used in Section 5.

Lemma 3.7.

Let $\Omega\subset(M,g)$ be an extremal domain.

• $v\in\mathcal{F}(\partial\Omega)$ is a Jacobi function of $\lambda_{1}$ if and only if there exist $c\in\mathbb{R}$ and a $(\Delta+\lambda_{1}(\Omega))$-extension $\widehat{v}$ of $v$ such that

  $$\dfrac{\partial\widehat{v}}{\partial\nu}+Hv=c\quad\text{on }\partial\Omega.$$

• If $\Omega$ is stable and $v\in\mathcal{F}(\partial\Omega)$ satisfies

  $$\int_{\partial\Omega}\left(\dfrac{\partial\widehat{v}}{\partial\nu}+Hv\right)v\,d\ell=0,$$

  then $v$ is a Jacobi function.

4. Instability of rotationally symmetric annuli in $\mathbb{S}^{2}$

Given $r_{0}\in(0,\pi/2)$, let $A_{r_{0}}\subset(\mathbb{S}^{2},g_{can})$ be the rotationally symmetric domain given by $A_{r_{0}}=\{p\in\mathbb{S}^{2}:d_{\mathbb{S}^{2}}(p,\gamma_{0})\leq r_{0}\}$, where $\gamma_{0}=\mathbb{S}^{2}\cap\{x_{3}=0\}$.

Since $A_{r_{0}}$ is rotationally symmetric and it is also symmetric with respect to $\gamma_{0}$, it is not difficult to see that $A_{r_{0}}$ is an extremal domain for all $r_{0}\in(0,\pi/2)$.

In this section, we will show that all these rotationally symmetric annuli are unstable extremal domains. We start with the following lemma.

Lemma 4.1.

Let $\Omega\subset\mathbb{S}^{2}$ be a stable extremal domain. If $\partial\Omega$ has at least two connected components and $\int_{\partial\Omega}x_{i}\,d\ell=0$ for all $i=1,2,3$, then $\lambda_{1}(\Omega)\leq 1$.

Proof.

By Proposition 3.6, the stability of $\Omega$ means that

(4.1) $$\int_{\partial\Omega}\kappa_{g}v^{2}\,d\ell+\int_{\Omega}|\nabla v|^{2}\,da-\lambda_{1}(\Omega)\int_{\Omega}v^{2}\,da\geq 0$$

for all $v\in C^{\infty}(\Omega)$ such that $\int_{\partial\Omega}v\,d\ell=0$, where $\kappa_{g}$ denotes the geodesic curvature of $\partial\Omega$.

Since $\int_{\partial\Omega}x_{i}\,d\ell=0$ for all $i=1,2,3$, we can use $x_{1},x_{2}$, and $x_{3}$ as test functions in (4.1). By summing over $i=1,2,3$, we get

(4.2) $$\int_{\partial\Omega}\kappa_{g}\,d\ell+2\operatorname{Area}(\Omega)\geq\lambda_{1}(\Omega)\operatorname{Area}(\Omega).$$

Let $r$ be the number of connected components of $\partial\Omega$. Since $r\geq 2$, we have by the Gauss-Bonnet Theorem that $\int_{\partial\Omega}\kappa_{g}\,d\ell=-\operatorname{Area}(\Omega)+2\pi(2-r)\leq-\operatorname{Area}(\Omega)$. Thus, by this inequality and by (4.2), we have

$$\operatorname{Area}(\Omega)\geq\lambda_{1}(\Omega)\operatorname{Area}(\Omega).$$

Therefore, $\lambda_{1}(\Omega)\leq 1$. ∎

Now we have:

Proposition 4.2.

The rotationally symmetric annuli $A_{r_{0}}\subset(\mathbb{S}^{2},g_{can})$ are not stable for any $r_{0}\in(0,\pi/2)$.

Proof.

First, note that we can write $g_{can}$ on $A_{r_{0}}$ as $g_{can}=dr^{2}+\cos^{2}r\,d\theta^{2}$ on $[-r_{0},r_{0}]\times\mathbb{S}^{1}$, where $r_{0}\in(0,\pi/2)$.

We note that the first eigenfunction of $\Delta_{g_{can}}$ on $A_{r_{0}}$ with Dirichlet boundary condition is a radial function, that is, it depends only on $r$.

If $\varphi=\varphi(r)$, then

$$\Delta_{g_{can}}\varphi=\varphi^{\prime\prime}-(\tan r)\varphi^{\prime}.$$

We would like to investigate the first eigenvalue of the following problem:

$$\varphi^{\prime\prime}-(\tan r)\varphi^{\prime}+\lambda\varphi=0,\quad\varphi(-r_{0})=\varphi(r_{0})=0.$$

Define $f(r)=\sqrt{\cos r}$. Since $f$ solves

(4.3) $$f^{\prime}=-\frac{1}{2}(\tan r)f,$$

we have, by a simple computation, that $w=f\varphi$ solves

$$w^{\prime\prime}-\frac{f^{\prime\prime}}{f}w+\lambda w=0,\quad w(-r_{0})=w(r_{0})=0.$$

As a consequence, we have

(4.4) $$\lambda_{1}(A_{r_{0}})=\inf\frac{\int_{-r_{0}}^{r_{0}}(w^{\prime})^{2}+\left(\frac{f^{\prime\prime}}{f}\right)w^{2}\,dr}{\int_{-r_{0}}^{r_{0}}w^{2}\,dr},$$

where the $\inf$ is taken over the set of functions $w$ defined on $[-r_{0},r_{0}]$ such that $w(-r_{0})=w(r_{0})=0$.

Our next goal is to find a lower estimate for $\lambda_{1}(A_{r_{0}})$.

First, it follows from (4.3) that

$$\frac{f^{\prime\prime}}{f}=-\frac{1}{2}-\frac{1}{4}\tan^{2}r.$$

Moreover, we have

$$\frac{\int_{-r_{0}}^{r_{0}}(w^{\prime})^{2}\,dr}{\int_{-r_{0}}^{r_{0}}w^{2}\,dr}\geq\left(\frac{\pi}{2r_{0}}\right)^{2}.$$

Note that for $|r|\leq r_{0}<\pi/3$, we have

(4.5) $$\left(\frac{f^{\prime\prime}}{f}\right)(r)\geq\left(\frac{f^{\prime\prime}}{f}\right)(r_{0})>-\frac{1}{2}-\frac{1}{4}\tan^{2}(\pi/3)=-\frac{5}{4}$$

and

(4.6) $$\left(\frac{\pi}{2r_{0}}\right)^{2}>\frac{9}{4}$$

for $|r|\leq r_{0}<\pi/3$.

Thus, it follows from (4.4) and estimates (4.5) and (4.6) that

$$\lambda_{1}\geq\left(\frac{\pi}{2r_{0}}\right)^{2}+\left(\frac{f^{\prime\prime}}{f}\right)(r_{0})>1.$$

From this and by using Lemma 4.1, we can conclude the instability of the extremal annuli $A_{r_{0}}\subset\mathbb{S}^{2}$ for all $r_{0}<\pi/3$.

Now, suppose that $r_{0}\geq\pi/3$ and define $\varphi(r)=\sin r/\sin r_{0}$ on $A_{r_{0}}$. Note that $\varphi(r_{0})=1$ and $\varphi(-r_{0})=-1$. Thus, $\int_{\partial A_{r_{0}}}\varphi\,d\ell=0$.

We have

$$\int_{\partial A_{r_{0}}}\kappa_{g}\varphi^{2}\,d\ell+\int_{A_{r_{0}}}|\nabla\varphi|^{2}\,da-\lambda_{1}(A_{r_{0}})\int_{A_{r_{0}}}\varphi^{2}\,da < \int_{\partial A_{r_{0}}}\kappa_{g}+\int_{A_{r_{0}}}|\nabla\varphi|^{2}\,da$$

$$= -\operatorname{Area}(A_{r_{0}})+\int_{A_{r_{0}}}|\nabla\varphi|^{2}\,da,$$

where we have used the Gauss-Bonnet Theorem in the last equality.

We have

(4.7) $$\operatorname{Area}(A_{r_{0}})=2\pi\int_{-r_{0}}^{r_{0}}\cos r\,dr=4\pi\sin r_{0}$$

(4.8) $$\int_{A_{r_{0}}}|\nabla\varphi|^{2}\,da=\frac{2\pi}{\sin^{2}r_{0}}\int_{-r_{0}}^{r_{0}}\cos^{3}r\,dr=\frac{4\pi}{\sin r_{0}}-\frac{4\pi}{3}\sin r_{0}.$$

By (4.7) and (4.8) we have

$$\int_{\partial A_{r_{0}}}\kappa_{g}\varphi^{2}\,d\ell+\int_{A_{r_{0}}}|\nabla\varphi|^{2}\,da-\lambda_{1}(A_{r_{0}})\int_{A_{r_{0}}}\varphi^{2}\,da<\frac{4\pi}{\sin r_{0}}\left(1-\frac{4}{3}\sin^{2}r_{0}\right)\leq 0,$$

since $r_{0}\geq\pi/3$.

Thus, we conclude that the annulus $A_{r_{0}}\subset\mathbb{S}^{2}$ is also unstable for $r_{0}\geq\pi/3$.

∎

5. Proof of Theorem 1.1

Let $\Omega\subset\mathbb{S}^{2}$ be a stable extremal domain and let $\partial^{1}\Omega,\partial^{2}\Omega,\ldots,\partial^{r}\Omega$, $r\geq 1$, be the connected components of $\partial\Omega$.

For each unit vector $a\in\mathbb{R}^{3}$, define the following vector field on $\mathbb{S}^{2}$: $V_{a}(x)=a\wedge x$, where $\wedge$ denotes the vector product in $\mathbb{R}^{3}$.

Now, define $\varphi_{a}:\partial\Omega\to\mathbb{R}$ by $\varphi_{a}(x)=\langle V_{a}(x),\nu(x)\rangle$, where $\nu$ denotes the outward unit normal vector along $\partial\Omega$. Since $\operatorname{div}_{\mathbb{S}^{2}}V_{a}=0$, we have

(5.1) $$\int_{\partial^{k}\Omega}\varphi_{a}\,d\ell=0,$$

for all $k=1,2,\ldots,r$.

The vector field $V_{a}$ generates a one-parameter family of rotations $f_{t}:\mathbb{S}^{2}\to\mathbb{S}^{2}$ around the vector $a$. Consider the local deformation of $\Omega$ in $\mathbb{S}^{2}$ given by $\Omega_{t}=f_{t}(\Omega)$. As $f_{t}$ is an isometry of $\mathbb{S}^{2}$ for all $t$, we have that $\lambda_{1}(\Omega_{t})=\lambda_{1}(\Omega)$ for all $t$. In particular,

$$\left.\dfrac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0}=0$$

and, by Proposition 3.2, this implies that

$$\int_{\partial\Omega}\left(\dfrac{\partial\widehat{\varphi_{a}}}{\partial\nu}+\kappa_{g}\varphi_{a}\right)\varphi_{a}\,d\ell=0.$$

Therefore, since $\Omega$ is stable, we have by Lemma 3.7 that $\varphi_{a}$ is a Jacobi function of $\lambda_{1}$, that is, there exists a function $\widehat{\varphi_{a}}\in C^{\infty}(\Omega)$ such that

(5.2) $$\left\{\begin{array}[]{rcl}\Delta\widehat{\varphi_{a}}+\lambda_{1}(\Omega)\widehat{\varphi_{a}}=0&\mbox{on}&\Omega,\vspace{0.2cm}\\ \dfrac{\partial\widehat{\varphi_{a}}}{\partial\nu}+\kappa_{g}\varphi_{a}=c&\mbox{on}&\partial\Omega.\end{array}\right.$$

After adding $\widehat{\varphi_{a}}$ to a constant multiple of $\varphi$, if necessary, where $\varphi$ is the first eigenfuncion of the Laplacian operator on $\Omega$ with Dirichlet boundary condition such that $\varphi>0$ and $\|\varphi\|_{L^{2}(\Omega)}=1$, we may assume without loss of generality that $c=0$ in the second equation above.

Next, fix $k\in\{1,2,\ldots,r\}$. We know that $\mathbb{S}^{2}\setminus\partial^{k}\Omega$ consists of two connected components, both homeomorphic to a disk. Let $W$ be one of these connected components. Note that $\partial W=\partial^{k}\Omega$. Now, let $D$ be the closed geodesic disk of largest radius such that $D\subset\overline{W}$. Because of the way $D$ was chosen, we have that $\partial D$ touches $\partial^{k}\Omega$ tangentially at least at two points. In particular, if $a\in D$ is the center of $D$, then $\varphi_{a}$ vanishes at least at two points on $\partial^{k}\Omega$. We claim that $\varphi_{a}$ vanishes at least at one more point of $\partial^{k}\Omega$. In fact, if $-a\in\partial^{k}\Omega$, we are done. If not, we let $W^{\prime}$ be the connected component of $\mathbb{S}^{2}\setminus\partial^{k}\Omega$ which contains $-a$ and consider $D^{\prime}$, the geodesic closed disk centered at $-a$ contained in $\overline{W^{\prime}}$ with the largest possible radius. In this case, we have that $\partial D^{\prime}$ touches $\partial^{k}\Omega$ tangentially at least at one point. Thus, this proves our claim.

Following ideas from Ros and Vergasta [25], we will prove in the following that $\Omega\setminus\widehat{\varphi_{a}}^{-1}(0)$ has at least three connected components. Let $M_{1},\ldots,M_{l}$ be the connected components of $\Omega\setminus\widehat{\varphi_{a}}^{-1}(0)$. By applying the Gauss-Bonnet Theorem to each $M_{i}$ we obtain

$$\operatorname{area}(M_{i})=2\pi\chi(M_{i})-\int_{\partial M_{i}}\kappa_{g}\,d\ell-\sum_{k=1}^{j_{i}}\theta^{i}_{k},$$

where $\theta^{i}_{k}$ are the external angles of $\partial M_{i}$.

By summing over $i=1,2,\ldots,l$, we get that

$$\operatorname{area}(\Omega)=2\pi\left(\sum_{i=1}^{l}\chi(M_{i})\right)-\int_{\partial\Omega}\kappa_{g}\,d\ell-\sum_{j}\theta_{j},$$

where $\sum_{j}\theta_{j}$ stands for the sum of the external angles of all nodal domains $M_{i}$.

By the Gauss-Bonnet Theorem applied to $\Omega$, we have

(5.3) $$2\pi(2-r)=2\pi\left(\sum_{i=1}^{l}\chi(M_{i})\right)-\sum_{j}\theta_{j}.$$

Now, it follows from (5.1) that $\varphi_{a}$ vanishes in at least two points of $\partial^{i}\Omega$, for all $i=1,2,\ldots,r$. Moreover, we know that for $\varphi_{a}$ vanishes in at least three points of $\partial^{k}\Omega$. This implies that

(5.4) $$\sum_{j}\theta_{j}\geq 2\pi\left(\dfrac{1}{2}+r\right).$$

Thus, by (5.3) and (5.4) we can conclude that

$$\sum_{i=1}\chi(M_{i})\geq\dfrac{5}{2}>2.$$

This implies that $\Omega\setminus(\widehat{\varphi_{a}})^{-1}(0)$ has at least three connected components.

But since $\Omega$ is a stable extremal domain and $\widehat{\varphi_{a}}$ solves (5.2) we have that $\Omega\setminus(\widehat{\varphi_{a}})^{-1}(0)$ has at most two connected components unless $\widehat{\varphi_{a}}\equiv 0$. Thus, it follows from the above argument that $\widehat{\varphi_{a}}\equiv 0$, which implies that $\Omega$ is rotationally symmetric.

Since the rotationally symmetric extremal annuli are unstable we have that $\Omega$ is a geodesic ball.

6. Proof of Theorem 1.2

Let $(M^{2},g)$ be an orientable Riemannian surface and let $\Omega\subset M$ be a stable extremal domain. Suppose that $\Omega$ has nonnegative total Gauss curvature, that is, $\int_{\Omega}K_{g}\,da\geq 0$.

By a result of Gabard [11, Théorème 7.2.], which improved a previous result due to Alfhors [1], there exists a proper conformal branched cover $\varphi=(\varphi_{1},\varphi_{2}):\Omega\to\mathbb{D}^{2}$, where $\mathbb{D}^{2}\subset\mathbb{R}^{2}$ is the closed unit disk, of degree at most $g_{\Omega}+r$. Moreover, after composing $\varphi$ with a conformal diffeomorphism of $\mathbb{D}^{2}$ if necessary, we may suppose that

$$\int_{\partial\Omega}\varphi_{i}\,d\ell=0,$$

for $i=1,2$.

Thus, by the stability of $\Omega$, we have that

$$\sum_{i=1}^{2}\int_{\partial\Omega}\kappa_{g}\varphi_{i}^{2}\,d\ell+\sum_{i=1}^{2}\int_{\Omega}|\nabla\varphi_{i}|^{2}\,da\geq\lambda_{1}(\Omega)\sum_{i=1}^{2}\int_{\Omega}\varphi_{i}^{2}\,da.$$

Since $\varphi$ is conformal and has degree at most $g_{\Omega}+r$,

$$\sum_{i=1}^{2}\int_{\Omega}|\nabla\varphi_{i}|^{2}\,da\leq 2\pi(g_{\Omega}+r).$$

Thus, it follows from this and from the Gauss-Bonnet Theorem that

$$2\pi(2-2g_{\Omega}-r)+2\pi(g_{\Omega}+r)\geq\lambda_{1}(\Omega)\sum_{i=1}^{2}\int_{\Omega}\varphi_{i}^{2}\,da+\int_{\Omega}K_{g}\,da>0.$$

Thus, $4\pi-2\pi g_{\Omega}>0$, that is, $g_{\Omega}<2$.

Now, in order to estimate the number $r$ of connected components of $\partial\Omega$, we use a similar, but different, balancing argument. Let $\overline{\Omega}$ denote a compact Riemannian surface obtained from $\Omega$ by attaching a conformal disk at any connected component of $\partial\Omega$, such that $\overline{\Omega}$ and $\Omega$ have the same genus. There exists a nonconstant holomorphic map $\psi=(\psi_{1},\psi_{2},\psi_{3}):\overline{\Omega}\to\mathbb{S}^{2}$ of degree at most $1+\left\lfloor(g_{\Omega}+1)/2\right\rfloor$ (see, for example, [14], p.261), where $\mathbb{S}^{2}\subset\mathbb{R}^{3}$ is the unit sphere centered at the origin. Again, after composing $\psi$ with a conformal diffeomorphism of $\mathbb{S}^{2}$ if necessary, we may assume that

$$\int_{\partial\Omega}\psi_{i}\,d\ell=0,$$

for $i=1,2,3$.

For each $i=1,2,3$, we define $\varphi_{i}=\psi_{i}|_{\Omega}$. Since $\sum_{i=1}^{3}\varphi_{i}^{2}=1$ in $\Omega$, it follows from the stability of $\Omega$ that

$$\int_{\partial\Omega}\kappa_{g}\,d\ell+8\pi\left(1+\left\lfloor\dfrac{g_{\Omega}+1}{2}\right\rfloor\right)\geq \int_{\partial\Omega}\kappa_{g}\,d\ell+\sum_{i=1}^{3}\int_{\overline{\Omega}}|\nabla\psi_{i}|^{2}\,da$$

$$> \int_{\partial\Omega}\kappa_{g}\,d\ell+\sum_{i=1}^{3}\int_{\Omega}|\nabla\varphi_{i}|^{2}\,da$$

$$\geq \,\lambda_{1}(\Omega)\operatorname{area}(\Omega)>0.$$

By the Gauss-Bonnet Theorem, we have that

$$12\pi-2\pi r-4\pi g_{\Omega}+8\pi\left\lfloor\dfrac{g_{\Omega}+1}{2}\right\rfloor>0.$$

Therefore, if $g_{\Omega}=0$, then $12\pi-2\pi r>0$ which implies that $r<5$. In the case $g_{\Omega}=1$, we have $16\pi-2\pi r>0$ and this implies $r<8$.

7. Proof of Theorem 1.3

Suppose that $\Omega\subset(\mathbb{S}^{n},g_{can})$ is a stable extremal domain such that $\partial\Omega$ is a compact minimal hypersurface, that is, $H_{\partial\Omega}=0$. It follows from a result due to Reilly [24, Theorem 4] that $\lambda_{1}(\Omega)\geq n$ with equality if and only if $\Omega$ is a hemisphere.

Since $\partial\Omega$ is minimal, it is well known that

$$\int_{\partial\Omega}x_{i}\,d\ell=0,$$

for all $i=1,2,\ldots,n+1$.

Thus, using the coordinate functions as test functions in Proposition 3.6, we get

$$\int_{\Omega}|\nabla x_{i}|^{2}\,da\geq\lambda_{1}(\Omega)\int_{\Omega}x_{i}^{2}\,da$$

for all $i=1,2,\ldots,n+1$.

By summing over $i=1,2,\ldots,n+1$ and using that $|\nabla x_{i}|^{2}=1-x_{i}^{2}$, we have

$$n\operatorname{vol}(\Omega)\geq\lambda_{1}(\Omega)\operatorname{vol}(\Omega).$$

This implies $\lambda_{1}(\Omega)\leq n$. Thus, $\lambda_{1}(\Omega)=n$ and $\Omega$ is a hemisphere.

Appendix A Proof of Proposition 3.2

Let $(M,g)$ be a Riemannian manifold and let $\Omega_{0}\subset M$ be an extremal domain. In this section, we will deduce the second variation formula (3.1) for the first eigenvalue functional $\Omega\mapsto\lambda_{1}(\Omega)$ at $\Omega_{0}$.

Let $\varphi_{0}\in C^{\infty}(\Omega_{0})$ be the first eigenfunction of the Laplacian operator on $\Omega_{0}$ with Dirichlet boundary condition such that $\varphi_{0}>0$ on $\Omega_{0}$ and $\|\varphi_{0}\|_{L^{2}(\Omega_{0})}=1$. Since $\Omega_{0}$ is extremal, there is a constant $c\in\mathbb{R}$ such that $\partial\varphi_{0}/\partial\nu_{0}=c$ on $\partial\Omega_{0}$.

For each smooth domain $\Omega\subset M$, define

$$J(\Omega)=\lambda_{1}(\Omega)+c^{2}\operatorname{vol}(\Omega).$$

Let $\Omega_{t}=f_{t}(\Omega_{0})$, $t\in(-\varepsilon,\varepsilon)$, be a local deformation of $\Omega_{0}$ in $M$ given by a smooth vector field $V\in\mathfrak{X}(M)$. It is well known that

$$\dfrac{d}{dt}\operatorname{vol}(\Omega_{t})=\int_{\partial\Omega_{t}}v_{t}\,d\ell_{t},\quad\forall\,t\in(-\varepsilon,\varepsilon),$$

where $v_{t}=\left\langle V,\nu_{t}\right\rangle$, $\nu_{t}$ denotes the outward unit normal vector along $\partial\Omega_{t}$, and $d\ell_{t}$ is the volume element of $\partial\Omega_{t}$ induced by $g$.

Therefore, by Proposition 2.2, we have

(A.1) $$\dfrac{d}{dt}J(\Omega_{t})=-\int_{\partial\Omega_{t}}v_{t}\left[\left(\dfrac{\partial\varphi_{t}}{\partial\nu_{t}}\right)^{2}-c^{2}\right]\,d\ell_{t},\quad\forall\,t\in(-\varepsilon,\varepsilon),$$

where $\varphi_{t}\in C^{\infty}(\Omega_{t})$ denotes the positive first eigenfunction of the Laplacian operator on $\Omega_{t}$ with Dirichlet boundary condition and such that $\|\varphi_{t}\|_{L^{2}(\Omega_{t})}=1$.

Since $\partial\varphi_{0}/\partial\nu_{0}=c$, it follows from (A.1) that $\Omega_{0}$ is a critical point of $J$ with respect to all local deformations $\Omega_{t}=f_{t}(\Omega_{0})$ of $\Omega_{0}$ in $M$, not only the volume-preserving ones. Moreover, we have

$$\left.\dfrac{d^{2}}{dt^{2}}J(\Omega_{t})\right|_{t=0} =-\int_{\partial\Omega_{0}}v_{0}\left[\left.\dfrac{d}{dt}\left(\frac{\partial\varphi_{t}}{\partial\nu_{t}}\right)^{2}\right|_{t=0}\right]\,d\ell_{0}$$

$$=-2\int_{\partial\Omega_{0}}v_{0}\dfrac{\partial\varphi_{0}}{\partial\nu_{0}}\left(\left.\dfrac{d}{dt}\dfrac{\partial\varphi_{t}}{\partial\nu_{t}}\right|_{t=0}\right)\,d\ell_{0}.$$

From now on, we will denote derivatives with respect to $t$ at $t=0$ by using dots.

Note that

$$\dot{\left(\frac{\partial\varphi_{t}}{\partial\nu_{t}}\right)} =\frac{d}{dt}\Big{|}_{t=0}\langle\nabla\varphi_{t},\nu_{t}\rangle$$

$$=\langle\nabla\dot{\varphi},\nu_{0}\rangle+\langle\nabla\varphi_{0},\dot{\nu}\rangle$$

$$=\frac{\partial\dot{\varphi}}{\partial\nu_{0}}+\frac{\partial\varphi_{0}}{\partial\dot{\nu}}.$$

Let $g_{t}=f_{t}^{*}(g)$ be the pullback metric of $g$ on $\Omega_{0}$. Let $h=\dot{g}$ be the derivative tensor at $t=0$. Note that $h=\mathcal{L}_{V}g$, where $\mathcal{L}$ denotes the Lie derivative.

Since $\nabla\varphi_{0}=\left(\partial\varphi_{0}/\partial\nu_{0}\right)\nu_{0}$ on $\partial\Omega_{0}$, we get $\langle\nabla\varphi_{0},\dot{\nu}\rangle=\left(\partial\varphi_{0}/\partial\nu_{0}\right)\langle\nu_{0},\dot{\nu}\rangle$. Moreover, since $\langle\nu_{t},\nu_{t}\rangle=1$ for all $t$, it follows directly that

$$\langle\nu_{0},\dot{\nu}\rangle=-\dfrac{1}{2}h(\nu_{0},\nu_{0}).$$

Therefore,

$$\left.\dfrac{d^{2}}{dt^{2}}J(\Omega_{t})\right|_{t=0} =-2\int_{\partial\Omega_{0}}v_{0}\dfrac{\partial\varphi_{0}}{\partial\nu_{0}}\dfrac{\partial\dot{\varphi}}{\partial\nu_{0}}\,d\ell_{0}+\int_{\partial\Omega_{0}}v_{0}\left(\dfrac{\partial\varphi_{0}}{\partial\nu_{0}}\right)^{2}h(\nu_{0},\nu_{0})\,d\ell_{0}$$

$$=-2c\int_{\partial\Omega_{0}}v_{0}\dfrac{\partial\dot{\varphi}}{\partial\nu_{0}}\,d\ell_{0}+c^{2}\int_{\partial\Omega_{0}}v_{0}h(\nu_{0},\nu_{0})\,d\ell_{0}.$$

Now, suppose that $\Omega_{t}=f_{t}(\Omega_{0})$, $t\in(-\varepsilon,\varepsilon)$, is a volume-preserving local deformation. Note that, in this case, we have

$$\left.\dfrac{d^{2}}{dt^{2}}J(\Omega_{t})\right|_{t=0}=\left.\dfrac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0}.$$

Moreover, we also have that $\int_{\partial\Omega_{0}}v_{0}\,d\ell_{0}=0$.

Let $\widehat{v}_{0}$ be a $(\Delta+\lambda_{1}(\Omega_{0}))$-extension of $v_{0}$, that is, $\widehat{v}_{0}$ solves the following problem (see Remark 3.3):

$$\left\{\begin{array}[]{rl}\Delta\widehat{v}_{0}+\lambda_{1}(\Omega_{0})\widehat{v}_{0}=0&\textrm{in }\Omega_{0},\\ \widehat{v}_{0}=v_{0}&\textrm{on }\partial\Omega_{0}.\end{array}\right.$$

Using that $\dot{\varphi}=0$ along $\partial\Omega_{0}$, the Green formula yields

$$\int_{\partial\Omega_{0}}v_{0}\frac{\partial\dot{\varphi}}{\partial\nu_{0}}\,d\ell_{0} =\int_{\Omega_{0}}\left(\widehat{v}_{0}\Delta\dot{\varphi}-\dot{\varphi}\Delta\widehat{v}_{0}\right)\,da_{0}$$

$$=\int_{\Omega_{0}}\widehat{v}_{0}\left(\Delta\dot{\varphi}+\lambda_{1}(\Omega_{0})\dot{\varphi}\right)\,da_{0},$$

where $da_{0}$ is the volume element of $\Omega_{0}$.

Now, differentiating the equation $\Delta_{g_{t}}\varphi_{t}+\lambda_{1}(\Omega_{t})\varphi_{t}=0$ we get

$$\dot{\Delta}\varphi_{0}+\Delta\dot{\varphi}+\dot{\lambda}_{1}\varphi_{0}+\lambda_{1}(\Omega_{0})\dot{\varphi}=0.$$

And so, since $\dot{\lambda}_{1}=0$ we have

$$\Delta\dot{\varphi}+\lambda_{1}(\Omega_{0})\dot{\varphi}=-\dot{\Delta}\varphi_{0}.$$

Combining these identities, we arrive at

(A.2) $$\left.\dfrac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0}=2c\int_{\Omega_{0}}\widehat{v}_{0}\dot{\Delta}\varphi_{0}\,da_{0}+c^{2}\int_{\partial\Omega_{0}}v_{0}h(\nu_{0},\nu_{0})\,d\ell_{0},$$

where $\dot{\Delta}\varphi_{0}=-\langle h,\nabla^{2}\varphi_{0}\rangle-\operatorname{div}h(\nabla\varphi_{0})+\frac{1}{2}\langle\nabla(\operatorname{tr}_{\Omega_{0}}h),\nabla\varphi_{0}\rangle$ (see [5, 8]).

Integrating by parts, we have

$$-\int_{\Omega_{0}}\langle h,\nabla^{2}\varphi_{0}\rangle\widehat{v}_{0}\,da_{0} =-\int_{\Omega_{0}}\operatorname{div}(h(\nabla\varphi_{0},\cdot)\widehat{v}_{0})\,da_{0}+\int_{\Omega_{0}}\operatorname{div}h(\nabla\varphi_{0})\widehat{v}_{0}\,da_{0}$$

$$\quad+\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}$$

$$=-\int_{\partial\Omega_{0}}h(\nabla\varphi_{0},\nu_{0})\widehat{v}_{0}\,d\ell_{0}+\int_{\Omega_{0}}\operatorname{div}h(\nabla\varphi_{0})\widehat{v}_{0}\,da_{0}$$

$$\quad+\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}.$$

Plugging this identity into (A.2) and noting that

$$h(\nabla\varphi_{0},\nu_{0})=\left(\partial\varphi_{0}/\partial\nu_{0}\right)h(\nu_{0},\nu_{0})=ch(\nu_{0},\nu_{0})\quad\text{on }\partial\Omega_{0},$$

we get

$$\left.\dfrac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0} =-2c\int_{\partial\Omega_{0}}h(\nabla\varphi_{0},\nu_{0})v_{0}\,d\ell_{0}+2c\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}$$

$$\quad+c\int_{\Omega_{0}}\widehat{v}_{0}\langle\nabla(\operatorname{tr}_{\Omega_{0}}h),\nabla\varphi_{0}\rangle\,da_{0}+c^{2}\int_{\partial\Omega_{0}}v_{0}h(\nu_{0},\nu_{0})\,d\ell_{0}$$

$$=-c^{2}\int_{\partial\Omega_{0}}vh(\nu_{0},\nu_{0})\,d\ell_{0}+2c\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}$$

$$\quad+c\int_{\Omega_{0}}\langle\nabla(\operatorname{tr}_{\Omega_{0}}h),\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}.$$

Using $h=\mathcal{L}_{V}g$, we have

$$\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0} =\int_{\Omega_{0}}\left(\mathcal{L}_{V}g\right)(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}$$

$$=\int_{\Omega_{0}}\left(\langle\nabla_{\nabla\varphi_{0}}V,\nabla\widehat{v}_{0}\rangle+\langle\nabla_{\nabla\widehat{v}_{0}}V,\nabla\varphi_{0}\rangle\right)\,da_{0}.$$

On the other hand,

$$\text{div}(\langle V,\nabla\widehat{v}_{0}\rangle\nabla\varphi_{0}) =\langle\nabla\langle V,\nabla\widehat{v}_{0}\rangle,\nabla\varphi_{0}\rangle+\langle V,\nabla\widehat{v}_{0}\rangle\Delta\varphi_{0}$$

$$=\langle\nabla_{\nabla\varphi_{0}}V,\nabla\widehat{v}_{0}\rangle+\langle V,\nabla_{\nabla\varphi_{0}}\nabla\widehat{v}_{0}\rangle-\lambda_{1}(\Omega_{0})\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0}$$

$$=\langle\nabla_{\nabla\varphi_{0}}V,\nabla\widehat{v}_{0}\rangle+\nabla^{2}\widehat{v}_{0}(V,\nabla\varphi_{0})-\lambda_{1}(\Omega_{0})\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0},$$

and similarly,

$$\text{div}(\langle V,\nabla\varphi_{0}\rangle\nabla\widehat{v}_{0}) =\langle\nabla_{\nabla\widehat{v}_{0}}V,\nabla\varphi_{0}\rangle+\nabla^{2}\varphi_{0}(V,\nabla\widehat{v}_{0})-\lambda_{1}(\Omega_{0})\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}.$$

Thus, we have that

$$\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}= \int_{\Omega_{0}}\text{div}(\langle V,\nabla\widehat{v}_{0}\rangle\nabla\varphi_{0})\,da_{0}-\int_{\Omega_{0}}\nabla^{2}\widehat{v}_{0}(V,\nabla\varphi_{0})\,da_{0}$$

$$+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\widehat{v}_{0}\rangle\varphi_{0}\,da_{0}+\int_{\Omega_{0}}\operatorname{div}(\langle V,\nabla\varphi_{0}\rangle\nabla\widehat{v}_{0})\,da_{0}$$

$$-\int_{\Omega_{0}}\nabla^{2}\varphi_{0}(V,\nabla\widehat{v}_{0})\,da_{0}+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}$$

$$= \int_{\partial\Omega_{0}}\left(\langle V,\nabla\widehat{v}_{0}\rangle\frac{\partial\varphi_{0}}{\partial\nu_{0}}+\langle V,\nabla\varphi_{0}\rangle\frac{\partial\widehat{v}_{0}}{\partial\nu_{0}}\right)\,d\ell_{0}$$

$$-\int_{\Omega_{0}}\left(\nabla^{2}\widehat{v}_{0}(V,\nabla\varphi_{0})+\nabla^{2}\varphi_{0}(V,\nabla\widehat{v}_{0})\right)\,da_{0}$$

$$+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0}\,da_{0}+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}.$$

Now, on $\partial\Omega_{0}$, we have $V=V^{T}+v_{0}\nu_{0}$ and $\nabla\varphi_{0}=c\nu_{0}$, where $V^{T}$ is the component of $V$ tangent to $\partial\Omega_{0}$. Moreover, note that

$$\nabla^{2}\widehat{v}_{0}(V,\nabla\varphi_{0})+\nabla^{2}\varphi_{0}(V,\nabla\widehat{v}_{0})=\langle\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle,V\rangle.$$

Thus

$$\int_{\Omega_{0}}h(\nabla\varphi_{0},\nabla\widehat{v}_{0})\,da_{0}= \int_{\partial\Omega_{0}}\left(\langle V^{T},\nabla\widehat{v}_{0}\rangle\frac{\partial\varphi_{0}}{\partial\nu_{0}}+v_{0}\frac{\partial\widehat{v}_{0}}{\partial\nu_{0}}\frac{\partial\varphi_{0}}{\partial\nu_{0}}+v_{0}\frac{\partial\varphi_{0}}{\partial\nu_{0}}\frac{\partial\widehat{v}_{0}}{\partial\nu_{0}}\right)\,d\ell_{0}$$

$$-\int_{\Omega_{0}}\langle\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle,V\rangle\,da_{0}+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0}\,da_{0}$$

$$+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}$$

$$= \,c\int_{\partial\Omega_{0}}\langle V^{T},\nabla\widehat{v}_{0}\rangle\,d\ell_{0}+2c\int_{\partial\Omega_{0}}v_{0}\frac{\partial\widehat{v}_{0}}{\partial\nu_{0}}\,d\ell_{0}$$

$$-\int_{\Omega_{0}}\langle\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle,V\rangle\,da_{0}+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0}\,da_{0}$$

$$+\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}.$$

Next, let us deal with the last integral in the formula of $\left.\frac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0}$ by applying integration by parts again and using that $\operatorname{tr}_{\Omega_{0}}h=2\operatorname{div}V$.

We have:

$$\int_{\Omega_{0}}\langle\nabla\operatorname{tr}_{\Omega_{0}}h,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}= \int_{\Omega_{0}}\operatorname{div}((\operatorname{tr}_{\Omega_{0}}h)\widehat{v}_{0}\nabla\varphi_{0})\,da_{0}-\int_{\Omega_{0}}(\operatorname{tr}_{\Omega_{0}}h)\operatorname{div}(\widehat{v}_{0}\nabla\varphi_{0})\,da_{0}$$

$$= \int_{\partial\Omega_{0}}(\operatorname{tr}_{\Omega_{0}}h)v_{0}\frac{\partial\varphi_{0}}{\partial\nu_{0}}\,d\ell_{0}-2\int_{\Omega_{0}}\operatorname{div}V\operatorname{div}(\widehat{v}_{0}\nabla\varphi_{0})\,da_{0}$$

$$= \,c\int_{\partial\Omega_{0}}(\operatorname{tr}_{\partial\Omega_{0}}h)v_{0}\,d\ell_{0}+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}$$

$$-2\int_{\Omega_{0}}\operatorname{div}V\operatorname{div}(\widehat{v}_{0}\nabla\varphi_{0})\,da_{0},$$

where in the last equality we have used that $\operatorname{tr}_{\Omega_{0}}h=\operatorname{tr}_{\partial\Omega_{0}}h+h(\nu_{0},\nu_{0})$. Now, since $V=\mathcal{L}_{V}g$, we have that

$$\int_{\Omega_{0}}\langle\nabla\operatorname{tr}_{\Omega_{0}}h,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}= \,2c\int_{\partial\Omega_{0}}(\operatorname{div}_{\partial\Omega_{0}}V)v_{0}\,d\ell_{0}+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}$$

$$-2\int_{\Omega_{0}}\operatorname{div}V(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle+\widehat{v}_{0}\Delta\varphi_{0})\,da_{0}$$

By using that $\Delta\varphi_{0}+\lambda_{1}(\Omega_{0})\varphi_{0}=0$, we obtain that

$$\int_{\Omega_{0}}\langle\nabla\operatorname{tr}_{\Omega_{0}}h,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}= \,2c\int_{\partial\Omega_{0}}(\operatorname{div}_{\partial\Omega_{0}}V)v_{0}\,d\ell_{0}+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}$$

$$-2\int_{\Omega_{0}}\operatorname{div}V(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle-\lambda_{1}(\Omega_{0})\widehat{v}_{0}\varphi_{0})\,da_{0}$$

$$= \,2c\int_{\partial\Omega_{0}}(\operatorname{div}_{\partial\Omega_{0}}V)v_{0}\,d\ell_{0}+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}$$

$$-2\int_{\Omega_{0}}\operatorname{div}(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle V)\,da_{0}+2\int_{\Omega_{0}}\langle V,\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle\rangle\,da_{0}$$

$$+2\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}(\operatorname{div}V)\widehat{v}_{0}\varphi_{0}\,da_{0}.$$

As $\operatorname{div}_{\partial\Omega_{0}}V=\operatorname{div}_{\partial\Omega_{0}}V^{T}+Hv_{0}$ on $\partial\Omega_{0}$, we obtain that

$$\int_{\Omega_{0}}\langle\nabla\operatorname{tr}_{\Omega_{0}}h,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0} =2c\int_{\partial\Omega_{0}}v_{0}\operatorname{div}_{\partial\Omega_{0}}V^{T}\,d\ell_{0}+2c\int_{\partial\Omega_{0}}H(v_{0})^{2}\,d\ell_{0}$$

$$\quad+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}-2\int_{\Omega_{0}}\operatorname{div}(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle V)\,da_{0}$$

$$\quad+2\int_{\Omega_{0}}\langle V,\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle\rangle\,da_{0}+2\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\operatorname{div}(\widehat{v}_{0}\varphi_{0}V)\,da_{0}$$

$$\quad-2\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}(\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}+\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0})\,da_{0}$$

$$=-2c\int_{\partial\Omega_{0}}\langle\nabla_{\partial\Omega_{0}}v_{0},V^{T}\rangle\,d\ell_{0}+2c\int_{\partial\Omega_{0}}H(v_{0})^{2}\,d\ell_{0}$$

$$\quad+c\int_{\partial\Omega_{0}}h(\nu_{0},\nu_{0})v_{0}\,d\ell_{0}-2\int_{\Omega_{0}}\operatorname{div}(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle V)\,da_{0}$$

$$\quad+2\int_{\Omega_{0}}\langle V,\nabla\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle\rangle\,da_{0}-2\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\varphi_{0}\rangle\widehat{v}_{0}\,da_{0}$$

$$\quad-2\lambda_{1}(\Omega_{0})\int_{\Omega_{0}}\langle V,\nabla\widehat{v}_{0}\rangle\varphi_{0}\,da_{0}.$$

Thus, substituting into the formula above for $\left.\frac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0}$ and canceling some terms, we have that

$$\left.\dfrac{d^{2}}{dt^{2}}\lambda_{1}(\Omega_{t})\right|_{t=0} =4c^{2}\int_{\partial\Omega_{0}}v_{0}\dfrac{\partial\widehat{v}_{0}}{\partial\nu_{0}}\,d\ell_{0}+2c^{2}\int_{\partial\Omega_{0}}H(v_{0})^{2}\,d\ell_{0}$$

$$\quad-2c\int_{\Omega_{0}}\operatorname{div}(\langle\nabla\widehat{v}_{0},\nabla\varphi_{0}\rangle V)\,d\ell_{0}$$

$$=2c^{2}\int_{\partial\Omega_{0}}\left(v_{0}\dfrac{\partial\widehat{v}_{0}}{\partial\nu_{0}}+H(v_{0})^{2}\right)\,d\ell_{0},$$

and this finishes the proof.

Acknowledgments

M.C. would like to express his gratitude to Princeton University for its hospitality, especially to Fernando Codá Marques, whose insightful questions inspired this work. This work was carried out while the authors were visiting ICTP - International Centre for Theoretical Physics, as associates. M.C. and I.N. acknowledge support from the ICTP through the Associates Programme (2022-2027 and 2019-2024, respectively). The authors are very grateful to the Institute and to Claudio Arezzo for their hospitality.

The authors were partially supported by the Brazilian National Council for Scientific and Technological Development (CNPq) under Grant 405468/ 2021-0. M.C. was also supported by CNPq under Grant 11136/ 2023-0 and by the Foundation for Research Support of the State of Alagoas (FAPEAL) under Grant 60030.0000000323/2023.