Charactarisation of distal actions of automorphisms on the space of one-parameter subgroups of Lie groups

Abstract.

For a connected Lie group $G$ and an automorphism $T$ of $G$, we consider the action of $T$ on Sub_G, the compact space of closed subgroups of $G$ endowed with the Chabauty topology. We study the action of $T$ on Sub${}^{p}_{G}$, the closure in Sub_G of the set of closed one-parameter subgroups of $G$. We relate the distality of the $T$-action on Sub${}^{p}_{G}$ with that of the $T$-action on $G$ and characterise the same in terms of compactness of the closed subgroup generated by $T$ in Aut$(G)$ when $T$ acts distally on the maximal central torus and $G$ is not a vector group. We extend these results to the action of subgroups ${\mathcal{H}}$ of Aut$(G)$ and equate the distal action of any closed subgroup ${\mathcal{H}}$ on Sub${}^{p}_{G}$ with that of every element in ${\mathcal{H}}$. Moreover, we show that a connected Lie group $G$ acts distally on Sub${}^{p}_{G}$ by conjugation if and only if $G$ is either compact or it is isomorphic to a direct product of a compact group and a vector group. Our results generalise some results proven by Shah and Yadav.

2020 Mathematics Subject Classification. Primary 37B05; Secondary 22E15, 22D45

Keywords. Distal actions of automorphisms, Lie groups, one-parameter subgroups, Chabauty topology.

1. Introduction

For a Hausdorff topological space, a homeomorphism $T$ of $X$ is said to be distal (equivalently, $T$ acts distally on $X$) if for any pair of distinct elements $x,y\in X$, the closure of the double orbit $\{(T^{n}(x),T^{n}(y))\mid n\in{\mathbb{Z}}\}$ in $X\times X$ does not intersect the diagonal, i.e. for $x,y\in X$ with $x\neq y$, $\overline{\{(T^{n}(x),T^{n}(y))\mid n\in{\mathbb{Z}}\}}\cap\{(a,a)\mid a\in X\}=\emptyset$. David Hilbert introduced the notion of distality to study non-ergodic actions on compact spaces (see Ellis [16] and Moore [29]). Distal actions on compact spaces, Lie groups as well as locally compact groups have been studied by many mathematicians in different contexts (see Abels [1, 2], Choudhuri et al [10], Ellis [16], Furstenberg [17], Palit-Shah [30], Palit-Prajapati-Shah [31], Raja-Shah [33, 34], Shah [36], Shah-Yadav [38, 39, 40], and references cited therein).

Let $G$ be a locally compact (Hausdorff) group with the identity $e$ and let ${\rm Sub}_{G}$ denote the set of all closed subgroups of $G$ endowed with the Chabauty topology. Note that ${\rm Sub}_{G}$ is compact and Hausdorff, and it is metrisable if $G$ is second countable [6]. Let ${\rm Aut}(G)$ denote the group of all (bicontinuous) automorphisms of $G$. There is a canonical group action of ${\rm Aut}(G)$ on ${\rm Sub}_{G}$; namely, $(T,H)\mapsto T(H)$, $T\in{\rm Aut}(G)$, $H\in{\rm Sub}_{G}$. Equivalently, we have a homomorphism from ${\rm Aut}(G)$ to ${\rm Homeo}({\rm Sub}_{G})$. As the image of ${\rm Aut}(G)$ under this action is a large subclass of homeomorphisms of ${\rm Sub}_{G}$, it is important to study the dynamics of the action of this special subclass in terms of distality. In [40], Shah and Yadav have studied the action of automorphisms of a connected Lie groups $G$ on ${\rm Sub}_{G}$ extensively and have shown in particular that for a large class of $G$ without the compact central subgroup of positive dimension, an automorphism $T$ of $G$ acts distally on ${\rm Sub}^{a}_{G}$, the space of closed abelian subgroups of $G$ if and only if $T$ generates a relatively compact subgroup of ${\rm Aut}(G)$. A similar characterisation also holds for $T$ belonging to the class (NC), i.e. those $T$ for which the closure of the $T$-orbit of any discrete cyclic group does not contain the trivial group $\{e\}$. Note that ${\rm Sub}^{a}_{G}$ is very large in many groups, e.g. it is the same as ${\rm Sub}_{G}$ if $G$ is abelian. It also contains all discrete cyclic subgroups. Shah, with Palit in [30], and with Palit and Prajapati in [31], has studied distal actions of automorphisms of a discrete group $G$ on ${\rm Sub}^{a}_{G}$, where $G$ is either polycyclic or a lattice in a connected Lie group.

Here, we would like to consider a closed subset of ${\rm Sub}^{a}_{G}$ containing smallest closed connected subgroups of $G$; namely, the closure in ${\rm Sub}_{G}$ of the set of all closed one-parameter subgroups of $G$; we denote this closed set by ${\rm Sub}^{p}_{G}$. Note that ${\rm Sub}^{p}_{G}$ is invariant under the action of ${\rm Aut}(G)$. We would like to study the dynamics of the actions of automorphisms on ${\rm Sub}^{p}_{G}$ and characterise them in terms of distality. In the case of ${\mathbb{R}}^{n}$, $n\geq 2$, ${\rm Sub}^{p}_{{\mathbb{R}}^{n}}$ is homeomorphic to ${\mathbb{RP}}^{n-1}\sqcup\{\{0\}\}$. It is indeed much smaller than ${\rm Sub}^{a}_{{\mathbb{R}}^{n}}={\rm Sub}_{{\mathbb{R}}^{n}}$. In general for a Lie group $G$, ${\rm Sub}^{p}_{G}$ does not contain any non-trivial discrete subgroup as it can not be a limit point of closed one-parameter subgroups (cf. Lemma 2.2). Theorem 4.1 of Shah and Yadav in [40] mainly deals with the action of automorphisms on discrete cyclic subgroups. Here, we are concentrating on the action of $T$ on ${\rm Sub}^{p}_{G}$, the smallest compact subset of ${\rm Sub}_{G}$ which contains all closed (connected) one-parameter subgroups of $G$. and get a characterisation for the distal action for a large class of connected non-abelian Lie groups $G$.

Theorem 1.1.

Let $G$ be a connected Lie group without any non-trivial compact connected central subgroup and let $T$ be an automorphism of $G$. Then the following hold:

1. If $G$ is abelian, then $G$ is isomorphic to ${\mathbb{R}}^{d}$ for some $d\in{\mathbb{N}}$, and $T$ acts distally on ${\rm Sub}^{p}_{G}$ if and only if $T\in{\mathcal{K}}{\mathcal{D}}$, where ${\mathcal{K}}$ is a compact subgroup of ${\rm GL}(d,{\mathbb{R}})$ and ${\mathcal{D}}$ is the center of ${\rm GL}(d,{\mathbb{R}})$.

2. If $G$ is not abelian, then $T$ acts distally on ${\rm Sub}^{p}_{G}$ if and only if $T\in{\mathcal{K}}$, a compact subgroup of ${\rm Aut}(G)$.

Note that the condition on the center in Theorem 1.1 holds if $G$ is simply connected, or more generally its nilradical is simply connected. It also holds for $G/M$ for any connected Lie group $G$, where $M$ is the maximal central torus in $G$ (the maximal compact connected central subgroup in $G$). The result in case of non-abelian groups $G$ in (2) as above gives a similar characterisation as in [40], even though the space ${\rm Sub}^{p}_{G}$ is much smaller than ${\rm Sub}^{a}_{G}$. In case $G$ as above is abelian, then $G={\mathbb{R}}^{d}$, a vector group, and ${\rm Sub}^{p}_{G}$ is homeomorphic to ${\mathbb{RP}}^{d-1}\sqcup\{\{0\}\}$ as noted earlier, and the Theorem 1.1 (1) shows that a larger class of $T$ act distally on ${\rm Sub}^{p}_{G}$.

In [40], it is shown for a connected Lie group without any compact central subgroup of positive dimension that if $T$ acts distally on ${\rm Sub}^{a}_{G}$, then it acts distally on $G$ (more generally, see Theorem 3.6 and Corollary 3.7 of [40]). We generalise this for Lie groups which are not vector groups; as in the later case, Theorem 1.1 (1) shows that it is possible to have automorphisms which act distally on ${\rm Sub}^{p}_{G}$ but not on $G$.

Theorem 1.2.

Let $G$ be a connected Lie group which is not a vector group and let $T\in{\rm Aut}(G)$ be such that it acts distally on the largest compact connected central subgroup of $G$. If $T$ acts distally on ${\rm Sub}^{p}_{G}$, then $T$ acts distally on $G$.

An automorphism $T$ of a connected Lie group $G$ is said to be unipotent if all the eigenvalues of ${\rm d\,}T$ are equal to 1, where ${\rm d\,}T$ is the corresponding Lie algebra automorphism of the Lie algebra of $G$. It is known that $T$ acts distally on $G$ if and only if all the eigenvalues of ${\rm d\,}T$ have absolute value 1 (cf. [1]). Any unipotent automorphism acts distally on $G$ and if it belongs to a compact subgroup of ${\rm Aut}(G)$, then it is trivial. Theorem 4.3 of [40] shows that no non-trivial unipotent automorphism acts distally on ${\rm Sub}^{a}_{G}$; generalising this statement we prove the following.

Theorem 1.3.

Let $G$ be a connected Lie group and let $T\in{\rm Aut}(G)$ be unipotent. Then $T$ acts distally on ${\rm Sub}^{p}_{G}$ if and only if $T={\rm Id}$, the identity map.

Theorem 1.3 illustrates in particular that the converse of Theorem 1.2 does not hold as the non-trivial unipotent automorphisms do not act distally on ${\rm Sub}^{p}_{G}$.

Let ${\rm Sub}^{c}_{G}$ denote the set of all closed discrete cyclic subgroups of $G$. It is invariant under the action of ${\rm Aut}(G)$ and so is its closure in ${\rm Sub}_{G}$. In [30] and [31], distal actions of automorphisms of ${\rm Sub}^{c}_{\Gamma}$ have been characterised, where $\Gamma$ is a certain type of discrete group, e.g. a discrete polycyclic group or a lattice in a connected Lie group. In a Lie group $G$, any closed one-parameter subgroup $H=\{x_{t}\}_{t\in{\mathbb{R}}}$ is the limit of $\{H_{n}\}_{n\in{\mathbb{N}}}$ as $n\to\infty$, where $H_{n}=\{x_{m/n}\mid m\in{\mathbb{Z}}\}$. Here, $H_{n}$ is a closed discrete cyclic group generated by $x_{1/n}$. If $H$ is compact, we may assume that $x_{1}=e$, the identity in $G$. Thus ${\rm Sub}^{p}_{G}\subset\overline{{\rm Sub}^{c}_{G}}$ (the closure of ${\rm Sub}^{c}_{G}$ in ${\rm Sub}_{G})$ and the latter is contained in ${\rm Sub}^{a}_{G}$. We should note that $\overline{{\rm Sub}^{c}_{G}}$ is much bigger than ${\rm Sub}^{p}_{G}$ as the latter contains all discrete cyclic subgroups, and these do not belong to ${\rm Sub}^{p}_{G}$ (see Lemma 2.2). It is easy to see that Theorems 1.1 (2), 1.2 and 1.3 remain valid if ${\rm Sub}^{p}_{G}$ is replaced by $\overline{{\rm Sub}^{c}_{G}}$ in the respective statement.

The following generalises a part of Theorem 4.1 of [40] and also Theorem 1.1 (2).

Theorem 1.4.

Let $G$ be a connected Lie group such that it is not a vector group. Let $T\in{\rm Aut}(G)$ be such that it acts distally on the largest compact connected central subgroup of $G$. Then the following are equivalent:

1. $T$ acts distally on ${\rm Sub}^{p}_{G}$.

2. $T$ acts distally on $\overline{{\rm Sub}^{c}_{G}}$.

3. $T$ acts distally on ${\rm Sub}^{a}_{G}$.

4. $T$ acts distally on ${\rm Sub}_{G}$.

5. $T$ is contained in a compact subgroup of ${\rm Aut}(G)$.

We may note that in case $G$ is a vector group, Theorem 1.1 (1) shows that $(1)$ and $(5)$ above are not equivalent; however, in this case, statements $(2-5)$ above are equivalent by Theorem 4.1 of [40] as (2) above implies that $T\in({\rm NC})$, i.e. $\{e\}$ is not a limit point of $\{T^{n}(H)\}_{n\in{\mathbb{Z}}}$ for any closed discrete cyclic subgroup $H$ of $G$.

Recall that a semigroup $\Gamma$ of homeomorphisms of a (Hausdorff) topological space $X$ acts distally on $X$ if $\overline{\{(\gamma(x),\gamma(y))\mid\gamma\in\Gamma\}}\cap\{(d,d)\mid\ d\in X\}=\emptyset$. Note that a homeomorphism $T$ of $X$ acts distally on $X$ if and only if the cyclic group $\{T^{n}\mid n\in{\mathbb{Z}}\}$ acts distally on $X$. For a Lie group $G$, any compact subgroup of ${\rm Aut}(G)$ acts distally on ${\rm Sub}_{G}$ (see e.g. Lemma 2.2 in [39]), It follows from Theorem 1.1 of [2] that ${\mathcal{H}}$ acts distally on $G$ if and only if ${\mathcal{H}}$ acts distally on the Lie algebra of $G$, and these statements are equivalent to the statement that every element of ${\mathcal{H}}$ acts distally on $G$ (see also [11]). The following result generalises Theorems 1.1 and 1.4 to the action of any subgroup ${\mathcal{H}}$ of ${\rm Aut}(G)$ and, if ${\mathcal{H}}$ is closed, then it equates the distal action of ${\mathcal{H}}$ on ${\rm Sub}^{p}_{G}$ with that of every element of ${\mathcal{H}}$. Let $\overline{{\mathcal{H}}}$ denote the closure of ${\mathcal{H}}$ in ${\rm Aut}(G)$. It is easy to see using Abel’s results in [1, 2] that ${\mathcal{H}}$ acts distally on $G$ if and only if so does $\overline{{\mathcal{H}}}$. It follows from Ellis’ result in [16] that ${\mathcal{H}}$ acts distally on any compact ${\mathcal{H}}$-invariant subspace of ${\rm Sub}_{G}$ if and only if so does $\overline{{\mathcal{H}}}$.

Theorem 1.5.

Let $G$ be a connected Lie group. Let ${\mathcal{H}}$ be a subgroup of ${\rm Aut}(G)$ such that it acts distally on the largest compact connected central subgroup of $G$. Consider the following statements:

1. Every element of $\overline{{\mathcal{H}}}$ acts distally on ${\rm Sub}^{p}_{G}$.

2. ${\mathcal{H}}$ acts distally on ${\rm Sub}^{p}_{G}$.

3. ${\mathcal{H}}$ acts distally on $\overline{{\rm Sub}^{c}_{G}}$.

4. ${\mathcal{H}}$ acts distally on ${\rm Sub}^{a}_{G}$.

5. ${\mathcal{H}}$ acts distally on ${\rm Sub}_{G}$.

6. $\overline{{\mathcal{H}}}$ is a compact group.

   Then the following hold:

   1. $(1)$ and $(2)$ are equivalent.

   2. $(3-6)$ are equivalent.

   3. If $G$ is not a vector group, then $(1-6)$ are equivalent.

   4. If $G={\mathbb{R}}^{d}$, for some $d\in{\mathbb{N}}$, then each of $(1)$ and $(2)$ is equivalent to the following:

7. ${\mathcal{H}}\subset{\mathcal{K}}{\mathcal{D}}$, where ${\mathcal{K}}$ is a compact subgroup of ${\rm GL}(d,{\mathbb{R}})$ and ${\mathcal{D}}$ is the center of ${\rm GL}(d,{\mathbb{R}})$.

Note that in Theorem 1.5, one can also have equivalent statements similar to $(1-2)$, where ${\rm Sub}^{p}_{G}$ is replaced by any of $\overline{{\rm Sub}^{c}_{G}}$, ${\rm Sub}^{a}_{G}$ or ${\rm Sub}_{G}$ for the action of a subgroup ${\mathcal{H}}$. Statement (6) in the theorem implies that that ${\mathcal{H}}$ acts distally on $G$, and hence the assertion (c) above implies that Theorem 1.2 remains valid for the action of a subgroup of ${\rm Aut}(G)$ instead of an automorphism. We may note here that there is a subgroup ${\mathcal{H}}$ of ${\rm SL}(4,{\mathbb{C}})$ in which every element of ${\mathcal{H}}$ is contained in a compact subgroup of ${\rm SL}(4,{\mathbb{C}})$ but the closure of ${\mathcal{H}}$ in ${\rm SL}(4,{\mathbb{C}})$ is connected and contains unipotent elements (cf. [5]). Hence, one can not replace $\overline{{\mathcal{H}}}$ by ${\mathcal{H}}$ in (1) of Theorem 1.5.

The condition that $T$ (resp. a subgroup ${\mathcal{H}}$ of ${\rm Aut}(G)$) acts distally on the maximal compact connected central subgroup (maximal central torus) of $G$ is satisfied easily if $T\in{\rm Aut}(G)^{0}$, (resp. ${\mathcal{H}}\subset{\rm Aut}(G)^{0}$), as ${\rm Aut}(G)^{0}$ acts trivially on any compact central subgroup. It is also satisfied if $T^{n}\in{\rm Aut}(G)^{0}$ for some $n\in{\mathbb{N}}$ (resp. ${\mathcal{H}}$ or ${\mathcal{H}}{\rm Aut}(G)^{0}$ has finitely many connected components). Here, ${\rm Aut}(G)^{0}$ denotes the connected component of identity in ${\rm Aut}(G)$.

Let ${\rm Inn}(G)$ denote the group of inner automorphisms of $G$. It is a (not necessarily closed) normal subgroup of ${\rm Aut}(G)$. The following corollaries generalises the first part of Corollaries 4.5 [40].

Corollary 1.6.

Let $G$ be a connected Lie group. Then the following are equivalent:

1. Every inner automorphism of $G$ acts distally on ${\rm Sub}^{p}_{G}$.

2. Every inner automorphism of $G$ acts distally on ${\rm Sub}_{G}$.

3. ${\rm Inn}(G)$ acts distally on ${\rm Sub}^{p}_{G}$.

4. ${\rm Inn}(G)$ acts distally on ${\rm Sub}_{G}$.

5. $G$ is compact or $G={\mathbb{R}}^{n}\times K$, for some $n\in{\mathbb{N}}$ and the maximal compact normal subgroup $K$ of $G$.

In Corollary 1.6, one can also write equivalent statements involving the action on $\overline{{\rm Sub}^{c}_{G}}$.

Note that ${\rm Sub}^{p}_{G}$ is contained in a larger compact set ${\rm Sub}^{co}_{G}$, which is the closure of the set of all closed connected subgroups of $G$. Hence it is easy to show that Theorems 1.1–1.3 are all valid if ${\rm Sub}^{p}_{G}$ is replaced by ${\rm Sub}^{co}_{G}$. In Theorems 1.4 and 1.5 and Corollary 1.6, the action of the given automorphism or a subgroup of ${\rm Aut}(G)$ is distal on ${\rm Sub}^{co}_{G}$ if and only if it is so on ${\rm Sub}^{p}_{G}$.

Ellis [16] has shown that a semigroup $\Gamma$ of homeomorphisms of a compact space $X$ acts distally on $X$ if and only if the closure of $\Gamma$ in $X^{X}$ is a group. Note that minimal distal actions on compact spaces are characterised by Furstenberg [17]. For many groups $G$, compact spaces ${\rm Sub}_{G}$, Sub${}^{a}_{G}$ and ${\rm Sub}^{c}_{G}$ are identified (see Baik-Clavier [3, 4], Bridson et al [8], Hamrouni and Kadri [19], Kloeckner [27] and also Pourezza and Habbard [32]); one can also identify ${\rm Sub}^{p}_{G}$ for some groups. The action of ${\rm Aut}(G)$ on some subspaces of ${\rm Sub}_{G}$ for the 3-dimensional Heisenberg group has also been described in detail [8]. The space of $G$-invariant measures on Sub_G and in particular, ergodic $G$-invariant measures on Sub_G for some $G$ have also been studied extensively (see Gelander [18] and the references cited therein). Since automorphisms of $G$ corresponds to a large subclass of homeomorphisms of ${\rm Sub}_{G}$ it is of significant interest to study the dynamics of the action of ${\rm Aut}(G)$ on closed (compact) invariant subspaces of ${\rm Sub}_{G}$.

Throughout, $G$ is a connected Lie group unless specified otherwise. Let $R$ denote the radical of $G$, i.e. the maximal connected solvable normal subgroup of $G$ and let $N$ denote the nilradical of $G$, i.e. the maximal connected nilpotent normal subgroup of $G$. Both the radical and the nilradical of $G$ are characteristic in $G$. Recall that $G$ is said to be semisimple if its radical is trivial. In particular, $G/R$ is semisimple. For a closed subgroup $H$ of $G$, let $H^{0}$ denote the connected component of the identity $e$ in $H$, $[H,H]$ denote the commutator subgroup of $H$, i.e. $[H,H]$ is generated by $\{xy,x^{-1}y^{-1}\mid x,y\in H\}$. Note that $H^{0}$, $[H,H]$ and its closure $\overline{[H,H]}$ are characteristic in $H$.

In §2, we study general properties of ${\rm Sub}_{G}$ and prove some elementary results on the structure of ${\rm Sub}^{p}_{G}$. In §3, we discuss some preliminary results on the action of ${\rm Aut}(G)$ on $G$ and ${\rm Sub}^{p}_{G}$ and prove main results stated above.

2. Actions of automorphisms of $G$ on ${\mathrm{Sub}}_{G}$ and ${\mathrm{Sub}}^{p}_{G}$

In this section, we discuss some basic properties of Chabauty topology on the space ${\rm Sub}_{G}$ of closed subgroups of $G$, the structure of ${\rm Aut}(G)$ and its action on ${\rm Sub}_{G}$. We also discuss some elementary results about the structure of ${\rm Sub}_{G}$ and ${\rm Sub}^{p}_{G}$.

Let $G$ be a locally compact Hausdorff topological group. For a compact set $K$ and an open set $U$ in $G$, we set $U_{1}(K)=\{H\in{\rm Sub}_{G}\mid H\cap K=\emptyset\}$ and $U_{2}(U)=\{H\in{\rm Sub}_{G}\mid H\cap U\neq\emptyset\}$. Here, $\{U_{1}(K)\mid K\mbox{ is compact}\}\cup\,\{U_{2}(U)\mid U\mbox{ is open}\}$ is a sub-basis of Chabauty topology on ${\rm Sub}_{G}$. As mentioned earlier, ${\rm Sub}_{G}$ is compact and Hausdorff, and if $G$ is second countable, then ${\rm Sub}_{G}$ is metrisable (cf.  [6], Lemma E.1.1). In particular, for a connected Lie group and its closed subgroups $G$, $Sub_{G}$ is metrisable.

Lemma 2.1.

Let $G$ be a locally compact second countable group. A sequence $\{H_{n}\}\subset{\rm Sub}_{G}$ converges to $H\in{\rm Sub}_{G}$ if and only if the following hold:

1. For $g\in G$, if there exists a subsequence $\{H_{n_{k}}\}$ of $\{H_{n}\}$ with $h_{k}\in H_{n_{k}}$, $k\in{\mathbb{N}}$, such that $h_{k}\to g$ in $G$, then $g\in H$.

2. For every $h\in H$, there exists a sequence $\{h_{n}\}_{n\in{\mathbb{N}}}$ such that $h_{n}\in H_{n}$, $n\in{\mathbb{N}}$, and $h_{n}\to h$.

The following lemma is well-known. We include a short proof for the sake of completeness.

Lemma 2.2.

Let $G$ be a second countable Lie group. Suppose $\{H_{n}\}_{n\in{\mathbb{N}}}$ in ${\rm Sub}_{G}$ is such that it converges to $H\in{\rm Sub}_{G}$. If $H$ is discrete, then $H_{n}$ is discrete for all large $n$. In particular, the set of discrete subgroups of $G$ is open in ${\rm Sub}_{G}$.

Proof.

It is enough to show that $H_{n}^{0}$ is trivial for all large $n$. As $G^{0}$ is open in $G$, we get that $H_{n}\cap G^{0}\to H\cap G^{0}$ and the latter is discrete. As $H_{n}^{0}\subset G^{0}$, we may replace $H_{n}$ by $H_{n}\cap G^{0}$, $H$ by $H\cap G^{0}$, and $G$ by $G^{0}$ and assume that $G$ is connected. As $G$ is a Lie group, there exists a neighbourhood $U$ of the identity $e$ in $G$ such that the only subgroup contained in $U$ is the trivial subgroup $\{e\}$. As $H$ is discrete, there exists a neighbourhood $W$ of the identity $e$ in $G$ such that $H\cap W=\{e\}$. Since $G$ is locally compact and Hausdorff, there exist an open set $V$ and a compact set $K$ such that $e\in V\subset K\subset U\cap W$. As $G$ is connected, $K\cap V^{c}$ is a nonempty compact set and

$$H\cap(K\cap V^{c})\subset H\cap W\cap V^{c}=\emptyset,$$

and hence $H\in U_{1}(K\cap V^{c})$ which is an open set in ${\rm Sub}_{G}$. As $H_{n}\to H$ in ${\rm Sub}_{G}$, we get that for all large $n$, $H_{n}\in U_{1}(K\cap V^{c})$, i.e. $H_{n}\cap(K\cap V^{c})=\emptyset$, and hence $H_{n}\subset V\sqcup K^{c}$. Thus for all large $n$, $H_{n}^{0}\subset V\subset U$, and it follows that $H_{n}^{0}=\{e\}$, and hence $H_{n}$ is discrete. The second assertion easily follows from the first. ∎

Recall that an automorphism of $G$ is a homomorphism and a homeomorphism, and ${\rm Aut}(G)$ denotes the group of all automorphisms of $G$. There is a natural group action of ${\rm Aut}(G)$ on ${\rm Sub}_{G}$ defined as follows:

$${\rm Aut}(G)\times{\rm Sub}_{G}\to{\rm Sub}_{G},\ (T,H)\mapsto T(H);\ T\in{\rm Aut}(G),\ H\in{\rm Sub}_{G}.$$

It is easy to see that the map $H\mapsto T(H)$ defines a homeomorphism of ${\rm Sub}_{G}$ for every $T\in{\rm Aut}(G)$, and the corresponding map is $\phi:{\rm Aut}(G)\rightarrow\mbox{Homeo}({\rm Sub}_{G})$, where $\phi(T)(H)=T(H)$. It is easy to check that $\phi$ is a group homomorphism.

Recall that for a locally compact Hausdorff group $G$, if ${\rm Aut}(G)$ is equipped with the compact-open topology, then

$${\rm Aut}(G)\times G\rightarrow G,(T,x)\mapsto T(x),T\in{\rm Aut}(G),x\in G,$$

is continuous. If we consider the modified compact-open topology on ${\rm Aut}(G)$, it is a topological group [42]. In Lemma 2.4 of [40], it is proved that the action of the group ${\rm Aut}(G)$ on ${\rm Sub}_{G}$ is continuous, implying that $\phi$ is continuous. Note that for a connected Lie group $G$, the compact-open topology and the modified compact-open topology on ${\rm Aut}(G)$ are same. In particular, every $T$ which generates a relatively compact group in ${\rm Aut}(G)$ acts distally on ${\rm Sub}_{G}$ (see e.g. [30], Lemma 4.2).

Let $G$ be a connected Lie group with the Lie algebra ${\mathfrak{g}}$. There is an exponential map $\exp:{\mathfrak{g}}\to G$ which is a continuous map and its restriction to a small open neighbourhood of $0$ in ${\mathfrak{g}}$ is a homeomorphism onto an open neighbourhood of the identity $e$ in $G$. Given $T\in{\rm Aut}(G),$ there exists a corresponding Lie algebra automorphism ${\rm d\,}T:{\mathfrak{g}}\to{\mathfrak{g}}$ such that $\exp\circ\,{\rm d\,}T=T\circ\,\exp$. The map ${\rm d\,}:{\rm Aut}(G)\rightarrow{\rm Aut}({\mathfrak{g}})$, $T\mapsto{\rm d\,}T$ is an injective continuous homomorphism and its image is closed, and in fact, ${\rm d\,}$ is a homeomorphism onto its image (cf. [9], see also [21]) It is known that ${\rm Aut}({\mathfrak{g}})$ is a closed subgroup of ${\rm GL}({\mathfrak{g}})$, and we can view ${\rm Aut}(G)$ as a closed subgroup of ${\rm GL}({\mathfrak{g}})$. The topology on ${\rm Aut}(G)$ when considered as a subspace topology of ${\rm GL}({\mathfrak{g}})$ coincides with the (modified) compact-open topology. We refer the reader to [22] for the basic structure theory of Lie groups.

Let $G$ be a connected Lie group. A (continuous real) one-parameter subgroup in $G$ is the image of a continuous homomorphism from $\mathbb{R}$ to $G$. Any one-parameter subgroup is of the form $\{\exp tX\}_{t\in{\mathbb{R}}}$ for some $X$ in the Lie algebra ${\mathfrak{g}}$ of $G$. Moreover, if $H$ is a one-parameter subgroup in $G$, then so is $T(H)$ for any $T\in{\rm Aut}(G)$. Let

$${\mathcal{P}}_{1}(G):=\{A\mid A\mbox{ is a closed one-parameter subgroup of }G\}.$$

Then ${\mathcal{P}}_{1}(G)\subset{\rm Sub}_{G}$ and it is invariant under the action of ${\rm Aut}(G)$. Let ${\rm Sub}^{p}_{G}$ denote the closure (in ${\rm Sub}_{G}$) of the set ${\mathcal{P}}_{1}(G)$ of all closed one-parameter subgroups in $G$, i.e.

$${\rm Sub}^{p}_{G}:=\overline{{\mathcal{P}}_{1}(G)}.$$

Then ${\rm Sub}^{p}_{G}$ is compact and it is also invariant under the action of ${\rm Aut}(G)$.

Note that if any one-parameter subgroup in a locally compact group is not closed, then its closure is compact and isomorphic to an $n$ dimensional torus $\mathbb{T}^{n}=(\mathbb{S}^{1})^{n}$ for some $n\in{\mathbb{N}}$ with $n\geq 2$. If it is closed and bounded (compact), then it is isomorphic to $\mathbb{S}^{1}$. If it is unbounded (not relatively compact), then it is closed and isomorphic to ${\mathbb{R}}$.

For brevity, for a set $\{x_{t}\}_{t\in{\mathbb{R}}}$ which may be a continuous one-parameter set or a one-parameter subgroup, we may just write $\{x_{t}\}$, i.e. we may omit the parameter set if there is no ambiguity.

Now we prove some elementary results about the structure of ${\rm Sub}^{p}_{G}$. The following lemma shows that if $G$ contains any non-trivial compact connected abelian subgroup and the dimension of $G$ is at least 2, then ${\mathcal{P}}_{1}(G)$ is not closed in ${\rm Sub}_{G}$.

Lemma 2.3.

Let $G$ be a connected Lie group and let ${\rm Sub}^{p}_{G}$ be the closure in ${\rm Sub}_{G}$ of all closed one-parameter subgroups. Then the following holds:

1. All compact connected abelian subgroups of $G$ belong to ${\rm Sub}^{p}_{G}$.

2. If $\{x_{t}\}\in{\mathcal{P}}_{1}(G)\subset{\rm Sub}^{p}_{G}$ and if $K$ is a compact connected abelian subgroup of $G$ which is centralised by each $x_{t}$. Then $\{x_{t}\}K\in{\rm Sub}^{p}_{G}$.

Proof.

Let $K$ be a compact connected abelian subgroup. Then the set $F$ of all finite order elements in $K$ is dense in $K$ and each such element of $F$ is contained in a closed one-parameter subgroup of $K$. Since $K$ is monothetic (i.e. it has a dense cyclic subgroup), we get that $K$ is a limit of a sequence of compact one-parameter subgroups of $K$ in ${\rm Sub}_{K}$. Therefore, $K\in{\rm Sub}^{p}_{G}$. Thus (1) holds.

Let $\{x_{t}\}\in{\mathcal{P}}_{1}(G)\subset{\rm Sub}^{p}_{G}$ be such that each $x_{t}$ centralises $K$. If $\{x_{t}\}$ is compact, then $\{x_{t}\}K$ is a compact connected abelian subgroup of $G$ and hence $\{x_{t}\}K\in{\rm Sub}^{p}_{G}$ from (1). Now suppose $\{x_{t}\}$ is isomorphic to ${\mathbb{R}}$. Let $k\in K$ be such that it generates a dense cyclic subgroup in $K$. Since $K$ is exponential, there exists a one-parameter subgroup $\{k_{t}\}$, which is dense in $K$ and $k=k_{1}$. Let $\{n_{m}\}$ be such that $k^{n_{m}}\to k$. Let $A_{m}=\{a_{t}(m)=x_{t/n_{m}^{2}}k_{t}\}_{t\in{\mathbb{R}}}$. Then each $A_{m}$ is a closed one-parameter subgroup in $G$, and passing to a subsequence if necessary, we get that $A_{m}\to A$ for some subgroup $A\in{\rm Sub}^{p}_{G}$. Note that $\{x_{t}\}\times K$ is a closed subgroup of $G$ and it contains each $A_{m}$. Therefore, $A\subset\{x_{t}\}\times K$. Since $a_{n_{m}}(m)=x_{1/n_{m}}k_{n_{m}}=x_{1/n_{m}}k^{n_{m}}\to k$, we get that $k\in A$. Therefore, $K\subset A$. For a fixed $t$, let $b_{m}=a_{tn_{m}^{2}}(m)=x_{t}k_{tn_{m}^{2}}\in A_{m}$, $m\in{\mathbb{N}}$. Passing to a subsequence of $\{b_{m}\}$ if necessary, we get that $b_{m}\to x_{t}h$ for some $h\in K$, and hence, $x_{t}\in A$. Now $\{x_{t}\}\subset A$. Hence, $A=\{x_{t}\}\times K$ and (2) holds. ∎

The following lemma relates the set ${\rm Sub}^{p}_{G/K}$ and a compact subset of ${\rm Sub}^{p}_{G}$ which is invariant under the action of ${\rm Aut}(G)$.

Lemma 2.4.

Let $G$ be a connected Lie group, $K$ be a compact connected central subgroup of $G$ and let $\pi:G\to G/K$ be the natural projection. Then the following hold:

1. $\{HK\mid H\in{\rm Sub}_{G}\}$ is a closed (compact) subset of ${\rm Sub}_{G}$ and $\{HK\mid H\in{\rm Sub}^{p}_{G}\}$ is a closed (compact) subset of ${\rm Sub}^{p}_{G}$, and they are $T$-invariant if $T(K)=K$, where $T\in{\rm Aut}(G)$.

2. ${\mathcal{P}}_{1}(G/K)=\{\pi(A)\mid A\in{\mathcal{P}}_{1}(G)\}$.

3. ${\rm Sub}^{p}_{G/K}=\{\pi(H)\mid H\in{\rm Sub}^{p}_{G}\}$.

4. ${\rm Sub}^{p}_{G/K}$ is homeomorphic to $\{HK\mid H\in{\rm Sub}^{p}_{G}\}$, and the latter is the same set as $\{H\in{\rm Sub}^{p}_{G}\mid HK=KH=H\}$.

Proof.

Let $K$ and $\pi$ be as above. Since $K$ is compact and central in $G$, $HK$ is a closed subgroup of $G$ for any closed subgroup $H$ of $G$. Also, if $H_{n}K\to L$ in ${\rm Sub}_{G}$, then since ${\rm Sub}_{G}$ and $K$ are compact, it is easy to see that for any limit point $H$ of $\{H_{n}\}$, $L=HK$. Therefore, $\{HK\mid H\in{\rm Sub}_{G}\}$ is closed. Now if $H\in{\rm Sub}^{p}_{G}$, then there exists $\{A_{n}\}\subset{\mathcal{P}}_{1}(G)$, such that $A_{n}\to H$. By Lemma 2.3 (2), $A_{n}K\in{\rm Sub}^{p}_{G}$. As $A_{n}K\to HK$ and ${\rm Sub}^{p}_{G}$ is closed, we get that $HK\in{\rm Sub}^{p}_{G}$. Now arguing as above, it is easy to show that $\{HK\mid H\in{\rm Sub}^{p}_{G}\}$ is a closed subset of ${\rm Sub}^{p}_{G}$. Moreover, if $T\in{\rm Aut}(G)$ with $T(K)=K$, then $T(HK)=T(H)K$ for any $H\in{\rm Sub}_{G}$. Therefore, the sets mentioned in (1) are invariant under the action of ${\rm Aut}(G)$ if $T(K)=K$. Thus (1) holds.

If $A\in{\mathcal{P}}_{1}(G)$, then $AK$ is closed and $\pi(A)\in{\mathcal{P}}_{1}(G/K)$. Conversely, suppose $B\in{\mathcal{P}}_{1}(G/K$). It is easy to see that $B_{1}:=\pi^{-1}(B)$ is a closed connected nilpotent Lie subgroup of $G$, and any one-parameter subgroup of $B_{1}$, which not contained in $K$, projects onto $B$. Let $U$ be an open neighbourhood of the identity $e$ in $B_{1}$ such that every $x\in U$ is contained in a one-parameter subgroup. As $B_{1}$ is connected, $U\neq K$ and $U\setminus K$ is open. Suppose $B_{1}$ is compact. Being connected and nilpotent, $B_{1}$ is abelian. Since the set of roots of unity in $B_{1}$ is dense and $U\setminus K$ is open in $B_{1}$, we can choose $x\in U\setminus K$ such that $x^{n}=e$ for some $n\in{\mathbb{N}}$. If $B_{1}$ is unbounded, then let $x\in U\setminus K$ be any element. Let $\{x_{t}\}\subset B_{1}$ be a one-parameter subgroup in $B_{1}$ such that $x_{1}=x$. Then $\pi(\{x_{t}\})=\{x_{t}\}K/K=B$. If $B_{1}$ is compact, then $\{x_{t}\}$ is closed as $x_{1}^{n}=e$. If $B_{1}$ is unbounded, then so is $\{x_{t}\}$, and hence it is closed. Therefore, ${\mathcal{P}}_{1}(G/K)=\{\pi(A)\mid A\in{\mathcal{P}}_{1}(G)\}$, i.e. (2) holds.

Let $H\in{\rm Sub}^{p}_{G}$. Then $A_{n}\to H$ for some $\{A_{n}\}\subset{\mathcal{P}}_{1}(G)$. By (2), we have that $\{\pi(A_{n})\}\subset{\mathcal{P}}_{1}(G/K)$, and as $K$ is compact, $\pi(A_{n})\to\pi(H)$. Therefore, $\pi(H)\in{\rm Sub}^{p}_{G/K}$. Conversely, let $B\in{\rm Sub}^{p}_{G/K}$. There exist $B_{n}\in{\mathcal{P}}_{1}(G/K)$, $n\in{\mathbb{N}}$, such that $B_{n}\to B$. By (2), there exist $A_{n}\in{\mathcal{P}}_{1}(G)$ such that $\pi(A_{n})=B_{n}$, $n\in{\mathbb{N}}$. Let $A$ be a limit point of $\{A_{n}\}$ in ${\rm Sub}_{G}$. Then $A\in{\rm Sub}^{p}_{G}$ and $B=\pi(A)$. Thus (3) holds.

Let ${\mathcal{S}}(K)=\{H\in{\rm Sub}^{p}_{G}\mid HK=KH=H\}$. It is easy to show that ${\mathcal{S}}(K)$ is closed in ${\rm Sub}^{p}_{G}$, and hence it is compact. Observe that the set ${\mathcal{S}}(K)$ is same as the set $\{HK\mid H\in{\rm Sub}^{p}_{G}\}$. For $H_{1},H_{2}\in{\rm Sub}_{G}$, if $\pi(H_{1})=\pi(H_{2})$, then $H_{1}K=H_{2}K$. Therefore, $H\mapsto\pi(H)$ from ${\mathcal{S}}(K)$ to ${\rm Sub}^{p}_{G/K}$ is injective, and it easily follows from (3) that it is surjective. Also if $H_{n}\to H$ in ${\rm Sub}_{G}$, then $\pi(H_{n})\to\pi(H)$ in ${\rm Sub}_{G/K}$. Since ${\mathcal{S}}(K)$ is compact and ${\rm Sub}^{p}_{G/K}$ is Hausdorff, we have that the map induced by $\pi$ from ${\mathcal{S}}(K)$ to ${\rm Sub}^{p}_{G/K}$ is a homeomorphism. Thus (4) holds. ∎

3. Characterisation of distal actions on the space of one-parameter subgroups

In this section we discuss the action of an automorphism as well as that of a subgroup of automorphisms of a connected Lie group $G$ on ${\rm Sub}^{p}_{G}$ and prove results stated in the introduction. We first set some notations and discuss some preliminary results which are useful.

For convenience, we define a class ${\mathcal{C}^{\prime}}$ of Lie groups as follows:

Definition 3.1.

A connected Lie group $G$ is said to be in class ${\mathcal{C}^{\prime}}$ if $G$ has no compact central subgroup of positive dimension $($i.e. if $G$ has no non-trivial compact connected central subgroup$)$.

The notation is inspired by the fact that the class ${\mathcal{C}^{\prime}}$ is a subset of the class $\mathcal{C}$ defined by Dani and McCrudden in [13]. The latter consists of connected Lie groups which admit a (real) linear representation with discrete kernel. Proposition 2.5 of [13] shows that $G$ is in class $\mathcal{C}$ if and only if for the radical $R$ of $G$, the intersection of $\overline{[R,R]}$ with the center of $G$ has no non-trivial compact subgroup. Therefore class ${\mathcal{C}^{\prime}}$ is a subset of class $\mathcal{C}$. Note that any connected Lie group $G$ admits a maximal compact connected central subgroup $M$, and $G/M$ has no non-trivial compact central subgroup and it belongs to class ${\mathcal{C}^{\prime}}$.

Recall that for a connected Lie group $G$ with the Lie algebra ${\mathfrak{g}}$, there is an exponential map $\exp:{\mathfrak{g}}\to G$, and for any $T\in{\rm Aut}(G)$, there is a Lie algebra automorphism ${\rm d\,}T:{\mathfrak{g}}\to{\mathfrak{g}}$ such that $\exp(({\rm d\,}T)(X))=T(\exp X)$, $X\in{\mathfrak{g}}$, and $T\mapsto{\rm d\,}T$ is a homomorphism and homeomorphism from ${\rm Aut}(G)$ onto its image in ${\rm GL}({\mathfrak{g}})$ which is closed; the image consists of Lie algebra automorphisms of ${\mathfrak{g}}$. If $G$ is in class ${\mathcal{C}^{\prime}}$, then ${\rm Aut}(G)$ is almost algebraic as a subgroup of ${\rm GL}({\mathfrak{g}})$, it is closed and has finitely many connected components (cf. [12]). If $H$ is a closed connected normal subgroup of $G$ with the Lie algebra ${\mathfrak{h}}$ which is a Lie subalgebra of ${\mathfrak{g}}$ and it is an ideal in ${\mathfrak{g}}$. Moreover, the Lie algebra of $G/H$ is isomorphic to ${\mathfrak{g}}/{\mathfrak{h}}$. Also, if $H$ is $T$-invariant and if $\overline{T}$ denotes the automorphism of $G/H$ induced by $T$. Then ${\rm d\,}\overline{T}=\overline{{\rm d\,}T}$, which is the Lie algebra automorphism on ${\mathfrak{g}}/{\mathfrak{h}}$ induced by ${\rm d\,}T$. Also ${\rm d\,}T|_{H}$ is the same as $({\rm d\,}T)|_{\mathfrak{h}}$. Note that the eigenvalues of ${\rm d\,}\overline{T}$ (as well as those of ${\rm d\,}T|_{H}$) are eigenvalues of ${\rm d\,}T$.

It is known that $T$ acts distally on $G$ if and only if ${\rm d\,}T$ acts distally on ${\mathfrak{g}}$, and the latter statement is equivalent to the statement that all the eigenvalues of ${\rm d\,}T$ have absolute value 1 (cf [2]). It is also known that if $G$ admits an automorphism $T$ such that ${\rm d\,}T$ has no eigenvalue of absolute value 1 (equivalently, if $T$ is expansive; see [37], see also [7]), then $G$ must be nilpotent (this is well-known, see e.g. [37]). We will need the following lemma which is slightly stronger. The lemma may be known, but we give a short proof for the sake of completeness.

Lemma 3.2.

Let $G$ be a connected non-nilpotent Lie group and let $T$ be an automorphism of $G$. Then at least one eigenvalue of ${\rm d\,}T$ is a root of unity.

Proof.

Let $M$ be the largest compact connected central subgroup of $G$. Then $G/M$ is in class ${\mathcal{C}^{\prime}}$. As $G$ is not nilpotent, $G/M$ is also not nilpotent. Since $M$ is characteristic in $G$, we may replace $G$ by $G/M$ and assume that $G\in{\mathcal{C}^{\prime}}$, i.e. it has no compact central subgroup of positive dimension. If $T$ has finite order, then $({\rm d\,}T)^{n}={\rm Id}$ for some $n\in{\mathbb{N}}$, and the assertion follows in this case. Now suppose $T$ does not have finite order.

Since $G\in{\mathcal{C}^{\prime}}$, ${\rm Aut}(G)$ is almost algebraic (cf. [12]), and hence it has finitely many connected components. Now we may replace $T$ by its suitable power and assume that $T\in{\rm Aut}(G)^{0}$, the connected component of the identity in ${\rm Aut}(G)$, and that $T$ is contained in a non-trivial one-parameter subgroup (say) $\{T_{t}\}$ of ${\rm Aut}(G)^{0}$ as $T=T_{1}$. We will show that 1 is an eigenvalue of ${\rm d\,}T$.

Suppose $G$ is solvable. Let $H=\{T_{t}\}\ltimes G$. Then $H$ is solvable and the nilradical of $H$, being contained in $G$, is the same as the nilradical $N$ of $G$. Therefore, $H/N$ is abelian. Thus $T$ acts trivially on $G/N$. As $G$ is not nilpotent, $G\neq N$, and hence 1 is an eigenvalue ${\rm d\,}T$.

Now suppose $G$ is not solvable. Let $R$ be the radical of $G$. Then $R$ is characteristic in $G$, and let $\overline{T}_{t}$ be the automorphisms of $G/R$ induced by $T_{t}$ for each $t$. It is enough to prove that 1 is an eigenvalue of ${\rm d\,}\overline{T}$. Hence we may replace $G$ by $G/R$ and $\{T_{t}\}$ by $\{\overline{T}_{t}\}$ and assume that $G$ is semisimple. Now ${\rm Aut}(G)^{0}$ consists of inner automorphisms of $G$ and $T_{t}={\rm inn}(x_{t})$, $t\in{\mathbb{R}}$, for some one-parameter subgroup $\{x_{t}\}\subset G$. If $x_{1}=e$, then $T=T_{1}={\rm Id}$. Now suppose $x_{1}\neq e$. Then $\{x_{t}\}$ is non-trivial. Let $X\in{\mathfrak{g}}$ be such that $\exp tX=x_{t}$, $t\in{\mathbb{R}}$. As $T(x_{t})=x_{t}$, $t\in{\mathbb{R}}$, we have that ${\rm d\,}T(X)=X$ and 1 is an eigenvalue of ${\rm d\,}T$. ∎

Remark 3.3.

The proof above also shows that for a connected solvable Lie group $G$ with the nilradical $N$, for any $T\in{\rm Aut}(G)$, $\overline{T}$ has finite order, where $\overline{T}$ is the automorphism of $G/N$ induced by $T$. In particular, all the eigenvalues of ${\rm d\,}\overline{T}$ on the Lie algebra of $G/N$ are roots of unity.

Note that the largest compact central subgroup and its connected component of the identity in a connected Lie group are characteristic. We note the following useful proposition which will enable us to work on Lie groups in class ${\mathcal{C}^{\prime}}$.

Proposition 3.4.

Let $G$ be a connected Lie group and let $T\in{\rm Aut}(G)$. Let $K$ be a $T$-invariant compact connected central subgroup of $G$ and let $\overline{T}$ denote the automorphism of $G/K$ induced by $T$. If $T$ acts distally on ${\rm Sub}^{p}_{G}$, then $\overline{T}$ acts distally on ${\rm Sub}^{p}_{G/K}$.

Proof.

Let $T\in{\rm Aut}(G)$ and let $\pi:G\to G/K$ be the natural projection. Note that $\overline{T}(\pi(H))=\pi(T(H))$ for any $H\in{\rm Sub}_{G}$ and ${\rm Sub}^{p}_{G/K}$ is $\overline{T}$ invariant. Let ${\mathcal{S}}(K)=\{H\in{\rm Sub}^{p}_{G}\mid HK=H\}$ which is a compact $T$-invariant subset of ${\rm Sub}^{p}_{G}$ and homeomorphic to ${\rm Sub}^{p}_{G/K}$ (cf. Lemma 2.4). Suppose $T$ acts distally on ${\rm Sub}^{p}_{G}$. Then the $T$-action on ${\mathcal{S}}(K)$ is distal. Let $L_{1},L_{2}\in{\rm Sub}^{p}_{G/K}$ be such that for some $\{n_{k}\}\subset{\mathbb{Z}}$, ${\overline{T}}^{n_{k}}(L_{i})\to L$, for some $L\in{\rm Sub}^{p}_{G/K}$, $i=1,2$. Then there exist $H_{1},H_{2}\in{\mathcal{S}}(K)$, such that $\pi(H_{i})=L_{i}$ and $T^{n_{k}}(H_{i})\to\pi^{-1}(L)$ in ${\mathcal{S}}(K)$, $i=1,2$ (cf. Lemma 2.4 (4)). As $T$ acts distally on ${\mathcal{S}}(K)$, we get that $H_{1}=H_{2}$, and hence that $L_{1}=L_{2}$. Thus $\overline{T}$ acts distally on ${\rm Sub}^{p}_{G/K}$. ∎

We also need the following elementary lemma.

Lemma 3.5.

Let $G$ be a simply connected nilpotent Lie group with the Lie algebra ${\mathfrak{g}}$ and let $T\in{\rm Aut}(G)$. Then the set ${\mathcal{P}}_{1}(G)$ consisting of all closed one-parameter subgroups of $G$ is closed in ${\rm Sub}_{G}$ and is the same as ${\rm Sub}^{p}_{G}$. Moreover, $T$ acts distally on ${\rm Sub}^{p}_{G}$ if and only if $dT$ acts distally on ${\rm Sub}^{p}_{\mathfrak{g}}$.

Proof.

In a simply connected nilpotent group, all one-parameter subgroups are closed and there is a one-to-one correspondence between non-trivial (closed) one-parameter subgroups of $G$ and one-dimensional subspaces of ${\mathfrak{g}}$. The correspondence is via the exponential map $\exp:{\mathfrak{g}}\to G$ with $\log$ as its inverse. Namely, for $\{x_{t}\}$, $x_{t}=\exp tX$, for some $X\in{\mathfrak{g}}$, and $\log x_{t}=tX$, $t\in{\mathbb{R}}$. Suppose $\{H_{n}\}\subset{\mathcal{P}}_{1}(G)$ is such that $H_{n}\to H$ in ${\rm Sub}_{G}$. Then $\log H_{n}\in{\mathcal{P}}_{1}({\mathfrak{g}})$ and using Lemma 2.1 and the continuity of the exponential map and its inverse, it is easy to show that $\{\log H_{n}\}$ converges to (say) $B$ in ${\rm Sub}^{p}_{\mathfrak{g}}$. and since ${\mathfrak{g}}$ is isomorphic to ${\mathbb{R}}^{n}$ for some $n$, ${\rm Sub}^{p}_{\mathfrak{g}}={\mathcal{P}}_{1}({\mathfrak{g}})$ which is closed and we have that $B\in{\mathcal{P}}_{1}({\mathfrak{g}})$. Then $H=\exp B\in{\mathcal{P}}_{1}(G)$ and, in particular, we get that ${\mathcal{P}}_{1}(G)$ is closed in ${\rm Sub}_{G}$, i.e. ${\mathcal{P}}_{1}(G)={\rm Sub}^{p}_{G}$. In particular, $H_{n}\to H$ in ${\rm Sub}^{p}_{G}$ implies that $\log H_{n}\to\log H$ in ${\rm Sub}^{p}_{\mathfrak{g}}$.

Conversely, if $L_{n}\to L$ in ${\mathcal{P}}_{1}({\mathfrak{g}})$, using Lemma 2.1 and the fact that $\exp$ is a homeomorphism from ${\mathfrak{g}}$ to $G$, it is easy to see that $\exp L_{n}\to\exp L$ in ${\mathcal{P}}_{1}(G)$. Now the exponential map defines a homeomorphism from ${\mathcal{P}}_{1}({\mathfrak{g}})$ to ${\mathcal{P}}_{1}(G)$ and $T$ (resp. ${\rm d\,}T$) defines a homeomorphism of ${\mathcal{P}}_{1}(G)$ (resp. ${\mathcal{P}}_{1}({\mathfrak{g}})$) such that $\exp\circ\,{\rm d\,}T=T\circ\exp$. Therefore, it is easy to see that $T$ acts distally on ${\mathcal{P}}_{1}(G)={\rm Sub}^{p}_{G}$ if and only if ${\rm d\,}T$ acts distally on ${\mathcal{P}}_{1}({\mathfrak{g}})={\rm Sub}^{p}_{\mathfrak{g}}$. ∎

Proof of Theorem 1.1 $(1)$ and $(2)$ for nilpotent groups.

Let $G$ be a (non-trivial) connected Lie group in class ${\mathcal{C}^{\prime}}$, i.e. $G$ does not have any non-trivial compact connected central subgroup. Let $T\in{\rm Aut}(G)$.

$(1):$ Suppose $G$ is a connected abelian group in class ${\mathcal{C}^{\prime}}$, i.e. $G$ does not have any non-trivial compact subgroup, and it is a vector group isomorphic to ${\mathbb{R}}^{d}$, for some $d\in{\mathbb{N}}$. Note that ${\mathcal{D}}=\{r\,{\rm Id}\mid r\in{\mathbb{R}}\setminus\{0\}\}$, where ${\rm Id}$ is the identity matrix in ${\rm GL}(d,{\mathbb{R}})$. If $d=1$, then ${\rm GL}(1,{\mathbb{R}})$ is abelian, and the assertion follows trivially. Now suppose $d\geq 2$. Then ${\rm Sub}^{p}_{G}$ is isomorphic to $\mathbb{RP}^{d-1}\sqcup\{\{e\}\}$ and $T\in{\rm GL}(d,{\mathbb{R}})$. Both $\mathbb{RP}^{d-1}$ and $\{e\}$ are $T$ invariant. Hence $T$ acts distally on ${\rm Sub}^{p}_{{\mathbb{R}}^{d}}$ if and only if $T$ acts distally on $\mathbb{RP}^{d-1}$. One can define an action of $T$ on $\mathbb{S}^{d-1}$, the boundary of unit sphere in ${\mathbb{R}}^{d}$, as $\overline{T}(x)=T(x)/\|x\|$, where $\|\cdot\|$ denotes the usual norm on ${\mathbb{R}}^{d}$. Here, $\overline{T}$ is a homeomorphism and $\overline{T}(-x)=-\overline{T}(x)$. As ${\mathbb{RP}}^{d-1}$ is a quotient space of $\mathbb{S}^{n-1}$, the action of $T$ on ${\mathbb{RP}}^{d-1}$ is the same as the action of $\overline{T}$ on ${\mathbb{S}}^{d-1}/\sim$, where $\sim$ is the equivalence relation which identifies $x$ with $-x$ for each $x\in\mathbb{S}^{d-1}$. It is easy to see that $T$ acts distally on ${\mathbb{RP}}^{d-1}$ if and only if $\overline{T}$ acts distally on $\mathbb{S}^{d-1}$. The latter statement is equivalent to the following: $T\in{\mathcal{K}}\mathcal{D}$, where ${\mathcal{K}}$ is a compact subgroup of ${\rm GL}(d,{\mathbb{R}})$ and $\mathcal{D}$ is the center of ${\rm GL}(d,{\mathbb{R}})$, (cf. [38], Theorem 1).

Conversely, for $G={\mathbb{R}}^{d}$ and $T\in{\mathcal{K}}\mathcal{D}$, with a compact set ${\mathcal{K}}\subset{\rm GL}(d,{\mathbb{R}})$ and the center ${\mathcal{D}}$ of ${\rm GL}(d,{\mathbb{R}})$, the action of $T$ on ${\rm Sub}^{p}_{G}$ is same as that of $S$ on ${\rm Sub}^{p}_{G}$ for some $S\in{\mathcal{K}}$ since $\mathcal{D}$ acts trivially on ${\rm Sub}^{p}_{G}$. As the compact subgroups of ${\rm Aut}(G)$ act distally on any closed invariant subspace of ${\rm Sub}_{G}$ (${\rm Sub}^{p}_{G}$ in this case), we have that $T$ acts distally on ${\rm Sub}^{p}_{G}$. This completes the proof of Theorem 1.1 (1).

$(2):$ Suppose $G$ belongs to the class ${\mathcal{C}^{\prime}}$ and it is non-abelian and nilpotent. Then $G$ is simply connected as $G$ has no compact central subgroup of positive dimension. Suppose $T$ acts distally on ${\rm Sub}^{p}_{G}$. We want to show that $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. Since $G$ belongs to the class ${\mathcal{C}^{\prime}}$, ${\rm Aut}(G)$ is almost algebraic as a subgroup of ${\rm GL}({\mathfrak{g}})$ (cf. [12]).

As $G$ is simply connected and nilpotent, by Lemma 3.5, ${\rm d\,}T$ acts distally on ${\rm Sub}^{p}_{\mathfrak{g}}$. Then from (1) above, we have that ${\rm d\,}T\in{\mathcal{K}}\{r{\rm Id}\mid r\neq 0\}$ for some compact subgroup ${\mathcal{K}}$ of ${\rm GL}({\mathfrak{g}})$ and ${\rm Id}$ is the identity map on ${\mathfrak{g}}$. As ${\rm d\,}T^{n}=\psi\,\circ\,r\,{\rm Id}$ for some $r\in{\mathbb{R}}\setminus\{0\}$ and $\psi\in{\mathcal{K}}$. If $r=\pm 1$, then ${\rm d\,}T^{2}\in{\mathcal{K}}$ and $T$ generates a relatively compact subgroup of ${\rm Aut}(G)$. If possible suppose $r\neq\pm 1$. Since ${\rm Aut}(G)$ is almost algebraic, there exists $n\in{\mathbb{N}}$ such that ${\rm d\,}T^{n}$ is contained in a connected almost algebraic subgroup ${\mathbb{R}}\times\mathbb{T}^{k}$ of ${\rm Aut}(G)^{0}$ (which is identified with its image in ${\rm GL}({\mathfrak{g}})$), where $\mathbb{T}^{k}=(\mathbb{S}^{1})^{k}$ for some $k\in{\mathbb{N}}$. Therefore, $\psi^{n}\,\circ\,r^{n}\,{\rm Id}\in{\mathbb{R}}\times\mathbb{T}^{k}\subset{\rm Aut}(G)^{0}$. Hence both $\psi^{n}$ and $r^{n}\,{\rm Id}\in{\rm Aut}(G)^{0}$. But $r^{n}\,{\rm Id}$ is not a Lie algebra automorphism of ${\mathfrak{g}}$ as $r^{n}\neq 1$, for if $r^{n}[X,Y]=r^{n}{\rm Id}[X,Y]=[r^{n}{\rm Id}(X),r^{n}{\rm Id}(Y)]=r^{2n}[X,Y]$, then $[X,Y]=0$ for all $X,Y\in{\mathfrak{g}}$, which leads to a contradiction as $G$ is not abelian. Thus, $r=\pm 1$ and $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. The converse statement is obvious. Thus (2) holds if $G$ is nilpotent. ∎

We will continue with the proof of Theorem 1.1 after proving Theorems 1.2 and 1.3 and some elementary results.

For a Lie group $G$ and $T\in{\rm Aut}(G)$, let $C(T)$ denote the contraction group of $T$ and it is defined as

$$C(T):=\{x\in G\mid T^{n}(x)\to e\mbox{ as }n\to\infty\}.$$

Note that $C(T)$ is a connected nilpotent subgroup of $G$ and it is simply connected if it is closed in $G$ (see [41], see also [20]). Both $C(T)$ and $C(T^{-1})$ are trivial if and only if $T$ acts distally on $G$ (cf.  [34], Theorem 4.1). For a connected Lie group $G$, $C(T)$ is trivial if and only if all the eigenvalues of ${\rm d\,}T$ have absolute value greater than or equal to 1 ([34, 2]).

We need the following lemma which can be proven easily by using Theorem 1.1 of [14], Theorem 2.4 and Corollary 2.7 of [20]. Note that the largest compact connected central subgroup of $G$ is characteristic in $G$. The lemma generalises a part of the statement in Proposition 4.3 of [34].

Lemma 3.6.

Let $G$ be a connected Lie group and let $T\in{\rm Aut}(G)$. Then the following are equivalent:

1. $T$ acts distally on the largest compact connected central subgroup of $G$.

2. The contraction groups $C(T)$ and $C(T^{-1})$ are closed, simply connected and nilpotent.

3. $T$ acts distally on every compact $T$-invariant subgroup of $G$.

Proof.

Let $M$ be the largest compact connected central subgroup of $G$. Let $\pi:G\to G/M$ be the natural projection and let $T^{\prime}\in{\rm Aut}(G/M)$ be the automorphism induced by $T$ on $G/M$. As $G/M$ does not have any non-trivial compact connected central subgroup, using Theorem 1.1 of [14], one can show that $C(T^{\prime})$ is closed (see the proof of Proposition 4.3 of [34] for more details). Hence $\pi^{-1}(C(T^{\prime}))=C_{M}(T):=\{x\in G\mid T^{n}(x)M\to M\mbox{ in }G/M\}$ is closed. Suppose (1) holds, i.e. $T$ acts distally on $M$. Then $C(T)\cap M=C(T|_{M})=\{e\}$. As $C_{M}(T)$ is closed, by Corollary 2.7 of [20], $C(T)$ is closed. Similarly, $C(T^{-1})$ is closed. The rest of the assertion in (2) follows from Corollary 2.4 in [41]. This proves $(1)\implies(2)$.

Now suppose (2) holds, i.e. $C(T)$ is closed. Let $K$ be any compact $T$-invariant subgroup of $G$. Then $C(T|_{K})=C(T)\cap K$ is a compact subgroup of $C(T)$. Hence $C(T|_{K})$ is trivial (cf. [20], Corollary 2.5). Similarly, $C((T|_{K})^{-1})$ is trivial and $T$ acts distally on $K$ (cf. [25]). Thus (3) holds. $(3)\implies(1)$ is obvious. ∎

Proof of Theorem 1.2.

Let $G$ be a connected Lie group which is not a vector group. Let $M$ be the largest compact connected central subgroup of $G$. Suppose $T\in{\rm Aut}(G)$ be such that it acts distally on $M$. Suppose $T$ acts distally on ${\rm Sub}^{p}_{G}$. Then we want to show that $T$ acts distally on $G$.

Step 1: Suppose $G$ is simply connected and nilpotent. Since $G$ is not a vector group, $G$ is non-abelian and by the proof of Theorem 1.1 (2) for the nilpotent case, $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. Hence $T$ acts distally on $G$.

Step 2: Now suppose $G$ is not a simply connected nilpotent group. By Theorem 4.1 of [34], it is enough to prove that the contraction groups $C(T)$ and $C(T^{-1})$ are trivial. As $T$ acts distally on the maximal compact connected central subgroup $M$, by Lemma 3.6, both $C(T)$ and $C(T^{-1})$ are closed, simply connected and nilpotent. Suppose $C(T)$ is non-trivial. If possible, suppose $C(T)$ is non-abelian. As it is $T$-invariant and $T|_{C(T)}$ acts distally on ${\rm Sub}^{p}_{C(T)}$, we get from Step 1 that $T|_{C(T)}$ acts distally on $C(T)$, a contradiction. Now suppose $C(T)$ is abelian. Then $C(T)$ is isomorphic to ${\mathbb{R}}^{d}$ as $C(T)$ is simply connected. By Theorem 1.1 (1), $T_{1}=T|_{C(T)}=\phi\circ\alpha\,I$ for some $\phi$ contained in a compact subgroup of ${\rm GL}(d,{\mathbb{R}})$ and $\alpha\in{\mathbb{R}}$, $0<|\alpha|<1$; here $I$ denotes the identity matrix in ${\rm GL}(d,{\mathbb{R}})$. We may replace $T$ by $T^{2}$ (and $T_{1}$ by $T_{1}^{2}$), and assume that $\alpha>0$. Here, $G\neq C(T)$ as $G$ is not a vector group.

Suppose $G$ is nilpotent but not simply connected. Then $M$ as above is non-trivial. Since $T$ acts distally on $M$, by Lemma 2.5 of [2], we have that $T^{k}|_{M}$ is unipotent for some $k\in{\mathbb{N}}$, i.e. ${\rm d\,}T$ has an eigenvalue which is a root of unity. If $G$ is not nilpotent, by Lemma 3.2, ${\rm d\,}T$ has an eigenvalue which is a root of unity. We may replace $T$ by a suitable power and assume that 1 is an eigenvalue of ${\rm d\,}T$.

Let $X\in{\mathfrak{g}}$, $X\neq 0$ be such that ${\rm d\,}T(X)=X$ and let $A=\{\exp tX\}$, a one-parameter subgroup, it is either closed or its closure $\bar{A}$ is compact. Since $T|_{A}={\rm Id}$, we have that $T|_{\bar{A}}={\rm Id}$ and it follows that $\bar{A}$ normalises $C(T)$ and $\bar{A}\cap C(T)=\{e\}$. Let $H=\overline{A\ltimes C(T)}$. Then $H$ is solvable. As $A$ normalises $C(T)$, so does $H$. Now $H/C(T)$ is connected and abelian as it is the closure of the image of the one-parameter subgroup $A$ in it. Then $H/C(T)$ is either isomorphic to ${\mathbb{R}}$ or it is compact. In the first case it follows that $H=A\ltimes C(T)$. In the second case, since $C(T)$ is a vector group, $H=K\ltimes C(T)$ for some (maximal) compact subgroup $K\subset H$ (cf. [24], Lemma 3.7). Here, $H$ is a closed $T$-invariant subgroup of $G$ and $T(h)\in h\,C(T)$ for all $h\in H$. If $A$ (resp. $\bar{A}$) is compact, then $H=A\ltimes C(T)$ (resp. $H=\bar{A}\ltimes C(T)$). Note that $C(T)$ is isomorphic to ${\mathbb{R}}^{d}$ and $T_{1}$, the restriction of $T$ to $C(T)$, is as described earlier.

Step 3: Suppose $H/C(T)$ is isomorphic to ${\mathbb{R}}$. Then so is $A$ and $H=A\ltimes C(T)$. Let $U,V$ be open neighbourhoods of the identity $e$ in $H$ such that $V^{2}\subset U$ and every element of $U$ is contained in a one-parameter subgroup. Let $x=ac$ for some $a\in A\cap V$ and $c\in C(T)\cap V$ with $c\neq e$. Then $x\in V^{2}\subset U$ and $T(x)=xy$ for $y=c^{-1}T(c)\in C(T)$ and $y\neq e$ as $c\in C(T)$ and $c\neq e$. Now $x$ is contained in a one-parameter subgroup (say) $\{x_{t}\}$ in $H$ as $x=x_{1}$. Then $T(x_{t})=x_{t}y_{t}$, $y_{t}\in C(T)$, $t\in{\mathbb{R}}$ and $y_{1}=y\neq e$. Here, $\{x_{t}\}$ is closed as $H$ is simply connected and solvable. As ${\rm Sub}^{p}_{H}$ is compact, we get that $T^{n_{k}}(\{x_{t}\})\to B$ for some $B\in{\rm Sub}^{p}_{H}$ and some unbounded sequence $\{n_{k}\}\subset{\mathbb{N}}$. By Lemma 2.2, $B\neq\{e\}$. Let $b\in B\setminus\{e\}$. Then $T^{n_{k}}(x_{t_{k}})\to b$ for some sequence $\{t_{k}\}\subset{\mathbb{R}}$. Here, $\{x_{t}\}C(T)/C(T)=H/C(T)$ is isomorphic to ${\mathbb{R}}$. Since $x_{t_{k}}C(T)\to b\,C(T)$ in $H/C(T)$, we get that $\{t_{k}\}$ is bounded. Passing to a subsequence if necessary, we may assume that $t_{k}\to t_{0}$. Then $x_{t_{k}}\to x_{t_{0}}$ and $y_{t_{k}}\to y_{t_{0}}$. As all the eigenvalues of $T|_{C(T)}=T_{1}=\phi\circ\alpha\,I$ have absolute value $\alpha$ with $0<\alpha<1$, we get that $I-T_{1}$ is invertible in ${\rm GL}(d,{\mathbb{R}})$, where $I$ is the identity matrix in ${\rm GL}(d,{\mathbb{R}})$. Using the addition as the group operation in ${\mathbb{R}}^{d}$, and the fact that $T_{1}^{n}(y)\to 0$ for all $y\in{\mathbb{R}}^{d}$, we get that

$$T^{n_{k}}(x_{t_{k}})=x_{t_{k}}\left(\sum_{i=0}^{n_{k-1}}T_{1}^{i}(y_{t_{k}})\right)\to x_{t_{0}}(I-T_{1})^{-1}(y_{t_{0}})=b$$

(as $n_{k}\to\infty$), where $T_{1}^{0}=I$, the identity in ${\rm GL}(d,{\mathbb{R}})$. Note that

$$I+T_{1}(I-T_{1})^{-1}=(I-T_{1})(I-T_{1})^{-1}+T_{1}(I-T_{1})^{-1}=(I-T_{1})^{-1}.$$

Therefore,

$$T(b)=x_{t_{0}}y_{t_{0}}T_{1}(I-T_{1})^{-1}(y_{t_{0}})=x_{t_{0}}(I+T_{1}(I-T_{1})^{-1})(y_{t_{0}})=b.$$

Thus $T(B)=B$. Now we get that $T^{n_{k}}(T(\{x_{t}\}))\to T(B)=B$ and hence that $T(\{x_{t}\})=\{x_{t}\}$ since $T$ acts distally on ${\rm Sub}^{p}_{G}$. As $T$ is an automorphism, we have that $T(x_{t})=x_{st}$ for some $s\in{\mathbb{R}}\setminus\{0\}$, and $xy=x_{1}y_{1}=x_{s}$. Therefore, $x_{s-1}=y_{1}=y\neq e$ and $s\neq 1$. Then $x_{s-1}\in C(T)$, and $\{x_{t}\}C(T)/C(T)$ is compact. This leads to a contradiction as $H=\{x_{t}\}C(T)$ and $H/C(T)$ is isomorphic to ${\mathbb{R}}$.

Step 4: Now we have that $H/C(T)$ is compact and $H=K\ltimes C(T)$, where $K$ is a maximal compact subgroup of $H$. We show that $T(x)=x$ for all $x\in K$. We know that $T(x)\in x\,C(T)$ for all $x\in K$. Since $K$ is connected and abelian, the set $F$ of roots of unity in $K$ is dense in $K$ and it is enough to show that $T(x)=x$ for all $x\in F$. If possible, suppose there exists $x\in F$ such that $T(x)\neq x$. Let $m\in{\mathbb{N}}$ be the order of $x$, i.e. $x^{m}=e$ and $m\neq 1$. Let $\{x_{t}\}$ be a one-parameter subgroup in $K$ such that $x_{1}=x$. As $x_{1}^{m}=e$, $\{x_{t}\}$ is closed and isomorphic to ${\mathbb{S}}^{1}$ and $T(x_{t})=x_{t}y_{t}$, where $y_{t}\in C(T)$, $t\in{\mathbb{R}}$ and $y_{1}=x^{-1}T(x)\neq e$. There exists an unbounded sequence $\{n_{k}\}\subset{\mathbb{N}}$ such that $T^{n_{k}}(\{x_{t}\})\to B$ for some $B\subset H$. Then $B\neq\{e\}$ as $x_{1}\neq e$ and $\{x_{t}\}$ is connected (cf. Lemma 2.2). Let $b\in B$ be such that $b\neq e$. Then $T^{n_{k}}(x_{t_{k}})\to b$ for some $\{t_{k}\}\subset{\mathbb{R}}$. Here, $\{x_{t}\}$ is periodic i.e. $x_{mn}=e$ and $x_{m+t}=x_{t}$ for all $n\in{\mathbb{Z}}$ and $t\in{\mathbb{R}}$, where $m$ is the order of $x_{1}=x$. Hence we can choose $\{t_{k}\}$ such that it is bounded. Then $\{x_{t_{k}}\}$, and hence, $\{y_{t_{k}}\}$ is bounded. Now arguing as in Step 3, we get that $T(b)=b$. Therefore, $T(B)=B$, and hence $T^{n_{k}}(T(\{x_{t}\}))\to B$. Since $T$ acts distally on ${\rm Sub}^{p}_{G}$, we get that $T(\{x_{t}\})=\{x_{t}\}$. As $T$ is an automorphism and $\{x_{t}\}$ is isomorphic to ${\mathbb{S}}^{1}$, we get that $T(x_{t})=x_{t}$ or $T(x_{t})=x_{t}^{-1}=x_{-t}$. In either case, we get that $y_{1}\in K\cap C(T)=\{e\}$ This leads to a contradiction. Hence $T(x)=x$, for all $x\in F$, and hence for all $x\in K$.

Step 5: We now show that $K$ centralises $C(T)$ and $H=K\times C(T)$. We have that $T|_{K}={\rm Id}$. Let $K_{c}=c^{-1}Kc$, $c\in C(T)$. Then $K_{c}$ is a compact connected abelian group and $H=K_{c}\ltimes C(T)$ and $T(x)\in x\,C(T)$ for all $x\in K_{c}$. Therefore, arguing as in Step 4 for $K_{c}$ instead of $K$, we get that $T|_{K_{c}}={\rm Id}$, i.e. $c^{-1}xc=T(c^{-1}xc)=T(c^{-1})x\,T(c)$ for all $c\in C(T)$ and $x\in K$. Thus $c\,T(c^{-1})=(I-T_{1})(c)$ centralises $K$, $c\in C(T)$, where $T_{1}=T|_{C(T)}$. As $I-T_{1}$ is invertible in $GL(d,{\mathbb{R}})$, we get that $C(T)$ centralises $K$. Thus $H=K\times C(T)$.

Step 6: Let $\psi,S:H\to H$ be such that $S|_{K}={\rm Id}$, $S|_{C(T)}=\alpha\,{\rm Id}$, $\psi|_{K}={\rm Id}$ and $\psi|_{C(T)}=\phi$, where $\alpha$ and $\phi$ are as in Step 2. Now $T|_{H}=S\psi=\psi S$, and $\psi$ is contained in a compact subgroup of ${\rm Aut}(H)$. Therefore, $S$ acts distally on ${\rm Sub}^{p}_{H}$, and $C(S)=C(T)$. Using that $C(T)=C(S)\neq\{e\}$, we show that $S$ does not act distally on ${\rm Sub}^{p}_{H}$, which would lead to a contradiction.

Let $\{z_{t}=x_{t}c_{t}\}$ be a closed one-parameter subgroup, where $\{x_{t}\}\subset K$ is isomorphic to ${\mathbb{S}}^{1}$ and $\{c_{t}\}\subset C(S)=C(T)$ is a non-trivial (closed) one-parameter subgroup such that $S(c_{t})=c_{\alpha t}$, $t\in{\mathbb{R}}$ (In additive notation on ${\mathbb{R}}^{d}$, we choose $c_{t}=tc$, $t\in{\mathbb{R}}$, for some nonzero $c\in{\mathbb{R}}^{d}$). Let $\{n_{k}\}$ be an unbounded sequence in ${\mathbb{N}}$ such that $S^{n_{k}}(\{z_{t}\})\to C$ for some $C$ in ${\rm Sub}^{p}_{H}$. Note that $\{z_{t}\}\subset\{x_{t}\}\times\{c_{t}\}$ and the latter is $S$-invariant and isomorphic to ${\mathbb{S}}^{1}\times{\mathbb{R}}$. So $C\subset\{x_{t}\}\times\{c_{t}\}$. For a fixed $t\in{\mathbb{R}}$, as $c_{t}\in C(S)=C(T)$, we get that

$$S^{n_{k}}(z_{t})=x_{t}S^{n_{k}}(c_{t})\to x_{t},$$

as $n_{k}\to\infty$. Therefore, $\{x_{t}\}\subset C$. Again for a fixed $t\in{\mathbb{R}}$,

$$S^{n_{k}}(z_{\alpha^{-n_{k}}t})=x_{\alpha^{-n_{k}}t}S^{n_{k}}(c_{\alpha^{-n_{k}}t})=x_{\alpha^{-n_{k}}t}c_{t}\to x_{s}c_{t}$$

along a subsequence of $\{n_{k}\}$, for some $s\in{\mathbb{R}}$. Therefore, $x_{s}c_{t}$, and hence, $c_{t}$ belongs to $C$. Thus $\{c_{t}\}\subset C$ and $C=\{x_{t}\}\times\{c_{t}\}$, and hence $C$ is $S$-invariant. This implies that $S^{n_{k}}(S(\{z_{t}\}))\to C$, and hence that $S(\{z_{t}\})=\{z_{t}\}$. As $\{z_{t}\}$ is isomorphic to ${\mathbb{R}}$, there exists $\beta\in{\mathbb{R}}\setminus\{0\}$ such that for all $t\in{\mathbb{R}}$,

$$S(z_{t})=z_{\beta t}=x_{\beta t}c_{\beta t}\ \mbox{ and }\ S(z_{t})=S(x_{t}c_{t})=x_{t}c_{\alpha t}.$$

As $\{x_{t}\}\times\{c_{t}\}$ is a direct product and $\{c_{t}\}$ is isomorphic to ${\mathbb{R}}$, it follows that $\beta=\alpha$ and $x_{t}=x_{\alpha t}$, i.e. $x_{(1-\alpha)t}=e$ for all $t\in{\mathbb{R}}$. As $\alpha\neq 1$, $x_{t}=e$ for each $t$, a contradiction. Hence, $C(S)=C(T)$ is trivial.

Similarly, we can show that $C(T^{-1})$ is trivial. Thus $T$ acts distally on $G$. ∎

The following corollary which extends Theorem 1.2 is an easy consequence of the theorem in view of Theorem 1.1 of [2]. We will omit the proof.

Corollary 3.7.

Let $G$ be a connected Lie group which is not a vector group and let $H$ be a subgroup of ${\rm Aut}(G)$ such that it acts distally on the largest compact connected central subgroup of $G$. If $H$ acts distally on ${\rm Sub}^{p}_{G}$, then $H$ acts distally on $G$.

Before proving Theorem 1.3, we first prove the following proposition which is a special case of the theorem.

Proposition 3.8.

Let $G$ be a connected Lie group, $T\in{\rm Aut}(G)$ and let $L$ be a closed connected nilpotent normal $T$-invariant subgroup. Suppose $T$ acts trivially on both $L$ and $G/L$. If $T$ acts distally on ${\rm Sub}^{p}_{G}$, then $T={\rm Id}$.

Proof.

Since $T$ acts trivially on $G/L$ and $L$, we have that $T$ is distal. In fact, all the eigenvalues of $T$ are equal to 1 and $T$ is unipotent. Suppose $T$ acts distally on ${\rm Sub}^{p}_{G}$. Let $U$ be a neighbourhood of the identity $e$ such that every element of $U$ is contained in a one-parameter subgroup. Since $G$ is connected, $U$ generates $G$ and it is enough to show that $T(x)=x$ for all $x\in U$.

Step 1: If possible, suppose $T(x)\neq x$ for some $x\in U$. There exists a one-parameter subgroup $\{x_{t}\}$ in $G$ such that $x_{1}=x$. Let $H=\overline{\{x_{t}\}L}$. Then $H$ is connected solvable and $H/L$ is abelian. Let $\pi:G\to G/L$ be the natural projection. Then $\pi(H)=\overline{\pi(\{x_{t}\})}$. If $\pi(\{x_{t}\})$ is unbounded, it is closed and it is isomorphic to ${\mathbb{R}}$ and in this case $\{x_{t}\}$ is also unbounded and closed. Moreover $H=\{x_{t}\}L$.

Now suppose $\pi(\{x_{t}\})$ is relatively compact. As $H$ is solvable, there exists a maximal compact (connected, abelian) subgroup $K$ in $H$ such that $H=KL$. As $T(x)\neq x$ and $T|_{L}={\rm Id}$, we have that $T(h)\neq h$ for some $h\in K$, and hence $T(h)\neq h$ for some finite order element $h$ in $K$. There exists a closed (compact) one-parameter subgroup $\{h_{t}\}$ in $K$ such that $h_{1}=h$. We may replace $\{x_{t}\}$ by $\{h_{t}\}$ and assume also in this case that $H=\{x_{t}\}L$.

Now we have that $H=\{x_{t}\}L$ is closed, where either $\pi(\{x_{t}\})$ (as well as $\{x_{t}\}$) is isomorphic to ${\mathbb{R}}$ or $\{x_{t}\}$ is isomorphic to ${\mathbb{S}}^{1}$ and $T(x_{1})\neq x_{1}$. As $T$ acts trivially on $G/L$, we have that $T(H)=H$ and $T(x_{t})=x_{t}y_{t}$, for some $y_{t}\in L$, $t\in{\mathbb{R}}$, where $y_{1}\neq e$.

Step 2: Let $A:=\{x_{t}\}$ and let $\{n_{k}\}\subset{\mathbb{N}}$ be an unbounded sequence such that $T^{n_{k}}(A)\to B$ for some $B\in{\rm Sub}^{p}_{H}$. By Lemma 2.2, $B$ is not discrete as $A$ is connected and non-trivial. Let $b\in B$ be such that $b\neq e$. There exists $t_{k}\in{\mathbb{R}}$, $k\in{\mathbb{N}}$, such that

$$T^{n_{k}}(x_{t_{k}})=x_{t_{k}}y_{t_{k}}^{n_{k}}\to b.$$

As $y_{t}\in L$, $t\in{\mathbb{R}}$, we get that $\pi(x_{t_{k}})\to\pi(b)$. If $H/L$ is isomorphic to ${\mathbb{R}}$, then we get that the sequence $\{t_{k}\}$ is bounded. In the second case where $\{x_{t}\}$ is compact, we can choose $\{t_{k}\}$ to be bounded. Passing to a subsequence if necessary, we may assume that $t_{k}\to t_{0}$. Therefore, $x_{t_{k}}\to x_{t_{0}}$ and $y_{t_{k}}\to y_{t_{0}}$ and $b=x_{t_{0}}y^{\prime}$, where $y_{t_{k}}^{n_{k}}\to y^{\prime}$ in $L$.

Step 3: We first assume that $L$ is simply connected. We show that $y_{t_{0}}=e$ in this case. Let ${\mathfrak{l}}$ be the Lie algebra of $L$. Since $L$ is simply connected and nilpotent, $\exp:{\mathfrak{l}}\to L$ is a homeomorphism with $\log$ as its inverse. Let $y_{t_{k}}=\exp Y_{k}$ for some $Y_{k}\in{\mathfrak{l}}$. Then $y_{t_{k}}^{n_{k}}=\exp(n_{k}Y_{k})$ and as $y_{t_{k}}^{n_{k}}\to y^{\prime}$, we have that $n_{k}Y_{k}\to\log y^{\prime}$ in ${\mathfrak{l}}$. As ${\mathfrak{l}}$ isomorphic to ${\mathbb{R}}^{d}$ for some $d\in{\mathbb{N}}$, it follows that $Y_{k}\to 0$ in ${\mathfrak{l}}$, and hence $y_{t_{k}}\to e$, i.e. $y_{t_{0}}=e$. Now

$$T(b)=T(x_{t_{0}})y^{\prime}=x_{t_{0}}y_{t_{0}}y^{\prime}=x_{t_{0}}y^{\prime}=b.$$

Thus $T(B)=B$. As $T$ acts distally on ${\rm Sub}^{p}_{G}$, we have that $T(A)=A$. Since $A$ is a one-parameter subgroup in $G$ and $T$ is unipotent, we get that $T(x_{t})=x_{t}$, $t\in{\mathbb{R}}$. This leads to a contradiction as $T(x_{1})\neq x_{1}$. Therefore, $T(x)=x$ for all $x\in U$, and hence, for all $x\in G$. Thus $T={\rm Id}$ if $L$ is simply connected.

Step 4: Suppose that $L$ is not simply connected. Let $M$ be the maximal compact (normal) subgroup of $L$. Then $M$ is central in $G$ and $L/M$ is simply connected. Let $\varrho:G\to G/M$ be the natural projection and let $\overline{T}$ be the automorphism on $G/M$ induced $T$. Then $\overline{T}$ acts trivially on both $\varrho(L)$ and $\varrho(G)/\varrho(L)$. By Proposition 3.4, $\overline{T}$ acts distally on ${\rm Sub}^{p}_{G/M}$. From Step 3, we get that $\overline{T}$ acts trivially on $G/M$, i.e. $T(g)\in gM$, $g\in G$. In particular, for $\{x_{t}\}$ as above, $T(x_{t})=x_{t}y_{t}$, where $y_{t}\in M$, for all $t\in{\mathbb{R}}$, and $y_{1}\neq e$. Since $M$ is central in $G$, $y_{t}x_{s}=x_{s}y_{t}$, $t,s\in{\mathbb{R}}$, and $\{y_{t}\}$ is a one-parameter subgroup of the compact central subgroup $M$. Note that $\{y_{t}\}$ is closed if $\{x_{t}\}$ is compact.

As ${\rm Sub}^{p}_{G}$ is compact, there exists an unbounded sequence $\{n_{k}\}$ in ${\mathbb{N}}$ such that $T^{n_{k}}(\{x_{t}\})\to C$ for some $C\in{\rm Sub}^{p}_{G}$. We show that $C=\{x_{t}\}\times\overline{\{y_{t}\}}$. As $\{x_{t}\}\subset\{x_{t}\}\times\overline{\{y_{t}\}}$, and the latter is $T$-invariant, we get that $C\subset\{x_{t}\}\times\overline{\{y_{t}\}}$. For a fixed $t\in{\mathbb{R}}$,

$$T^{n_{k}}(x_{t/n_{k}})=x_{t/n_{k}}y_{t/n_{k}}^{n_{k}}=x_{t/n_{k}}y_{t}\to y_{t}$$

as $n_{k}\to\infty$. Therefore, $y_{t}\in C$ for each $t$ and $\overline{\{y_{t}\}}\subset C$ as $C$ is closed. Also, $T^{n_{k}}(x_{t})=x_{t}y_{t}^{n_{k}}\to x_{t}y$ along a subsequence of $\{n_{k}\}$ for some $y\in\overline{\{y_{t}\}}\subset C$. Hence, $x_{t}\in C$, $t\in{\mathbb{R}}$. Thus $\{x_{t}\}\times\overline{\{y_{t}\}}\subset C$. Now $T(C)=C$, and we get that $T^{n_{k}}(T(\{x_{t}\}))\to C$. as $T$ acts distally on ${\rm Sub}^{p}_{G}$, we have that $T(\{x_{t}\})=(\{x_{t}\})$. Since $T$ is unipotent, we have that $T(x_{t})=x_{t}$ for each $t$. But $T(x_{1})=x_{1}y_{1}\neq x_{1}$. This leads to a contradiction. Therefore, $T(x)=x$ for each $x\in U$, and hence, for each $x\in G$. Thus $T={\rm Id}$. ∎

Proof of Theorem 1.3.

Let $G$ be a connected Lie group and let $T\in{\rm Aut}(G)$. Suppose $T$ is unipotent and it acts distally on ${\rm Sub}^{p}_{G}$. We show that $T={\rm Id}$.

Step 1: Suppose $G$ is nilpotent. If possible, suppose $T\neq{\rm Id}$. Since $T$ is unipotent, by Proposition 3.10 of [40], there exists an increasing sequence of closed connected normal T-invariant subgroups $\{e\}=G_{0}\subset G_{1}\subset\cdots\subset G_{n}=G$, $n\geq 2$, such that $T$ acts trivially on each successive quotient group $G_{i}/G_{i-1}$, $1\leq i\leq n$, and $T$ does not act trivially on $G_{i+1}/G_{i-1}$, $1\leq i\leq n-1$. Then $G_{2}\neq G_{1}$. As $T$ acts distally on ${\rm Sub}^{p}_{G}$, it acts distally on ${\rm Sub}^{p}_{G_{2}}$. Now by Proposition 3.8, $T|_{G_{2}}={\rm Id}$, which leads to a contradiction. Therefore, $T={\rm Id}$ when $G$ is nilpotent.

Step 2: Suppose $G$ is not nilpotent. Let $N$ be the nilradical of $G$. Then $T|_{N}={\rm Id}$. Suppose $G$ is solvable. As noted in Remark 3.3, we have that the automorphism induced by $T$ on $G/N$ is a finite order automorphism, and it is trivial, since it is unipotent. Now by Proposition 3.8, $T={\rm Id}$.

Step 3: Suppose $G$ is not solvable. Let $R$ be the the radical of $G$. Then as $T$ acts distally on ${\rm Sub}^{p}_{R}$, we have from from Step 2 that $T|_{R}={\rm Id}$.

We first assume that $G$ belongs to class ${\mathcal{C}^{\prime}}$, i.e.  $G$ has no compact subgroup of positive dimension. (For this case, we will follow the proof of Theorem 4.1 in [40] to some extent and use Step 1 instead of Proposition 4.2 of [40]). Now ${\rm Aut}(G)$ is almost algebraic and it has finitely many connected components (see [12]). Thus some power of $T$ belongs to a one-parameter subgroup of ${\rm Aut}(G)^{0}$. Since $T$ is unipotent, we get that $T$ itself belongs to a one-parameter subgroup $\{T_{t}\}$ consisting of unipotent elements as $T=T_{1}$. Also, since $\{T^{n}\}_{n\in{\mathbb{Z}}}$ is co-compact in $\{T_{t}\}$, it is easy to see that each $T_{t}$ also acts distally on ${\rm Sub}^{p}_{G}$.

Let $G^{\prime}=G/R$ and let $T^{\prime}_{t}$ be the automorphism of $G^{\prime}$ induced by $T_{t}$, $t\in{\mathbb{R}}$. Here, $T^{\prime}_{t}$ is also unipotent. Note that $G^{\prime}$ is semisimple and ${\rm Aut}(G^{\prime})^{0}$ is the group of inner automorphisms of $G^{\prime}$. There exists a one-parameter subgroup $\{u_{t}\}$ consisting of Ad-unipotent elements in $G^{\prime}$, such that $T^{\prime}_{t}={\rm inn}(u_{t})$, the inner automorphisms by $u_{t}$, $t\in{\mathbb{R}}$. As shown in [40] we can choose a Iwasawa decomposition $G^{\prime}=KAU$ such that $\{u_{t}\}\subset U$, and $U$ (resp. $AU$) is a closed simply connected nilpotent (resp. solvable) group. Let $\pi:G\to G^{\prime}=G/R$ denote the natural projection. Then $B:=\pi^{-1}(AU)$ is a closed connected solvable group, $T_{t}(B)=B$ and $T_{t}$ acts distally on ${\rm Sub}^{p}_{B}$. This implies from Step 2, that $T_{t}|_{B}={\rm Id}$. In particular, ${\rm Inn}(u_{t})$ acts trivially on $AU$, i.e. $\{u_{t}\}$ is contained in the center of $AU$. Let $\mathcal{A}$ (resp. $\mathcal{U}$) be the Lie algebra of $A$ (resp. $U$). Then $u_{t}=\exp tX$, $t\in{\mathbb{R}}$, for some $X\in\mathcal{U}$, and hence $X$ centralises $\mathcal{A}$. Using the restricted root system and Lemma 6.50 of [28], it follows that the centraliser of $\mathcal{A}$ in $\mathcal{A}\,\mathcal{U}$ is contained in $\mathcal{A}$. Therefore, $X\in\mathcal{A}\cap\mathcal{U}=\{0\}$. This implies that for each $t$, $u_{t}=e$, and hence that $T^{\prime}_{t}={\rm Id}$.

Let $G=SR$ be a Levi decomposition, where $S$ is a Levi subgroup (maximal connected semisimple subgroup) of $G$. Let $\pi_{1}:G\to G/N$ be the natural projection. Then $\pi_{1}(S)$ and $\pi_{1}(R)$ centralise each other. Since $T=T_{1}$ and $T^{\prime}_{1}={\rm Id}$ on $G/R$, we have that $T(x)\in xR$, $x\in G$. Let $x,y\in S$. Then $\pi_{1}(T(xyx^{-1}y^{-1}))=\pi_{1}(xyx^{-1}y^{-1})$. Since $[S,S]=S$, we get that $T(x)\in xN$ for all $x\in S$. As $N$ is closed, we get that $T(x)\in xN$ for all $x\in\overline{SN}$. As $T$ acts distally on ${\rm Sub}^{p}_{\overline{SN}}$ and $T|_{N}={\rm Id}$, by Proposition 3.8, we get that $T(x)=x$ for all $x\in S$. Since $T|_{R}={\rm Id}$. We get that $T={\rm Id}$.

We have shown that under the conditions stated in the proposition, $T={\rm Id}$ if $G$ is either solvable or if $G$ belongs to class ${\mathcal{C}^{\prime}}$.

Step 4: Let $M$ be the largest compact connected central subgroup of $G$. Suppose $M$ is non-trivial. Then $G/M$ belongs to class ${\mathcal{C}^{\prime}}$, $M$ is $T$-invariant and by Proposition 3.4, $T$ acts distally on ${\rm Sub}^{p}_{G/M}$. Since the automorphism of $G/M$ induced by $T$ is also unipotent, we get from Step 3 that $T$ acts trivially on $G/M$. That is, $T(x)\in xM$ for all $x\in G$. As $M$ is abelian, by Step 1, $T$ acts trivially on $M$. Now by Proposition 3.8, $T={\rm Id}$.

The converse statement in the theorem is obvious. ∎

Proof of Theorem 1.1 $(2)$ for non-nilpotent case.

Let $G$ be a connected Lie group in class ${\mathcal{C}^{\prime}}$. Then ${\rm Aut}(G)$ is almost algebraic as a subgroup of ${\rm GL}({\mathfrak{g}})$ (via the correspondence $T\leftrightarrow{\rm d\,}T$) (cf. [12]). Suppose $T\in{\rm Aut}(G)$ acts distally on ${\rm Sub}^{p}_{G}$. Suppose $G$ is not nilpotent. In particular, it is not a vector group. By Theorem 1.2, $T$ acts distally on $G$. Then for some $n\in{\mathbb{N}}$, $T^{n}$ is contained in a connected abelian almost algebraic subgroup of ${\rm Aut}(G)$ and it has the form ${\mathcal{K}}\times\,{\mathcal{U}}$, where ${\mathcal{K}},\,{\mathcal{U}}\subset{\rm Aut}(G)$, ${\mathcal{K}}$ is an abelian compact connected group and ${\mathcal{U}}$ is an abelian unipotent connected group; i.e. it consists of unipotent elements (cf. [1], Corollaries 2.3 and 2.5). Now $T^{n}=S\circ\psi=\psi\circ S$, where $S\in{\mathcal{U}}$ is unipotent and $\psi\in{\mathcal{K}}$. Hence, $S$ acts distally on ${\rm Sub}^{p}_{G}$ (cf. [39], Lemma 2.2). By Theorem 1.3, $S={\rm Id}$. Thus $T^{n}=\psi\in{\mathcal{K}}$ and hence $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. The converse is obvious (cf. [30], Lemma 4.2). ∎

Now we want to prove Theorem 1.4. We need an elementary result about certain automorphisms keeping a maximal compact abelian subgroups (maximal tori) invariant. Maximal compact subgroups in a connected Lie group $G$ are connected and conjugate to each other (see [24]). Any maximal compact subgroup of a connected solvable Lie group is abelian. Any maximal compact abelian subgroup of $G$ is connected (in fact, it is a Cartan subgroup of the maximal compact group in which it is contained). A maximal compact (connected) abelian subgroup of a connected Lie group $G$ is called a maximal torus and maximal tori are conjugate to each other. Any compact abelian subgroup is contained in a maximal torus. If $A$ is a maximal torus of $G$, then so is $T(A)$, for any $T\in{\rm Aut}(G)$, and $T(A)$ is a conjugate of $A$.

In [15], any subgroup of ${\rm Aut}(G)$ which fixes elements of a maximal torus are shown to be almost algebraic. Given any automorphism $T$ of $G$, it need not keep any torus invariant. However, this is the case if $T$ is contained in a compact connected abelian subgroup of ${\rm Aut}(G)$ as shown in the following elementary lemma, which also holds if we replace the radical $R$ by any closed connected characteristic subgroup of $G$.

Lemma 3.9.

Let $G$ be a connected Lie group. If ${\mathcal{K}}$ is a compact connected abelian subgroup of ${\rm Aut}(G)$, then there exists a maximal torus of $G$ (resp. of the radical of $G$) which is invariant under every automorphism of $G$ which belongs to ${\mathcal{K}}$.

Proof.

Let $A$ be any maximal torus in a connected Lie group $G$. For any closed connected normal subgroup $L$ of $G$, since the maximal tori of $G$ (resp. $L$) are conjugate to each other in $G$ (resp. $L$), we get that $A$ contains a maximal torus of $L$ and hence, $A\cap L$ is a maximal torus of $L$.

Let ${\mathcal{K}}$ be a compact connected abelian subgroup of ${\rm Aut}(G)$ and let $H={\mathcal{K}}\ltimes G$; where the action of ${\mathcal{K}}$ on $G$ is by automorphisms. Then $H$ is a connected Lie group and $G$ is normal in $H$. Let $R$ be the radical of $G$. Then $R$ is characteristic in $G$ and hence normal in $H$. Let $A$ be a maximal torus in $H$ containing ${\mathcal{K}}$. Arguing as above, we get that $A\cap G$ (resp.  $A\cap R$) is a maximal torus in $G$ (resp. $R$). As $A\cap G$ (resp. $A\cap R$) is normal in $A$ and ${\mathcal{K}}\subset A$, we get that $A\cap G$ (resp. $A\cap R$) is invariant under every automorphism which belongs ${\mathcal{K}}$. ∎

Proof of Theorem 1.4.

Let $G$ be a connected Lie group and let $T\in{\rm Aut}(G)$. The statements $(5)\implies(4)\implies(3)\implies(2)\implies(1)$ are obvious. Now it is enough to show that $(1)\implies(5)$ when $G$ is not a vector group and $T$ acts distally on the largest compact connected central subgroup (say) $M$ of $G$. Suppose (1) holds, i.e. $T$ acts distally on ${\rm Sub}^{p}_{G}$. Then we want to show that $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. If $M$ is trivial, then $G$ belongs to the class ${\mathcal{C}^{\prime}}$ and the assertion follows from Theorem 1.1 (2). Now suppose $M$ is non-trivial. By Theorem 1.2, $T$ acts distally on $G$. Now $\overline{T}$ also acts distally on $G/M$ (cf. [33], Theorem 3.1).

Suppose $G=M$. Since $T$ acts distally on $M$, by Lemma 2.5 of [2], $T^{n}$ is unipotent for some $n\in{\mathbb{N}}$. As $T$ acts distally on ${\rm Sub}^{p}_{G}$, so does $T^{n}$, and by Theorem 1.3, $T^{n}={\rm Id}$. Hence the assertion holds when $G$ is a compact and abelian.

Now suppose $M$ is non-trivial and $G\neq M$. We show that there exists a maximal torus of $G$ which is $T^{k}$-invariant for some $k\in{\mathbb{N}}$. Let $\pi:G\to G/M$ be the natural projection. By Proposition 3.4, $\overline{T}$ acts distally on ${\rm Sub}^{p}_{G/M}$, where $\overline{T}$ is the automorphism of $G/M$ induced by $T$.

Suppose $G/M$ is a vector group. Then by Theorem 1.1 (1), $\overline{T}=\psi\circ\,\alpha\,{\rm Id}$, for some $\alpha\in{\mathbb{R}}\setminus\{0\}$ and $\psi$ generates a relatively compact subgroup in ${\rm Aut}(G/M)$. Since $\overline{T}$ acts distally on $G/M$, we get that $|\alpha|=1$, and hence $\overline{T}$ is contained in a compact subgroup of ${\rm Aut}(G/M)$. Now suppose $G/M$ is not a vector group. As $G/M$ belongs to the class ${\mathcal{C}^{\prime}}$, by Theorem 1.1 (2), $\overline{T}$ is contained in a compact subgroup of ${\rm Aut}(G/M)$. So in either case we have that $\overline{T}$ is contained in a compact subgroup of ${\rm Aut}(G/M)$.

Let ${\mathcal{K}}$ be the connected component of the identity in the closed subgroup generated by $\overline{T}$ in ${\rm Aut}(G/M)$, then $(\overline{T})^{k}\in{\mathcal{K}}$ for some $k\in{\mathbb{N}}$. Now by Lemma 3.9, $\overline{T}$ keeps a maximal torus (say), $A$ of $G/M$ invariant. Let $K=\pi^{-1}(A)$. Then $K$ is a maximal torus of $G$ and $T^{k}(K)=K$.

Note that $K$ is compact, connected and abelian. Moreover, as $T$ acts distally on $G$, so does $T^{k}$ and the restriction of $T^{k}$ to $K$ is also distal. Arguing as above for $K$ instead of $M$, we get that for some $m\in{\mathbb{N}}$, $T^{m}|_{K}$ is unipotent, and as it acts distally on ${\rm Sub}^{p}_{K}$, it acts trivially on $K$ (by Theorem 1.3). Replacing $T$ by $T^{m}$, we may assume that $T$ fixes every element of $K$, i.e. $T\in F_{K}(G)=\{\psi\in{\rm Aut}(G)\mid\psi(x)=x\mbox{ for all }x\in K\}$. We know that $F_{K}(G)$ is almost algebraic (cf. [15]). As $T$ is distal and $T\in F_{K}(G)$, for some $n\in{\mathbb{N}}$, $T^{n}$ is contained in a connected almost algebraic subgroup ${\mathcal{B}}\times{\mathcal{U}}$, where ${\mathcal{B}}$ is a compact group and ${\mathcal{U}}$ consists of unipotent elements (cf. [1], Corollaries 2.3 and 2.5). Now we have that $T=\phi\circ S=S\circ\phi$ such that $\phi\in{\mathcal{B}}$ and $S\in{\mathcal{U}}$. As ${\mathcal{B}}$ is compact, we have that $S$ acts distally on ${\rm Sub}^{p}_{G}$. By Theorem 1.3, $S={\rm Id}$. Thus $T=\phi\in{\mathcal{B}}$ and (5) holds. ∎

Using Theorems 1.1 and 1.4 and some known results, we prove Theorem 1.5.

Proof of Theorem 1.5.

Let $G$ be a connected Lie group and let ${\mathcal{H}}$ be a subgroup of ${\rm Aut}(G)$. Suppose ${\mathcal{H}}$ acts distally on the largest compact connected central subgroup of $G$. Then so does $\overline{{\mathcal{H}}}$, which is a closed subgroup of ${\rm Aut}(G)$. Note that ${\mathcal{H}}$ acts distally on a compact ${\mathcal{H}}$-invariant subspace of ${\rm Sub}_{G}$ if and only if so does $\overline{{\mathcal{H}}}$ (both the above statements follow from [16]). Also (7) holds for ${\mathcal{H}}$ if and only if it holds when ${\mathcal{H}}$ is replaced by $\overline{{\mathcal{H}}}$, as ${\mathcal{K}}{\mathcal{D}}$ is closed in ${\rm GL}(d,{\mathbb{R}})$ for ${\mathcal{K}},{\mathcal{D}}$ as in (7). Therefore to prove the assertions, we can assume that ${\mathcal{H}}$ is closed. Note that every compact subgroup of ${\rm Aut}(G)$ acts distally on ${\rm Sub}_{G}$ (see e.g. [39], Lemma 2.2). Now $(6)\implies(5)\implies(4)\implies(3)\implies(2)\implies(1)$ are obvious.

First suppose that $G$ is not a vector group. We show that $(1)\implies(6)$. As in the hypothesis, Suppose (1) holds, i.e. every $T\in{\mathcal{H}}$ acts distally on ${\rm Sub}^{p}_{G}$. By Theorem 1.4, every $T\in{\mathcal{H}}$ is contained in a compact subgroup of ${\rm Aut}(G)$. Recall that ${\rm Aut}(G)$ is identified with a closed subgroup of ${\rm GL}({\mathfrak{g}})$ via the map $T\to{\rm d\,}T$, where ${\mathfrak{g}}$ is the Lie algebra of $G$ and it is a $d$-dimensional vector space for some $d\in{\mathbb{N}}$ (cf. [9], see also [21]). Therefore, ${\mathcal{H}}$ is also a closed subgroup of ${\rm GL}({\mathfrak{g}})$ and the latter is isomorphic to ${\rm GL}(d,{\mathbb{R}})$, and we get by Proposition 2 of [38] that ${\mathcal{H}}$ is compact. Thus (6) holds and $(1-6)$ are equivalent. Therefore, statements (a), (b) and (c) hold in case $G$ is not a vector group.

Now suppose $G={\mathbb{R}}^{d}$ for some $d\in{\mathbb{N}}$. Suppose (3) holds, i.e. ${\mathcal{H}}$ acts distally on $\overline{{\rm Sub}^{c}_{G}}$. Then every $T\in\rm{(NC)}$. Since $G$ belongs to the class ${\mathcal{C}^{\prime}}$, by Theorem 4.1 of [40], $T$ is contained in a compact subgroup of ${\rm Aut}(G)$. Since ${\mathcal{H}}$ is closed, arguing as above using Proposition 2 of [38], we get that ${\mathcal{H}}$ is compact. Hence (6) holds. Thus $(3-6)$ are equivalent and hence (c) holds when $G$ is a vector group. Now we show that (1), (2) and (7) are equivalent. Suppose (7) holds. Note that ${\mathcal{D}}=\{r\,{\rm Id}\in{\rm GL}(d,{\mathbb{R}})\mid r\in{\mathbb{R}}\setminus\{0\}\}$. Therefore, ${\mathcal{D}}$ acts trivially on ${\rm Sub}^{p}_{G}$ and the action of ${\mathcal{K}}{\mathcal{D}}$ on ${\rm Sub}^{p}_{G}$ is same as that of the compact group ${\mathcal{K}}$ on ${\rm Sub}^{p}_{G}$. Hence the action of ${\mathcal{K}}{\mathcal{D}}$, and hence, the action of ${\mathcal{H}}$ on ${\rm Sub}^{p}_{G}$ is distal. Thus $(7)\implies(2)$. We have already noted that $(2)\implies(1)$.

Now we show that $(1)\implies(7)$. Suppose every $T\in{\mathcal{H}}$ acts distally on ${\rm Sub}^{p}_{G}$. Let $T\in{\mathcal{H}}$ be fixed. By Theorem 1.1 (1), $T\in{\mathcal{B}}{\mathcal{D}}$, for some compact group ${\mathcal{B}}\subset{\rm GL}(d,{\mathbb{R}})$. Let ${\mathcal{H}}_{1}:=\overline{[{\mathcal{H}},{\mathcal{H}}]}$ .Then ${\mathcal{H}}_{1}\subset{\mathcal{H}}$ as the latter is closed. Let $T\in{\mathcal{H}}_{1}$. Then $\det T=1$. From above we have that $T=\phi\,\circ\,r\,{\rm Id}$ for some $\phi\in{\mathcal{B}}$ and some $r\in{\mathbb{R}}\setminus\{0\}$. Since ${\mathcal{B}}$ is a compact group, $\det\phi=\pm 1$, and hence $r=\pm 1$. Therefore, $T$ generates a relatively compact group, for every $T\in{\mathcal{H}}_{1}$. As ${\mathcal{H}}_{1}$ is a closed subgroup of ${\rm GL}(d,{\mathbb{R}})$ by Proposition 2 of [38], ${\mathcal{H}}_{1}$ is compact. Now ${\mathcal{H}}_{1}$ is a compact normal subgroup of ${\mathcal{H}}$. Let $N({\mathcal{H}}_{1})$ be the normaliser of ${\mathcal{H}}_{1}$ in ${\rm GL}(d,{\mathbb{R}})$. Then $N({\mathcal{H}}_{1})$ is a closed algebraic subgroup of ${\rm GL}(d,{\mathbb{R}})$ and ${\mathcal{H}},{\mathcal{D}}\subset N({\mathcal{H}}_{1})$. Moreover, $N({\mathcal{H}}_{1})\subset N({\mathcal{H}}_{1}{\mathcal{D}})$. Being algebraic, $N({\mathcal{H}}_{1})$ has finitely many connected components, and hence so does $N({\mathcal{H}}_{1})/{\mathcal{H}}_{1}{\mathcal{D}}$. In particular any closed abelian subgroup of $N({\mathcal{H}}_{1})/{\mathcal{H}}_{1}{\mathcal{D}}$ is compactly generated (cf. [23]) Let $\rho:N({\mathcal{H}}_{1})\to N({\mathcal{H}}_{1})/{\mathcal{H}}_{1}{\mathcal{D}}$ be the natural projection. Then $\rho({\mathcal{H}})$ is abelian, and hence so is $\overline{\rho({\mathcal{H}})}$. Then $\overline{\rho({\mathcal{H}})}$ is compactly generated, and it has a unique compact subgroup, say $C$. Now let $T\in{\mathcal{H}}$. Then $T=\psi\,\circ\,r\,{\rm Id}$ for some $\psi$ which generates a relatively compact group and $r\,{\rm Id}\in{\mathcal{D}}$. Then $\rho(T)=\rho(\psi)$ also generates a relatively compact group in $\overline{\rho({\mathcal{H}})}$, and hence it is contained in $C$. Thus $\rho({\mathcal{H}})\subset C$ and $\overline{\rho({\mathcal{H}})}$ is compact. As ${\mathcal{H}}_{1}$ is compact, it follows that $\overline{{\mathcal{H}}{\mathcal{D}}}/{\mathcal{D}}$ is compact. Since ${\mathcal{D}}$ has two connected components, we get that $\overline{{\mathcal{H}}{\mathcal{D}}}/{\mathcal{D}}^{0}$ is also compact and it is a Lie group. As ${\mathcal{D}}^{0}$ is a vector group which is central in $\overline{{\mathcal{H}}{\mathcal{D}}}$, by Lemma 3.7 of [24], $\overline{{\mathcal{H}}{\mathcal{D}}}={\mathcal{K}}\times{\mathcal{D}}^{0}$, where ${\mathcal{K}}$ is the maximal compact subgroup of $\overline{{\mathcal{H}}{\mathcal{D}}}$. In particular, ${\mathcal{H}}\subset{\mathcal{K}}{\mathcal{D}}^{0}$. Thus (7) holds. Therefore, if $G$ is a vector group, (1), (2), and (7) are equivalent and (a) holds and (d) holds. This completes the proof. ∎

Now we state two corollaries which are easy to deduce using results proven above and a result in [1].

Corollary 3.10.

Let $G$ be a connected Lie group and let ${\mathcal{H}}$ be a subgroup of ${\rm Aut}(G)$. Then ${\mathcal{H}}$ acts distally on both $G$ and ${\rm Sub}^{p}_{G}$ if and only if $\overline{{\mathcal{H}}}$ is compact.

Corollary 3.10 is a direct consequence of Theorem 1.5 in case $G$ is not a vector group. If $G$ is a vector group, then as ${\mathcal{H}}$ acts distally on $G$, the eigenvalues of every element in ${\mathcal{H}}$ have absolute value 1 (cf. [1]), and this together with the statement (7) in Theorem 1.5 is equivalent to the statement that $\overline{{\mathcal{H}}}$ is compact. We omit the details.

The following corollary is an extension of Corollary 4.6 of [40] for a particular class of Lie groups. It can be proven easily using Theorems 1.1 and 1.5 and Proposition 3.4. We omit the proof.

Corollary 3.11.

Let $G$ be a connected Lie group such that $G/M$ is not a vector group, where $M$ is the maximal compact connected central subgroup of $G$. Let $T\in{\rm Aut}(G)$ (resp. ${\mathcal{H}}$ be a subgroup of ${\rm Aut}(G)$). If $T$ (resp. ${\mathcal{H}}$) acts distally on ${\rm Sub}^{p}_{G}$, then $T$ (resp. ${\mathcal{H}}$) has bounded orbits; i.e. $\{T^{n}(x)\mid n\in{\mathbb{Z}}\}$ (resp. ${\mathcal{H}}x$) is relatively compact in $G$, for every $x\in G$.

Now we prove Corollary 1.6. The statement in the corollary which requires a proof is $(1)\implies(5)$ and can be proven along the lines of proof of $(1)\implies(5)$ of Corollary 4.5 in [40]; instead of Corollary 4.3 and Theorem 4.3 there, one has to use Theorems 1.2 and 1.3. We will give a proof here for the sake of completeness.

Proof of Corollary 1.6.

Let $G$ be a connected Lie group. The statement in (5) implies that ${\rm Inn}(G)$ is compact, and hence it follows that $(5)\implies(4)$. Also $(4)\implies(3)\implies(1)$ and $(4)\implies(2)\implies(1)$ are obvious. It is enough to show that $(1)\implies(5)$.

Suppose (1) holds; i.e. every inner automorphism of $G$ acts distally on ${\rm Sub}^{p}_{G}$. If $G$ is either compact or a vector group, then (5) holds trivially. Now suppose $G$ neither compact nor a vector group. Since every inner automorphism acts trivially on the center of $G$, we have by Theorem 1.2 that it acts distally on $G$. Therefore, $G$ is distal (cf. [35]), i.e. the action of ${\rm Inn}(G)$ on $G$ is distal. It follow from Theorem 9 of [35] and Corollary 2.1 (ii) of [26] that $G/R$ is compact, where $R$ is the radical of $G$. Let $N$ be the nilradical of $G$ and let $x\in N$. Then ${\rm inn}(x)$ is unipotent, and by Theorem 1.3, ${\rm inn}(x)={\rm Id}$. This implies that $x$ is central in $G$ for every $x\in N$. Then $N$ is abelian and central in $G$. Now for the radical $R$, we know that $[R,R]\subset N$ is central in $G$, and hence $R$ is nilpotent and it is same as $N$ and it is abelian and central in $G$. Now $G/N$ is compact, where $N$ is central in $G$, i.e. $N={\mathbb{R}}^{n}\times M$, for some $n\in{\mathbb{N}}$, where $M$ is compact. Here, $V={\mathbb{R}}^{n}$ is a vector subgroup which is central. By Lemma 3.7 of [24], $G=K\times{\mathbb{R}}^{n}$, where $K$ is the maximal compact subgroup of $G$. Thus (5) holds. ∎

The distal actions of automorphisms $T$ of a connected Lie group $G$ on ${\rm Sub}_{G}$ and its closed $T$-invariant subspaces were studied if $G$ is in class ${\mathcal{C}^{\prime}}$ or $T$ is unipotent; here we have studied a more general case when $T$ acts distally on the maximal central torus of $G$. We have also studied the action of subgroups of ${\rm Aut}(G)$ on ${\rm Sub}_{G}$ and ${\rm Sub}^{p}_{G}$ for a connected Lie group $G$ under a condition that the action on the maximal central torus is distal. It would be interesting to characterise general automorphisms of $G$ which act distally on ${\rm Sub}_{G}$ or its closed invariant subspace for any connected Lie group $G$ without the condition involving the action on the central torus.

Acknowledgement. The authors would like to acknowledge the support in part by the International Centre for Theoretical Sciences (ICTS) during a visit for participating in the program - ICTS Ergodic Theory and Dynamical Systems (code: ICTS/etds-2022/12).