On Almost-Sure Intention Deception Planning that Exploits Imperfect Observers

Abstract

Intention deception involves computing a strategy which deceives the opponent into a wrong belief about the agent’s intention or objective. This paper studies a class of probabilistic planning problems with intention deception and investigates how a defender’s limited sensing modality can be exploited by an attacker to achieve its attack objective almost surely (with probability one) while hiding its intention. In particular, we model the attack planning in a stochastic system modeled as a Markov decision process (MDP). The attacker is to reach some target states while avoiding unsafe states in the system and knows that his behavior is monitored by a defender with partial observations. Given partial state observations for the defender, we develop qualitative intention deception planning algorithms that construct attack strategies to play against an action-visible defender and an action-invisible defender, respectively. The synthesized attack strategy not only ensures the attack objective is satisfied almost surely but also deceives the defender into believing that the observed behavior is generated by a normal/legitimate user and thus failing to detect the presence of an attack. We show the proposed algorithms are correct and complete and illustrate the deceptive planning methods with examples.

Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-22-1-0034.

Appendices

A Proof of Proposition 2 and the Construction of ASW Region and ASW Strategies

Proof

First, we provide the algorithm to solve the ASW region $\textsf{ASW}(\varphi )$ and ASW strategy $\pi$ for the task $\varphi {:}{=}\lnot U \, \textsf{U}\,F$ where $U \cap F = \emptyset$.

• 1. Initiate $X_0=F$ and $Y_0=S {\setminus } U$. Let $i=j=1$.

• 2. Let $X_{i+1} = \{X_i\} \cup \{s \in Y_j {\setminus } X_i\mid \exists a\in A(s), \textsf{Post}(s,a) \cap X_i \ne \emptyset \text { and } \textsf{Post}(s,a)\subseteq Y_j\}$

• 3. If $X_{i+1}\ne X_i$, then let $i=i+1$ and go to step 2; else, let $n=i$ and go to step 4.

• 4. Let $Y_{j+1}=X_i$. If $Y_{j+1}= Y_j$, then $\textsf{ASW}(\varphi )=Y_j$. Return $\{X_i, i=1,\ldots , n\}$ computed from the last iteration. Else, let $j=j+1$ and $i=0$ and go to step 2.

The algorithm returns a set of level sets $X_i, i=0,\ldots , n$ for some $n\ge 0$ and the ASW region $\textsf{ASW}(\varphi )$. Recall that $\textsf{Allowed}: S\rightarrow 2^A$ is defined by $\textsf{Allowed}(s) = \{a\in A(s)\mid \textsf{Post}(s,a)\subseteq \textsf{ASW}(\varphi )\}$. The following property holds: For each $s\in X_i\setminus X_{i-1}$, there exists an action $a\in \textsf{Allowed}(s)$ that ensures, with a positive probability, the next state is in $X_{i-1}$ and with probability one, the next state is in $\textsf{ASW}(\varphi )$. The strategy $\pi : \textsf{ASW}(\varphi )\rightarrow \mathcal {D}(A)$ is almost-sure winning if for every state $s\in \textsf{ASW}(\varphi )$, $\text{ Supp }(\pi (s)) = \textsf{Allowed}(s)$. That is, for every permissible action a at state $s \in \textsf{ASW}(\varphi )$, $\pi (s)$ selects that action with a non-zero probability. The ASW strategy may not be unique.

Let’s define a function $\textsf{Prog}: \textsf{ASW}(\varphi )\rightarrow 2^A$ such that for each $s\in X_i{\setminus } X_{i-1}$, $\textsf{Prog}(s)=\{a\in \textsf{Allowed}(s)\mid \textsf{Post}(s,a)\cap X_{i-1}\ne \emptyset \}$. Intuitively, the set $\textsf{Prog}(s)$ is a set of actions, each of which ensures that a progress (to a lower level set) can be made with a positive probability.

Therefore, by following a policy $\pi$ that selects any action in $\textsf{Allowed}(s)$ with probability $> 0$, the probability of starting from a state $s\in X_i{\setminus } X_{i-1}$ and not reaching a state in $X_0=F$ in i steps is less than $(1-p)^i$ where $p = \min _{0<i \le n, s\in S, a\in \textsf{Prog}(s)} \pi (s,a)P(s' \mid s,a)$ and is nonzero. If in the i-th step, the set $X_0$ is not reached, the agent will reach a state $s'\in \textsf{ASW}(\varphi )$ from which an action in $\textsf{Prog}(s')$ will be selected with a nonzero probability. Thus, the probability of never reaching a state in F is $\lim _{k\rightarrow \infty } (1-p)^k =0$. In other words, the policy $\pi$ ensure F is eventually reached with probability one. At the same time, because the set $Y_j \cap U =\emptyset$ for all $j >0$ during iterations, $\textsf{ASW}(\varphi )\cap U = \emptyset$ and thus the probability of reaching a state in U is zero by following the policy $\pi$.

B Proof of Theorem 2

Proof

We show that the ASW policy $\widehat{\pi }_1: S\times 2^S\rightarrow \mathcal {D}(A)$ obtained from the augmented MDP is qualitatively observation-equivalent to an ASW policy for the user.

Consider a history $h = s_0a_0s_1a_1\ldots s_n$ which is sampled from the stochastic process $M_{\widehat{\pi }_1}$ and satisfies $s_i \notin F_1 \cup U_1$ for $0\le i <n$ and $s_n \in F_1$. The history is associated with a history in the augmented MDP, $\widehat{h} = (s_0, B_0)a_0(s_1,B_1)a_1\ldots (s_n,B_n)$ where $B_0=\textsf{DObs}_S(s_0)$ is the initial belief for the defender. Due to the construction of the augmented MDP, for all $0\le i \le n$, $B_i\ne \emptyset$.

By the definition of qualitatively observation-equivalence, we only need to show that there exists $h' =s_0'a'_0s_1'a'_1 \ldots s_n'$ where $s_i'\in B_i$, for all $i=0,\ldots , n$, such that $\Pr (h', M_{\pi _0})>0$ where $M_{\pi _0}$ is the Markov chain induced by a user’s ASW policy $\pi _0$ from the MDP M. It is observed that $a_i$ and $a'_i$ may not be the same. By way of contradiction, suppose, for any state-action sequence $h' =s_0'a'_0s_1'a'_1 \ldots s_n'$ where $s_i'\in B_i$ and $a'_i \in A(s_i')$ for $0\le i \le n$, it holds that $\Pr (h', M_{\pi _0})=0$. When $\Pr (h', M_{\pi _0})=0$, there are two possible cases: First case: there exists some $i\ge 0$ such that for all $s\in B_i$, $\textsf{Allowed}(s)=\emptyset$; second case: there does not exists an action a enabled from the belief $B_i$ and a state $s'\in B_{i+1}$ such that $P(s'|s,a)>0$.

The first case is not possible because when for all $s\in B_i$, $\textsf{Allowed}(s)=\emptyset$, then the next state reached will be $(s_i,\emptyset )$ which is a sink state, contradicting the fact that $\widehat{h}$ satisfies the reach-avoid objective. In the second case, if for any state $s\in B_i$, for any action a enabled from s, $\textsf{Post}(s,a) \cap B_{i+1}=\emptyset$, then $B_{i+1}=\emptyset$, this again contracts that $\widehat{h}$ visits a state in $\widehat{F}$.

Thus, it holds that there exists $h'$ such that $\textsf{DObs}_S(h)=\textsf{DObs}_S(h')$ and $\Pr (h', M_{\pi _0})>0$.