Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

Heterogeneous multi-agent collaboration tasks leverage the diverse capabilities and complementary attributes of the robots to enhance the scalability and robustness of the multi-agent system. There are already some heterogeneous robot combinations proposed such as aerial-ground collaboration [21, 53], main-picket collaboration [13], and humanoid-quadruped collaboration [43]. However, challenges in coordination, planning, and execution are also presented. Some methodologies that focus on fixed robot combinations have been proposed. Zhao et al. [56] generate coordination strategies for robots with asymmetric information and influence. Decentralized cooperative transportation by aerial and ground robots equipped with manipulators is discussed in [21]. Haldane et al. [13] undertake joint locomotion and perception tasks by legged robots of varying sizes and capabilities. There is also work dealing with multi-task collaboration scenarios involving a variety of heterogeneous robots and humans [25]. Our research focuses on improving collaboration within any team, irrespective of the diversity in capabilities, policies, and team size.

Ad hoc teamwork involves designing a collaboration policy for agents to collaborate with unknown teammates without prior coordination, which was initially proposed by Stone et al. [40] and further detailed by Mirsky et al. [28]. Early research mostly focuses on matrix games, which assume knowledge of partner actions [1, 39]. Two common methods for ad hoc teamwork include type inference, which models teammates as defined types and uses past interactions to estimate type probabilities for learning algorithms [2, 3], and experience recognition, which compares current and past observations to identify the best actions, with PLASTIC-Policy as an influential example [4]. Deep learning has been applied to both methods [5, 29, 57], and some variations of ad hoc teamwork have been explored, including partial observability settings [12, 30] and open environment settings with fluctuating team sizes [11, 33]. However, most ad hoc teamwork research uses data-driven learning methods, which are computationally expensive and lack transparency. To address this, an architecture is introduced that uses non-monotonic logical reasoning with prior commonsense domain knowledge and models learned from limited examples to predict other agents’ behavior [9]. This approach is further developed into a framework that supports non-monotonic logical reasoning with prior commonsense domain knowledge [10]. Our work expands existing ad hoc teamwork research by considering settings where agents have different arrival times.

Large Language Models (LLMs) have been increasingly utilized to solve embodied collaboration tasks among multiple robots. In [7], four distinct communication mechanisms based on LLMs have been proposed, and their efficiency and scalability have been studied in 2D multi-agent tasks. In the Overcooked-AI game, the ProAgent leverages LLMs for cooperative reasoning to enhance collaboration [54]. For 3D embodied visual tasks, LLMs empower embodied agents to communicate, plan, and collaborate with other agents or humans [55]. The multi-robot manipulator benchmark, RoCoBench, introduces LLMs to facilitate communication, generate subtask plans, and perform multi-arm motion planning [26]. In the context of homogeneous multi-agent visual semantic navigation, Co-NavGPT employs LLMs as a central planner to predict the next navigation frontier for each agent [51]. Also, SMART-LLM uses LLMs for central task decomposition and task allocation in fully observable heterogeneous multi-agent tasks, considering the different abilities and states of agents [20]. These works highlight the versatility and potential of LLMs in facilitating effective multi-agent collaboration.

For convenience, we describe the needed information in the proposed framework under the tidying-up setting in advance as follows

• •

  State information ($\mathcal{S}_{0}$): the location of the agent, the object-grasping state, the observation, and key detection information, as well as the task-relevant state in the current time step.

• •

  Message information ($\mathcal{M}_{0}$): all messages the agent has sent and received in the current time step.

• •

  Capability information ($\mathcal{C}_{0}$): agent height, manipulation ability, payload (the maximum mass of objects that can be picked up), and total number of steps the battery power can initially support.

• •

  Action information ($\mathcal{A}_{0}$): the action the agent takes in the current time step.

• •

  Memory information ($\mathcal{H}_{0}$): all state, message, and action information in all previous time steps.

The ad hoc agent utilizes the communication module to exchange effective information with the original team. When the ad hoc agent joins the original team, it will broadcast its state and capability information to teammates and receive reflected messages. The reflected messages refer to the state and capability information of teammates that are transmitted from each teammate to the ad hoc agent. Team members are expected to transmit the corresponding messages back. When planning the next sub-task, the ad hoc will broadcast its intention and utilize the received messages from teammates to complete the planning. The task-relevant state contains the current sub-task and sub-skill planning results. When the ad hoc agent has a new task-relevant state, specifically when it finishes or switches to new sub-tasks or sub-skills, it will transmit the information to teammates. The ad hoc agent can also receive the corresponding information from other teammates to do subsequent planning. After execution of each step, the ad hoc agent can reason and judge whether the currently detected object is misplaced and if it can be the receptacle for other misplaced objects. The key detection information refers to these reasoning results. When the ad hoc agent obtains the new key detection information, it will broadcast this information to other teammates. The examples of the above communication are shown in Fig. 3. We have created an interface for different policies of teams to adapt to the ad hoc agent. This interface extracts capability and state information from the message format of teammates’ models, accommodating their varying action policies.

The visual semantic perception module detects the objects in the current view of the ad hoc agent, generates the room-object aware semantic map, and obtains the spatial relationships among all detected objects. At each step, the ad hoc agent generates the semantic point clouds based on the obtained RGB-D image, instance segmentation result, and the current pose. The type of room where the agent is located is reasoned according to the categories of currently detected objects. The semantic point clouds are voxelized and flattened to generate a top-down object-room-aware semantic map, which is utilized for exploration and navigation. The hierarchical spatial scene graph is constructed based on the coordinates of each detected object and receptacle as well as the predicted room location, which is utilized to plan the subsequent sub-task and sub-skill. The ad hoc agent updates the semantic map and scene graph with new detection information at each step.

The LLM is employed as the dynamic sub-task planner to predict the subsequent sub-task for the ad hoc agent. This sub-task is structured to encapsulate the agent’s overall intention within a specific time frame. The LLM is utilized for planning due to its ability to analyze and rationalize the sub-tasks that need to be executed based on teammates’ capabilities and state information. It also offers flexibility in collaborating with teammates possessing varied skill sets. When joining and collaborating with the original team, the ad hoc agent is likely to have multiple options of sub-tasks to execute when planning, and performing different sub-tasks may have different effects on the original team. The ad hoc agent needs to find the most suitable sub-task to make the collaboration more efficient. Due to the limited information the ad hoc agent obtains when joining the team, the communication messages from teammates are beneficial for the ad hoc agent to make decisions. To solve this complex situation in the interactive environment, we propose the new method Interactive Reflection of Thoughts (IRoT) on the basis of the LLM, which introduces the communication reflection mechanism to evaluate the generated candidate sub-tasks with obtained communication messages and determine the next sub-task for the ad hoc agent.

In the context of the heterogeneous multi-agent tidying-up task, the sub-tasks are categorized into two types:

• •

  Explore $\left\langle ID_{room}\right\rangle$: Explore the room $ID_{room}$ to search for misplaced objects and obtain information of the room.

• •

  Re-place $\left\langle ID_{object}\right\rangle$ $\left\langle Type_{receptacle}\right\rangle$ $\left\langle Type_{room}\right\rangle$: Put the detected misplaced object $ID_{object}$ to the reasonable type of receptacle $Type_{receptacle}$ in the reasonable type of room $Type_{room}$.

• •

  Stop: Stop at the current location.

This approach contains three stages, which are demonstrated in Fig. 4. The first stage is Generation of Candidate Sub-Tasks, in which the LLM generates $n_{irot}$ candidate sub-tasks and the corresponding reasoning analysis with the provided prompts. The second stage is Evaluation and Ranking, in which the LLM evaluates the generated sub-tasks and ranks them based on the execution rationality with the candidate sub-tasks and evaluation prompts as input. The third stage is Interactive Reflection. In this stage, ranked sub-tasks are sequentially selected and shared with teammates through communication as the ad hoc agent’s intention. The LLM judges the feasibility of the selected sub-task based on the received feedback from teammates. If the selected one is judged to be unreasonable, the next sub-task is selected for another round of this stage. If the selected one is reasonable, the selected sub-task is the planning results of the sub-task planner. The key prompts of the IRoT are shown in Fig. 5, and the output example is shown in the Appendix.

The dynamic sub-skill planner generates the next sub-skill with LLM based on the CoT method for the ad hoc agent. The execution of a sub-task can be decomposed into a list of sub-skills within a certain time sequence. The candidate sub-skills include GoToObject $\left\langle ID_{object}\right\rangle$, GoToPoint $\left\langle\Delta x,\Delta y\right\rangle$, GoToRoom $\left\langle ID_{room}\right\rangle$, PickupObject $\left\langle ID_{object}\right\rangle$, PutObject $\left\langle ID_{object}\right\rangle$ $\left\langle ID_{receptacle}\right\rangle$ $\left\langle ID_{room}\right\rangle$, Explore, Stop. The sub-skills are the meta skills that can be executed by the controller. A detailed description of these sub-skills is illustrated in Appendix. We provide the state, capability, communication, and memory information, as well as the in-context examples in the prompts to the LLM. Specifically, the current sub-task and sub-skill plan of each teammate can be obtained from the state and memory information. The key prompts of planning sub-skills are shown in Fig. 6, and the output example is shown in Appendix.

The sub-skill execution module generates the low-level actions to execute the sub-skill. This module executes the planned sub-skill and updates the ever-changing state information, informing both the sub-task and sub-skill planners of necessary adjustments. We can obtain the execution policies for each sub-skill in advance. The execution policies can be generated through rule-based methods or trained by imitation learning (IL) and reinforcement learning (RL).

The state assessment module evaluates the success of an action and the state information after executing the corresponding low-level action. The evaluation results would provide more effective information and are likely to lead to the re-planning of sub-tasks or sub-skills. When evaluating the success of the action execution, this module considers whether the ability of the ad hoc agent can match the executed action and whether the action can successfully interact with the environment. For example, the Pickup action would fail if the ad hoc agent lacks the manipulation ability, the mass of the object exceeds the maximum mass that the agent can pick, or the agent is far away from the target object when executing the Pickup action. If the agent encounters obstacles when performing the navigation action, the action execution would also fail. If the action execution fails, this module would identify the failure reason and provide feedback to the planner and the action execution module to re-generate the reasonable sub-skills and feasible actions. Meanwhile, the agent also judges whether newly detected objects are misplaced or likely to be the target receptacles. If misplaced objects are found, the ad hoc agent would inform its teammates through communication and re-plan the sub-task as well as the sub-skill to execute in the next step. If potential receptacles are identified, the ad hoc agent would share the corresponding information with teammates through communication and re-plan the sub-skill in the next step.

The hallucinations of the sub-skill planner refer to the generated unreasonable plans of sub-skills but are deemed syntactically plausible. The state assessment module evaluates the action execution state, which can detect the hallucinations of the sub-skill planner and help re-plan the next sub-task as well as the sub-skill.

The proposed framework with the LLM-based hierarchical planner can also be applied to other different tasks. When applying our model to different tasks, the structure of the model remains unchanged. The dynamic sub-task planner with IRoT and the dynamic sub-skill planner are still utilized to generate the next sub-task and the sub-skill to execute. To adapt to different tasks, it is required to design corresponding reasonable candidate sub-tasks and sub-skills for the task, update the description of the new task, modify the descriptions of sub-tasks and sub-skills in the corresponding prompts, and keep the structure as well as other contents in the prompts of the sub-task and sub-skill planner unchanged.

As for the sub-task planner, the whole process of IRoT is kept unchanged and the candidate sub-tasks are designed according to the nature of the corresponding tasks. Then it is necessary to update the description of the new task and modify the description of candidate sub-tasks in the prompts of the sub-task planner as shown in Fig. 5 to the description corresponding to the newly defined sub-tasks. The other contents in the prompts of the sub-task planner can be kept unchanged. The modified model can generate corresponding sub-tasks in new tasks. Meanwhile, as for more complex tasks, if all possible candidate sub-tasks cannot be completely pre-defined, we can remove the sub-task descriptions in the prompts without setting the sub-task types in advance and let the LLM directly generate reasonable sub-tasks according to current information. Then the generated sub-task is input to the sub-skill planner to generate the corresponding sub-skills. The candidate sub-skills are pre-defined and the output of the sub-skill planner must come from the pre-defined types of sub-skills. The structure and other contents in the prompts of the sub-task planner can be kept unchanged.

The sub-skill planner generates the basic skill for the agent to execute according to the current sub-task. In common indoor tasks such as visual navigation [48], visual rearrangement [47], and ALFRED [38], the manipulation and interaction on objects can be eventually decomposed into the sub-skills of navigation, picking and placing, so the seven candidate sub-skills proposed in our model can cover basic skills in common tasks. When applying our model to different tasks, the task description needs to be updated and other contents in the prompts of the sub-skill planner can remain unchanged. If more complex tasks require other sub-skills to be executed, it is required to design new sub-skills and provide the low-level action strategy under the corresponding sub-skills. Then the sub-skill descriptions in the prompts as shown in Fig. 6 need to be modified to the descriptions of newly defined sub-skills. Other contents in the prompts can be kept unchanged in the sub-skill planner. The sub-skill planner can generate the next sub-skill to execute from the candidate sub-skills.

To apply our model to other tasks, we only need to design the candidate sub-tasks and sub-skills according to the properties of the new task, update the description of the new task, and modify descriptions of the sub-task as well as the sub-skill in the prompts of the sub-task planner and the sub-skill planner.

As far as we know, there isn’t any existing benchmark dataset that can evaluate heterogeneous ad hoc teamwork. Considering the tidying-up is an ideal task to evaluate the performance of heterogeneous ad hoc teamwork, we build a benchmark testing dataset for the heterogeneous ad hoc teamwork on the tidying-up task with the ProcTHOR-10K [8] dataset.

Scenario generation: We randomly select 90 houses from the testing houses in ProcTHOR-10K for evaluation experiments, among which every 10 houses are with 1-8, and 10 rooms respectively (there is no house with 9 rooms in ProcTHOR-10K). We generate 5 tidying-up scenarios for each house, where $k$ pickupable objects are randomly selected and misplaced to unreasonable receptacles or dropped in a random location in the house ($k=1,2,3,4,5$).

Heterogeneous teammates generation: The characteristics of the teammate are described by a tuple of $(\alpha_{nav},\alpha_{manip},h,m,step)$ where $\alpha_{nav}$ and $\alpha_{manip}$ indicate the navigation ability and manipulation ability of the agent, $h$ is the height of the agent, $m$ denotes the payload which means the maximum weight of the object that can be picked up by the agent, and $step$ is the max steps that the agent’s battery can support. For each tidying-up scenario, we consider situations of having 3, 4, and 5 teammates, respectively, and we randomly generate the values for each teammate’s characteristics. It is guaranteed that all generated agents have the navigation ability and at least one agent has manipulation ability in order to accomplish the tidying up task. Additionally, the initial positions of each teammate are randomly generated from the reachable positions in the corresponding task.

Ad hoc agent generation: Similar to the process of generating heterogeneous teammates. We generate one ad hoc agent for each tidying-up task. It is noted that the ad hoc agent does not have to have the manipulation ability. Its initial position is also randomly generated.

Finally, we obtain 1350 tidying-up tasks for the heterogeneous ad hoc teamwork. We divide these tasks into three difficulty categories: Easy, Medium and Difficult, in which houses with 1-3 rooms are Easy tasks, houses with 4-6 rooms are Medium tasks, and houses with 7-10 rooms are Difficult tasks. The heterogeneous agent and scene settings are shown in Fig. 7. The process of generating the data can be applied and extended to the whole testing set in ProcTHOR-10K. More detailed information can be found in Appendix.

It is worth noting that the tidying-up task is different from the rearrangement task. In the rearrangement task, the agent only needs to re-place objects according to the given initial state, while in the tidying-up task, the agent is required to infer the reasonable locations of each object by itself without extra guidance and re-place misplaced objects to reasonable locations, which puts forward higher requirements for scene understanding and commonsense reasoning.

Before the ad hoc agent joins the team, the original heterogeneous teammates collaborate with each other with a certain policy. The proposed ad hoc teamwork framework can adapt to different heterogeneous teammates no matter what policy they employ. In the experiment, we consider the following different common policies for the original heterogeneous teammates’ collaboration.

For the ad hoc agent, we compare the proposed method with several baselines as follows.

We call our method based on IRoT as the IRoT-LLM model. All the LLM-based baselines and our IRoT-LLM model utilize the gpt-4-1106-preview to complete the planning process. The parameter Temperature is set to be 0.3 in our experiments. The maximum limited number of tokens in the prompt is 128,000. If the maximum limit is exceeded in tasks, we would move out some of the earliest information from the historical communication messages in the prompt to adapt to the maximum limit.

The effectiveness of an ad hoc policy can be quantified by the difference in performance between the original heterogeneous team and the new team after the ad hoc agent joins. We denote $N_{epi}$ as the number of tasks. In the context of our experiment, we utilize the evaluation metrics as follows:

1) %Suc: The success rate of tasks. $\%Suc=\frac{1}{N_{epi}}\sum_{i=1}^{N_{epi}}R_{i}$, where $R_{i}=1$ if all misplaced objects are re-placed in reasonable locations, otherwise $R_{i}=0$. It denotes the strict success rate.

2) %PS: The partial success rate of tasks. $\%PS=\frac{1}{N_{epi}}\sum_{i=1}^{N_{epi}}\frac{K^{suc}_{i}}{K_{i}}$, where ${K_{i}}$ is the number of misplaced objects in the $i$-th task, and $K^{suc}_{i}$ is the number of misplaced objects that are re-placed to reasonable locations. It reflects the proportion of misplaced objects that are successfully tidied.

3) #TS: The average temporal steps of task completion. $\#TS=\frac{1}{N_{epi}}\sum_{i=1}^{N_{epi}}TS_{i}$, where $TS_{i}$ is the number of temporal steps when completing the $i$-th task, the value equals the maximum steps among all agents’ steps when they stop. If agents do not successfully complete the task, $TS_{i}$ is the maximum number of steps. It evaluates the temporal efficiency of completing the task. The fewer $\#TS$, the higher the efficiency.

4) #AS: The total action steps of all agents in tasks. $\#AS=\frac{1}{N_{epi}}\sum_{i=1}^{N_{epi}}AS_{i}$, where $AS_{i}$ is the sum of action steps of all agents when they stop in the $i$-th task. If agents do not successfully complete the task, the action step of each agent is the maximum number of steps. It can evaluate the total energy consumption after the ad hoc agent joins the team. The fewer $\#AS$, the less energy consumption of the collaboration.

5) %SE: Success rate weighted by efficiency improvement. $SE=\frac{1}{N_{epi}}\sum_{i=1}^{N_{epi}}\frac{max(0,TS_{i}^{team}-TS_{i}^{adhoc})}{TS_{i}^{team}}{R_{i}^{adhoc}}$, where $TS_{i}^{team}$ is the number of temporal steps when only teammates completing the $i$-th task, and $TS_{i}^{adhoc}$ is the number of temporal steps when the ad hoc agent joins the team to complete the $i$-th task. It can evaluate the efficiency improvement of the ad hoc agent collaboration compared with the existing team.

We evaluate the performance of our ad hoc framework collaborating with different policies of teammates at different times; the results of joining the team at $t_{0}=0$, $t_{0}=50$ $t_{0}=100$ are illustrated in Table I, II and III respectively. We call our method based on IRoT as the IRoT-LLM policy. We calculate the value improvement percentage for the metrics that are better with the higher value ($\%Suc$, $\%PS$), and calculate the decline percentage for the metrics that are better with the lower value ($\#TS$, $\#AS$), and put this value in brackets. The performance shows the influence of different ad hoc methods on the original team when the ad hoc agent joins the team and collaborates with teammates, and from the results we obtain the following findings:

1) Our framework can obviously improve the success rate and efficiency of the original team of different policies compared with no ad hoc agent joining the team. Our method improves $\%Suc$, $\%PS$, $\%SE$ and reduces $\#TS$, $\#AS$ when collaborating with the heuristic-based, learning-based, and LLM-based teams, which demonstrates that our policy can adapt to different policies of teams. The improvements on the original team of our policy differ in collaborating with a team of different policies, and the improvement can be larger when the team policy itself is worse. The performance of different ad hoc policies in $\%Suc$ collaborating with the LLM-based team at different times is also illustrated in Fig. 8(a).

2) Our framework can improve the performance of the original teams of different policies when joining them at different times of $t_{0}=0$, $t_{0}=50$, and $t_{0}=100$. It is noticed in the Fig. 8(a), the later the ad hoc agent joins the original team, the less improvement of the ad hoc agent on this team. This is because teammates have already cooperated with each other and completed more parts of the task when the ad hoc agent joins the team later. The results demonstrate that the ad hoc agent with the proposed policy can still adapt to the team no matter when it joins the team and improve the performance of the team in the subsequent execution of the task.

3) Our framework performs better than other ad hoc policies when collaborating with different teams. The ad hoc policies based on the LLM including Naive-LLM, GPL-LLM, CoT-LLM and IRoT-LLM achieve higher $\%Suc$, $\%PS$ and $\%SE$ as well as lower $\#TS$, $\#AS$ than the learning-based baseline PLASTIC, since LLM-based methods can deal with unseen complex circumstance more flexible, and have better generalization ability. Naive-LLM performs similarly to Heuristic-based policy, which to some extent shows that planning in the embodied environment is more difficult compared with a simple static planning problem. IRoT-LLM performs the best among all LLM-based ad hoc policies, demonstrating the effectiveness of the IRoT method with the communication messages. The performance of GPL-LLM shows that although most of the existing learning methods for ad hoc teamwork are difficult to be directly transferred to our 3D task, combining the key idea with the LLM can still achieve moderate results. When the ad hoc agent joins the original team, it would improve the efficiency of the team but also might bring more energy consumption due to the addition of one more agent. The $\#AS$ of IRoT-LLM when joining the team at $t_{0}=100$ is even smaller than the original team, which demonstrates the proposed method can effectively improve the efficiency of the existing team.

More specifically, we separately compare the results of the proposed policy in three difficulty categories when collaborating with the LLM-based teammates, which is shown in Fig. 8(b). The comparison performance illustrates that the proposed model achieved better improvement after the joining of the ad hoc agent in the Medium tasks since the moderate number of rooms provides a good chance for the newly joined ad hoc agent to effectively help the original team and avoid conflicts with the original team cooperation policies. In Easy tasks, the existing team can already achieve better performance, so the performance improvement is not obvious. In Difficult tasks, the large number of rooms makes it more difficult to find and tidy all misplaced objects, so the improvement of ad hoc teamwork is also limited. Meanwhile, we further compare the influence on the existing LLM-based team and whether the joined IRoT-based ad hoc agent has the manipulation ability, which is illustrated in Fig. 8(c). The proposed framework can adapt to ad hoc agents with different abilities, and the results demonstrate that the ad hoc agent with the manipulation ability can better help the original teams complete the task, obviously improving efficiency.

We show the sample of the ad hoc agent collaborating with the LLM-based original team and compare the collaboration performance of IRoT-LLM ad hoc policy with Naive-LLM policy, which is demonstrated in Fig. 9. The original team contains three heterogeneous agents and completes the task with three misplaced objects at the 207-th step. The ad hoc agent joins the team at the 50-th step, and the actions of teammates are the same as the original team before the ad hoc agent joins the team. The new team with the ad hoc agent completes the task earlier than the original team. The new team with IRoT-LLM ad hoc agent finishes the same task earlier than the Naive-LLM policy, since Naive-LLM generates the executable sub-task without evaluation and reflection when re-planning, which demonstrates the effectiveness of the IRoT method. We also show the failure samples of the dynamic sub-task planner of the ad hoc agent in the Appendix.

We conduct ablation experiments to evaluate the effectiveness of the proposed hierarchical dynamic planner and IRoT-based sub-task planner. The comparison performance of joining the team at different times and collaborating with LLM-based teammates are shown in Table IV. The method w/o STP. removes the dynamic sub-task planner and directly generates the sub-task with the LLM. The performance has decreased obviously compared with our framework, which demonstrates that hierarchical planning is effective in decision-making and can avoid invalid planning. We also discuss the effectiveness of different components of the proposed IRoT methods in the sub-task planning. IRoT w/o Ev. & Rf. does not utilize the evaluation & ranking as well as the interactive reflection steps, and it directly generates $n_{irot}$ candidate sub-task and then chooses to execute them in the order of generation. IRoT w/o Ev. and IRoT w/o Rf. remove the evaluation and reflection steps respectively. The comparison results show that the three steps of the proposed IRoT method are all useful, and each step can provide effective information for sub-task planning. IRoT(GPT-3.5) utilizes the gpt-3.5-turbo-1106 model to complete the planning, and the performance demonstrates that the reasonable planning method is crucial for the LLM to fulfill its remarkable potential in planning and decision-making. The results of the ablation experiments show the effectiveness and efficiency of the proposed hierarchical planning with the IRoT planner.