In-Context Abstraction Learning (ICAL) aims at automating the acquisition of generalizable examples and knowledge for in-context agents. ICAL operates by receiving a language instruction I with a noisy trajectory of observations and actions, denoted ξ_noisy = {o₀, a₀, …, o_T, a_T} in a new task domain D.

A new domain D represents changes in task variables not captured in VLM pretraining, such as a different environment (e.g., kitchen #1 vs. kitchen #2), task (e.g., "add the cheapest red bike to my wish list"), or user preference (e.g., "I prefer the red cup for coffee"). The core aim of is to abstract each noisy trajectory into a single example e, which then forms part of a memory set M. Each example e ∈ M represents an optimized trajectory ξ_optimized with generalizable language abstractions L. The objective is to ensure that M collectively encapsulates examples that, when used in a VLMs context window, increase the likelihood of successful task execution in the new domain, while also containing knowledge that is transferable across similar tasks and contexts. This can be encapsulated as:

\[ \max_{M} \mathbb{E}\left[ \sum_{t=0}^T r_t(o_t, a_t) \,\Big|\, a_t \sim \pi(a \mid h_t, M, I), D \right] \]

where:

• \( M \) represents the memory or model parameters that influence the policy \(\pi\).
• \( \mathbb{E}[\cdot] \) denotes the expectation over the random variables involved in the decision-making process.
• \( r_t(o_t, a_t) \) is the reward function at time \( t \), dependent on the observation \( o_t \) and action \( a_t \).
• \( a_t \sim \pi(a \mid h_t, M, I) \) indicates that the action \( a_t \) is sampled from a policy \(\pi\), conditioned on the history \( h_t \), memory \( M \), and additional input \( I \).
• \( h_t = \{o_0, a_0, \ldots, o_{t-1}, a_{t-1}, o_t\} \) is the history up to time \( t \).
• \( D \) represents the new domain or task to learn.
• \( T \) is the time horizon over which the reward is summed.

1. Task and Causal Abstractions: Essential principles and actions are identified to achieve goals, explaining how elements interconnect through cause and effect.
2. State Changes: Crucial for decision-making, predicted state changes during a demonstration are annotated, showing the dynamic environment, such as a bowl becoming clean.
3. Task Decomposition and Subgoals: Tasks are broken down into steps and subgoals with natural language annotations detailing each step and summarizing actions to facilitate understanding.
4. State Abstraction: Focuses on selecting only relevant state variables interacted with during a demonstration, and suggesting additional relevant state variables, avoiding overwhelming detail and enhancing performance.

ICAL processes noisy/suboptimal/incorrect trajectories, coming from a human or agent policy, in two phases:

(1) the abstraction phase where a VLM is prompted to correct action errors and generate initial programs of thought: \[ F_{abstract}: (\xi_{noisy}, I, \{e^1, \ldots, e^k\}) \rightarrow (\xi_{optimized}, L) \] (2) the human-in-the-loop phase, where the actions and programs of thought are refined guided by human feedback \(H(a_t, o_t)\). The model update can be represented as \(\Xi_{\text{update}}\): \[ \Xi_{update}(\xi_{optimized}, H(a_t, o_t), L, I, \{e^1, ..., e^k\}) \rightarrow \xi'_{optimized}, L' \] Successful trajectories are stored in a repository to assist the agent in learning and responding to new instructions and environments.

After the ICAL examples have been learned, ICAL uses retrieval-augmented generation or supervised fine-tuning to teach the agent to perform the new tasks.

Given the learned example set M and a new instruction \(I\), we prompt the VLM to carry out the instruction by producing action sequences \(\{a_0, ..., a_T\} \in A\) from an action API that describes the skills set \(A\) (e.g., go_to(X), pickup(X)), by retrieving the top \(K\) examples from M to include in the prompt based on their textual and visual similarity with the current scene.

The aggregated similarity score \(s\) for each example \(e\) reads:

\(s = \lambda_{I} \cdot s^I + \lambda_{\text{textual}} \cdot s^{\text{textual}} + \lambda_{\text{visual}} \cdot s^{\text{visual}}\)

where \(s^I\), \(s^{\text{textual}}\), and \(s^{\text{visual}}\) are the similarity scores for the input text instruction, textual state, and visual state, respectively, computed via cosine similarity using embeddings from OpenAI's text-embedding-ada-002 model and CLIP ViT-B/32 model. The coefficients \(\lambda_{I}\), \(\lambda_{\text{textual}}\), and \(\lambda_{\text{visual}}\) are weighting hyperparameters chosen in each domain by a held out validation set.

The VLM prompt contains the new instruction \(I\), the current webpage image for VisualWebArena or 12 video frames for Ego4D annotated with Set-of-Marks, a textual state description \(x_t\) describing the objects and their attributes for embodied agents and HTML elements for web agents, the action API \(A\), and the retrieved set of in-context examples \(\{e^1,...,e^k\} \in \textit{M}\).

Supervised fine-tuning trains the multimodal LLM, Qwen2-VL, to directly predict the ICAL programs of thought and corresponding action sequences for new tasks. The training dataset pairs task instructions \(I\), multimodal inputs \(S\) (e.g., visual frames, textual descriptions), and ground truth outputs, including intermediate reasoning steps and final actions.

The model learns to generate structured ICAL outputs and actions end-to-end, enabling it to perform tasks without relying on retrieval from learned examples.