The explicit modeling of the input modality is typically required for deep learning inference. For instance, by encoding picture patches into vectors, Vision Transformers (ViTs) directly model the 2D spatial organization of images. Similarly, calculating spectral characteristics (like MFCCs) to transmit into a network is frequently involved in audio inference. A user must first decode a file into a modality-specific representation (such as an RGB tensor or MFCCs) before making an inference on a file that is saved on a disc (such as a JPEG image file or an MP3 audio file), as shown in Figure 1a. There are two real downsides to decoding inputs into a modality-specific representation.

It first involves manually creating an input representation and a model stem for each input modality. Recent projects like PerceiverIO and UnifiedIO have demonstrated the versatility of Transformer backbones. These techniques still need modality-specific input preprocessing, though. For instance, before sending picture files into the network, PerceiverIO decodes them into tensors. Other input modalities are transformed into various forms by PerceiverIO. They postulate that executing inference directly on file bytes makes it feasible to eliminate all modality-specific input preprocessing. The exposure of the material being analyzed is the second disadvantage of decoding inputs into a modality-specific representation.

Think of a smart home gadget that uses RGB photos to conduct inference. The user’s privacy may be jeopardized if an enemy gains access to this model input. They contend that deduction can instead be carried out on inputs that protect privacy. They make notice that numerous input modalities share the ability to be saved as file bytes to solve these shortcomings. As a result, they feed file bytes into their model at inference time (Figure 1b) without doing any decoding. Given their capability to handle a range of modalities and variable-length inputs, they adopt a modified Transformer architecture for their model.

Different people tend to have unique needs and preferences—particularly when it comes to cleaning or tidying up. Home robots, especially robots designed to help humans with house chores, should ideally be able to complete tasks in ways that account for these individual preferences.

Researchers at Princeton University and Stanford University recently set out to personalize the assistance offered by home robots using large language models (LLMs), a class of artificial intelligence models that are becoming increasingly popular after the release of ChatGPT. Their approach, presented in a paper pre-published on arXiv, was initially tested on a mobile robot called TidyBot engineered to tidy up indoor environments.

“For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios,” Jimmy Wu, Rika Antonova and their colleagues wrote in their paper. “In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away.”