my work

[1,087 words, 5-minute read] Three products emerged from our study in scaling up multi-modal late-interaction retrievers: * The Multi-task Multi-modal Knowledge Retrieval benchmark (M2KR) totaling 4.4M training examples for training and comprehensively evaluating knowledge retrievers on question-to-doc, image-to-doc, and question+image-to-doc tasks. * The Pretrained Fine-grained Late-interaction Multi-modal Retriever (PreFLMR)

[898 words, 3-minute read] Hateful memes are captioned images promoting hostility towards specific social groups. Most hateful memes detection systems are logistic classifiers built on the embedding space of pre-trained visual-langauge model (e.g., CLIP). However, we find that under these embedding spaces, hateful memes and belign memes are located

Knowledge-based Visual Question Answering (KBVQA) aims to answer a question related to an image that requires some world knowledge. Here's an example. Our NeurIPS paper takes the retrieval-augmented approach to tackles KBVQA. We first retrieve relevant documents from an external database and then generate answers based on the

Minimum Bayes Risk (MBR) decoding generally outperforms temperature sampling and beam search. But it is expensive computationally. We can train the model on the MBR decoding outputs so that cheaper decoding methods perform on par with MBR. Google calls it "MBR fine-tuning". Our recent work introduces a more