Jinghong Chen

Sign in Subscribe

Latest

The Six Elements of NLP Experiments. Part 1: Datasets

Experiments are the foundations of Natural Languge Processing (NLP). Just like experimental sciences, we test our proposed methods and verify our theories with experiments. Describing, planning, and conducting experiments are therefore the core skills of every NLP researcher. This article concerns describing NLP experiments. There are six basic elements that

Telling AI from Human: DeepFake Texts in the era of Large Language Models. Tutorial Notes @NAACL 2024

Telling AI from Human: DeepFake Texts in the era of Large Language Models. Tutorial Notes @NAACL 2024

This blog is NOT written or revised by AI. Human can be held responsible for what they write, whereas AI cannot. With capable language models such as ChatGPT, writing irresponsibly has become easier than ever. In the NAACL 2024 Tutorial by Adaku Uchendu et al. titled Catch Me if You

Control-DAG: Constraining Non-Autoregressive Text Generation with Weighted Finite State Automata (WFSA)

Control-DAG: Constraining Non-Autoregressive Text Generation with Weighted Finite State Automata (WFSA)

[4-minute read] TL;DR. Non-autoregressive (NAR) models generate texts much faster than auto-regresssive (AR) models. However, we find previous NAR approaches, largely developed for Machine Translation, fail harshly when faced with Task-Oriented Dialogue and Data-to-Text. Our NAACL 2024 paper introduces Control-DAG, a constrained decoding algorithm that uses Weighted Finite State

PreFLMR: SoTA Open-sourced Multi-modal Knowledge Retriever from Scaling Up FLMR

[1,087 words, 5-minute read] Three products emerged from our study in scaling up multi-modal late-interaction retrievers: * The Multi-task Multi-modal Knowledge Retrieval benchmark (M2KR) totaling 4.4M training examples for training and comprehensively evaluating knowledge retrievers on question-to-doc, image-to-doc, and question+image-to-doc tasks. * The Pretrained Fine-grained Late-interaction Multi-modal Retriever (PreFLMR)

Deriving Speculative Sampling Intuitively

[538 words, 2-minute read] A family of lossless LLM inference acceleration techniques has been developed based on speculative sampling (review here). Proposed by Google and Deepmind, speculative sampling is the following three-step procedure: 1. Draft: a small model (draft model, \(p(\cdot|\text{context})\)) quickly generates a K-token draft. 2.

Retrieval-Augmented Image Synthesis: A Researcher's Guide

[1029 words, 4-minute read] Retrieval-Augmented Image Synthesis (RA-IS) first retrieves relevant images from a database and then generates the image grounding in the retrieved images. RA-IS has been shown to (1) enhance image quality; (2) guide image styles; and (3) help faithfully generate specific objects (e.g., The Oriental Pearl

3-minute Pitch: Retrieval Guided Contrastive Learning for Hateful Memes Detection

[898 words, 3-minute read] Hateful memes are captioned images promoting hostility towards specific social groups. Most hateful memes detection systems are logistic classifiers built on the embedding space of pre-trained visual-langauge model (e.g., CLIP). However, we find that under these embedding spaces, hateful memes and belign memes are located

[Paper Express] Data Selection for Language Models via Importance Resampling (DSIR)

[Paper Express] Data Selection for Language Models via Importance Resampling (DSIR)

README. Data Selection (DS) aims to select a given number of samples from a large, unlabeled dataset for training a capable model in a target domain. In the case of training langauge models, practical DS methods need to efficiently select from raw text corpus containing trillions of tokens. This paper,

Catch up on Speculative Decoding in 5 minutes: a survey for researchers as of December 2023

Speculative decoding speeds up LLM inference without any loss of generation quality. As of December 2023, researchers have reported ~2x speed-up from applying speculative decoding to 3B to 1T models. This survey explains the latest speculative decoding methods that enable lossless speed-up, examines reported experimental results, and suggests future research

Estimate LLM inference speed and VRAM usage quickly: with a Llama-7B case study

You can estimate Time-To-First-Token (TTFT), Time-Per-Output-Token (TPOT), and the VRAM (Video Random Access Memory) needed for Large Language Model (LLM) inference in a few lines of calculation. I will show you how with a real example using Llama-7B. LLM Inference Basics LLM inference consists of two stages: prefill and decode.

MultiLoRA explained in 3 minutes: Democratizing LoRA for Better Multi-Task Learning

Executive Summary: LoRA (Low-Rank Adaptation) fine-tunes a low-rank weight update matrix instead of the whole weight matrix. MultiLoRA modifies LoRA to better learn multiple tasks simultaneously. A MultiLoRA module can be viewed as several LoRA modules connected in parallel and weighted by learnable scaling factors. Finetuning LLaMA with MultiLoRA enhances

Papers with Practical Values for Vision-Language Research @NeurIPS 2023 Day 5.

These 9 papers below offer practical solutions or guidance for vision-language research. I describe each work in 5 sentences. Invited Talk: Systems and Foundation Models (FM). General-purpose FM solves niche problems such as data cleaning better than dedicated algorithms. Christopher Ré shares two directions to make FMs more efficient from

(Vision-Language Researcher) Selected Papers @NeurIPS 2023

Here are some papers that we, who mainly work on vision-language models, think are interesting on Day 4 of NeurIPS 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Tree of Thoughts (ToT) is a decoding scheme for auto-regressive Transformer. A thought is defined as a coherent piece

Efficient Learning Orals @NeurIPS 2023

We will walk through 3 oral presentations on Efficient Learning at NeurIPS 2023 in 2 minutes. Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture Presentor: Dan Fu@Stanford. Paper. Attention and MLP layers scale quadratically in sequence length and model dimension. This work proposes Monarch Mixer, an alternative architecture with sub-quadratic