안녕하세요, lora1379입니다!
저는 NLP, LLM 분야에 관심이 있어서 LLM 관련 서베이 논문을 가볍게 읽어보고 있었습니다.
제가 읽던 서베이 논문은 다음 링크를 참고 부탁드립니다.
논문 링크: A Survey of Large Language Models
A Survey of Large Language Models
Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has
arxiv.org
NLP 분야 초심자로서 이 서베이 논문을 가볍게라도 읽다 보니 LLM 분야에서 읽어보면 좋을 것 같은 연구논문 혹은 자료가 무엇이 있을지 볼 수 있었습니다.
LLM 서베이 논문은 해당 분야의 흐름이나 주된 기술에 대해 소개하는 논문이라 여기 reference 된 논문들은 LLM 분야의 기둥이자 뼈대가 되는 논문이라고 생각했습니다.
물론 아직 일반 NLP도 아니고 LLM 연구논문을 이해하기는 쉽지 않겠지만 추후 LLM을 본격적으로 공부하게 됐을 때 읽어보면 좋을 논문이나 자료를 머릿속으로만 흘려보내기보다 블로그에 기록해두면 어떨까 싶더라구요.
이런 생각으로 본 포스팅을 작성하게 되었습니다!
참고로 본 포스팅에 제시된 리스트는 최초 작성한 이후에도 변동될 수 있습니다.
즉, 최초로 작성하고 나서 추가하면 좋을 논문이나 자료가 더 보이면 이 글을 수정해서 리스트를 업데이트 한다는 의미입니다.
자, 그럼 리스팅 시작하겠습니다.
* 중복이 있을 수 있습니다.
Transformer 등장 이후 LLM 등장하기 바로 전에 있던 PLMs (Pre-trained Language Models)
[Encoder-only PLMs]
- BERT (link) - BERT: pre-training of deep bidirectional transformers for language understanding, 2019
- RoBERTa (link) - Roberta: A robustly optimized bert pretraining approach, 2019
- ALBERT (link) - Albert: A lite bert for self-supervised learning of language representations, 2019
- DeBERTa (link) - Deberta: Decoding-enhanced bert with disentangled attention, 2020
[Decoder-only PLMs]
- GPT-1 (link) - Improving language understanding by generative pre-training, 2018
- GPT-2 (link) - Language models are unsupervised multitask learners, 2019
[Encoder-Decoder PLMs]
- MASS (link) - Mass: Masked sequence to sequence pre-training for language generation, 2019
- BART (link) - Bart: Denoising sequence-tosequence pre-training for natural language generation, translation, and comprehension, 2019
LLMs
GPT 계열
- GPT-3 (link) - Language models are few-shot learners, 2020
- CODEX (link) - Evaluating large language models trained on code, 2021
- WebGPT (link) - Webgpt: Browser-assisted question-answering with human feedback, 2021
- InstructGPT (link) - Training language models to follow instructions with human feedback, 2022
- ChatGPT (link) - Introducing chatgpt (OpenAI Blog, November 2022), 2022
- GPT-4 (link) - Gpt-4 technical report, 2023
LLaMA 계열
- LLaMA (link) - Llama: Open and efficient foundation language models, 2023
- LLaMA-2 (link) - Llama 2: Open foundation and fine-tuned chat models, 2023
- Alpaca (link) - Alpaca: A strong, replicable instruction-following model, 2023
- Koala (link) - Koala: A dialogue model for academic research, 2023
- Mistral-7B (link) - Mistral 7b, 2023
- (이 뒤는 후순위로 고려)
- Code LLaMA (link) - Code llama: Open foundation models for code, 2023
- Gorilla (link) - Gorilla: Large language model connected with massive apis, 2023
- Giraffe (link) - Giraffe: Adventures in expanding context lengths in llms, 2023
- Vigone (link) - Vigogne: French instruction-following and chat models, 2023
- Tulu 65B (link) - How far can camels go? exploring the state of instruction tuning on open resources, 2023
- Long LLaMA (link) - Focused transformer: Contrastive training for context scaling, 2023
- Stable Beluga2 (link) - Stable beluga models
PaLM 계열
- PaLM (link) - Palm: Scaling language modeling with pathways, 2022
- U-PaLM (link) - Transcending scaling laws with 0.1% extra compute, 2022
- Flan-PaLM (link) - Scaling instruction-finetuned language models, 2022
- PaLM-2 (link) - Palm 2 technical report, 2023
- Med-PaLM (link) - Large language models encode clinical knowledge, 2022
- Med-PaLM 2 (link) - Towards expert-level medical question answering with large language models, 2023
기타 (빨갛게 표시한 논문 우선 읽어볼 예정이고 나머지는 미룰 예정)
- T5, 논문에 따라 PLM 취급되기도 함 (link) - Exploring the limits of transfer learning with a unified text-to-text transformer, 2020
- FLAN (link) - Finetuned language models are zero-shot learners, 2022
- Gopher (link) - Scaling language models: Methods, analysis & insights from training gopher, 2021
- T0 (link) - Multitask prompted training enables zero-shot task generalization, 2021
- ERNIE 3.0 (link) - Ernie 3.0: Large-scale knowledge enhanced pretraining for language understanding and generation, 2021
- RETRO (link) - Improving language models by retrieving from trillions of tokens, 2022
- GLaM (link) - Glam: Efficient scaling of language models with mixture-of-experts, 2022
- LaMDA (link) - Lamda: Language models for dialog applications, 2022
- OPT (link) - Opt: Open pre-trained transformer language models, 2022
- Chinchilla (link) - Training compute-optimal large language models, 2022
- Galactica (link) - Galactica: A large language model for science, 2023
- CodeGen (link) - Codegen: An open large language model for code with multi-turn program synthesis, 2022
- AlexaTM (link) - Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model, 2022
- Sparrow (link) - Improving alignment of dialogue agents via targeted human judgements, 2022
- Minerva (link) - Solving quantitative reasoning problems with language models, 2022
- MoD (link) - Unifying language learning paradigms, 2022
- BLOOM (link) - Bloom: A 176b-parameter open-access multilingual language model, 2022
- GLM (link) - Glm-130b: An open bilingual pre-trained model, 2022
- Pythia (link) - Pythia: A suite for analyzing large language models across training and scaling, 2023
- Orca (link) - Orca: Progressive learning from complex explanation traces of gpt-4, 2023
- StarCoder (link) - Starcoder: may the source be with you!, 2023
- KOSMOS (link) - Language is not all you need: Aligning perception with language models, 2023
- Gemini (link) - Gemini: a family of highly capable multimodal models, 2023
GPT 관련 강화학습 (Human alignment)
- Deep RL from human preferences (link) - Deep reinforcement learning from human preferences.
- PPO (link) - Proximal Policy Optimization Algorithms
- Learning to summarize from human feedback (link) - Learning to summarize from human feedback
- RLHF in InstructGPT (link) - Training language models to follow instructions with human feedback
Prompting
- CoT (link) - Chain of thought prompting elicits reasoning in large language models, 2022
Instruction following 관련
- paper (link) - Multitask prompted training enables zero-shot task generalization, 2022
- paper (link) - Training language models to follow instructions with human feedback, 2022
- paper (link) - Finetuned language models are zero-shot learners, 2022
Tool manipulation 관련
- 계산기 활용 LLM (link) - Toolformer: Language models can teach themselves to use tools, 2023
- 검색엔진 활용 LLM (link) - Webgpt: Browser-assisted question-answering with human feedback, 2021
Cost-Effective 테크닉 관련
- ZeRO (link) - Zero: Memory optimizations toward training trillion parameter models, 2020
- RWKW (link) - Rwkv: Reinventing rnns for the transformer era, 2023
- LoRA (link) - Lora: Low-rank adaptation of large language models, 2021
- Distilling step-by-step (link) - Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, 2023
- MiniLLM (link) - MiniLLM: Knowledge Distillation of Large Language Models, 2024
참고한 논문 및 자료
수정사항 (시각은 대략적인 정보입니다)
2024-10-22 07:20 최초 작성
2024-11-26 02:10 GPT 관련 강화학습 (Human alignment) 부분 추가
다 읽기는 쉽지 않겠지만 NLP 기초 공부하고 LLM 연구논문 읽게 되면 fundamental하게 보이는 논문일수록 먼저 읽어보려 합니다.
LLM을 다룬다면 효율이나 경량화 관련해서 관심 있다 보니 아마 fundamental한 모델이나 방법론에 대한 논문을 읽고 나면 마이너한 모델에 대한 논문보다는 비용 효율 관련된 논문(LoRA 등)으로 넘어갈 것 같네요.
수정사항 있으면 그때그때 수정해보겠습니다.
'Large Language Model' 카테고리의 다른 글
| [읽어보기] A Survey of Large Language Models - Overview (2) (2) | 2024.11.26 |
|---|---|
| [읽어보기] A Survey of Large Language Models - Overview (1) (12) | 2024.10.22 |
| [읽어보기] A Survey of Large Language Models - Introduction (0) | 2024.08.01 |
| [읽어보기] A Survey of Large Language Models - Abstract (0) | 2024.07.30 |