Modelpublic

INC4AI/gpt-j-6b-sparse

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

1likes34downloads

Model Card

Sparse GPT-J 6B

Model Description

The sparse version of GPT-J 6B is a pruned variant derived from the original GPT-J 6B model and the vast majority of linear layers maintain a 40% unstructured sparsity (except for the 'lm_head').

Hyperparameter	Value
\\(n_{parameters}\\)	6053381344
\\(n_{layers}\\)	28&ast;
\\(d_{model}\\)	4096
\\(d_{ff}\\)	16384
\\(n_{heads}\\)	16
\\(d_{head}\\)	256
\\(n_{ctx}\\)	2048
\\(n_{vocab}\\)	50257/50400&dagger; (same tokenizer as GPT-2/3)
Positional Encoding	Rotary Position Embedding RoPE
RoPE Dimensions	64

<figcaption>&ast; Each layer consists of one feedforward block and one self attention block. &dagger; Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer.</figcaption></figure>

The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model dimension is split into 16 heads, each with a dimension of 256. Rotary Position Embedding (RoPE) is applied to 64 dimensions of each head. The model is trained with a tokenization vocabulary of 50257, using the same set of BPEs as GPT-2/GPT-3.

Evaluation results

Evaluating the accuracy of the sparse model of gpt-j-6b using the lambadaopenai dataset in lmeval, providing the accuracy fluctuation under two precisions: FP32 and BF16. <figure>

Sparsity	Dataset	Precision	Dense Acc ↑	Sparse Acc ↑	Acc fluctuations
40%	Lambada_openai	FP32	0.6831	0.6922	+1.33%
40%	Lambada_openai	BF16	0.6771	0.6874	+0.63%