Parameter Efficient Fine Tuning

GPT2 on GLUE

  • GLUE

    • The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems.

  • GPT2

    • GPT-2 is a Hugging Face transformers model pretrained on a very large corpus of English data in a self-supervised fashion.

  • Parameter Efficient Fine Tuning

    • PEFT approaches enable you to get performance comparable to full fine-tuning while only having a small number of trainable parameters.

  • Over 5 experiments with different random initializations, adding dendrites to the LoRA layers of Hugging Face’s PEFT library achieved 16% error reduction than just training LoRA layers alone.