Knowledge Distillation of Black-Box Large Language Models

Authors: Hongzhan Chen, Runjun Chen, Yuqi Yi, Xiaojun Quan (Sun Yat-sen University), Chenliang Li, Ming Yan, Ji Zhang (Alibaba Group)

Abstract

Proposes Proxy-KD, which uses an intermediate proxy model to transfer knowledge from proprietary black-box LLMs (like GPT-4) to smaller student models. The proxy first aligns with the teacher using the teacher's outputs, then preference optimization (DPO) further refines alignment. During KD, the proxy generates a dense distribution approximating the teacher's, with a sample-level weight reflecting alignment quality. The authors claim Proxy-KD outperforms both black-box and white-box KD methods.

Method

Problem Setup

• Black-box teacher π_t (GPT-4), proxy π_p, student π_s
• Dataset split: 10% warm-up (𝒟_w), 45% proxy alignment (𝒟_p), 45% student distillation (𝒟_s)

Proxy Alignment

1. Supervised fine-tuning on teacher outputs via NLL loss
2. DPO preference optimization: for each input, teacher response y and proxy response ŷ form preference pair. DPO iteratively adjusts proxy to prefer teacher responses. Runs for k=16 iterations.

Knowledge Distillation

• Hard-label: NLL on teacher-generated sequences
• Soft-label (weighted KL): Uses aligned proxy's output distributions
• Sample-level weight: w(x,y) = σ[(log π_p(y|x) − μ) / γ] — downweights samples where proxy disagrees with teacher
• Overall student loss: ℒ_Student = ℒ_Student-NLL + α·ℒ_Weight-KL (α=100)

Experiments

• Teacher: GPT-4. Proxy: Llama-2-70B / Llama-2-13B. Student: Llama-1-7B / Llama-2-7B
• Training data: OpenOrca + Nectar (1M GPT-4 outputs) + synthetic data
• Benchmarks: BBH, AGIEval, ARC-challenge, MMLU, CSQA, GSM8K
• 8×A100 80GB GPUs, LoRA training

Key Results

• Proxy-KD (Llama-2-7B student): 56.78% average vs. vanilla black-box KD 53.66%, white-box KD 51.27%
• Strong on BBH (53.40) and GSM8K (53.07), outperforming larger models
• Removing the proxy drops average 3.12 points; removing DPO alignment drops further
• Proxy-KD outperforms white-box KD even when the aligned proxy is used as the teacher directly — suggesting closed-source teacher capabilities are more beneficial than open-source, even after alignment