Hi @hiyouga

After attempting to integrate AutoRound, I’ve encountered several issues, and one in particular that I’d like to discuss with you first.

Quantization Order: I’ve noticed challenges related to quantization with GPTQ on multiple GPU devices. Specifically, the torch parallelization seems to distribute the quantization process across different devices, which doesn’t appear optimal. Additionally, I’ve encountered exceptions after the quantization process."ValueError: You can't train a model that has been loaded with device_map='auto' in any distributed mode. Please rerun your script specifying --num_processes=1 or by launching with python {{myscript.py}}."

To avoid conflicts between settings for models during quantization and fine-tuning, I’ve opted to quantize the model first, followed by fine-tuning. However, I believe there’s room for improvement in the code.

Which approach do you prefer? Would you rather merge the quantization process into the model loading phase, or is the naive method I proposed acceptable? I would appreciate your input.

Thank you!

Hi @hiyouga， could you kindly take a look? Once I have your feedback, I can continue refining my code. Thank you!