magnum-72b-v1

This is the first i a series of models desiged to replicate the prose quality of the Claude 3 models, specifically Soet ad Opus. This model is fie-tued o top of Qwe-2 72B Istruct.

Promptig

Model has bee Istruct tued with the ChatML formattig. A typical iput would look like this:

"""<|im_start|>user
Hi there!<|im_ed|>
<|im_start|>assistat
Nice to meet you!<|im_ed|>
<|im_start|>user
Ca I ask a questio?<|im_ed|>
<|im_start|>assistat
"""

Credits

This model has bee a team effort, credits go to:

• Sao10K for help with (ad cleaig up!) the dataset.
• alpidale for the traiig.
• kalomaze for helpig with the hyperparameter tuig.
• Various other people for their cotiued help as we tued the parameters, restarted failed rus. I o particular order: Doctor Shotgu, Lucy, Nopm, Mago, ad the rest of the Silly Tilly.

Ad last but ot least, we'd like to thak Kearm for sposorig the compute eeded to trai this model.

Traiig

The traiig was doe with 55 millio tokes of high-quality RP data, over 1.5 epochs. We used 8x AMD Istict™ MI300X Accelerators for the full-parameter fie-tuig of the model.

Safety

…

This is the first in a series of models designed to replicate the prose quality of the Claude 3 mode