Llama-3-Instruct-8B-SPPO-Iter3-GGUF

Llamacpp imatrix Quatizatios of Llama-3-Istruct-8B-SPPO-Iter3

Usig llama.cpp release b3197 for quatizatio.

Origial model: https://huggigface.co/UCLA-AGI/Llama-3-Istruct-8B-SPPO-Iter3

All quats made usig imatrix optio with dataset from here

Prompt format

<|begi_of_text|><|start_header_id|>system<|ed_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|ed_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistat<|ed_header_id|>

Dowload a file (ot the whole brach) from below:

Dowloadig usig huggigface-cli

First, make sure you have huggiface-cli istalled:

pip istall -U "huggigface_hub[cli]"

The, you ca target the specific file you wat:

huggigface-cli dowload bartowski/Llama-3-Istruct-8B-SPPO-Iter3-GGUF --iclude "Llama-3-Istruct-8B-SPPO-Iter3-Q4_K_M.gguf" --local-dir ./

If the model is bigger tha 50GB, it will have bee split ito multiple files. I order to dowload them all to a local folder, ru:

huggigface-cli dowload bartowski/Llama-3-Istruct-8B-SPPO-Iter3-GGUF --iclude "Llama-3-Istruct-8B-SPPO-Iter3-Q8_0.gguf/*" --local-dir Llama-3-Istruct-8B-SPPO-Iter3-Q8_0

You ca either specify a ew local-dir (Llama-3-Istruct-8B-SPPO-Iter3-Q8_0) or dowload them all i place (./)

Which file should I choose?

A great write up with charts showig various performaces is provided by Artefact2 here

The first thig to figure out is how big a model you ca ru. To do this, you'll eed to figure out how much RAM ad/or VRAM you have.

If you wat your model ruig as FAST as possible, you'll wat to fit the whole thig o your GPU's VRAM. Aim for a quat with a file size 1-2GB smaller tha your GPU's total VRAM.

If you wat the absolute maximum quality, add both your system RAM ad your GPU's VRAM together, the similarly grab a quat with a file size 1-2GB Smaller tha that total.

Next, you'll eed to decide if you wat to use a 'I-quat' or a 'K-quat'.

If you do't wat to thik too much, grab oe of the K-quats. These are i format 'QXKX', like Q5KM.

If you wat to get more ito the weeds, you ca check out this extremely useful feature chart:

llama.cpp feature matrix

But basically, if you're aimig for below Q4, ad you're ruig cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quats. These are i format IQXX, like IQ3M. These are ewer ad offer better performace for their size.

These I-quats ca also be used o CPU ad Apple Metal, but will be slower tha their K-quat equivalet, so speed vs performace is a tradeoff you'll have to decide.

The I-quats are ot compatible with Vulca, which is also AMD, so if you have a AMD card double check if you're usig the rocBLAS build or the Vulca build. At the time of writig this, LM Studio has a preview with ROCm support, ad other iferece egies have specific builds for ROCm.

Wat to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Llamacpp imatrix Quantizations of Llama-3-Instruct-8B-SPPO-Iter3 Using llama.cpp release b3197 for q