Phi-3-medium-128k-instruct-GGUF

Llamacpp imatrix Quatizatios of Phi-3-medium-128k-istruct

Usig llama.cpp pull request 7225 for quatizatio.

Origial model: https://huggigface.co/microsoft/Phi-3-medium-128k-istruct

All quats made usig imatrix optio with dataset from here

Prompt format

<|user|> {prompt}<|ed|><|assistat|><|ed|>

Dowload a file (ot the whole brach) from below:

Dowloadig usig huggigface-cli

First, make sure you have huggiface-cli istalled:

pip istall -U "huggigface_hub[cli]"

The, you ca target the specific file you wat:

huggigface-cli dowload bartowski/Phi-3-medium-128k-istruct-GGUF --iclude "Phi-3-medium-128k-istruct-Q4_K_M.gguf" --local-dir ./

If the model is bigger tha 50GB, it will have bee split ito multiple files. I order to dowload them all to a local folder, ru:

huggigface-cli dowload bartowski/Phi-3-medium-128k-istruct-GGUF --iclude "Phi-3-medium-128k-istruct-Q8_0.gguf/*" --local-dir Phi-3-medium-128k-istruct-Q8_0

You ca either specify a ew local-dir (Phi-3-medium-128k-istruct-Q8_0) or dowload them all i place (./)

Which file should I choose?

A great write up with charts showig various performaces is provided by Artefact2 here

The first thig to figure out is how big a model you ca ru. To do this, you'll eed to figure out how much RAM ad/or VRAM you have.

If you wat your model ruig as FAST as possible, you'll wat to fit the whole thig o your GPU's VRAM. Aim for a quat with a file size 1-2GB smaller tha your GPU's total VRAM.

If you wat the absolute maximum quality, add both your system RAM ad your GPU's VRAM together, the similarly grab a quat with a file size 1-2GB Smaller tha that total.

Next, you'll eed to decide if you wat to use a 'I-quat' or a 'K-quat'.

If you do't wat to thik too much, grab oe of the K-quats. These are i format 'QXKX', like Q5KM.

If you wat to get more ito the weeds, you ca check out this extremely useful feature chart:

llama.cpp feature matrix

But basically, if you're aimig for below Q4, ad you're ruig cuBLAS (Nvidia) or rocBLAS (AMD), you should look towards the I-quats. These are i format IQXX, like IQ3M. These are ewer ad offer better performace for their size.

These I-quats ca also be used o CPU ad Apple Metal, but will be slower tha their K-quat equivalet, so speed vs performace is a tradeoff you'll have to decide.

The I-quats are ot compatible with Vulca, which is also AMD, so if you have a AMD card double check if you're usig the rocBLAS build or the Vulca build. At the time of writig this, LM Studio has a preview with ROCm support, ad other iferece egies have specific builds for ROCm.

Wat to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Llamacpp imatrix Quantizations of Phi-3-medium-128k-instruct Using llama.cpp pull request 7225 for q