Code Llama

Code Llama is a collectio of pretraied ad fie-tued geerative text models ragig i scale from 7 billio to 34 billio parameters. This is the repository for the base 13B versio i the Huggig Face Trasformers format. This model is desiged for geeral code sythesis ad uderstadig. Liks to other models ca be foud i the idex at the bottom.

Model Use

Model capabilities:

• [x] Code completio.
• [x] Ifillig.
• [ ] Istructios / chat.
• [ ] Pytho specialist.

import torch
from modelscope import Model, AutoTokeizer


model = Model.from_pretraied("AI-ModelScope/CodeLlama-13b-hf", revisio='v1.0.1', device_map='cuda:0', torch_dtype=torch.float16)
tokeizer = AutoTokeizer.from_pretraied("AI-ModelScope/CodeLlama-13b-hf", revisio='v1.0.1')

prompt = 'import socket\\def pig_expoetial_backoff(host: str):'
iputs = tokeizer(prompt, paddig=False, add_special_tokes=False, retur_tesors="pt")

# Geerate
geerate_ids = model.geerate(
    iputs.iput_ids.to(model.device), 
    attetio_mask=iputs['attetio_mask'].to(model.device), 
    do_sample=True,
    top_k=10,
    temperature=0.1,
    top_p=0.95,
    um_retur_sequeces=1,
    eos_toke_id=tokeizer.eos_toke_id,
    max_legth=200)
prit(tokeizer.batch_decode(geerate_ids, skip_special_tokes=True, clea_up_tokeizatio_spaces=False)[0])

Model Details

*Note: Use of this model is govered by the Meta licese. Meta developed ad publicly released the Code Llama family of large laguage models (LLMs).

Model Developers Meta

Variatios Code Llama comes i three model sizes, ad three variats:

• Code Llama: base models desiged for geeral code sythesis ad uderstadig
• Code Llama - Pytho: desiged specifically for Pytho
• Code Llama - Istruct: for istructio followig ad safer deploymet

All variats are available i sizes of 7B, 13B ad 34B parameters.

This repository cotais the base versio of the 13B parameters model.

Iput Models iput text oly.

Output Models geerate text oly.

Model Architecture Code Llama is a auto-regressive laguage model that uses a optimized trasformer architecture.

Model Dates Code Llama ad its variats have bee traied betwee Jauary 2023 ad July 2023.

Status This is a static model traied o a offlie dataset. Future versios of Code Llama - Istruct will be released as we improve model safety with commuity feedback.

Licese A custom commercial licese is available at: https://ai.meta.com/resources/models-ad-libraries/llama-dowloads/

Research Paper More iformatio ca be foud i the paper "Code Llama: Ope Foudatio Models for Code".

Iteded Use

Iteded Use Cases Code Llama ad its variats is iteded for commercial ad research use i Eglish ad relevat programmig laguages. The base model Code Llama ca be adapted for a variety of code sythesis ad uderstadig tasks, Code Llama - Pytho is desiged specifically to hadle the Pytho programmig laguage, ad Code Llama - Istruct is iteded to be safer to use for code assistat ad geeratio applicatios.

Out-of-Scope Uses Use i ay maer that violates applicable laws or regulatios (icludig trade compliace laws). Use i laguages other tha Eglish. Use i ay other way that is prohibited by the Acceptable Use Policy ad Licesig Agreemet for Code Llama ad its variats.

Hardware ad Software

Traiig Factors We used custom traiig libraries. The traiig ad fie-tuig of the released models have bee performed Meta’s Research Super Cluster.

Carbo Footprit I aggregate, traiig all 9 Code Llama models required 400K GPU hours of computatio o hardware of type A100-80GB (TDP of 350-400W). Estimated total emissios were 65.3 tCO2eq, 100% of which were offset by Meta’s sustaiability program.

Traiig Data

All experimets reported here ad the released models have bee traied ad fie-tued usig the same data as Llama 2 with differet weights (see Sectio 2 ad Table 1 i the research paper for details).

Evaluatio Results

See evaluatios for the mai models ad detailed ablatios i Sectio 3 ad safety evaluatios i Sectio 4 of the research paper.

Ethical Cosideratios ad Limitatios

Code Llama ad its variats are a ew techology that carries risks with use. Testig coducted to date has bee i Eglish, ad has ot covered, or could it cover all scearios. For these reasos, as with all LLMs, Code Llama’s potetial outputs caot be predicted i advace, ad the model may i some istaces produce iaccurate or objectioable resposes to user prompts. Therefore, before deployig ay applicatios of Code Llama, developers should perform safety testig ad tuig tailored to their specific applicatios of the model.

Please see the Resposible Use Guide available available at https://ai.meta.com/llama/resposible-user-guide.

Code Llama Code Llama is a collection of pretrained and fine-tuned generative text models ranging in