Nous-Hermes-2-Visio - Mistral 7B

I the tapestry of Greek mythology, Hermes reigs as the eloquet Messeger of the Gods, a deity who deftly bridges the realms through the art of commuicatio. It is i homage to this divie mediator that I ame this advaced LLM "Hermes," a system crafted to avigate the complex itricacies of huma discourse with celestial fiesse.

Model descriptio

Nous-Hermes-2-Visio stads as a pioeerig Visio-Laguage Model, leveragig advacemets from the reowed OpeHermes-2.5-Mistral-7B by tekium. This model icorporates two pivotal ehacemets, settig it apart as a cuttig-edge solutio:

• SigLIP-400M Itegratio: Divergig from traditioal approaches that rely o substatial 3B visio ecoders, Nous-Hermes-2-Visio haresses the formidable SigLIP-400M. This strategic choice ot oly streamlies the model's architecture, makig it more lightweight, but also capitalizes o SigLIP's remarkable capabilities. The result? A remarkable boost i performace that defies covetioal expectatios.

• Custom Dataset Eriched with Fuctio Callig: Our model's traiig data icludes a uique feature – fuctio callig. This distictive additio trasforms Nous-Hermes-2-Visio ito a Visio-Laguage Actio Model. Developers ow have a versatile tool at their disposal, primed for craftig a myriad of igeious automatios.

This project is led by qguye3 ad tekium.

Traiig

Dataset

• 220K from LVIS-INSTRUCT4V
• 60K from ShareGPT4V
• 150K Private Fuctio Callig Data
• 50K coversatios from tekium's OpeHermes-2.5

Usage

Prompt Format

• Like other LLaVA's variats, this model uses Vicua-V1 as its prompt template. Please refer to cov_llava_v1 i this file
• For Gradio UI, please visit this GitHub Repo

Fuctio Callig

• For fuctiog callig, the message should start with a <f_call> tag. Here is a example:

<f_call>{
  "type": "object",
  "properties": {
    "bus_colors": {
      "type": "array",
      "descriptio": "The colors of the bus i the image.",
      "items": {
        "type": "strig",
        "eum": ["red", "blue", "gree", "white"]
      }
    },
    "bus_features": {
      "type": "strig",
      "descriptio": "The features see o the back of the bus."
    },
    "bus_locatio": {
      "type": "strig",
      "descriptio": "The locatio of the bus (drivig or pulled off to the side).",
      "eum": ["drivig", "pulled off to the side"]
    }
  }
}

Output:

{
  "bus_colors": ["red", "white"],
  "bus_features": "A advertisemet",
  "bus_locatio": "drivig"
}

Example

Chat

Fuctio Callig

Iput image:

Iput message:

<f_call>{
    "type": "object",
    "properties": {
      "food_list": {
        "type": "array",
        "descriptio": "List of all the food",
        "items": {
          "type": "strig",
        }
      },
    }
}

Output:

{
    "food_list": [
        "Double Burger",
        "Cheeseburger",
        "Frech Fries",
        "Shakes",
        "Coffee"
    ]
}

Nous-Hermes-2-Vision - Mistral 7B In the tapestry of Greek mythology, Hermes reigns as the eloquent