Skip to main content

Quickstart

Local Installation​

Cortex has a Local Installer with all of the required dependencies, so that once you've downloaded it, no internet connection is required during the installation process.

Start a Cortex Server​

This command starts the Cortex's' API server at localhost:39281.


cortex start

Pull Models​

This command allows users to download a model from these Model Hubs:

It displays available quantizations, recommends a default and downloads the desired quantization.

The following two options will show you all of the available models under those names. Cortex will first search on its own hub for models like llama3.3, and in huggingface for hyper specific ones like bartowski/Meta-Llama-3.1-8B-Instruct-GGU.


cortex pull llama3.3

or,


cortex pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

Run a Model​

This command downloads the default gguf model (if not available in your file system) from the Cortex Hub, starts the model, and chat with the model.


cortex run llama3.3

info

All model files are stored in the ~/cortex/models folder.

Using the Model​

API​


curl http://localhost:39281/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1:8b-gguf",
"messages": [
{
"role": "user",
"content": "Hello"
},
],
"stream": true,
"max_tokens": 1,
"stop": [
null
],
"frequency_penalty": 1,
"presence_penalty": 1,
"temperature": 1,
"top_p": 1
}'

Refer to our API documentation for more details.

Show the System State​

This command displays the running model and the hardware system status (RAM, Engine, VRAM, Uptime).


cortex ps

Stop a Model​

This command stops the running model.


cortex models stop llama3.3

Stop a Cortex Server​

This command stops the Cortex.cpp API server at localhost:39281 or whichever other port you used to start cortex.


cortex stop

What's Next?​

Now that Cortex is set up, you can continue on to any of the following sections:

  • Adjust the folder path and configuration using the .cortexrc file.
  • Explore the Cortex's data folder to understand how data gets stored.
  • Learn about the structure of the model.yaml file in Cortex.