> ## Content Index
> Fetch the complete content index at: https://blog.meganova.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# Choosing the Right AI Model Setup with LibraOS: BYOK, Local & Hosted
- URL: https://blog.meganova.ai/choosing-the-right-ai-model-setup-with-libraos-byok-local-hosted/
- Published: 2026-11-02T10:15:00.000Z
- Updated: 2026-11-02T10:15:00.000Z
- Author: Tracy Giang
- Tags: LibraOS

Every AI app has to answer one question early: where does the model run, and who provides it? The answer decides where your prompts go, who bills you for inference, and how much infrastructure you operate.

With LibraOS, the model is a choice you make separately from the rest of the stack. Your app handles the experience, the LibraOS Runtime handles the execution (agents, memory, knowledge and tools), and the model endpoint handles the inference. You can use the Runtime with your own model provider, a local model server, or hosted inference. This guide compares the three setups and shows how to choose.

## **The three setups**

| Setup                      | How it works                                                                                         | What you keep                                                                       | Good when                                                                       |
| -------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Bring your own key         | Connect an OpenAI-compatible endpoint with your own credentials                                      | Your provider relationship; you choose models and manage billing with your provider | You already trust a provider and its terms                                      |
| Keep inference local       | Connect a compatible model server on your machine or private network, alongside your LibraOS Runtime | Control of model and data placement                                                 | Prompts and data must stay inside your network, or you need an air-gapped setup |
| Hosted inference (Gateway) | Use MegaNova's OpenAI-compatible model APIs                                                          | Less to operate; models served for you                                              | You'd rather not run inference yourself                                         |

Two things to know about the hosted option. Gateway is MegaNova's model API, and its account, model access, usage billing and limits are separate from LibraOS Cloud and Runtime licensing. Check [model availability and pricing](https://docs.meganova.ai/inference-models/model-list?ref=blog.meganova.ai) in the Gateway documentation before you plan around a specific model.

Because the model sits behind its own endpoint, the first choice doesn't lock you in. You can start with a provider key during development, for example, and move to local inference when a customer requires it.

## **How to choose**

Work through these questions in order. The first one that gives a hard answer usually decides the setup.

| Question                                                                         | If the answer is yes                                                                                         |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Must prompts and data stay inside our network, or is the environment air-gapped? | Keep inference local                                                                                         |
| Does a customer or regulator require a specific placement for model traffic?     | Local, or a provider whose placement you've confirmed                                                        |
| Do we already have a provider we trust, with terms we've reviewed?               | Bring your own key                                                                                           |
| Do we lack the hardware or the team to run models ourselves?                     | Hosted inference (Gateway) or your own provider                                                              |
| Do we have specific latency or cost targets?                                     | Compare options against your workload; for Gateway, review serving, capacity, latency and cost with MegaNova |

Whichever you pick, read the data terms of whoever runs the model: whether prompts are stored, for how long, and whether they're used for training. For local inference, select models that fit your hardware and validate network access and connectors.

## **Choose two models, not one**

An app built on LibraOS usually needs two kinds of model:

- **A chat model** writes the responses your users see.
- **An embedding model** turns documents and questions into vectors, which is how knowledge retrieval finds the right passages.

With your own key, you configure both chat and embedding endpoints. Choose each for your workload rather than defaulting to the largest model available.

Then test compatibility with your real workflow before committing. Your chosen model powers the responses; LibraOS connects those responses to persistent context, knowledge and actions. A model that writes well can still struggle with the tool calls or structured output your app depends on, so run your actual tasks against it.

## **Start with the model card**

If you run open-weight models, read the model card before choosing weights. Check the base model, the publisher, the license, any modifications, and the deployment requirements. LibraOS publishes its model repositories on [Hugging Face](https://huggingface.co/libraos?ref=blog.meganova.ai).

The labels on a model tell you what was done to it:

| Label      | What it means                                                  | What to check                                              |
| ---------- | -------------------------------------------------------------- | ---------------------------------------------------------- |
| Fine-tuned | Additional training adapted the model                          | Who performed it, the intended use, and evaluation results |
| Quantized  | Reduced precision changes the memory and hardware trade-offs   | The quantization method and its author                     |
| Mirror     | A hosted copy that may preserve the original weights unchanged | That it's a copy, not a new fine-tune                      |

The license matters as much as the benchmark. A model you can't use commercially, or whose terms don't fit your customer's contract, isn't an option however well it performs.

## **Checklist before you commit**

- ☐ **Placement.** You know where prompts and data go for each setup you're considering.
- ☐ **Terms.** You've read the provider's data terms, or the Gateway terms if you use hosted inference.
- ☐ **Billing.** You know who bills you: your provider for your own key, MegaNova for Gateway, and your own infrastructure costs for local.
- ☐ **Two models.** A chat model and an embedding model, each chosen for the workload.
- ☐ **Model card.** For open weights: license, modifications and deployment requirements checked.
- ☐ **Hardware.** For local inference: models fit your hardware, and network access and connectors are validated.
- ☐ **Compatibility.** You've run your real tasks, including tool calls and structured output, against the model.

## **Conclusion**

LibraOS keeps the model choice separate from the rest of your app. Bring your own key when you already trust a provider, keep inference local when data must stay inside your network, and use Gateway when you'd rather not run inference yourself. Choose a chat model and an embedding model for your workload, read the model card, and test with your real tasks before you commit.

> **Explore LibraOS:** [**libraos.com**](https://libraos.com/?ref=blog.meganova.ai)

## **Sources**

- [Models & Inference](https://libraos.com/models/?ref=blog.meganova.ai)
- [Gateway documentation](https://docs.meganova.ai/?ref=blog.meganova.ai)
- [Gateway model availability and pricing](https://docs.meganova.ai/inference-models/model-list?ref=blog.meganova.ai)
- [LibraOS model repositories on Hugging Face](https://huggingface.co/libraos?ref=blog.meganova.ai)

### **What’s Next?**

[**Sign up**](https://console.meganova.ai/register?referral=e0602d28&utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fmn%5Flaunch&utm%5Fcontent=body%5Flink&ref=blog.meganova.ai) and explore now.

🔍 **Learn more:** Visit [**our blog**](https://blog.meganova.ai/?utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fmn%5Flaunch&utm%5Fcontent=body%5Flink) and [**documents**](https://docs.meganova.ai/?utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fmn%5Flaunch&utm%5Fcontent=body%5Flink) for more insights or [contact us](https://www.meganova.ai/contact-us?utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fmn%5Flaunch&utm%5Fcontent=body%5Flink) for a demo of how MegaNova can power your AI products.

📬 **Get in touch**: Join our [Discord community](https://discord.com/invite/QBTnGurDw4?utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fai%5Flaunch&utm%5Fcontent=body%5Flink) for help or [Contact Us](https://www.meganova.ai/contact-us?utm%5Fsource=blog&utm%5Fmedium=referral&utm%5Fcampaign=blog%5Fmn%5Flaunch&utm%5Fcontent=body%5Flink).