Hide navigation sidebar
Hide table of contents sidebar
Back to top
Toggle table of contents sidebar

# Concept | LLM connections

Dataiku provides connections to a variety of generative AI services, both hosted and self-hosted, that are required to build Large Language Model (LLM) applications.

The LLM connections can then be leveraged with Dataiku’s:

- Prompt Studio

- LLM visual recipes, such as the Prompt recipe, text processing recipes (summarization, classification), and the Embed recipe for Retrieval-Augmented Generation (RAG)

- Python API for usage in code recipes, code studios, notebooks, and web applications.

Instance administrators can configure connections to LLMs in:

- Launchpad > Connections menu > Add a connection > LLM Mesh (for Dataiku Cloud).

- Administration menu > Connections > New Connection > LLM Mesh (for a self-managed instance).

## Types of connections

In Dataiku’s LLM Mesh, you can connect to models that are hosted by service providers, hosted locally, or developed locally.

### Hosted LLM connections

Supported connections to Generative AI services include models such as OpenAI, Hugging Face, Cohere, etc.

Connection settings vary depending on the requirements of the providers.

### Hugging Face local connections

Users can host, manage, and use open source Huffing Face models locally with Dataiku’s Hugging Face connection. This connection includes containerized execution and integration with NVIDIA GPUs to make the model execution simpler. Users can start experimenting with Hugging Face models without downloading the model into the code environment’s resources.

Administrators can configure which Hugging Face models are available for different tasks, and other settings such as containerized execution, in the connection settings.

### Custom LLM connections

Users who have developed their own LLMs with API endpoints can integrate these custom services into Dataiku via the Custom LLM connection. Setting up a Custom LLM connection requires developing a specific plugin using the Java LLM component to map the endpoint’s behavior.

## LLM costs

Cost is a major consideration for any business using LLM services. Dataiku provides solutions to assess and reduce costs.

### Cost assessment

You can use the Compute resource usage (CRU) solution to keep track of the cost, thanks to the logs from the LLM connection (see Auditing LLM connections).

It enables you to monitor performance, diagnose issues, build fine-grained cost reports, aggregate costs per application, recipe, user or project, and define appropriate mitigation measures such as alerts.

### Using cache to reduce costs

The LLM Mesh provides caching capabilities that an administrator can configure at the LLM connection level. By caching responses to common queries, the LLM Mesh avoids unnecessary re-generation of answers, reducing avoidable costs.

To enable the use of cache:

1. In the Connections menu from the Launchpad, select your LLM connection or create a new one (New connection > LLM Mesh).

   Note

   On a self-managed instance, go to the Administration > Connections menu of the top navigation bar.

2. Under Usage Control, check the Caching option.

## Auditing LLM connections

Dataiku administrators can configure the level of auditing and logging to keep a complete record of queries and responses. They can either:

- Disable the audit.

- Log only the metadata.

- Log all the queries/responses.

Logs are then available for download from the administration panel or can be used directly in a Dataiku project for further analysis, for example, to ensure extensive cost control and monitoring. For further information, see the Cost assessment section above.