Accessing Novita AI API through Portkey AI Gateway

Portkey AI Gateway has revolutionized the way developers interact with various AI models, including Novita AI. This powerful tool offers a unified interface for accessing multiple language models, ensuring fast, reliable, and secure routing. In this comprehensive guide, we'll explore how to leverage Portkey AI Gateway to access Novita AI API, streamlining your AI development process and enhancing your application's performance.

1. Understanding Portkey AI Gateway

Portkey AI Gateway serves as a robust intermediary between your application and various AI models, including Novita AI. It's designed to simplify the integration process, offering a unified API that supports over 1600 language, vision, audio, and image models. With its lightweight, open-source nature, Portkey allows developers to integrate any language model in under two minutes.

Key characteristics of Portkey AI Gateway include:

Blazing fast performance with less than 1ms latency
Tiny footprint of only 122kb
Battle-tested infrastructure processing over 10 billion tokens daily
Enterprise-ready features including enhanced security and custom deployments

2. Key Features and Benefits

Portkey AI Gateway offers a plethora of features that significantly enhance the development and deployment of AI applications:

Unified API: Integrate multiple AI providers, including Novita AI, through a single, consistent interface.
Automatic Retries and Fallbacks: Prevent downtimes and ensure high availability of your AI services.
Load Balancing and Conditional Routing: Scale your AI applications efficiently by distributing requests across multiple providers or models.
Guardrails: Implement content filters and output validation to maintain quality and safety standards.
Multi-modal Capabilities: Extend beyond text-based models to include vision, audio, and image processing.
Agentic Workflow Integrations: Explore advanced AI agent frameworks for complex applications.

These features collectively contribute to faster development cycles, improved reliability, and enhanced performance of AI-powered applications.

3. Setting Up Portkey AI Gateway

Setting up Portkey AI Gateway is a straightforward process that involves three main steps: setting up the gateway, making your first request, and configuring routing and guardrails.

Setup your AI Gateway

To run the gateway locally, you'll need Node.js and npm installed. Then, execute the following command:

npx @portkey-ai/gateway

Once running, you'll see two important URLs:

The Gateway: http://localhost:8787/v1
The Gateway Console: http://localhost:8787/public/

Make your first request

First, install the Portkey AI Python library:

pip install -qU portkey-ai

Then, use the following Python code to make your first request:

from portkey_ai import Portkey

# OpenAI compatible client
client = Portkey(
    provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
    Authorization="sk-***" # the provider API key
)

# Make a request through your AI Gateway
client.chat.completions.create(
    messages=[{"role": "user", "content": "What's the weather like?"}],
    model="gpt-4o-mini"
)

You can monitor all your local logs in one place using the Gateway Console at http://localhost:8787/public/.

Routing & Guardrails

Portkey AI Gateway allows you to set up routing rules, add reliability features, and implement guardrails. Here's an example configuration:

config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# In this example, the guardrail denies all replies containing "Apple", so the response would always be "Bat". The retry configuration would attempt the request up to 5 times before giving up.

AI Gateway

4.Integrating Novita AI API with Portkey

To access Novita AI API through Portkey AI Gateway, follow these steps:

Install the Portkey SDK

Add the Portkey SDK to your application to interact with Novita AI's API through Portkey's gateway.

Node.JS

npm install --save portkey-ai

Python

pip install portkey-ai

Initialize Portkey with the Virtual Key

To use Novita AI with Portkey, get your LLM API key from Novita AI, then add it to Portkey to create the virtual key.

Node.JS SDK

import Portkey from 'portkey-ai'

const portkey = new Portkey({
  apiKey: "PORTKEY_API_KEY",  // Replace with your Portkey API key
  virtualKey: "VIRTUAL_KEY"   // Replace with your virtual key for Novita AI
})

Python SDK

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    virtual_key="VIRTUAL_KEY"   # Replace with your virtual key for Novita AI
)

Invoke Chat Completions with Novita AI

Use the Portkey instance to send requests to Novita AI. You can also override the virtual key directly in the API call if needed.

Node.JS SDK

const chatCompletion = await portkey.chat.completions.create({
  messages: [{ role: 'user', content: 'Say this is a test' }],
  model: 'Nous-Hermes-2-Mixtral-8x7B-DPO'
});

console.log(chatCompletion.choices);

Python SDK

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'reka-core'
)

print(completion)

5. Useful Resources

To further enhance your understanding and usage of Portkey AI Gateway with Novita AI, consider exploring these valuable resources:

Portkey AI Gateway GitHub Repository: Stay updated with the latest pull requests, features, and improvements.
Portkey AI Documentation for Novita AI Integration: Comprehensive guide on integrating Novita AI with Portkey, including SDK usage and supported models.
Portkey AI Blog: Access insightful articles, tutorials, and updates related to AI integration and best practices.
Novita AI API Documentation: Refer to the official Novita AI documentation for detailed information on their LLM API and model specifications.

These resources provide a wealth of information to help you maximize the potential of Novita AI through Portkey AI Gateway, ensuring you stay at the forefront of AI integration and development.

6.Conclusion

Integrating Novita AI API with Portkey AI Gateway offers a seamless and efficient way to enhance your AI applications. By following the steps outlined in this guide, you can leverage the powerful features of Portkey AI Gateway to streamline your development process, improve performance, and ensure reliability. Whether you are working with text, vision, audio, or image models, Portkey AI Gateway provides a unified interface that simplifies the integration of multiple AI providers.

By staying updated with the latest resources and best practices, you can continue to optimize your AI applications and stay ahead in the rapidly evolving field of AI integration and development. Embrace the capabilities of Portkey AI Gateway and Novita AI to build innovative and robust AI solutions.

Originally published at Novita AI

Novita AI is the All-in-one cloud platform that empowers your AI ambitions. Integrated APIs, serverless, GPU Instance — the cost-effective tools you need. Eliminate infrastructure, start free, and make your AI vision a reality.

Accessing Novita AI API through Portkey AI Gateway: A Comprehensive Guide

Table of contents