Unleashing the Power of Streaming: Revolutionizing Real-Time Interactions with AI

In the rapidly evolving landscape of LLM-powered applications, smooth experience is key as users’ attention spans are diminishing, and tolerance for delays is below zero. In this context, the quest for real-time, dynamic interactions has led developers to embrace streaming as a core part of almost any fully functional application. If you’re building a product coupled with the OpenAI API, read on to learn more about integrating efficient AI capabilities and keeping UX  top-notch through implementing streaming. 

What is Streaming?

Streaming is not just a feature; it's a catalyst for unlocking real-time possibilities. It facilitates dynamic exchanges between developers and the OpenAI API, allowing responses to be received incrementally as data becomes available. This real-time interaction introduces a level of flexibility and responsiveness that is indispensable in scenarios where standard request-response mechanisms fall short. If you’re aiming to build an LLM-powered application with real-time experience, then Streaming is a mandatory feature for success.

Why is real-time experience important in LLM apps?

If a developer chooses not to use streaming in their LLM-powered chatbot, for example, they would typically rely on standard request-response mechanisms when interacting with the OpenAI API. What would that mean for the end users? It might be a throbber that just never disappears and makes your app look broken, or blank screens and users hitting “refresh” or directly the “exit” button. What could be worse than that?!

Here is a table we’ve prepared of common use cases, where real-time experience is a must and what happens if you fail or omit implementing it:

Streaming VS Standard Request-Response

‍

How to stream the response using GPTBoost?

Now, I believe you should be already firmly convinced that if you’re building an application with OpenAI API integration aiming to deliver real-time experience, you must implement Streaming. So, in the next few lines, I’ll show you how to do that in a smooth and easy manner.

GPTBoost has a special built-in service called GPTBoost Turbo Proxy that supports OpenAI streaming. All you need to do is simply add stream=True to your request to receive the response in chunks as they're generated. Here is how:

Python

import openai

openai.api_base = "https://turbo.gptboost.io/v1"
openai.api_key = "$OPENAI_API_KEY"

for chunk in openai.ChatCompletion.create(
    model='gpt-3.5-turbo',
    messages=[{
        'role': 'user',
        'content': "Tell me an interesting fact about Chile"
    }],
    stream=True
):
    if chunk["choices"][0].delta:
        content = chunk["choices"][0].get("delta", {}).get("content")
        print(content)

cURL

curl --request 'https://turbo.gptboost.io/v1/chat/completions' \
--header 'Authorization: Bearer $OPENAI_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gpt-3.5-turbo",
    "messages": [
        {
            "role": "user", 
            "content": "Tell me an interesting fact about koalas!"
        }
    ], 
    "stream": true
}'

NodeJS

// This code is for v4 of the openai package: npmjs.com/package/openai
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://turbo.gptboost.io/v1"
})

async function generateStream(prompt) {
  const stream = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{"role": "user", "content": prompt}],
    stream: true,
  });
  for await (const part of stream) {
    console.log(part.choices[0].delta.content);
  }
}

Upgrade your development experience with GPTBoost!

Developing awesome OpenAI applications doesn’t have to be a hassle. It can be done faster and easier with the right tools. Enhance your projects with the power of streaming and take your real-time AI interactions to the next level. We can't wait to see the incredible applications and experiences you build using GPTBoost and Streaming.

Get started today by exploring our documentation and integrating streaming into your OpenAI API requests. 

Happy streaming! 🚀✨