Ready to dive into the lake?
lakeFS is currently only
available on desktop.

For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!

lakeFS Community
Amit Kesarwani
Amit Kesarwani Author

Amit heads the solution architecture group at Treeverse, the company...

Published on March 26, 2024

TL;DR

In the previous blog, Introducing the LangChain lakeFS Loader, and sample notebook, we explained and demonstrated integration of lakeFS with LangChain and LLM models (specifically OpenAI models). In this blog, we will explore a new beta feature from OpenAI that enables reproducible responses from a model.

Introduction

Language models are Stochastic models (stochastic refers to a random or probabilistic process) which means their behavior is non-deterministic in nature and involves chance or probability. They generate outputs based on probability distributions, not deterministic rules. This makes parsing their output more difficult since the outputs can vary each time.

For conversational applications in general, this is not a bad thing, since models improve, as do the responses, and the level of flexibility is also convenient for any chatbot or voicebot implementation, since a message abstraction layer is not required.

But determinism has always been a big request from user communities when using LLMs. For instance, when granted the capability of getting deterministic numerical results, users can unlock quite a bit of use cases that are sensitive to numerical changes.

Model level controls for deterministic outputs

Now OpenAI offers some control towards deterministic outputs by giving you access to the seed parameter. OpenAI always includes a system_fingerprint in the response that helps developers understand changes in their system that will affect determinism.

To receive (mostly) deterministic outputs across OpenAI API calls, you can:

  • Set the seed parameter to any integer of your choice and use the same value across requests you’d like deterministic outputs for.
  • Ensure all other parameters (like prompt or temperature) are the exact same across requests.
  • In the response, check the system_fingerprint field. The system fingerprint is an identifier for the current combination of model weights, infrastructure, and other configuration options used by OpenAI servers to generate the completion. It changes whenever you change request parameters, or OpenAI updates numerical configuration of the infrastructure serving their models (which may happen a few times a year).

If the seed, request parameters, and system_fingerprint all match across your requests, then model outputs will mostly be identical. There is a small chance that responses differ even when request parameters and system_fingerprint match, due to the inherent non-determinism of computers.

Example

If you want to run this example on your own then go to OpenAI Playground and enter the same user message twice. We used the “what is lakeFS” user message in our example. As expected, we received varying non-deterministic responses:

OpenAI Playground deterministic response 'what is lakeFS'

Below is a sample Python code to demonstrate deterministic outputs. We set the seed parameter to 123. The seed number is an arbitrary number, which creates an association between the user input, and the system_fingerprint.

pip install openai

import os
os.environ['OPENAI_API_KEY'] = str("your api key goes here")

from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates short stories."
        },
        {
            "role": "user",
            "content": "what is lakeFS"
        }
    ],
    temperature=1,
    max_tokens=150,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0,
    seed=123
)
print(response)

First Execution

Below is the first execution with the seed number defined:

ChatCompletion(id='chatcmpl-8VZuMdB7MW7ok31MrIyftl0PAot6Z', 
choices=[Choice(finish_reason='length', index=0, 
message=ChatCompletionMessage(content='LakeFS is an open-source Git-like version control system specifically designed for data lakes. It provides a unified layer of control and governance for data stored in cloud object storage, allowing data teams to easily manage, version, and collaborate on large-scale datasets.\n\nOnce upon a time, in a land known for its bountiful data, there was a majestic lake called LakeFS. This lake, unlike any other, held an immense collection of valuable and diverse data. Its calm waters mirrored the shimmering clouds above, creating a serene atmosphere.\n\nPeople from all walks of life would come to LakeFS, seeking knowledge and insights hidden within its depths. They would dive in, their curious minds exploring the vastness of information stored there, hoping to uncover new discoveries', role='assistant', function_call=None,
tool_calls=None))], created=1702536890, model='gpt-3.5-turbo-0613', 
object='chat.completion', system_fingerprint=None, 
usage=CompletionUsage(completion_tokens=150, prompt_tokens=25, total_tokens=175))

Second Execution

Below is the second execution, with the exact same response:

ChatCompletion(id='chatcmpl-8VZuguUai2eAOC7VAQLSo2eVbM9h3', 
choices=[Choice(finish_reason='length', index=0, 
message=ChatCompletionMessage(content='LakeFS is an open-source Git-like version control system specifically designed for data lakes. It provides a unified layer of control and governance for data stored in cloud object storage, allowing data teams to easily manage, version, and collaborate on large-scale datasets.\n\nOnce upon a time, in a land known for its bountiful data, there was a majestic lake called LakeFS. This lake, unlike any other, held an immense collection of valuable and diverse data. Its calm waters mirrored the shimmering clouds above, creating a serene atmosphere.\n\nPeople from all walks of life would come to LakeFS, seeking knowledge and insights hidden within its depths. They would dive in, their curious minds exploring the vastness of information stored there, hoping to uncover new discoveries', role='assistant', function_call=None,
tool_calls=None))], created=1702536910, model='gpt-3.5-turbo-0613', 
object='chat.completion', system_fingerprint=None, 
usage=CompletionUsage(completion_tokens=150, prompt_tokens=25, total_tokens=175))

Recap

We demonstrated how to use a fixed integer seed to generate consistent outputs from the OpenAI model. By making use of the seed parameter the outputs are reproducible. However, it’s important to note that while the seed ensures consistency, it does not guarantee the quality of the output. For instance, in the example provided, we used the same seed to generate a short story about a journey to Mars. Despite querying the model multiple times, the output remained consistent, demonstrating the effectiveness of using this model level control for reproducibility.

This feature assists in simulations for debugging, benchmarking/evaluating the performance of different prompts or models, creating comprehensive unit tests and achieving a higher degree of control over model behavior.

Git for Data – lakeFS

  • Get Started
    Get Started