ZenNest Chatbot Assistant

By Jason Pitts


Overview


Welcome to the ZenNest Chatbot assistant! This project leverages OpenAI’s API to create a customer service chatbot for the ZenNest home system. The ZenNest home system is a futuristic home automation system that integrates with a chatbot assistant that aims to solve user issues quickly and efficiently. This chatbot was trained using Python, JSON training data, and integrated with Weights and Biases to record training results. Following along with this project, you will gain a basic understanding of working with LLMs, specifically OpenAI’s API, to achieve desired results. For detailed instructions, check out the readme on GitHub.


ZenNest Mockup I’m no artist, but I enjoyed making this mockup


Where to Start?


This project started out as a relatively simple idea: create a customer service chatbot. The fun comes in finding a key objective to focus on as an LLM such as OpenAI’s ChatGPT already has a robust foundation to work with. For this purpose, I devised a home automation company, ZenNest, with a chatbot, Sarah, that can assist customers with almost anything.


The first place to start is training the model based on that idea so I started grinding away, creating training data to cover several basic objectives that the chatbot should begin to cover.

  • Main Objective - The purpose and identity statements of the bot
  • Knowledge Base Integration - Real world knowledge of the product
  • Contextual Understanding - The base model is strong but our product is specific with related jargon
  • User Interaction Design - How users are expected to interact with the bot
  • Multi-turn Dialogue Handling - How the bot will respond in extended conversations
  • Error Handling - Unexpected user inputs can cause unexpected responses that we must account for


json
{
    "messages": [
        {
            "role": "system",
            "content": "Sarah is a chatbot that assists customers having trouble with their ZenNest home system."
        },
        {
            "role": "user",
            "content": "What is a ZenNest home system?"
        },
        {
            "role": "assistant",
            "content": "ZenNest is a smart home system blending technology with Zen principles for serene living. It offers security, energy management, and personalized settings for a harmonious home environment."
        }
    ]
}

An example of Main Objective training

What were the results?


With my first set of training data, I split it into 80% training and 20% validation data. The validation data is crucial as we hold it back from the bot to see how it will perform on unseen conversations. The initial results looked very promising even though we are mainly working with qualitative data.


Training loss
0.8562
Validation loss
0.7708
Full validation loss
0.7356


Our training loss of 0.8562 is a good indicator that our first training is performing moderately well with only 80 lines of training. OpenAI typically sees large sizeable improvements with 50-100 interactions trained. The validation loss coming in lower at .7708 is a great indicator that we are also not overfitting. Finally, the full validation loss is even lower, indicating that our initial objectives are being met. So what does it look like?


TrainingRound2 Base GPT Model on the left; Trained Model on the right.


Trouble in Paradise


So the first response looks great, with the system message set to our training message the model is already integrating into our ZenNest home system and “instantly” solving problems. However, a few common problems are arising.


TrainingRound3 The bot is incredibly easy to get off topic


Further Fine-Tuning


So how do we fix this? Well, more training. I returned to the drawing board knowing my first attempt wouldn’t be perfect as it never is. Almost doubling the training data to account for common pitfalls of LLM chatbots I started the second set of training. The second set of results started to look miles better after hours of prompt engineering.


Training loss
0.5983
Full validation loss
0.8023


As a bonus, more than common pitfalls, industry ‘Zen Jargon’ were trained, as well as a myriad of situations and longer conversations. Our second round of training starts off with an incredible looking .5983 training loss indicating that our model is trending towards reasonably well performance. The full validation loss of .8023 is a bit higher but this can be expected as the LLM learns more and begins to cover a wide variety of subjects. You’ll also notice that we are missing validation loss, which is due to the number of epochs compared to the data amount. Given a greater spread of epochs (3 were used in this training pass) our validation data will be more stable. We have increased the model’s performance and knowledge base at the cost of possible outlier answers. So how is Sarah doing?


TrainingRound4 The latest and greatest model staying on task; new Sarah to the right


Weights & Biases


wandbPreview Comparing the training jobs in the WandB dashboard


Conclusion and Considerations


Training a custom chatbot, especially one that is an expert in a specialized industry, is a large undertaking. Training the ZenNest Chatbot to completely reach a tranquil level of understanding is out of the scope of the project, but you are now equipped with the tools necessary to fully train your own chatbot.


View the full project on GitHub.