I Trained an LLM on 75K of My Own Messages So It Would Stop Writing Like a Chatbot

training.

Key Takeaways and Future Directions

1. Oversampling Formal Writing: Since your corpus is heavily skewed towards casual messages, it would be beneficial to include more formal writing samples (emails, professional documents) in the training data. This will help balance the model's output across different contexts.

2. Style-Routing Prompt: Implementing a system prompt that specifies the context or style of the response can improve adaptability. For example:

   • "Email voice: Write an email to your manager."
   • "Chat voice: Respond casually to a friend."

3. Longer-Form Writing: Include more long-form writing samples in the training data to ensure the model can handle extended text generation effectively.

4. Contextual Fine-Tuning: Consider fine-tuning on specific contexts (e.g., professional, casual) separately and then merging them into a single adapter or using conditional prompts for better control over output style.

Example of Style-Routing Prompt

Here’s an example of how you might implement a style-routing prompt in your training data:

• Prompt: "Write a response to the following message in email voice: Can you review this PR when you get a chance?"
  • Response: "

Read the full article at DEV Community

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.