As we try to enter the same earlier prompt that we’ve used for GPT-4, we receive an honest answer, namely, “I don’t know”. We do expect this behavior since flan-alpaca lacks the knowledge of Scala from pre-training, unlike GPT-4.
As of May 2023, the open-source modeling landscape is evolving swiftly, with the emergence of fresh models occurring daily. For instance, two new models, Databricks’ Dolly model and StabilityAI’s StableLM-Tuned-Alpha model, were introduced while we were writing this article. That is why it is essential to remain up-to-date with the newest developments in the field.
As natural language processing (NLP) solutions such as GPT-4, and open-source models continue to revolutionize the way we interact with technology, the question of how to maximize the value of these models have become increasingly important.
We’ve compared the answer quality of these models and explored the differences in their approaches to generating answers.
The real challenge lies in leveraging hundreds or thousands of documents and automatically detecting the relevant context. We suggest downloading the documentation and using the raw documents as a source to create a solution that performs three essential functions:
- Finding the most relevant documents as a context.
- Building a prompt based on the context and specific instructions.
- Generating the model’s answer to the question.
With these steps, we only need to insert the question, without specifying which part of the documentation can answer the question.
When it comes to finding relevant documents for a specific question, relying solely on word overlap renders it ineffective, especially when dealing with synonyms. However, there are now more advanced methods available to sustain efficiency.
One such method involves using vector representations, or embeddings, to search for documents similar to the query. By converting each document into a vector and comparing it to the vector representation of the query, we can identify the most similar documents in the vector space.
Hugging Face’s Sentence Transformers help to make this process easier than ever. For example, we can use this approach to compare the benchmark sentence “How to convert case class Person(name: String, surname: String) into Json?” to various Scala documentation extracts, including ones on serialization, macros, and varargs.
Let’s have a look at msmarco-distilbert-base-v4 API: