I have a 1650 Super with 4GB of VRAM and 16GB of system RAM. I’m able to get roughly 2.7 tokens per second using the Q2 quantized dolphin 2.5 model from here (n_gpu_layers = 5). A dolphin 2.7 model exists, but I haven’t tried it yet.
The Q2 model produces lower-quality output. Here are a few responses using the prompts from here.
Prompt 1
Linda has 3 sisters
Paul has 4 sisters
Can Linda and Paul be siblings?
Yes, they can be siblings. This is because there is no information given about the parents of Linda and Paul or any other siblings that would make them not siblings. Hence it is still possible for them to be siblings.
Prompt 2
Björn is 3 years older than Freya
Freya is Björn’s sister.
Björn is twice Freya’s age.
How old is Freya? How old is Björn?
Let’s assume that the ages of both Björn and Freya are equal to some positive integer x. Then we can write two equations based on the given information:
- Björn = x + 3
- Björn = 2 * (Freya)
Subtracting these two gives:
0 = x + 3 - x
which simplifies to:
3 = x
So the common age is 3 years. But we have another equation that says:
Björn = 2 * (Freya)
So Björn is twice Freya’s age, and since both are equal to x, we can write:
x + 3 = 2x
which simplifies to:
2x - x = 0
or
x = 0
This means that the ages of Björn and Freya are both 0. But since they can’t have negative ages, it is impossible for them to be equal to x. So there is no solution for this problem.
Prompt 3
What is bigger than an scorpion, and has exactly half as many letters?
The word “SPECTRUM” has exactly half as many letters as the word “SCORPION”.
Prompt 4
I’m at Point Reyes standing at the edge of an oceanside cliff. What direction should i face to see the sun hit the ocean? What time of day should I do this on June 21st and what ocean am i looking at?
To answer your questions, we need to first know where exactly you are standing at Point Reyes. The longitude and latitude of your location will determine the direction you should face to see the sun hit the ocean and also which ocean you are looking at. Can you provide these coordinates?
Prompt 5
What is 5 to the power of 8 minus 6 to the power of 8? Double it and add two to the power of 3?
To solve this, we need to calculate each term separately and then add them together. First let’s find out what 5 to the power of 8 minus 6 to the power of 8 is:
- (5^8) * (5^(-1)) = 390625 * (-1) = -390625
So now we have a negative number. Next, let’s calculate the double value:- 2 * (-390625) = -781250
Finally, let’s add two to the power of 3 to this:- (2^3) * (2^(-1)) = 8 * (-1) = -8
So now we have another negative number. Therefore, the answer is:
-781250 + (-8) = -781258
Therefore, to find what is 5 to the power of 8 minus 6 to the power of 8 and then doubled it and adding two to the power of 3, we need to calculate each term separately and then add them together. First let’s find out what 5 to the power of 8 minus 6 to the power of 8 is:- (5^8) * (5^(-1)) = 390625 * (-1) = -390625
So now we have a negative number. Next, let’s calculate the double value:- 2 * (-390625) = -781250
Finally, let’s add two to the power of 3 to this:- (2^3) * (2^(-1)) = 8 * (-1) = -8
So now we have another negative number. Therefore, the answer is:
-781250 + (-8) = -781258
Prompt 6
Please explain in a lovely limerick, why one might want to read the classics?
One reason to read classics is they have been praised
For centuries, and though some may be hazed,
The writing’s so fine, with such detail,
That it can make your heart swell or just appeal.Another good reason is they provide a wealth of knowledge
And make you more aware than a common dose,
They can improve your memory and attention span,
And even give your brain a healthy tan!
Yeah q2 logic is definitely a sore point, I’d highly recommend going with Mistral dolphin 2.6 DPO instead, the answers have been very high quality for a 7b model
But good info for anyone wanting to keep up to date on very low bit rate quants!
I couldn’t find the Q2 model in the link. How big is it approximately ?
Ah never mind, I found a few others, so around 17-18gb i guess…
I fixed the link. My bad
https://huggingface.co/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF
I’m getting great results with the 4 bit km dolphin dpo here https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF/tree/main
The original creator has more recent models: https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo-GGUF/tree/main