How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine?
How do the smallest quants of this compare to Bonsai 2?
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Awesome! I wonder how it does at Q5+ compared with v3.0
4 comments:
How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine?
How do the smallest quants of this compare to Bonsai 2?
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Awesome! I wonder how it does at Q5+ compared with v3.0