42x Faster Prompt Lookup Drafting in llama.cpp

jadidbourbaki.github.io

cross-posted to:
[email protected]

42x Faster Prompt Lookup Drafting in llama.cpp

jadidbourbaki.github.io

cm0002@literature.cafe to

AI - Artificial intelligence@programming.devEnglish · 2 months ago

cross-posted to:
[email protected]

Four changes to the n-gram caches of llama.cpp make drafting up to 41.6x faster, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.

You must log in or register to comment.

Chat

AI - Artificial intelligence@programming.dev

Aii@programming.dev

Create a post

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]

AI related news and articles.

Rules:

No Videos.
No self promotion: Don’t post links to your articles.

Visibility: Public

This community can be federated to other instances and be posted/commented in by their users.

9 users / day
23 users / week
137 users / month
156 users / 6 months
1 local subscriber
410 subscribers
127 Posts
15 Comments
Modlog