One environment variable made our llama.cpp server up to 21% faster
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Tensward reports that setting the glibc environment variable GLIBC_TUNABLES=glibc.malloc.hugetlb=1 on a llama.cpp server raised throughput by up to 21% and sped up first-token times in most tests, with no code changes.