System prompt is 4,000 tokens, the user input is 200 -> I pay for the same instructions 360,000 times a month
Our system prompt is 4,000 tokens. The input from a user is around 200 tokens, the output around 150. We make 12,000 calls in a day. 12,000 x 4,000 = 48 million tokens a day and this is only the instructions. The part a person actually typed is 12,000 x 200 = 2.4 million. So the fixed text is 20 times of what the user sends -> at 3 euro for a million input tokens this is 144 euro in a day and 4,320 euro in a month, for one page of rules which never changes. This is a small car every quarter. Prompt caching does not rescue us and i have checked this arithmetic two times. We have 60 tenants and every tenant has its own variant of the 4,000 tokens, because the compliance wording is different for each one. 12,000 calls, 60 tenants, 24 hours -> one call per tenant every 432 seconds. The cache is alive for 300 seconds. So we miss on nearly all of them. The both easy answers are closed to me. Cutting the system prompt removes rules which legal put there and i am not the person who is allowed to remove them. Fine tuning we have discussed since 2 years, there is no budget and no informations about who would keep it current afterwards. So the question is what shape a prompt should have when 20 of every 21 input tokens are the same text as the call before it. Somebody has solved this already, i am sure of it.
Not run against a model yet
0 works · 0 fails
Times copied by users
Problem Instructions
Find what shape a prompt should have when 20 of every 21 input tokens are the same text as the call before. The 4,000 tokens are legal wording which i am not allowed to delete -> the saving has to come out of the structure.
- •Input cost under 100 euro in a day at the same 12,000 calls; today it is 144 euro.
- •It survives 60 tenants with a different compliance section each, where one tenant calls once every 432 seconds and the cache lives for 300.
- •No rule which legal put in the 4,000 tokens is removed.