community llm benchmark questions ( brainstorm ) continual development

a post to donate types of prompts that could be used as a benchmark.

criteria guideline:

problems that llm had struggled to solve. ( and somewhat desirable )
valuable usecases
centric to various other problems ( with value )
located in a domain that underperforms on all tasks related to it

* soft push for actual personally useful problems. not toy demonstrations/tests . * no criticizing/debating . this is write-only
* soft focus for smaller models ( rationale being: no really hard problems that are cutting-edge math,science,engineering ) . eg ok to be using large models but for not cutting-edge problems that a smaller model definitely could never handle.
* one post per user
* vague description of problem/prompts is ok.

-1 points · 1 comments · view on lemmy.world

1 Comments

leanleft@lemmy.ml · 1 pts · 13d

homeless knowledge ; ( as you would expect.. poor knowledge across the board )
perl5 coding ; ( not as good as python ).
usa knowledge and usa society ; ( you would expect alot better )
simple geographic trip planning with constraints andor weighted decision making, that involves graph
ptsd response ; ( failed )
notetaking ; ( not as good as you would expect. also gives cheesy obvious answers )
android OS
firefox about:config
OSM
relationships ( not bad but overconfident and not a careful thinker. prompting is sensitive. also sometimes cheesy obvious answers)
meaning of life
designing new things with reasoning ( not just reproducing a synthesis or variation ) ; feels similar to multishot overfitting problem.
problem solving ; ( not as good as you would expect )
certain security problems ; ( outperformed by human commonsense )
telecom products/companys ; ( blatantly wrong or censored info )
BSD
data ; ( regarding special case: non-enterprise )
sexy chat ; ( models that *i* use.. failed. cheesy answers. refused to get kinky. im sure theres a couple out there. )
crime scene investigation/forensic ; ( not an expert. however i detected multiple missing explainations which i think are fairly plausible and worth mentioning )