It may be able to code one (the code is relatively short and well known) and give training program, and then you would need to spend a few trillion tokens to make it generate data.
You can generate synthetic data matching the distribution your transformer learned. You can use this dataset to train another model. As of now, that's about it.
8 Comments
keepthepace@slrpnk.net · 7 pts · 2y
Not directly no.
It may be able to code one (the code is relatively short and well known) and give training program, and then you would need to spend a few trillion tokens to make it generate data.
Anticorp@lemmy.world · 1 pts · 2y
Where can we see this well known code? I'd like to see how it works.
keepthepace@slrpnk.net · 3 pts · 2y
Here is an implementation in pytorch:
https://github.com/lyeoni/gpt-pytorch/blob/master/model.py
Here is one in pure C that karpathy started:
https://github.com/karpathy/llm.c
Anticorp@lemmy.world · 1 pts · 2y
Thanks!
Audalin@lemmy.world · 5 pts · 2y
You can generate synthetic data matching the distribution your transformer learned. You can use this dataset to train another model. As of now, that's about it.
balder1993@programming.dev · 2 pts · 2y
Yep, this is called model distillation.
over_clox@lemmy.world · 3 pts · 2y
Don't give them any ideas.. 😂
wargreymon@sh.itjust.works · 0 pts · 2y
ok... LOL