[plug] [a bit OT but OSS] Building LLM with expert datasets

[Harry McNally]

Hello

I read an O'Reilly (I think) Radar or other newsletter an item linked to 
discussion about creating federated sets of LLMs (hosted by various companies) 
that are selected based on the query.

This would have to be of interest to O'Reilly because their published books 
and other resources would be higher quality training data.

When I spoke to James Bromberger about this his feedback was that, as Perl 
evolved, he had to buy each of the new O'Reilly books as they were 
significantly different with each new Perl version. Fair comment but it would 
be necessary for any programming LLM to have to not have "version" 
hallucination (or it isn't useful).

I have searched through emails and can't re-discover the reference or 
discussion. If anyone read a similar reference to federated sets of LLM that 
would be helpful to me.

I restarted thinking about this having got distracted today reading:

https://techstartups.com/2025/01/31/deepseek-r1-reproduced-for-30-berkeley-researchers-replicate-deepseek-r1-for-30-casting-doubt-on-h100-claims-and-controversy/

that led through TinyZero on GitHub:

https://github.com/Jiayi-Pan/TinyZero

to the veRL project and this talk (three months ago so old news to others I 
guess):

https://www.youtube.com/watch?v=MrhMcXkXvJU

and paper:

https://arxiv.org/abs/2409.19256

The whole flurry and upset seems to be because: "We haven't got the money, so 
we've got to think." https://www.azquotes.com/author/12817-Ernest_Rutherford

or in this case "We have lesser GPU's, so we've got to optimise".

All the best
Harry