u/BankApprehensive7612
[PAPER] GPQA, MMLU-Pro, and MMMU-Pro were audited for broken questions, and up to 12% of them had to be removed. New drop in clean versions released
I was very curious why all the models were topping out on GPQA-Diamond around 92 or 93% (AA) and spent the last few weeks pouring over GPQA (Diamond and Extended), and then expanded to auditing MMLU-Pro and MMMU-Pro.
It was quite frankly shocking just how many questions were malformed, had wrong answer keys, or questions with more than one realistic answer. In fact, on GPQA-Extended, MMLU-Pro and MMMU-Pro, ~12% of questions were verifiably broken! Once fixed, the top models hit around 98%.
Full paper released here: https://github.com/adamallcock/answer-key-audit
As part of this process, I have also shopped -Clean versions of all four benchmarks with the broken items removed, but also a full flagged-candidate ledger so you can see exactly why. I've also included dual original-vs-cleaned scoring, lm-eval-harness tasks and Hugging Face datasets.
I would love any feedback you have on the paper, and what benchmarks I should look at next. PS: There are some verbatim examples of broken questions on page 28 if you want to take a look.
Kimi-K3 weights are available on HuggingFace in official release
Moonshot's latest model Kimi-K3 weights are available on HuggingFace to download for personal and commercial. Moonshot has published their model on HuggingFace
It's a 2.8T-parameters model by Moonshot.ai with 1 million tokens context window. The architecture is mixture-of-experts (896 experts) with 108B active parameters
It's available via vLLM, SGLang and TokenSpeed
License: Kimi K3 License. It permits commercial use with some limitations. For Model-as-a-Service it has $20M/year limit and after reaching the limit the license requires to make additional agreement with Moonshot (see license for more details)
Link to the model
Model could be downloaded from the HuggingFace by the link:
Kimi-K3 is published on HuggingFace
Moonshot's latest model Kimi-K3 is available on HuggingFace since today. And it's another good news for open-weight AI and for the future of open-source AI
It's a 2.8T-parameters Moonshot's SOTA model with 1 million tokens context window. The architecture is mixture-of-experts (896 experts) with 108B active parameters
It's available via vLLM, SGLang and TokenSpeed
License: Kimi K3 License. It allows commercial use with some limitations. For Model-as-a-Service it has $20M/year limit and after reaching the limit the license requires to make additional agreement with Moonshot. Details: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
Link to the model
Model could be downloaded from the HuggingFace:
Linux has officially won
Actually it happened in June of 2025, but the process has completed recently, though. After Apple had announced the support of OCI-compatible containers in the June '25 it took a year to complete development and implement full support of continers. Apple had published 1.0 version of own container manager (https://github.com/apple/container). And Microsoft had announced native support of containerization without Docker in Windows 11 (https://devblogs.microsoft.com/commandline/wsl-container-is-now-available-for-public-preview/). Now Linux is a part of any major platform: Windows, MacOS, BSD and Linux itself. Knowledge of Linux is now part of learning any of these systems, at least for developers. And now you can rely on Linux based containers running everywhere. What it is if not a win!?
What's also interesting. Linux can run other Linux distros and with this Alpine Linux could become the most popular version of Linux in the World
It's the biggest win for the whole open-source software and I believe it should get into history books of technological progress
⚡️ Electron v43 is out. Faster boots. More performant IPC and events. Polished DX, UX and OS integrations. With all the Chromium 149 and 150 features as a bonus
Electron version 43 has just landed and is now available to download. It contains performance optimizations and os integration improvements
Release on Github: https://github.com/electron/electron/releases/tag/v43.0.0
Notable Changes
- The main process now boots from an embedded snapshot provided by Node.js
- Improved performance of native event emission, IPC dispatch,
- Improved performance of webRequest header conversions
- Improved performance of Linux and Windows release builds. It's based on link-time optimization and improves loading of the Electron binary on startup
- New option allowExtensions added to protocol.registerSchemesAsPrivileged(). It enables Chrome extensions on custom protocols.
Breaking changes
- File downloads now open in the Downloads folder
- Frameless windows have rounded corners on Linux by default
- Removed: showHiddenFiles from dialog API on Linux
Read about other changes in detailed report on the Electronjs website
Detailed report: https://releases.electronjs.org/release/v43.0.0
Fugu is a probable competitor to Fable 5 by Japanese startup Sakana
According to the release Fugu reaches greater score in LiveCodeBench and TerminalBench than Fable 5. In the same time Fugu has different architecture and it's a routing agent built around models less capable than Fable 5 itself
LiveCodeBench
| Model | Score |
|---|---|
| Fugu Ultra | 93.2 |
| Fugu | 92.9 |
| Fable 5 | 89.8 |
| Gemini 3.1 Pro | 88.5 |
| GPT 5.5 | 85.3 |
| Opus 4.8 | 87.8 |
Terminal Bench 2.1
| Model | Score |
|---|---|
| Fugu Ultra | 82.1 |
| Fugu | 80.2 |
| Fable 5 | 80.4 |
| Gemini 3.1 Pro | 70.3 |
| GPT 5.5 | 78.2 |
| Opus 4.8 | 74.6 |
Release: https://sakana.ai/fugu-release/
Fable 5 is not the number 1 anymore!? Fugu by Sakana beats it in some cases, according to the company announcement
Fugu beats Fable 5 at least in LiveCodeBench (by 3%) and Terminal Bench 2.1 (by ~1.7%) according to their corporate blog
Official release: https://sakana.ai/fugu-release/
The numbers has not been independently confirmed yet
ASI is coming though! Fugu🐡 beats one Fable 5 in some coding benchmarks, according to Sakana.ai's own claim
Japanese AI startup Sakana.ai has published new release of their own agent Fugu. And according to benchmarks they beat Fable5 in LiveCodeBench with over than 3% up to 93,2%. The model itself is in a closed beta to the moment
Release: https://sakana.ai/fugu-release/
Worth noting, the numbers are to be independently confirmed yet. I hope it will happen soon
Fable 5 has been beaten by Sakana.ai's Fugu🐡 in some tasks, claims Japanese startup in the new release
According to the official announcement the company has reached parity or even has beaten Fable 5 by Anthropic on some tasks
Release: https://sakana.ai/fugu-release/
The benchmarks require independent confirmation, but their previous works include one with Nvidia give the reason to believe the numbers
HuggingFace: Probably it's time to move from LoRA 👋
HuggingFace team published an article on their blog that PEFT (Parameter Efficient Fine-Tuning) techniques (including LoRA) had evolved. And according to their benchmarks there are more efficient methods (like Orthogonal Fine-Tuning or OFT) to train graphic generation models. OFT provides higher quality (0.708 vs LoRA's 0.697) and less memory consumption (9.01Gb vs LoRA's 9.97Gb) according to their benchmarks
The article: https://huggingface.co/blog/peft-beyond-lora
They invite developers to try and customize their PEFT library and training methods: https://github.com/huggingface/peft/tree/main/method_comparison#creating-new-experiments
Also PEFT methods could be applied to different types of models, not only image generating one, so it could worth to know about and experimenting with it
Unsloth: DiffusionGemma runs on 2000 tokens/second speed
According to Unsloth.ai's post on HuggingFace, they increased local inference up to 1.8x times on local machines consuming 18GB RAM
Post and video by Unsloth: https://huggingface.co/posts/danielhanchen/743111551583952
New QUERY method is about to join GET, POST, PUT, DELETE and PATCH and become part of HTTP standard 🎉
URL: https://www.rfc-editor.org/info/rfc10008/
New method named QUERY would receive data from a server with a data sent in request body but unlike POST would not mutate server's data. All the details are in the RFC draft text
Actually it's quite unexpected after years of silence. It felt like HTTP is in a low maintenance mode. But here it is the new method!
Open Knowledge Format has just been announced as a new Knowledge Base format for AI agents made by Google
It's based on a simple idea by Andrej Karpathy just to put everything in a Wiki (read about llm-wiki.md here)
What Google engineers propose is to put everything into a folder named bundles with cross-linking markdown files. Producers should create wiki-bundles and consumers turn them into something else like a website or a PDF, etc.
Any agent can use it. As the standard doesn't specify special tools. Actually it's pretty simple and proposes mostly the format and the way to organize things
I made a Yes/No classification dataset. And it has been downloaded 49 times
It contains two rows and has classification for "Yes" and "No" only. I think it actually has some degree of usefulness, while being extremely simple, and now I'm deciding to develop it into something more sophisticated and practical
Could you share your experience of dataset creation? What's your journey and way to build a good dataset? What could be the way to develop it further?
Google has release API for pkg.go.dev to support AI coding
The API is in beta stage. Now it allows to:
- List packages and modules
- Search modules
- List vulnerabilities
- Inspect package symbols (functions, structs and variables exported by the package)
Google adapts Go for AI-coding era and to compete with other platforms and languages
UPD. The release has been postponed
>We have identified an issue in our OSX Release builds related to the Temporal API. Until this issue is solved, we won't be able to release Node.js v26.0.0. Therefore, I'm postponing this release (again) to May 4th (next Tuesday)
RafaelGSS commented
Announcement
The latest release of Node.js (v26.0) is full of small improvements, and bug fixes of different severity, and tweaks here and there across the modules and core. Even the upgrade of V8 to the version 14.6 is nothing big. There are module version changes to match with Electron, so some native modules would require rebuilding, it means that for those who uses native modules it probably would be useful to test them against the new Node.js before upgrading
The promised cherry. The most notable thing is the removal of --experimental-transform-types flag, so now TypeScript is not experimental nor optional. Since default support of TypeScript since the v25 it's only a symbolic change
Here are some of the changes:
- update V8 to v14.6.202.33
- update NODE_MODULE_VERSION to 147
- Temporal API is enabled by default. Also it has been improved with V8's update
- Upsert proposal support: map.getOrInsert() and map.getOrInsertComputed()
- Iterator concatenation: Iterator.concat()
- better Rust support, from crate's CLI flags to ENV variables
- sqlite: enabled percentile extension required for statistics with such functions as median and percentile
Seems like the biggest changes are about to be made to the next LTS release
The project has codename Pacquet. Its a rewrite to Rust after the fresh release of the v11. Don't expect it soon though. There is no clear schedule behind the rewrite. What's might be interesting the Rust version was abandoned for about 2 years and now the development has restarted
For those of you who might not know, PNPM is a notable game-changing package manager for Node.js. It stores dependencies once using hardlinks and doesn't download things twice when you start a new project with the same or similar structure. It would download newer versions of the packages if there are and the new ones. It's very space efficient and fast
With the latest Vite 8's Rust overwrite, it seems obvious that Rust has become the favorite language of the Webdev community and I'm curious what would be the next project to migrate