u/DSByTitleOnly

OpenAI Is Pissing Off a Bunch of Mathematicians—Again
▲ 237 r/mathminds+1 crossposts

OpenAI Is Pissing Off a Bunch of Mathematicians—Again

>[C]ompany representatives assured attendees [OpenAI] would not release the solutions all at once—an assurance OpenAI spokesperson Lindsay McCallum says the company is “not aware of”—and, worried by how the community might react, sought advice on how to publish the findings.

>"We are working to responsibly release the next math results from our model, drawing on advice and public recommendations from the Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study to inform how we release these results. We have not set a release time,” said McCallum.

Of course, just hours after this article was published, OpenAI dumped 377 "proofs" on github. But don't worry about the burden created by dumping 377 slop papers on weary mathematicians laps for them to make sense of. OpenAI has it covered:

>To promote scientific transparency and openness, we are also publishing additional details about how we obtained the results in the repository. These include 10 summaries of the model’s reasoning, estimations of compute spent in terms of Pro usage on ChatGPT, and statistics about the number of attempted problems.

Wow, that will be really helpful! And they didn't even mention that they also included 10 (excerpts) of the original prompts!

https://archive.ph/m51FF

Another take on the whole mess: https://gizmodo.com/openai-dumps-377-new-math-results-on-github-publishes-hand-wringing-blog-post-2000822613

wired.com
u/DSByTitleOnly — 7 days ago

OpenAI didn't prove Navier-Stokes, they contrived a narrow, impractical counterexample

OpenAI claims they resolved the Millennium Prize problem with a "proof." This is misleading; what they actually provided was a specific "disproof by counterexample". I'll let Terence Tao, who laid out a pretty explicit roadmap for finding a Navier-Stokes counterexample in this paper, explain the limits of such a result:

>It is worth pointing out, however, that even if this program is successful, it would only demonstrate blowup for a very specific type of initial data (and tiny perturbations thereof), and is not necessarily in contradiction with the belief that one has global regularity for most choices of initial data (for some carefully chosen definition of “most”, e.g. with overwhelming (but not almost sure) probability with respect to various probability distributions of initial data).

What we're not getting are new techniques, new physics tacked on to the Navier-Stokes equations, or anything we can reproduce and study in the lab. OpenAI's result might be quite the feat, even if it was part of a ruthless attempt to scoop a competitor, burned many tens of millions of dollars, and relied on intellectual theft from the many scientists who have worked on this and similar classes of counterexamples. But, like OpenAI's other recent math results, it's not likely to have much effect beyond increasing the company's IPO chances and disheartening the mathematics community.

reddit.com
u/DSByTitleOnly — 1 month ago

I'd be surprised if they didn't "solve" ARC-AGI-3.

Between the many billion-dollar AI data curation companies (ScaleAI alone has an almost 30 billion valuation) and the scores of domain researchers employed by the frontier model makers, and with ARC-AGI-3 being out since late March, it's more than plausible that OpenAI trained Astra on orders of magnitude more games than exist in ARC's public training and private evaluation sets.

There are even open-source datasets with ARC-like puzzles that show just how many answer keys can be created for cheating on the test:

I'd wager that Astra will score poorly on ARC-AGI-4 when it comes out. A better benchmark would be for someone to finally show Ed a useful product they built with an LLM.

u/DSByTitleOnly — 1 month ago

Prompt injection attacks reduced to 0.00000000000% on Claude Auto Mode

The title might be perceived as rage bait but I hope the sarcasm comes through with so many zeros.

Claude-Code-Auto-Mode has mitigation "layers" that "stack" so well as to reduce indirect prompt injection to "~0," according to Boris Cherny and based on test attacks run by 3rd party vendor Trajectory labs. Those tests, on 72 scenarios, had an impressively low 0.00% success rate at impressively false precision (1/72 is already 1.39%).

Security researcher wunderwuzzi (the link) came up with a dead simple attack that got auto mode running Opus 5 to install and run real malware. Worth a read, but here's the author's TLDR (emphasis/zeros on the 80% mine):

In A Nutshell

I got attack success rates up to 80.00% using a small sample size.

The attack chain is as follows:

  1. First, we nudge Claude from using the WebFetch tool into using curl directly
  2. Redirects it to a ZIP archive with files in a special encoding, there is also a native decoder
  3. Claude correctly refuses to execute the binary and writes its own Python decoder instead
  4. But it runs that decoder inside the attacker-controlled directory (unzipped archive)
  5. There a malicious struct.py shadows Python’s standard implementation
  6. So, when Claude imports the base64 module it triggers the poisoned struct.py, and
  7. BOOM.

wunderwuzzi notes that the attack could have been prevented by Anthropic if they ran python with the -I (isolated mode) flag. I can't help but be reminded of what the fix was on Hugging Face's end to patch the OpenAI attack vector: a single line of code

>record_batch.validate(full=True)

There is a narrative going around right now that the models are getting so powerful and so scary and many people are shitting their pants about it. But I think it's worth considering the other extreme that the labs are just running monkeys with terminals and autocomplete against their own unguarded targets.

embracethered.com
u/DSByTitleOnly — 2 months ago