What do you do with your sealed test set if you find a bug after validating your model against that test set

I have a model that I've been tuning on various train-validate splits, picked the best one and then evaluated against a sealed hold out set. While checking the results I noticed something off, and after reviewing my code I noticed a small bug in my code.

But now I'm not sure if I'm allowed to use the same sealed test to report my numbers. Because I used the results of the last run to improve my model, which introduces a small bias. If I had a different test set I probably would not have noticed the bug.

The bug in question was an off-by-one indexing error in a data ingestion stage.

I plan to write one sentence in my paper for transparency, but not sure if the editors/reviewers will flag this as a concern.

reddit.com
u/PhilosopherNext1448 — 11 days ago

Is the majority of research vibecoded these days?

I recently got to the final round for a research engineer position at Google. They rejected me in the end and the feedback I received that I've been out of touch with research for too long. I finished my PhD almost ten years ago and since then did various data science / software engineering / AI engineering stuff.

So the past few months I've been reaching out to old colleagues and people I've met at conferences to see if we can collaborate on a few research papers. I joined three projects with different people, and I'm quite shocked at that everyone is offloading everything to AI with barely any human intervention. All three projects followed basically this premise:

  1. Ask AI to do deep research, and come back with ten ideas for a research paper, with the disclaimer that it should focus on easy gaps to fill.

  2. Ask AI create a detailed spec with hypothesis, stages and gates. Use AI to refine and validate.

  3. Ask AI to implement each stage. Ask AI to review the code and if it matches the requirements in the specs.

  4. Ask AI to run experiments (our experiments are just training/finetuning/optimizing agents on expensive GPUs).

  5. Ask AI to review the results and create pretty graphs.

  6. Ask AI to draft a paper.

Then you notice that a lot of shit in the paper don't make sense and you end up spending a week asking Claude to fix bugs, rerun experiments, redo analysis,... etc.

The whole process from ideation to submitting takes 1-2 weeks. I asked my co authors if this is normal and they all said that everyone is doing this these days.

Is that the status in academia now, or was I just unlucky with the people that I got involved with.

reddit.com
u/PhilosopherNext1448 — 2 months ago

advice on reviewing code written in languages you know little about

I have 15 years of experience in Python, C# and C++ and recently joined a team that writes code exclusively in Go (they made a full switch from python to go shortly after I signed the contract).

Everything is written by Codex, and all the team does day in day out is prompting and pushing out new pull requests every few hours. I got hired as a senior engineer and while I can definitely help out with system designs, architecture, sprint plannings, stakeholder management I'm a bit at a loss when I'm asked to review my colleagues code. Apart from the fact that the pull requests are way too big (1000-2000 line changes), and mix multiple features, bug fixes and refactors together, there's an expectation to have it reviewed within a few hours because a PR that sits still for a day cannot be merged anymore because of too many merge conflicts.

So I'm trying to address the review culture a bit, but I'm more struggling with how to approach code reviews in this genAI age when you're not that familiar with the language itself.

Any ideas or experience would be helpful :)

reddit.com
u/PhilosopherNext1448 — 2 months ago

How likely are you to be reconsidered at google if you manage to go through all the rounds, but did poorly only on coding?

So I went through all the technical, behavioral and cultural rounds for a research engineer position at Google, but I did quite poorly on my coding interview. I didn't get an offer and the feedback was that I did very well in every round, but that the coding was insufficient. I received the advice to focus on that and try again in a year. The recruiter told me that if he comes across an opening that he thinks is a fit for me he will definitely reach out to me earlier, so I shouldn't lose all hope.

But I wonder if he was being polite or not. When I search online I didn't really see cases where people get a 2nd opportunity to redo the coding interview on a shorter notice. Any have any experience or heard of similar stories?

Regarding the coding interview: it was fairly simple where I had to do various array manipulations while the requirements and features kept changing. At one point I did a binary search on a 2D grid, by flattening the array but then they wanted me to do it without using additional memory. So I said easy, I just map (i,j) coordinates to a scalar i*N + j, do binary search on that and map it back to (i,j). And then I spent 20 minutes fumbling with modulo and div operations and turned everything into one big mess. very sad.

reddit.com
u/PhilosopherNext1448 — 2 months ago

I accidentally threw out something very expensive in an underground container. Any chance I can get it back? :')

I was cleaning the house and threw out a box of what I thought was trash, but turns out my wife kept a lot valuable stuff inside. I threw it in the underground container next to my house. The municipality is unavailable till Monday and I don't know what to do.

update: apparently its really easy to open the side door even if you don't have a D-sleutel. just a screw driver works fine. I lowered a TikTok ring-light, and I actually saw what I threw out (bottom left). I used a leaf scraper to pick it up, but I dropped it and the contents fell out. Will ask neighbors tomorrow morning to see if someone has something a bit longer or more friendly than a tuinhark. And in case someone thinks I will climb inside: I don't think that's a good idea.

update 2: with the help of my neighbor I climbed in and out. I put some cardboard on the edge of the entrance and slided in, feet first face down. Dropping yourself is a little bit scary because its 2m deep. To climb out I used a small step ladder that i balanced on top of trash. In and out took about 5 minutes.

update 3: (in case anyone finds themselves in a similar situation, if you call 14070 and go through the entire tape of options, you can get through an emergency line, which connects you to the police. They can send someone over to help even if its weekend, but in my case it wasn't emergency enough.)

reddit.com
u/PhilosopherNext1448 — 2 months ago

Woning verhuren voor 1 jaar terwijl ik in het buitenland zit

Ik heb een tijdelijke baan gekregen in het buitenland en ga binnenkort verhuizen. Ik heb mijn huis recent gekocht en vind het zonde hem weer te verkopen wetend dat ik over een jaar waarschijnlijk terugkom.

Ik dacht om mijn woning tijdelijk te verhuren, maar als ik in het buitenland woon wordt het misschien lastig met sleuteloverdracht, contracten ondertekenen of als er een keer iets stuk gaat in het huis. Ik heb goed en regelmatig contact met de buren, dus ik zat te denken om hen te vragen om als huisbaas te spelen terwijl ik in het buitenland zit, en als vergoeding een deel van de ontvangen huur met hun te delen als compensatie.

Mijn partner vindt het echter onzin want het enige wat de buren moeten doen is aan het begin van het jaar een sleutel af te geven, en als wij een jaar later terug komen doen we een inspectie voordat we de sleutel terugvragen van de huurders.

Wat denken jullie? 0 euro en een mooie fles wijn als we terug zijn, of iets van 10-20% van de huur?

reddit.com
u/PhilosopherNext1448 — 3 months ago

waterlelies planten in kanaal?

hoi! Wij wonen in een buurt waar iedereen's tuin tegen een kanaal aan ligt. Een aantal straten verderop ligt een kanaal en die zit vol met waterlelies en ik dacht ziet er mooi uit. Ik vroeg me af of iemand ervaring heeft met het aanplanten van water lelies in openbare kanalen? Ik vermoed dat dit niet mag, maar tegelijkertijd, we hebben geen boot verkeer want het is een gesloten systeem en ik verwacht niet dat de gemeente ooit ons kanaal op komt. De bodem van het kanaal is ongeveer 50-100cm diep.

reddit.com
u/PhilosopherNext1448 — 3 months ago

for the first time in my ten year career, I had to implement a binary expression tree and because of leetcode practice I did it by hand in about 15 minutes

title.

For context, I'm part of a quantum software team and most of my engineering time is spent on hacking compiler optimizations. I needed to optimize the number of assembly instructions when a user tries to compile a quantum program that contains simple arithmetic. It's simply moving minus signs from a register to an immediate.

The solution consisted of building a binary expression tree, sort the sub trees involving multiplications, and a stack.

thanks leetcode

reddit.com
u/PhilosopherNext1448 — 3 months ago

What is a good rule of thumb of how often you should remind/email recruiters regarding updates in the interview process?

I've been in an interview process for almost three months now, and I get moved to the next stage once in a while. I sometimes send emails to the recruiter with questions about the status, or when i get moved to the next stage send questions on how to prepare because I get invited to "talk to a senior research engineer from the XYZ department", but I don't know how to prepare.

I've sent 5 emails by now, 2 times asking for an update and 3 times clarification on what the next interview round holds, but never got any answer. Is there a scenario where I might come across as too pushy which reduces my chances of getting hired?

reddit.com
u/PhilosopherNext1448 — 3 months ago

boekjes/woordenlijsten om woorden met de letter "r" te oefenen

Hoi! Ik spreek redelijk Nederlands, maar woorden en namen met de letter "r" gaan me soms moeilijk af. Vooral als ze in het midden van een woord staan. Adriaan, Juriaan, schrijven, Utrecht, .. noem maar op.

Ik heb een aantal boekjes thuis, maar die gaan vooral op spelling/grammatica en woordenschat. Ik zoek iets oefeningen die ik elke dag kan doen die er voor zorgen dat mijn "r" meer Nederlands wordt, maar ook die voor zorgen dat mijn mond minder in de knoop raakt.

reddit.com
u/PhilosopherNext1448 — 4 months ago
▲ 25 r/Klussen

Tips om vlonderdek onkruidvrij te houden?

Wij hebben in onze achtertuin een vlonderdek liggen, en ieder jaar is het nogal gedoe om al het onkruid tussen de planken eruit te trekken. Het dek is ruim ruim tien jaar geleden aangelegd bovenop een worteldoek. Maar ondertussen is dat worteldoek vol met gaten, en genoeg grond tussen het doek en dek gekomen dat onkruid z'n gangetje kan gaan.

Het is een dek van 30 vierkante meter en ik heb niet echt zin om alles eruit te halen, een nieuw doek neer te leggen en het dek weer op te bouwen.

Ik zat te denken om misschien siliconenkit tussen de planken te doen, maar weet niet zeker hoe weersbestendig dat is.

Iemand goeie ideeen?

(foto is van nadat ik onkruid eruit heb getrokken)

u/PhilosopherNext1448 — 4 months ago

I failed the coding interview at my dream job and now I'm sad

I won't mention the company because of the NDA, but it's one of the few that do things on the intersection of quantum and AI.

We started with an ML fundamentals, which went quite okay. About 30 questions in a 1 hour interview. 80% definitions: what is precision, what is KNN, what is a transformer, etc. And a few that requires a little bit of understanding: how are learnable embeddings trained, how do you train a diffusion model. It was really a rapid fire of random questions and quite tiring, but overall not too difficult. Two questions I didn't get right (based on his facial expression): how do you deal with multimodality in your data (in hindsight, he was asking about multimodal distributions and gaussian mixtures, and not multimodal data (images + text)) and how to deal with out-of-memory during training (i mentioned pruning, smaller batch size, quantization, parameter efficient fine-tuning (lora) + adapters, gradient checkpointing). For the last one it was clear he was waiting for one specific keyword, but no clue what he was looking for.

Then an hour later we had the coding interview. Throughout the interview I was clueless what the interviewer wanted from me. He introduced the problem by saying that they have a slow database connection that needs to be optimized. For ten minutes I explained you solve this with concurrency, but I kept getting pushed back until I realized he wanted me to optimize another function. The goal was simply, given an array of integers, create an array of ranges. e.g. [1,2,3,6,5,9] => [(1,4), (5,7), (9,10)]. Depending on the number of ranges you do this linearly, or with a binary search. Easy, except that I didn't get that the end points of each range should be exclusive, not inclusive. Throughout the interview I would get new/modified requirements, but I kept misinterpreting those and at the end I was so confused my mind went blank when I needed to write new code.

needless to say, I didn't get the job and they told me they won't consider my future applications for another 12 months. :'(

reddit.com
u/PhilosopherNext1448 — 4 months ago