u/AntiAmericanismBrit

Background: OCI Always Free instances can be reclaimed if CPU is idle, currently defined as below 20% for 95% of any 7-day period. (ARM instances can also be marked as non-idle by RAM usage, but the "buffer-cache counted" loophole has been closed.)

Folding@Home is a distributed science project which Oracle said nice things about during the pandemic, so I assumed it would be a legitimate load to stop my instance being reclaimed.

Folding@Home is however understandably "picky" about which work units can be sent to your hardware depending on its capability. Recently they haven't been sending out enough ARM-compatible, CPU-only, low-core-count work units to keep a 4-core system busy all day, and I don't yet know about a 2-core system as I've not yet downsized. (On x86, Folding@Home stopped getting work on OCI's free-tier 1/8 OCPUs in mid-2024.) So for the ARM, I wanted a script to monitor the situation and to warn me in enough time if the work level drops low enough for me to need to set up something else (World Community Grid perhaps?)

The Python script below reads the logs left by version 8 of Folding@Home to put a lower bound on the time periods when the server has been computing, and works out how many hours you'd currently have left to do something if the computation stopped. You can use the --hours option in a cron job or whatever to report it only if it falls below a threshold.

Regarding teams, I suggest NOT joining any team that gets involved with currency because that might be borderline violating the no-mining policies. This is strictly science for its own sake. (On the other hand you're all welcome to join my "M3GAN's secret lab" Folding@Home team which is just for fun and the team number is 1064908. Or any other fun-only team or no team at all.)

#!/usr/bin/env python3
from optparse import OptionParser
parser = OptionParser()
parser.add_option("--log-dir",default="/var/log/fah-client",help="Location of Folding@Home client (v8) log files")
parser.add_option("--daycount",type=int,default=7,help="Number of days (before now) which count in the 'box idle' calculation")
parser.add_option("--days",type=int,default=7,help="Number of days (before now) to analyse (can be different from daycount if you're curious about earlier activity)")
parser.add_option("--idle",type=int,default=95,help="Percentage of the time over idle-calculation period which, if not computing, would make the box count as idle")
parser.add_option("--hours",type=int,default=0,help="If non-0, suppress output unless the true margin to idleness-counting is less than this number of hours")
options, args = parser.parse_args()
globals().update(options.__dict__)

import os,re,sys,glob
from datetime import datetime, timedelta, timezone, time as dtime

def parse_logs():
    files=sorted(glob.glob(os.path.join(log_dir,"log*.txt")))
    if not files: print(f"No log files found in {log_dir}", file=sys.stderr),sys.exit(1)
    events = []
    for filepath in sorted(files):
        currentDate,prevTime = None,None
        for line in open(filepath,"r",errors="replace"):
            line=line.strip()
            m=re.search(r'Log Started (\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)',line)
            if m:
                dt = datetime.strptime(m.group(1), "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
                currentDate = dt.date()
                prevTime = dt.time()
            elif not currentDate: continue # lines before Log Started?
            m_ts = re.match(r'^(\d{2}:\d{2}:\d{2}):',line)
            if not m_ts: continue
            parts = m_ts.group(1).split(":")
            lineTime = dtime(int(parts[0]),int(parts[1]),int(parts[2]))
            if prevTime and lineTime < prevTime: currentDate += timedelta(days=1)
            prevTime = lineTime
            rest = line[m_ts.end():]
            m = re.search(r'(WU\d+):Completed \d+ out of \d+ steps \(\d+%\)',rest)
            if m: events.append((datetime.combine(currentDate, lineTime, tzinfo=timezone.utc),m.group(1)))
            elif 'Machine state pause' in rest: events.append((datetime.combine(currentDate, lineTime, tzinfo=timezone.utc),None))
    if not events: print("No Completed events found in any FAH log file."),sys.exit(1)
    wu_times = {} ; pauses,lastWid = [],None
    for dt, wu_id in events:
        if wu_id: wu_times.setdefault(wu_id,[]).append(dt)
        else: pauses.append((dt,lastWid))
        lastWid = wu_id or lastWid
    wu_details = []
    for wu_id, times in wu_times.items():
        start, end = min(times),max(times)
        if end > start:
            for dt,wid in pauses:
                if wid==wu_id and start<=dt<=end:
                    wu_details.append((start, dt, wu_id))
                    start=min(t for t in times if t>dt)
            wu_details.append((start, end, wu_id))
    wu_details.sort()
    merged = [] # in case overlapping work units
    for start, end, wu_id in wu_details:
        if merged and start <= merged[-1][1]: merged[-1] = (merged[-1][0], max(merged[-1][1], end))
        else: merged.append((start, end))
    return merged, wu_details

def checkBusy(window_start,now,merged,needHours):
    total_busy = timedelta(0)
    needAchievedFrom = None
    for start, end in reversed(sorted(merged)):
        start = max(start, window_start)
        total_busy2 = total_busy + max(start,min(end,now)) - start
        if total_busy.total_seconds() < needHours*3600 and total_busy2.total_seconds() >= needHours*3600:
            needAchievedFrom = end+(total_busy-timedelta(hours=needHours))
        total_busy = total_busy2
    return total_busy, needAchievedFrom

def main():
    merged, wu_details = parse_logs()
    now = datetime.now(timezone.utc)
    window_start = now-timedelta(days=daycount)
    needHours = (100-idle)*24*daycount/100
    total_busy, needAchievedFrom = checkBusy(window_start, now, merged, needHours)
    hours_busy = total_busy.total_seconds() / 3600
    if not needAchievedFrom:
        print("Unrecoverable!  We are BELOW our needed hours: this box could be reclaimed IMMEDIATELY")
        needAchievedFrom = window_start
    true_margin = (needAchievedFrom-window_start).total_seconds()/3600 # The computations from needAchievedFrom to now are enough to give us needHours, and their non-idleness effect will last from needAchievedFrom to needAchievedFrom + window size (which is needAchievedFrom - window_start + now) and *after* that point it's too late, but *at* that point if we start a computation then every new flop that comes in makes up for one being dropped out of the window, so true_margin is the number of hours before we *have* to start something if for some reason Folding@Home stopped working now and didn't pick up again at all.
    if hours and true_margin > hours: return
    print("FAHv8 log analysis:")
    print(f"Busy {hours_busy:.1f} hours ({(total_busy.total_seconds() / (now - window_start).total_seconds()) * 100:.1f}%) over last {daycount} days ({window_start:%Y-%m-%d %H:%M} to {now:%Y-%m-%d %H:%M})")
    print(f"Needed {needHours:.1f} hours ({100-idle}%), excess {hours_busy - needHours:.1f} hours")
    print(f"We have {true_margin:.1f} hours to start another calculation before box counts as idle{' if we stop now' if true_margin > hours else ''}")
    if not daycount==days: # curiosity branch
        window_start = now-timedelta(days=days)
        total_busy, _ = checkBusy(window_start, now, merged, needHours)
        print(f"\nBusy {total_busy.total_seconds()/3600:.1f} hours ({(total_busy.total_seconds() / (now - window_start).total_seconds()) * 100:.1f}%) over last {days} days ({window_start:%Y-%m-%d %H:%M} to {now:%Y-%m-%d %H:%M})")
    visible = [(s, e, wu) for s, e, wu in wu_details if e >= window_start]
    if visible:
        print(f"\n{len(visible)} work units in this window:")
        for start, end, wu_id in visible: print(f"  {wu_id:6s}  {start:%m-%d %H:%M} to {end:%m-%d %H:%M}  ({(min(end,now)-max(start,window_start)).total_seconds()/3600:5.1f}h)")

if __name__ == "__main__": main()

I'm on an Always Free account taken out in 2022, with a 4-core 24GB ARM instance. I don't actually need 4 cores (it's mostly used for Folding@Home science to stop idle reclamation) and I understand new instances are now limited to 2 cores and they'd like it very much if we can downsize. Being public-spirited I'm happy to downsize if I can do so without risk.

​​The problem is I'm not sure if downsizing an ARM instance is a risk-free thing to do. If the hypervisor just "hands in" the cores I don't want, and reboots the instance without needing to migrate it anywhere else, that should in theory be OK. But what if it turns out to be a more "clunky" script that returns the entire core allocation to the free pool and then puts the request for the reduced number of cores at the end of some queue, meaning I might be hit by an "out of capacity" error and have to write one of those scripts to keep trying for weeks to get my server back (or upgrade to PAYG, face constant anxiety over bill shock due to quiet rule changes and possibly still have to script a way to get my server back).

I wonder if anybody can get someone at Oracle to confirm if downsizing a running instance is implemented as an atomic operation so cannot result in you losing your allocation altogether. When I tried to search for this on Google their Gemini-based "AI mode" chipped in that yes downsizing an instance does risk losing it altogether, although none of its cited sources backed up that claim and I had to give it a point for point rebuttal over two more interactions before it stood down and said it didn't actually know, so it would be nice to get something "on the record" for next time somebody searches about this, and especially if Oracle does actually want hobbyists to downsize they might find them more willing to do so if given more reassurance that there is no risk in losing your allocation.

(Edit: fix typo)