u/hunxai69

so after seeing all the "are AI bots ignoring robots.txt" threads i wanted to actually test it instead of guessing from logs.

the problem with guessing is anyone can set any user agent, and a lot of these fetchers don't come from published ip ranges. so every AI got its own secret url, nothing linked to it anywhere, meaning whatever hit that url had to be that AI. cloudflare worker in front logging everything. then in a fresh chat i asked each one to open the link and tell me a code that's on the page.

claude checked robots.txt before touching anything on a new site. when i disallowed the page it just refused and said it wouldn't try to get around it. kinda polite honestly

chatgpt was the interesting one. first visit to a brand new site it went straight to the disallowed page without reading robots.txt. about 76 seconds later OAI-SearchBot shows up and reads robots.txt. next time i asked, chatgpt refused the page without even making a request. so it learns the rules after the first visit lol

gemini comes in with a user agent that's literally just "Google" and never read robots.txt once. the ips weren't on any of google's published lists either

grok is where it got weird. every request, 9 to 12 different ips hit the page, most within a few seconds, all claiming to be chrome or safari. hosting companies, small isps in brazil, mexico, poland. a couple came through a network registered to oxylabs (the proxy company). 59 ips over 6 requests and it never reused one. never touched robots.txt

perplexity only sent its crawler to read my robots.txt and homepage, then told me the link might be phishing

to be fair openai and google both say fetches a user asks for can skip robots.txt, so those two are doing what they say. and it's a small test, few runs each, one day. but blocking by user agent clearly only stops the bots that tell you their name

also the first time i ran this i opened all the links in my own browser before pasting them, so the only visitor in my logs was me. took me a minute to figure out why every AI apparently used brave on windows

anyone else seeing bursts of chrome/safari traffic from random isps right after someone asks grok about a page?

so after seeing all the "are AI bots ignoring robots.txt" threads i wanted to actually test it instead of guessing from logs.

the problem with guessing is anyone can set any user agent, and a lot of these fetchers don't come from published ip ranges. so every AI got its own secret url, nothing linked to it anywhere, meaning whatever hit that url had to be that AI. cloudflare worker in front logging everything. then in a fresh chat i asked each one to open the link and tell me a code that's on the page.

claude checked robots.txt before touching anything on a new site. when i disallowed the page it just refused and said it wouldn't try to get around it. kinda polite honestly

chatgpt was the interesting one. first visit to a brand new site it went straight to the disallowed page without reading robots.txt. about 76 seconds later OAI-SearchBot shows up and reads robots.txt. next time i asked, chatgpt refused the page without even making a request. so it learns the rules after the first visit lol

gemini comes in with a user agent that's literally just "Google" and never read robots.txt once. the ips weren't on any of google's published lists either

grok is where it got weird. every request, 9 to 12 different ips hit the page, most within a few seconds, all claiming to be chrome or safari. hosting companies, small isps in brazil, mexico, poland. a couple came through a network registered to oxylabs (the proxy company). 59 ips over 6 requests and it never reused one. never touched robots.txt

perplexity only sent its crawler to read my robots.txt and homepage, then told me the link might be phishing

to be fair openai and google both say fetches a user asks for can skip robots.txt, so those two are doing what they say. and it's a small test, few runs each, one day. but blocking by user agent clearly only stops the bots that tell you their name

also the first time i ran this i opened all the links in my own browser before pasting them, so the only visitor in my logs was me. took me a minute to figure out why every AI apparently used brave on windows

anyone else seeing bursts of chrome/safari traffic from random isps right after someone asks grok about a page?

i've spent the last few months talking to equipment lenders, factoring companies, sba brokers and private lenders about what they collect from borrowers. i kept a running list. posting it here because i couldn't find one anywhere when i started looking and it seems dumb to sit on it.

this is business purpose lending, not residential mortgages.

every file, more or less:

photo id for each owner at 20 percent or more

articles of organization or incorporation

operating agreement or bylaws

ownership breakdown by percentage

ein letter from the irs

certificate of good standing

business licence if your trade needs one

w9

beneficial ownership certification

business bank statements, usually 3 to 6 months

merchant processing statements if you take cards

business tax returns, 2 years

personal tax returns, 2 years, for owners at 20 percent or more

irs form 4506c for tax transcripts

year to date p and l and balance sheet

debt schedule of what you already owe

certificate of insurance, general liability and workers comp

signed application and credit pull authorization

voided check or bank details for funding

proof of business address, lease or utility bill

if there's real estate behind the loan, add:

purchase contract, or settlement statement on a refinance

payoff statement from the current lender

appraisal or bpo

scope of work and budget if there's rehab

contractor licence, insurance and w9 if there's rehab

articles of organization or incorporation

operating agreement or bylaws

ownership breakdown by percentage

ein letter from the irs

certificate of good standing

business licence if your trade needs one

w9

beneficial ownership certification

business bank statements, usually 3 to 6 months

merchant processing statements if you take cards

business tax returns, 2 years

personal tax returns, 2 years, for owners at 20 percent or more

irs form 4506c for tax transcripts

year to date p and l and balance sheet

debt schedule of what you already owe

certificate of insurance, general liability and workers comp

signed application and credit pull authorization

voided check or bank details for funding

proof of business address, lease or utility bill

if there's real estate behind the loan, add:

purchase contract, or settlement statement on a refinance

payoff statement from the current lender

appraisal or bpo

scope of work and budget if there's rehab

contractor licence, insurance and w9 if there's rehab

title commitment

insurance binder

proof of funds for down payment and reserves

rent roll and leases for income property

entity docs for the borrowing llc

schedule of past projects

exit strategy, sale or refinance

no lender asks for all of it. the real estate half only shows up if there's property involved. but if you're going in cold, assume the first list.

if you've been through this recently and something here is wrong or missing, say so and i'll fix it.