u/error-dgn

So I have made a project which goes to different company websites, and get back the bio/about of people in the teams page.
I am facing an issue there.
Currently I was dealing with dynamic components/modals using the below method:-

admin-ajax.php

- Going to the page using playwright
- Using GET command for all XHR and Fetch and Document on that page.
This was very generic, I did not have to make different concepts for different dropdowns, or sections etc.

But now there is this one site where I am facing issue since the request is AJAX based. What happens is I will HAVE to interact with the picture in order to get the payload for the requests.

I do not want to click on the components, it makes the code very hardcoded, and agentic fails, cuz this pipeline will have to run for MANY companies.

This ajax request works on different payloads of sections AND queries.

And the site contains different section:

Directors | Partners | Investors | Investor Relations | etc.

I want every single person of every single section without making it hardcoded. It makes the code messy.

Sometimes the section is of document type, so I call XHR and Fetch network requests AGAIN in order to get all people. but for this particular page, EVERYTHING is ajax based, its a POST request which demands for the query id and the person id. This asks for the code to be brittle which I cannot afford.

So I have made a project which goes to different company websites, and get back the bio/about of people in the teams page.
I am facing an issue there.
Currently I was dealing with dynamic components/modals using the below method:-

admin-ajax.php

- Going to the page using playwright
- Using GET command for all XHR and Fetch and Document on that page.
This was very generic, I did not have to make different concepts for different dropdowns, or sections etc.

But now there is this one site where I am facing issue since the request is AJAX based. What happens is I will HAVE to interact with the picture in order to get the payload for the requests.

I do not want to click on the components, it makes the code very hardcoded, and agentic fails, cuz this pipeline will have to run for MANY companies.

This ajax request works on different payloads of sections AND queries.

And the site contains different section:

Directors | Partners | Investors | Investor Relations | etc.

I want every single person of every single section without making it hardcoded. It makes the code messy.

Sometimes the section is of document type, so I call XHR and Fetch network requests AGAIN in order to get all people. but for this particular page, EVERYTHING is ajax based, its a POST request which demands for the query id and the person id. This asks for the code to be brittle which I cannot afford.

So I have made a project which goes to different company websites, and get back the bio/about of people in the teams page.
I am facing an issue there.
Currently I was dealing with dynamic components/modals using the below method:-

- Going to the page using playwright
- Using GET command for all XHR and Fetch and Document on that page.
This was very generic, I did not have to make different concepts for different dropdowns, or sections etc.

But now there is this one site where I am facing issue since the request is AJAX based. What happens is I will HAVE to interact with the picture in order to get the payload for the requests.

I would send the site here but I think that would be against the policies of this subreddit...

Please help me out. I do not want to click on the components, it makes the code very hardcoded, and agentic fails, cuz this pipeline will have to run for MANY companies.

This ajax request looks like this:

admin-ajax.php

And the site contains different section:

Directors | Partners | Investors | Investor Relations | etc.

I want every single person of every single section without making it hardcoded. It makes the code messy.

Sometimes the section is of document type, so I call XHR and Fetch network requests AGAIN in order to get all people. but for this particular page, EVERYTHING is ajax based, its a POST request which demands for the query id and the person id. This asks for the code to be brittle which I cannot afford.

Please help me out. Please help me out, I want a generic script which works for all the site. I will focus on cleaning later, No matter how big the dump be, but atleast it should have all the data.

HTTPX_RATE_LIMIT   = 1   # seconds between httpx requests
HTTPX_CONCURRENCY  = 3     # max parallel httpx fetches

please let me know if this is safe, and my IP won't be blocked. I am using the requests on company websites....

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of an open source browser and API Key of an LLM. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of Browser Use and OpenAI's API Key. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of Browser Use and OpenAI's API Key. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of Browser Use and OpenAI's API Key. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of Browser Use and OpenAI's API Key. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.