Parse the documents using Crawlers WITHOUT ai?
I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).
I am looking for some advice or an open source crawler that I can take inspiration from.
My main issues are:
Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result
- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue
Any advice is greatly appreciated.