u/Educational-Text1934

How can one reliably extract Windows a11y tree, that is not just at the desktop level, but also inside app whatever is rendered as a part of the UI?

More over, I intend to safely interact and manipulate with that tree regardless of its size. I do understand that a11y tree sometimes if too big can hung up the system by blocking thread, but there must be some optimised, reliable, way to retrieve, access, and manipulate the a11y tree?

I am building self-driving computer agents. I am stumbled upon an issue whereby i am not able to accurately move cursor to a correct location on Windows.

Let me explain in detail: Imagine you want you agent to open outlook on your desktop (outlook desktop shortcut) and send an email to someone. It is not able to accurately move to the cursor to correct location. Tried dozens of variations including sending raw un-compressed image, attempted to include parameters like DPI, scale, 2K/4K resolution etc. and yet not able to accurately make it work. Any clues on how to approach this?