My biggest mistake as an RPA developer
&quot;Most native RPA automation is GUI based. But let us take a moment so that this sinks in. GUI based automation involves instructing a bot to communicate with another program via the UI. This is analogous to forcing two native speakers to communicate via charades. GUI based automation is always a compromise because there is invariably a more efficient way to perform the same task under the hood. This brings me to my second reason.&quot;<p>The author isn&#x27;t very clear here and seems to be themselves unclear on how these RPA technologies actually &quot;see&quot; an application.<p>Every Robotic Operating Model I&#x27;ve ever seen or worked on has always held firm that &quot;surface automation&quot; (think Citrix Receivers and applications built in Silverlight or Flash) should be outside the scope of any RPA solution.<p>What&#x27;s left are browser or desktop based &quot;physical&quot; applications that actually have an underlying model used to describe and render a GUI. This is actually what is being utilized by (most) RPA clients.<p>While correct that the clients do interact with the UI of a program, depending on the application they actually interpret (or &quot;see&quot; it) via the application COM (component object model), or in the case of a webpage, the DOM (document object model). Since these are essentially used to describe what is rendered on screen -usually in a more detailed way then what is actually rendered on screen- the RPA client is able to efficiently and confidently interact with the application.<p>Take a webpage with a red button to click for example:<p>Automating purely via UI&#x2F;surface automation:
- Capture 150px by 150px at screen coordinates x and y
- Is captured image red with the words &quot;Click me&quot; on it?
- Go to x coordinates on screen
- Go to y coordinates on screen
- Send key &#x27;Right mouse button click&quot;
- Pray<p>Automating via DOM:
- Attach to process `Chrome titled &quot;My webpage&quot;`
- Is element `&lt;button enabled=true id=&quot;superUnique_superDependable&quot; class=&quot;clickMe&quot; onClick=&quot;navigate(MyOtherWebpage)&quot;&gt;Click me&lt;&#x2F;button&gt;` present?
- (to browser client) Send key `Right mouse button click` to element id = &quot;superUnique_superDependable&quot;
- Wait for process `Chrome titled &quot;My other webpage&quot;`<p>OR<p>Automating via DOM:
- Attach to process `Chrome titled &quot;My webpage&quot;`
- Run function `navigate(MyOtherWebpage)`
- Wait for process `Chrome titled &quot;My other webpage&quot;`