Answers.Body: <p>Two main approaches are used for AI to recognize elements and understand the purpose of webpages:</p>
  <p>Computer Vision with Deep Learning: This is the leading technique and relies heavily on deep learning models, a type of artificial neural network particularly adept at image recognition.expand_more  These models are trained on vast amounts of labeled webpage data, allowing them to identify specific elements on a webpage like buttons, menus, or text sections.  Some popular deep learning architectures for object recognition on webpages include Faster R-CNN and YOLO.expand_more</p>
  <p>Natural Language Processing (NLP):  This approach focuses on analyzing the text content of a webpage. By understanding the language used, NLP techniques can classify the webpage's purpose.  For instance, an NLP system might identify keywords related to e-commerce on a webpage, suggesting it's an online store.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-31T07:12:08.327Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T12:03:14.663Z
Answers.LastActivityDate: 2024-05-24T12:03:14.663Z
Answers.UserDisplayName: Emerson Sanchotene
Answers.Body: <p>The branch of AI that is devoted to image-related processing is <strong>Computer Vision</strong> (powered by deep learning, indeed). In particular for this project you probably need to train an <strong>object recognition</strong> model (e.g. Faster R-CNN, YOLO), able to find relevant parts of the websites like buttons and forms. An object detector finds the <em>location</em> (bounding box, i.e. a rect) and <em>class</em> (which kind of object is) of each detection. In alternative, you can look at <strong>semantic</strong> or even <strong>instance segmentation</strong> (look <a href="https://ai.stackexchange.com/questions/40310/what-is-panoptic-segmentation">here</a> for an explanation) model (e.g. Mask R-CNN) that instead provides segmentation masks which can handle, in principle, arbitrary complex shapes.</p>
  <ul>
  <li>To train such a model, you have to collect a dataset of <code>(images, labels)</code>, where the <code>labels</code> comprise: bounding box or segmentation mask of the page's object of interest, and its class (e.g. button, form, video, header, etc).</li>
  </ul>
  <p>I think it would be interesting to train a <em>multi-modal</em> model that at the same time learns from images of webpage, and their source code as well. In this case you need to pair CV with NPL (natural language processing). In order of complexity, code (text in general) can be understood by recurrent neural networks, attention models, and large language models (e.g. BERT, GPT, etc.)</p>
  <ul>
  <li>In this other solution, you have two neural networs: one processes the image, the other the text/code; you have to combine both at some point, then have the output layer.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-29T17:58:32.853Z
Answers.LastActivityDate: 2023-05-29T20:02:30.720Z
Answers.LastEditDate: 2023-05-29T20:02:30.720Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I'm curious to know about the capabilities of AI today in 2022. I know that AI has become pretty good at recognizing things like objects in photos. But what about when it comes to elements in HTML? Would it be feasible to use AI to determine things like:</p>
  <ul>
  <li>Is there a call-to-action? Basically a button or main action that directs the user somewhere. The text in the call to action can obviously contain a variety of different text.</li>
  <li>Is there a form on the page for the user to fill out?</li>
  </ul>
  <p>The last time I tried running a rendered image of a website through image recognition software, such as Google Vision or Amazon's Rekognition, it didn't detect these things, which didn't surprise me. However, maybe there's a better or alternate way, such as using the source code? But the end goal would be to determine if the page is meant to capture leads, and the form elements are <em>some</em> of the criteria we'd be looking for. Maybe this can also be seen as a categorization type of task too.</p>
  <p>As I understand, AI is a broad term. So, if this was a feasible project, I'd also be curious to know what branch of AI would be the correct one to explore.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T20:03:27.443Z
Comments.Text: I think image recognition with screenshots of a webpage is the wrong approach for this sort of task. I think you can leverage the syntax in the source code of the webpage using a custom NLP model to create a much more powerful classification model. I've read a few papers that do this for fraud detection. But could you clarify what you specifically are looking for? A pretrained model? A network structure?
Comments.UserDisplayName: Lars
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T22:28:52.660Z
Comments.Text: Thanks for the feedback. Honestly, I'm not exactly sure what I'm looking for, which is part of the problem. The thing I'm looking to solve is determining if the intent of a page is to gather leads. I had asked something similar to this elsewhere and NLP was also mentioned there.
Comments.UserDisplayName: kenshin9
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-04T13:36:28.967Z
Comments.Text: You could do it with the current set of vision models
Comments.UserDisplayName: Anshuman Kumar
Tags: <neural-networks><reference-request><applications><state-of-the-art>
Title: What are the state-of-the-art AI methods to recognize elements on webpages or the purpose of webpage?
UserDisplayName: kenshin9
ThreadText: # What are the state-of-the-art AI methods to recognize elements on webpages or the purpose of webpage?
  
  (Asked by: username_0 on 2022-03-11)
  
  I'm curious to know about the capabilities of AI today in 2022. I know that AI has become pretty good at recognizing things like objects in photos. But what about when it comes to elements in HTML? Would it be feasible to use AI to determine things like:
  
  * Is there a call-to-action? Basically a button or main action that directs the user somewhere. The text in the call to action can obviously contain a variety of different text.
  * Is there a form on the page for the user to fill out?
  
  The last time I tried running a rendered image of a website through image recognition software, such as Google Vision or Amazon's Rekognition, it didn't detect these things, which didn't surprise me. However, maybe there's a better or alternate way, such as using the source code? But the end goal would be to determine if the page is meant to capture leads, and the form elements are *some* of the criteria we'd be looking for. Maybe this can also be seen as a categorization type of task too.
  
  As I understand, AI is a broad term. So, if this was a feasible project, I'd also be curious to know what branch of AI would be the correct one to explore.
  
  **Tags:** neural-networks, reference-request, applications, state-of-the-art
  
  > **Comments:**
  > **username_1**: I think image recognition with screenshots of a webpage is the wrong approach for this sort of task. I think you can leverage the syntax in the source code of the webpage using a custom NLP model to create a much more powerful classification model. I've read a few papers that do this for fraud detection. But could you clarify what you specifically are looking for? A pretrained model? A network structure?  
  > **username_0**: Thanks for the feedback. Honestly, I'm not exactly sure what I'm looking for, which is part of the problem. The thing I'm looking to solve is determining if the intent of a page is to gather leads. I had asked something similar to this elsewhere and NLP was also mentioned there.  
  > **username_2**: You could do it with the current set of vision models  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-05-24. Score: 0
  
  Two main approaches are used for AI to recognize elements and understand the purpose of webpages:
  
  Computer Vision with Deep Learning: This is the leading technique and relies heavily on deep learning models, a type of artificial neural network particularly adept at image recognition.expand\_more These models are trained on vast amounts of labeled webpage data, allowing them to identify specific elements on a webpage like buttons, menus, or text sections. Some popular deep learning architectures for object recognition on webpages include Faster R-CNN and YOLO.expand\_more
  
  Natural Language Processing (NLP): This approach focuses on analyzing the text content of a webpage. By understanding the language used, NLP techniques can classify the webpage's purpose. For instance, an NLP system might identify keywords related to e-commerce on a webpage, suggesting it's an online store.
  
  > **Comments:**
  > **username_4**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  ### Answer by username_5 on 2023-05-29. Score: 0
  
  The branch of AI that is devoted to image-related processing is **Computer Vision** (powered by deep learning, indeed). In particular for this project you probably need to train an **object recognition** model (e.g. Faster R-CNN, YOLO), able to find relevant parts of the websites like buttons and forms. An object detector finds the *location* (bounding box, i.e. a rect) and *class* (which kind of object is) of each detection. In alternative, you can look at **semantic** or even **instance segmentation** (look [here](https://ai.stackexchange.com/questions/40310/what-is-panoptic-segmentation) for an explanation) model (e.g. Mask R-CNN) that instead provides segmentation masks which can handle, in principle, arbitrary complex shapes.
  
  * To train such a model, you have to collect a dataset of `(images, labels)`, where the `labels` comprise: bounding box or segmentation mask of the page's object of interest, and its class (e.g. button, form, video, header, etc).
  
  I think it would be interesting to train a *multi-modal* model that at the same time learns from images of webpage, and their source code as well. In this case you need to pair CV with NPL (natural language processing). In order of complexity, code (text in general) can be understood by recurrent neural networks, attention models, and large language models (e.g. BERT, GPT, etc.)
  
  * In this other solution, you have two neural networs: one processes the image, the other the text/code; you have to combine both at some point, then have the output layer.
  
  
  ---
  
  