Answers.Body: <blockquote>
  <p>can it be generally claimed that the learning on this dataset with the use of GPTs will be more effective than training by other deep learning methods?</p>
  </blockquote>
  <p>No. The only way to find out which would be better for your purposes here is to try both. A very critical factor here will be amount of training data. An LLM may perform better when you have little or no training data, whilst a deep learning model may be far more data hungry but capable of performing better above a certain threshold of data.</p>
  <p>If your use case was closer to natural language comprehension, resulting in language manipulation outputs, then a tuned LLM may be better even with large amounts of training data for deep learning, since the LLMs capabilities may be a good superset of your task. However, the more specialist and unique your data analysis needs to be, the worse a general-purpose LLM will be at working with it, and there are strong limitations on how much you can tune such a model to perform other functions.</p>
  <blockquote>
  <p>What exactly should be considered for this comparison?</p>
  </blockquote>
  <p>Before starting tests, select a metric that measures the utility of results in a way that you care about for the task. Common abstract metrics for classification are accuracy, F1 score, area under ROC. If you can customise a metric to measure true rewards/costs to your project for correct and incorrect classifications, that is best. You may also want to consider cost/benefits of training effort, if there is a limited project budget to create your model, and cost per inference to run the models in production.</p>
  <p>Ensure you have enough correctly labelled test data to train with, and plot learning curves for performance versus amount of training data used. These plots can be used to decide whether collecting more training data would be worthwhile, because hitting a plateau in performance versus training data means that more training data would not be likely to improve a model. Whilst conversely, if doubling the amount of training data improves your metrics noticeably then it may be worth the effort to double it again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T15:03:08.063Z
Answers.LastActivityDate: 2024-01-02T15:03:08.063Z
Answers.UserDisplayName: Neil Slater
Body: <p>Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the dynamic functionality of the file, with a specific format), can it be generally claimed that the learning on this dataset with the use of GPTs will be more effective than training by other deep learning methods?
  What exactly should be considered for this comparison?
  An example of assembly code of a binary file may be this simple code:</p>
  <pre><code>global    _start
  
            section   .text
  _start:   mov       rax, 1                  ; system call for write
            mov       rdi, 1                  ; file handle 1 is stdout
            mov       rsi, message            ; address of string to output
            mov       rdx, 13                 ; number of bytes
            syscall                           ; invoke operating system to do the write
            mov       rax, 60                 ; system call for exit
            xor       rdi, rdi                ; exit code 0
            syscall                           ; invoke operating system to exit
  
            section   .data
  message:  db        &quot;Hello, World&quot;, 10      ; note the newline at the end
  </code></pre>
  <p>And an example of features that a report can present are:
  hash of binary file, sections of file (.text , etc.), strings, network connections, activities carried out in the registry, etc.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T08:24:57.870Z
Comments.Text: Welcome to AI stack exchange. It is not clear what kind of information you are hoping to extract from binary files, and what kind of training data you have access to. Please give a more concrete example of the function(s) you want to compare, including inputs and outputs.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T09:31:23.713Z
Comments.Text: @Neil_Slater Thank you very much, consider the problem in this way that our inputs consist of a set of pairs containing a binary file (the content of the assembly code of a program that is compiled from a high-level language) and a report file corresponding to this binary file. This file describes both the static characteristics of the file and its dynamic characteristics.
Comments.UserDisplayName: user16385455
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T09:32:00.683Z
Comments.Text: @Neil_Slater Static properties can include things like used libraries, system calls, etc. Dynamic properties also include what can be extracted from the execution of the file on a system.
  Generally, GPT will use different parameters for learning. But consider the final goal that we intend to identify the binary files that are in a specific category (after which categories are considered). That is, after learning, when a binary file is given to the model, it will tell in which category it will be placed.
Comments.UserDisplayName: user16385455
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T10:20:21.083Z
Comments.Text: Thanks. Please give more concrete examples to go with this description (e.g. a snippet of the binary file, and of the report, and labels of some of the classes), and use [edit] to insert them into the question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T10:41:24.393Z
Comments.Text: @NeilSlater It has been edited.
Comments.UserDisplayName: user16385455
Tags: <deep-learning><transformer><generative-model><large-language-models><gpt>
Title: Comparing the performances of GPTs with deep learning in the field of binary files and their related reports
UserDisplayName: user16385455
ThreadText: # Comparing the performances of GPTs with deep learning in the field of binary files and their related reports
  
  (Asked by: username_0 on 2024-01-02)
  
  Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the dynamic functionality of the file, with a specific format), can it be generally claimed that the learning on this dataset with the use of GPTs will be more effective than training by other deep learning methods?
  What exactly should be considered for this comparison?
  An example of assembly code of a binary file may be this simple code:
  
  ```
  global    _start
  
            section   .text
  _start:   mov       rax, 1                  ; system call for write
            mov       rdi, 1                  ; file handle 1 is stdout
            mov       rsi, message            ; address of string to output
            mov       rdx, 13                 ; number of bytes
            syscall                           ; invoke operating system to do the write
            mov       rax, 60                 ; system call for exit
            xor       rdi, rdi                ; exit code 0
            syscall                           ; invoke operating system to exit
  
            section   .data
  message:  db        "Hello, World", 10      ; note the newline at the end
  
  ```
  
  And an example of features that a report can present are:
  hash of binary file, sections of file (.text , etc.), strings, network connections, activities carried out in the registry, etc.
  
  **Tags:** deep-learning, transformer, generative-model, large-language-models, gpt
  
  > **Comments:**
  > **username_1**: Welcome to AI stack exchange. It is not clear what kind of information you are hoping to extract from binary files, and what kind of training data you have access to. Please give a more concrete example of the function(s) you want to compare, including inputs and outputs.  
  > **username_0**: @username_1 Thank you very much, consider the problem in this way that our inputs consist of a set of pairs containing a binary file (the content of the assembly code of a program that is compiled from a high-level language) and a report file corresponding to this binary file. This file describes both the static characteristics of the file and its dynamic characteristics.  
  > **username_0**: @username_1 Static properties can include things like used libraries, system calls, etc. Dynamic properties also include what can be extracted from the execution of the file on a system. Generally, GPT will use different parameters for learning. But consider the final goal that we intend to identify the binary files that are in a specific category (after which categories are considered). That is, after learning, when a binary file is given to the model, it will tell in which category it will be placed.  
  > **username_1**: Thanks. Please give more concrete examples to go with this description (e.g. a snippet of the binary file, and of the report, and labels of some of the classes), and use [edit] to insert them into the question.  
  > **username_0**: @username_1 It has been edited.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-02. Score: 1
  
  > can it be generally claimed that the learning on this dataset with the use of GPTs will be more effective than training by other deep learning methods?
  
  No. The only way to find out which would be better for your purposes here is to try both. A very critical factor here will be amount of training data. An LLM may perform better when you have little or no training data, whilst a deep learning model may be far more data hungry but capable of performing better above a certain threshold of data.
  
  If your use case was closer to natural language comprehension, resulting in language manipulation outputs, then a tuned LLM may be better even with large amounts of training data for deep learning, since the LLMs capabilities may be a good superset of your task. However, the more specialist and unique your data analysis needs to be, the worse a general-purpose LLM will be at working with it, and there are strong limitations on how much you can tune such a model to perform other functions.
  
  > What exactly should be considered for this comparison?
  
  Before starting tests, select a metric that measures the utility of results in a way that you care about for the task. Common abstract metrics for classification are accuracy, F1 score, area under ROC. If you can customise a metric to measure true rewards/costs to your project for correct and incorrect classifications, that is best. You may also want to consider cost/benefits of training effort, if there is a limited project budget to create your model, and cost per inference to run the models in production.
  
  Ensure you have enough correctly labelled test data to train with, and plot learning curves for performance versus amount of training data used. These plots can be used to decide whether collecting more training data would be worthwhile, because hitting a plateau in performance versus training data means that more training data would not be likely to improve a model. Whilst conversely, if doubling the amount of training data improves your metrics noticeably then it may be worth the effort to double it again.
  
  
  ---
  
  