title: basic lexical analyzer that reads a text file as input and writes all the "tokens" to an output file line by line
body: <pre><code>     writing a very basic lexical analyzer in standard
      C++ (std usage is allowed) that reads a text file
      as input and writes all the "tokens" to an output
      file line by line
      ● All items that are separated by white space are
      tokens
      ● White space includes
      ● Space(s)
      ● Newlines
      ● Tabs
  
      Input file having this
  
          if (x &gt; 50)
          x=0
  
      Output file should be like this
  
          if
  
          (
  
          x
  
          &gt;
  
          50
  
          )
  
          x
  
          =
  
          0
  
  
  
  Edit 3
  </code></pre>
  
  <p>Ok so im almost there i have made an input file that takes whatever is written in it and then tokenizes it and shows on the screen. an output file is generated bt when it shows this output Output tokens 0033F8C8  i thinks its the array address. now all i need to do is store the output being shown on the screen should go into the output text file and on the screen it should print see your output file your inputs are tokenized.</p>
  
  <p>code edit 3  </p>
  
  <pre><code>#define _CRT_SECURE_NO_WARNINGS
  #include &lt;iostream&gt;
  #include &lt;fstream&gt;
  using std::ifstream;
  
  #include &lt;cstring&gt;
  using namespace std;
  
  const int MAX_CHARS_PER_LINE = 512;
  const int MAX_TOKENS_PER_LINE = 20;
  const char* const DELIMITER = " ";
  
  int main()
  {
      // create a file-reading object
      ifstream infile;
      infile.open("input.txt"); // open a file
      if (!infile.good())
          return 1; // exit if file not found
  
      ofstream outfile;
      outfile.open("output.txt");
      if (!outfile)
      {
          cout &lt;&lt; "error opening fiLe";
      }
  
      // read each line of the file
      while (!infile.eof())
      {
          // read an entire line into memory
          char buf[MAX_CHARS_PER_LINE];
          infile.getline(buf, MAX_CHARS_PER_LINE);
  
          // parse the line into blank-delimited tokens
          int n = 0; // a for-loop index
  
          // array to store memory addresses of the tokens in buf
          const char* token[MAX_TOKENS_PER_LINE] = {}; // initialize to 0
  
          // parse the line
          token[0] = strtok(buf, DELIMITER); // first token
          if (token[0]) // zero if line is blank
          {
              for (n = 1; n &lt; MAX_TOKENS_PER_LINE; n++)
              {
                  token[n] = strtok(0, DELIMITER); // subsequent tokens
                  if (!token[n]) break; // no more tokens
              }
          }
  
          // process (print) the tokens
          for (int i = 0; i &lt; n; i++) // n = #of tokens
              cout &lt;&lt; "Token[" &lt;&lt; i &lt;&lt; "] = " &lt;&lt; token[i] &lt;&lt; endl;
          cout &lt;&lt; endl;
  
  
  
          outfile&lt;&lt; "Output tokens" &lt;&lt;token;
          infile.close();
          outfile.close();
  
  
      }
  
  
      system("pause");
      return 0;
  }
  </code></pre>