title: Disassemble to identify encryption algorithm
body: <p><strong>Goal (General)</strong>
  My ultimate (long term) goal is to write an importer for a binary file into another application</p>
  
  <p><strong>Question Background</strong>  </p>
  
  <ul>
  <li>I am interested in two fields within a binary file format. One is
  encrypted, and the other is compressed and possibly also encrypted
  (See how I arrived at this conclusion <a href="https://stackoverflow.com/questions/9403762/base-64-decode-variation">here</a>).</li>
  <li>I have a viewer program (I'll call it viewer.exe) which can open these files for viewing.  I'm hoping this can offer up some clues.</li>
  <li>I will (soon) have a correlated deciphered output to compare and have values to search for.</li>
  <li>This is the most relevant <a href="https://stackoverflow.com/questions/1607227/encryption-encoding-algorithm">stackoverflow Q/A</a> I have found</li>
  </ul>
  
  <p><strong>Question Specific</strong><br>
  What is the best strategy given the resources I have to identify the algorithm being used?  </p>
  
  <p><strong>Current Ideas</strong>  </p>
  
  <ul>
  <li><p>I realize that without the key, identifying the algo from just data is practically impossible</p></li>
  <li><p>Having a file and a viewer.exe, I must have the key somewhere.  Whether it's public, private, symmetric etc...that would be nice to figure out.</p></li>
  <li><p>I would like to disassemble the viewer.exe using OllyDbg with the findcrypt plugin as a first step.  I'm just not proficient enough in this kind of thing to accomplish it yet.</p></li>
  </ul>
  
  <p><strong>Resources</strong><br>
  full <a href="http://dl.dropbox.com/u/2586482/Mesh%20Data%20Demo/model2.mesh" rel="nofollow noreferrer">example file</a><br>
  <a href="http://dl.dropbox.com/u/2586482/Mesh%20Data%20Demo/model2_base64vertices.data" rel="nofollow noreferrer">extracted binary</a> from the field I am interested in<br>
  <a href="http://dl.dropbox.com/u/2586482/decrypted%20data.zip" rel="nofollow noreferrer">decrypted data</a>  In this zip archive there is a binary list of floats representing x,y,z (model2.vertices) and a binary list of integers (model2.faces).  I have also included an "stl" file which you can view with many free programs but because of the weird way the data is stored in STL's, this is not what we expect to come out of the original file.</p>
  
  <p><strong>Progress</strong><br>
  1.  I disassembled the program with Olly, then did the only thing I know how to do at this poing and "searched for all referenced text" after pausing the porgram right before it imports of of the files.  Then I searched for words stings like "crypt, hash, AES, encrypt, SHA, etc etc."  I came up with a bunch of things, most notably "Blowfish64" which seems to go nicely with the fact that mydata occasionally is 4 bytes too long (and since it is guranteed to be mod 12 = 0) this to me looks like padding for 64 bit block size (odd amounts of vertices result in non mod 8 amounts of bytes). I also found error messages like...</p>
  
  <p>“Invalid data size, (Size-4) mod 8 must be 0"  </p>
  
  <p>After reading Igor's response below, here is the output from signsrch. I've updated this image with green dot's  which cause no problems when replaced by int3, red if the program can't start, and orange if it fails when loading a file of interest. No dot means I haven't tested it yet.</p>
  
  <p><img src="https://i.stack.imgur.com/jUm6I.png" alt="Signsrch results annotated"></p>
  
  <p><strong>Accessory Info</strong>  </p>
  
  <ul>
  <li>Im using windows 7 64 bit</li>
  <li>viewer.exe is win32 x86 application</li>
  <li>The data is base64 encoded as well as encrypted</li>
  <li>The deciphered data is groups of 12 bytes representing 3 floats (x,y,z coordinates)</li>
  <li>I have OllyDb v1.1 with the findcrypt plugin but my useage is limited to following along with this guys <a href="http://www.youtube.com/watch?v=Ve6aSCeKRNQ&amp;feature=plcp&amp;context=C3d9d54eUDOEgsToPDskLiiH8p1Gb8RKuf0fLltwE1" rel="nofollow noreferrer">youtube videos</a></li>
  </ul>