Tempering Expectations for GPT-3 and OpenAI’s API
I was going to give a talk at Thotcon 2020 (pushed back to 2021) about generating fake tweets by refining GPT-2. The whole purpose of my experiment was to see that if OpenAI was right in 
their statement that GPT-2 could be used in nefarious ways. I saw a bunch of tutorials but first I read gwerns tutorial and that made me understand the basics of using GPT-2.<p>If you see Gwerns experiments with GPT-2 you notice that his websites are actually just extremely large samples of text &#x2F; image data. Essentially the design is that the whole website is practically statically generated.<p>Refining on AWS was costing me $100 a day.<p>I also met Shawn (<a href="https:&#x2F;&#x2F;github.com&#x2F;shawwn" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;shawwn</a>) who decided to attempt to refine &#x2F; train large models using TPUs. That sounded interesting but since I figured out that I would waste time trying to understand TPUs because I have a day job 
I instead just bought a Titan RTX.<p>My first experiment was that refinement of GPT-2 with your own data would somewhat work. At first I used Donald Trump because he is a very active person on twitter that I believed that people would have some kind of ability of detecting whether the tweets are fake or not.<p>The above was a bad choice for the following unique reasons.
1. People would generally believe that Trump would say nearly anything.
2. I noticed a weird situation where someone basically duplicated everything that I did had commenters which were apparently much better at figuring out that the person performing the tweets were fake.<p>Since that experiment sort of failed I created an expanded refinement of GPT-2 on general twitter data. It was 200 MB of tweets from <a href="https:&#x2F;&#x2F;www.kaggle.com&#x2F;kazanova&#x2F;sentiment140" rel="nofollow">https:&#x2F;&#x2F;www.kaggle.com&#x2F;kazanova&#x2F;sentiment140</a>.<p>I then repeated the experiment and eventually figured out that some people were really bad at figuring out fake tweets and I found a person who used twitter a great deal would actually perform better. I&#x27;m not sure if its that you tweet a bunch or just read twitter enough. I had a low sample (n=5) where I carried out the test. So my test could be completely biased.<p>The test procedure that I followed was to make a set of 10-20 questions and then have the user pick which one was fake and which one wasn&#x27;t.<p>I was also doing these experiments on a Titan RTX (I&#x27;m thinking about getting a V100 or maybe just another Titan RTX so I can train two models at the same time). I accidentally upgraded the memory to 32GB which worked for a bit but instead you should probably get at least 1 or two multiples of your VRAM so that you can keep your operating system running and having the dataset loaded into memory.<p>Also, during refinement I don&#x27;t think my loss ratio was improving. I think that either I wasn&#x27;t using the system long enough.<p>But, as a conclusion I figured out that there is a huge shortcut that I never considered. It would be MUCH easier to just take any refined GPT-2 model or even use GPT-3 as an API above and then make it LOOK like a tweet. Just adding a hashtag and a t.co link would work. (the funny part is that GPT-2 actually seems to have some notion of a t.co link and will happly generate t.co links that don&#x27;t work. Removing t.co links before refinement would be one way to get this out.<p>I did some of these experiments using Google CoLab initially. But as I used it I got out of memory options. I asked someone at pyOhio who worked for google if there was a way to connect Google Colab to a paid instance. They responded to refer me to Jake Vanderplas <a href="https:&#x2F;&#x2F;twitter.com&#x2F;jakevdp" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;jakevdp</a> . They said no at the time. Then a few months after Google came out with Google Colab Pro. But then I was able to run out of system memory in the high memory instances. I upgraded my Deep Learning Rig with a AMD 3600 and I am now waiting for my 64 GB memory that is in the mail.<p>The latest thing that I have done is use local voice synthesis and recognition so that you can talk to GPT-2 locally. My tutorial is at <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=d6Lset0RFAw&amp;t=2s" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=d6Lset0RFAw&amp;t=2s</a>