Modern Data Practice and the SQL Tradition
Alright here&#x27;s a relevant question I&#x27;ve been having in terms of this. Let&#x27;s say I have the code to gather &#x2F; scrape &#x2F; load some stats into postgres, but then want to run projections on them.<p>For example, if I&#x27;m trying to predict the next day&#x27;s stats by using stats in the past by simply taking the average, how many days in the past should I look at results and take the average of? Is the last 3 day average the best? 5 days? 8? 10?<p>There are clearly a couple ways to do it. One is by getting a data frame of all the stats for all the objects, write the python logic to loop through the days, get back the stats from the past X days not including the current day, taking the average and then storing that back to postgres in a projections table. A function like set_projections(5) where 5 is the number of days in the past I&#x27;m taking the average of.<p>Second way to do this is write that function as a plpgsql function where uses subqueries to find the past X day stats for the players and then creates or updates the projections table all in sql so we can run `select set_projections(5)` and that&#x27;ll do it itself.<p>So the question becomes, which ones is &quot;best&quot;? I have to imagine it&#x27;s mostly a case by case basis. Since it&#x27;s only me here, I&#x27;ve been doing it in postgres alone since it can be done in one query (with multiple sub queries, yes), but that&#x27;s it. With python, it&#x27;d involve many more steps. On the other hand, the sql looks huge and then I&#x27;ve been running into the issue of do I split some of the sub queries into sub functions since that&#x27;s what I&#x27;d be doing in python? If there were more people involved, would it be bad to have larger cases like that in postgres since we wouldn&#x27;t know the skill of the others, where mostly they&#x27;d be coders and could write the projections in languages they&#x27;d want?<p>Another example of this tradeoff is how should I interact with the database? I have a rails background, and ActiveRecord is an incredibly good ORM, whereas SqlAlchemy hasn&#x27;t done it for me. In either case, there&#x27;s a ton of overhead to getting the connections running with the correct class variables. So instead, I kind of created my own ORM &#x2F; interface where I can quickly write queries on my own and use those in the code. This is especially easy since most the queries can be written in postgres functions so the strings of those queries is incredibly tiny!<p>What I&#x27;ve learned from this project I&#x27;m doing is that sql is very, very powerful in what it does. I&#x27;ve shifted more to using it for actions than I would have in the past, and pretty much all thinking I do is making as little code as possible.<p>Anyone make it through reading this and have comments about what I&#x27;m doing and what they like to do?