Ever since David Moutray introduced me to the concept of cargo cult programming, it’s been running round and round in my head. I’m actually coming to believe that this is a major issue in all walks of life. True or not, we’re absolutely seeing it in the data professionals space. Don’t believe me? OK. A few questions.
Do you work for one of the organizations that absolutely requires NOLOCK on all queries (even INSERT/DELETE/UPDATE)? Why? Because it runs faster? Why don’t you just use READ_UNCOMMITTED for your isolation level? What’s that? Never heard of isolation levels? I’ll bet you also don’t know what is meant by “dirty reads” then either. READ_COMMITTED_SNAPSHOT anyone? In short, you’re guilty of being in a cargo cult. You’ve got this form in your head of how things ought to be, but you don’t have the content that defines that form which will give you the necessary information to make good decisions about your code.
Do you shrink your databases on a automated basis? How about, do you have your databases in FULL recovery, but you don’t have log backups. Instead, every night you switch to simple and then shrink the database again? You’re operating from the bare bones of knowledge of how SQL Server works but without the fundamentals that, again, are driving you towards making uninformed decisions.
Do you do every single possible step of all parts of your database management with the GUI? You too are suffering from a cargo cult disease of form over content that is keeping you from getting to the underlying strengths that T-SQL (in combination with PowerShell) will allow you to automate your tasks, making you a more efficient and knowledgeable DBA.
Do you look at Windows Azure SQL Database with all it’s limitations (“Great Gu! You can’t take a backup”) and declare it unworkable and insane… but you’ve never tested it, you don’t fully understand how it works and what it does, AND most of your databases under management are less than 150 gb in size and you’re not operating under HIPAA requirements? In short, you’re a perfect candidate for saving money and increasing efficiency by taking advantage of WASD. You’re starting to sound like a cargo cultist to my ears.
Do you think that NOSQL means NO gal-darned Ess Q Ell dag-nabit, because that T-SQL set-based stuff is difficult to understand and slowing me down? Do you think that your ORM tool solves every single problem ever presented by your DBA because if you can just have an Information Persistence Layer instead of one of those dirty databases, then that relational dragon will be slain? Conversely, do you absolutely insist that there is no way, ever, that any ORM tool will sully the sainted purity of your databases because… Heck I still don’t understand why people are against ORM tools if they’re properly used. You just might be a cargo cultist (and yes, I realize I’m starting to sound like Jeff Foxworthy, minus the humor).
If you’re answering yes to these questions, or even thinking about answering yes to these questions, you might just be one of those cargo cultists. If any of these questions makes you angry, you are absolutely a cargo cultist. Nothing makes people angrier than having their faith tested.
I also think that this pervasive cargo cult mentality is what leads to my experiences while interviewing hundreds of people. How is it possible that you’ve worked with SQL Server for five or more years, but you don’t know the difference between a clustered index and a non-clustered one or what a recovery model is, or how to do a point in time restore? Why? Because, you’ve learned the basic forms of being a DBA or a database developer or a report writer in 6 months to a year, then you’ve just repeated those ritualistic patterns over and over and over. Heck, most of you who really, really need to get this message aren’t even reading this blog because you don’t even search for things on the internet until your data house is figuratively (or even literally) burning down around your ears.
In short, for every single one of the people who got better at what they did on Saturday, there were a greater number who are still insisting on building the equivalent of a stick and leaf version of an airplane in the hopes that the great cargo gods will deliver to them good performing, protected databases that won’t cause their entire business to go under.
Surely, in the 21st century, with all the means of communication at our disposal there is a way to reach these people, to get into their faces, to scream WAKE UP! But, then again, as I said before, I’m seeing evidence of the cargo cult mentality everywhere. Maybe it’s just fundamental to the human condition, a coping mechanism that we can’t possibly get beyond. But, at least in theory, those of us smart enough to make computers work, surely, surely, we’re capable of seeing just a tad beyond the end of our own noses.
For all three of you that have made it this far, here’s your assignment. You know who the cultists closest to you are. For one of those cultists, find a good SQL Server article (or a programming article or a personal development/community article) that will benefit them. Print it out. Give it to them. Don’t try to send them a link. Hand them an article. Suggest that they pass it on. Ask them to make notes in the margins. We’ll try communicating using the old samizdat approach. Let’s see if that works.
If anyone else has any ideas for how to reach these people, post it in the comments. We have to try to make a difference.
One thing I’ve learned to do is present within my company – send out meeting invites, do demos, show fellow DBAs, developers, data analysts what actually happens when that query is executed, when a GUID is chosen as a clustered index, how NOLOCK on a busy system can give inconsistent results. And show, don’t just tell.
I used to do the same thing at my previous company. Red Gate already does exactly that sort of thing, so I couldn’t even introduce it here.
At work, I have made it a principle that I never, ever say no when someone asks for help. I am never too busy, and I am never working on something else (even if I really am). My fear is that, if I say that I am busy at the moment, they will go away and never, ever come back! This does impact my personal productivity, but because of this, I am the “go to” guy in our data warehouse team.
However, I would dearly, dearly love to help our .NET and web developers with their database problems. Unfortunately, they don’t seem to realize that they HAVE problems. I see their problems, because I am trying to consume (and cleanse) the data that their applications create. I am struggling to find a way to communicate our “back end” problems with out “front end” developers.
Oh, and I should mention that I absolutely believe that programming a .NET application or a web application is MUCH harder that working in T-SQL. I’d LOVE to do an “information exchange” – you teach me ASP.NET and I’ll teach you SQL, but I don’t know how pique anyone’s interest!
Excellent advice David. And your first blog post really. See, it’s not that hard.
Grant, I’d like to think I’m not one of these cultists, but maybe I am. I periodically look at tools and processes and all the rest, but I can’t do it every day or even every week. I suspect there is a time and place for most solutions and I know that we tend to use the ones we know – more than that, we tend to use the ones we know work. Unless you’re from the other camp, the one that will use ANYTHING new because it’s new, regardless of risk or contrary opinions.
In my mind the best thing we – bloggers, pundits, etc – can do is speak candidly about successes and failures, lessons learned good and bad, and try to model the behavior we seek. We won’t change everyone, but we’ll get some!
Looks like I’m the last of the three people that made it that far! One question for you – do you find that it’s people who started out their careers in the IT field that are more prone to “cargo cult” programming, or those (like me) who followed non-linear paths to their current positions (whatever those may be)? Or is career progression completely irrelevant to this phenomenon?
I’m pretty sure you don’t fall into that camp having spent some time around you watching you work. I’m not so much advocating for new, New, NEW. It’s more a question of knowledge. The understanding of what the choices you are making are doing to your systems, the knowledge that there are choices and trade-offs and that “I’ve always done it this way” is not a valid answer. You are not one of those people.
No, I don’t think it’s something from either direction. I’ve seen people credentialed to a fare-thee-well who swore up and down that NOLOCK is magic fairy dust for databases and I’ve seen people who were the most accidental of DBAs just doing the things they were told in a rote fashion because they were told to do it that way.
I don’t have a quantifiable measure for how this occurs. Just the sure and steady knowledge that I see it everywhere.
Grant,reading this blog post did two things for me. One, it made up my mind to make a 2 hour drive after work tonight to attend a SQL Server users group meeting. And two, it made me take a long hard look at myself and some of my fellow DBAs in my current job. I’m sure there are cargo cultist in many, if not most, IT organizations. Last week I received the following response to my suggestion of changing some of the way we monitor our SQL Server environment.
Fritch, Again you captured the essence of what we as database professionals have to deal with daily, large divided silos of information. Most of the time the concrete data silos are the easiest to deal with, understand the problem implement the solution, however I find the knowledge silo the most difficult to deal with. Humans tend to cling on to what makes us comfortable, after all how often have you heard “Its what we have always done, we did it this way for years and we are not about to change it now, so can you help us make this go fast but don’t change anything.” I find I spend a vast amount of time engaging in sometimes heated discussions about best practices or where I think the industry is going and I feel I can participate in them because I have been there, and I have done that. I have seen what works and what doesn’t and that is what makes us valuable. Data professionals can’t stop learning and growing, once you do you will become that person in the corner who watches a collection of dashboards about SQL Server health waiting for something to happen -SO not for me!
Thanks for posting on the blog. If you stopped and thought, you’re already out of the cargo cult. Nice job.
Also, I’m not saying that everything needs to be rethought all the time. I’m still a huge advocate of backups and backup testing and will, assuming no extenuating circumstances, be pretty darned conservative in my approach to those. Why? Because it works. And because I’ve used my methods for real emergencies to successfully get back online. It all comes back to understanding why we’re doing something, not just doing it because.
Right there with you. Let’s face it, we’ve seen how easy it is to have one year of experience multiplied 25 times. I absolutely refuse to be one of those guys. You’re the same. So, we won’t be. And that is a very good thing.
Great seeing you at Summit.
After I read a post by Eric Dietrich called http://www.daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner The Rise of the Expert Beginner. I was determined not to become one.
This post adds to that motivation, thank you.
This post is why I took 30 minutes earlier this week walking around to various desks with a paper copy of the #sqlsat265 schedule of sessions talking it up. Unfortunately there is a mindset of any training related to what you do at work should be paid for by your employer.
Yep. 100% an investment in your future. Well done!