Big Data Coming in Faster Than Biomedical Researchers Can Process It
One point: Most Bio type people have not had multi-variable calculus, and many have not had calculus at all. So, it&#x27;s not really that they <i>can&#x27;t</i> process it fast enough, it&#x27;s that their techniques for processing it are stereotypical. They figured out how to do that T-Test (or something else) once, and they stick with it, because they really don&#x27;t know he math behind it.<p>Also, though there is a TON of &#x27;data&#x27; coming in, most of it is not useful. For example, I have a 500Gb file of a stack of .tiff images per fish that I have imaged in a confocal microscope. I have a GFP filter on the scope and therefore only get the green part of the .tiff files exposed, the red and blue are just background noise. Also, most of the image is the dish I have the fishes in. I tickle the fish, they flick their tails, and I see this all in 120fps. Now, I measure how much of an angle the fish made their tails flick, all in 3-D, because that&#x27;s what the scope records in. I have a half TB per fish to comb through, and I have ~20 fishes, say ~10TB. At the end, I get a single graph comparing the fish with some gene to those without it, and I have 10TB of &#x27;data&#x27; left over. Yeah, someone <i>could</i> comb through it all and find something else to look at. But i forgot to record the precise temperatures, the orientation of the fish, the fish that I knew later died, etc. I had that all in my head. And, hey, what do you know?, the p-value is ~.45 and therefore there is no &#x27;real&#x27; difference in the fish and we can&#x27;t include this in a paper. Now all that &#x27;data&#x27; is being kept on a drive on some computer somewhere and is counted towards the budget that the lab has on the shared spaces. It&#x27;s not really &#x27;data&#x27; anymore, in that it is useful to advancing knowledge for anyone (it counts as practice I guess), but it still clogs up space.