How to deal with large data sets
So i'm very new to this, forgive my silly questions.
I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job.
I have .csv files containing data for every seconds over a year period. The data consists simlpy of timestamp and valueX.
I need to analyse these values to see if values X goes below a certain value, and if it does I have various calculations to carry out. 650
Value A is proportional to X and just instantaneous, then value B is a counter proportional to A.
Could you suggest a method/ language / software that would be the best and most accessible to do this? I don't have much experience in data analytics/big data apart from using excel and a bit of matlab and python.
thanks in advance
EDIT: Thanks for the replies, my data is like this...
I have a text file for January values.
01/02/2016 00:00:00,<PHONE_NUMBER>
01/02/2016 00:00:01,49.9690056
01/02/2016 00:00:02,<PHONE_NUMBER>
01/02/2016 00:00:03,<PHONE_NUMBER>
01/02/2016 00:00:04,<PHONE_NUMBER>
Different text files for each month, some files have text or notes at the top which I'd need to clean up.
For some context the second value is grid frequency, If it goes below a certain threshold, it constitutes an grid fault. I need to know how long these faults occur for, how often, how far the f value falls etc.
EDIT 2: Thanks for the tips folks, I've been able to read in my data, clean it up and combine it into one dataframe. I'm now try to add some extra column to the dataframe that are dependent of the col 1 value but having some trouble.
"Read in month csv files and create dataframes"
df_jan_short = pd.read_csv('2016_01_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)
df_feb_short = pd.read_csv('2016_02_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)
"Combine dataframes into one dataframe for year" months_short = [df_jan_short, df_feb_short] year_short = pd.concat(months_short)
"Repalce NaN values in df with 50.00 Hz"
year_short.fillna(50.00,inplace = True)
"Check what range freq is in"
year_short['Case'] = 'No Case'
year_short['Case'](year_short['Freq'] > Fc & year_short['Freq'] < Ff ) = 'A' year_short['Case'](year_short['Freq'] > Fa & year_short['Freq'] < Fc ) = 'B'
year_short['Case'](year_short['Freq'] < Fa ) = 'C'
year_short['Case'](year_short['Freq'] > Ff & year_short['Freq'] < Fh ) = 'G'
year_short['Case'](year_short['Freq'] > Fh ) = 'H'
But I get the following error...
runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-74-3a0fbc1f23f6>", line 1, in <module>
runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4. 373