[MSNoise] Station list
Thomas Lecocq
Thomas.Lecocq at seismology.be
Wed Sep 21 07:51:06 UTC 2016
Hello both,
Wow ! The dream of a program developer: One user helping another user !
Thanks to both of you, I'll update the doc with this info too!
Cheers,
Thomas
On 21/09/2016 08:38, Crowder, Emily wrote:
> Thank you! This addition to the my.cnf file has finally fixed my problem too!
>
> I was almost about to reinstall Ubuntu 14, but now it seems that MySQL is working like normal with this fix. Thanks!
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Emily Crowder MSc BSc
> PhD Student
> School of Geosciences
> University of Aberdeen
> Aberdeen AB24 3UE
> Scotland
> Em: emily.crowder at abdn.ac.uk
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
> ________________________________
> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of VICIC BLAZ [PHD0900016] <BLAZ.VICIC at phd.units.it>
> Sent: 20 September 2016 19:50:11
> To: msnoise at mailman-as.oma.be
> Subject: Re: [MSNoise] Station list
>
> You were right... i found the solution:
>
>
> If someone else will have this problem.
>
> On Ubuntu 16.04 u need to add this at the end of my.cnf file:
>
>
> [mysqld]
> sql_mode="TRADITIONAL,NO_AUTO_CREATE_USER"
>
>
>
> ________________________________
> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of Thomas Lecocq <Thomas.Lecocq at seismology.be>
> Sent: Tuesday, September 20, 2016 8:08:11 PM
> To: msnoise at mailman-as.oma.be
> Subject: Re: [MSNoise] Station list
>
> Blaz,
>
> The solution I gave worked for me. Ask you sysadmin or someone who knows
> about linux systems, it might be a different setting on different
> flavours of linux... It's not a MSNoise problem, I'll try to come up
> with a workaround later but for now I'm sure you'll find a solution.
>
> Thomas
>
>
> On 20/09/2016 17:29, VICIC BLAZ [PHD0900016] wrote:
>> The problem was actually in PyTables... I reinstalled it with apt-get and it did not work...then i checked with anaconda. it was not installed there, so i installed it, and now it works.
>>
>>
>> Next problem is, as i saw the same, as Emily had last month... was it resolved?
>>
>> i try to stack and i get:
>>
>>
>> Lets STACK !
>> 2016-09-20 18:21:23 [DEBUG] Starting the ref stack
>> []
>> 2016-09-20 18:21:24 [DEBUG] Processing SL_CEY:SL_JAVS-ZZ-2
>> Traceback (most recent call last):
>> File "/home/blaz/miniconda2/bin/msnoise", line 11, in <module>
>> sys.exit(run())
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 614, in run
>> cli(obj={})
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/click/core.py", line 716, in __call__
>> return self.main(*args, **kwargs)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/click/core.py", line 696, in main
>> rv = self.invoke(ctx)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
>> return _process_result(sub_ctx.command.invoke(sub_ctx))
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/click/core.py", line 889, in invoke
>> return ctx.invoke(self.callback, **ctx.params)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/click/core.py", line 534, in invoke
>> return callback(*args, **kwargs)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 301, in stack
>> main('ref', interval)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/s04stack.py", line 203, in main
>> updated_days = updated_days_for_dates(db, start, end, pair.replace('_', '.'), jobtype='CC', interval=datetime.timedelta(days=interval),returndays=True)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/api.py", line 1189, in updated_days_for_dates
>> days = session.query(Job).filter(Job.pair == pair).filter(Job.day >= date1).filter(Job.day <= date2).filter(Job.jobtype == jobtype).filter(Job.lastmod >= lastmod).group_by(Job.day).order_by(Job.day).all()
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all
>> return list(self)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
>> return self._execute_and_instances(context)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances
>> result = conn.execute(querycontext.statement, self._params)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
>> return meth(self, multiparams, params)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
>> return connection._execute_clauseelement(self, multiparams, params)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
>> compiled_sql, distilled_params
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
>> context)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
>> exc_info
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
>> reraise(type(exception), exception, tb=exc_tb, cause=cause)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
>> context)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
>> cursor.execute(statement, parameters)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
>> result = self._query(query)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
>> conn.query(q)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/connections.py", line 835, in query
>> self._affected_rows = self._read_query_result(unbuffered=unbuffered)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/connections.py", line 1019, in _read_query_result
>> result.read()
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/connections.py", line 1302, in read
>> first_packet = self.connection._read_packet()
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/connections.py", line 981, in _read_packet
>> packet.check_error()
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
>> err.raise_mysql_exception(self._data)
>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
>> raise errorclass(errno, errval)
>> sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1055, u"Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'msnoise.jobs.ref' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by") [SQL: u'SELECT jobs.ref AS jobs_ref, jobs.day AS jobs_day, jobs.pair AS jobs_pair, jobs.jobtype AS jobs_jobtype, jobs.flag AS jobs_flag, jobs.lastmod AS jobs_lastmod \nFROM jobs \nWHERE jobs.pair = %(pair_1)s AND jobs.day >= %(day_1)s AND jobs.day <= %(day_2)s AND jobs.jobtype = %(jobtype_1)s AND jobs.lastmod >= %(lastmod_1)s GROUP BY jobs.day ORDER BY jobs.day'] [parameters: {u'day_2': datetime.date(2016, 9, 20), u'lastmod_1': datetime.datetime(2016, 9, 19, 18, 21, 24, 191369), u'day_1': datetime.date(1970, 1, 1), u'jobtype_1': 'CC', u'pair_1': 'SL.CEY:SL.JAVS'}]
>>
>>
>>
>> So, as you mentioned, i added sql_mode=TRADITIONAL line in my.cnf, and the same as for Emily,
>> sudo service mysql restart did not work. i added [mysql] in front of the line that i added, and now restart works, but I get the same error as before when calling
>>
>> msnoise stack -r -m
>>
>> any idea? or should i install older version of mysql?
>>
>> thanks
>>
>>
>> ________________________________
>> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of Thomas Lecocq <Thomas.Lecocq at seismology.be>
>> Sent: Tuesday, September 20, 2016 5:10:46 PM
>> To: msnoise at mailman-as.oma.be
>> Subject: Re: [MSNoise] Station list
>>
>> Hi,
>>
>> you probably checked "keep all" parameter. it's really not properly
>> tested. just avoid it for the moment.
>>
>> Thomas
>>
>>
>> On 20/09/2016 15:50, VICIC BLAZ [PHD0900016] wrote:
>>> Sorry, but there is another problem...
>>>
>>> i ran:
>>>
>>> msnoise -t 2 scan_archive --init
>>>
>>> msnoise new_jobs
>>>
>>> without problems.
>>>
>>>
>>> then i tried with
>>>
>>> msnoise -t 4 compute_cc
>>>
>>>
>>> it starts to work, but after some time it produces this error:
>>>
>>> ...
>>>
>>> 2016-09-20 16:42:18 [DEBUG] Azimuth=258.9
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> File "/home/blaz/miniconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
>>> self.run()
>>> File "/home/blaz/miniconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
>>> self._target(*self._args, **self._kwargs)
>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/s03compute_cc.py", line 683, in main
>>> export_allcorr(db, ccfid, allcorr[ccfid])
>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/msnoise/api.py", line 825, in export_allcorr
>>> df.to_hdf(os.path.join(path, date+'.h5'), 'data')
>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 1101, in to_hdf
>>> return pytables.to_hdf(path_or_buf, key, self, **kwargs)
>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pandas/io/pytables.py", line 259, in to_hdf
>>> complib=complib) as store:
>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/pandas/io/pytables.py", line 389, in __init__
>>> 'importing'.format(ex=str(ex)))
>>> ImportError: HDFStore requires PyTables, "No module named tables" problem importing
>>>
>>> I tried reinstalling PyTables without luck.
>>>
>>>
>>> Thanks!
>>>
>>>
>>> ________________________________
>>> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of VICIC BLAZ [PHD0900016] <BLAZ.VICIC at phd.units.it>
>>> Sent: Tuesday, September 20, 2016 3:43:22 PM
>>> To: msnoise at mailman-as.oma.be
>>> Subject: Re: [MSNoise] Station list
>>>
>>> Thanks, it works now.
>>>
>>>
>>> Blaž
>>>
>>> ________________________________
>>> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of Thomas Lecocq <Thomas.Lecocq at seismology.be>
>>> Sent: Tuesday, September 20, 2016 3:01:07 PM
>>> To: msnoise at mailman-as.oma.be
>>> Subject: Re: [MSNoise] Station list
>>>
>>> Hi,
>>>
>>> You have to import data_structure too :
>>>
>>> from msnoise.data_structures import data_structure
>>>
>>>
>>> otherwise it'll fail (your custom.py file is not valid, fails at import).
>>>
>>> Thomas
>>>
>>>
>>> On 20/09/2016 12:59, VICIC BLAZ [PHD0900016] wrote:
>>>> Thank you.
>>>>
>>>>
>>>> Which brings me to the 2nd question...
>>>>
>>>> I tried to populate with this command, but of course it is not working...
>>>>
>>>>
>>>> What I tried:
>>>>
>>>>
>>>> in the folder
>>>>
>>>> msnoise/data I have put a miniseed file of one station of one day.
>>>>
>>>> Name of the miniseed is in the format:
>>>>
>>>> STA.NET.CHAN.YEAR.DAY
>>>>
>>>>
>>>> as data_folder in settings I have set this folder.
>>>>
>>>>
>>>> in the data_structures.py I added this line:
>>>>
>>>> data_structure['SEI'] = "STA.NET.CHAN.YEAR.DAY"
>>>>
>>>>
>>>> i created a custom.py script:
>>>>
>>>>
>>>> from obspy import read
>>>>
>>>> data_structure['SEI'] = "STA.NET.CHAN.YEAR.DAY"
>>>>
>>>> def populate():
>>>> st = read("*")
>>>> st.merge()
>>>>
>>>> stations = []
>>>>
>>>> for tr in st:
>>>> sta = str(tr.stats.station)
>>>> net = str(tr.stats.network)
>>>> stations.append(str("%s_%s") % (net, sta))
>>>>
>>>> return stations
>>>>
>>>> So,now i have main folder with sub folder data, where I have miniseed file. in the main folder I put custom.py and db.ini and, if I understand right, run msnoise populate from here. In the browser I set data_folder to where my miniseed is and data_structure as SEI.
>>>>
>>>> When i run the populate command, i get error:
>>>>
>>>>
>>>> ()
>>>>>> Populating the Station table
>>>> ()
>>>> Can't parse the archive for format SEI !
>>>> trying to import local parser (should return a station list)
>>>> ()
>>>> No file named custom.py in the /home/blaz/Documents/msnoise folder
>>>>
>>>> (custom.py IS in /home/blaz/Documents/msnoise)
>>>> I hope I was clear enough.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: msnoise-bounces at mailman-as.oma.be <msnoise-bounces at mailman-as.oma.be> on behalf of Thomas Lecocq <Thomas.Lecocq at seismology.be>
>>>> Sent: Tuesday, September 20, 2016 12:56:56 PM
>>>> To: msnoise at mailman-as.oma.be
>>>> Subject: Re: [MSNoise] Station list
>>>>
>>>> Hi,
>>>>
>>>> Yes, this bug was recently reported by another user and I haven't had
>>>> time to fix it.
>>>>
>>>> The best way to add the station currently is to use the "msnoise
>>>> populate" command.
>>>>
>>>> Cheers,
>>>>
>>>> Thomas
>>>>
>>>>
>>>> On 20/09/2016 11:51, VICIC BLAZ [PHD0900016] wrote:
>>>>> Dear all
>>>>>
>>>>>
>>>>> I would like to try to use msnoise. I installed it on Ubuntu 16.04, and as far as I know, everything was installed as planed.
>>>>>
>>>>> Now, I open msnoise admin and try to manually add stations in the browser. I set NET, STA, X, Y, ALT, COORD and click save. When I do that, I get a lot of errors:
>>>>>
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
>>>>>
>>>>> return self.wsgi_app(environ, start_response)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
>>>>>
>>>>> response = self.make_response(self.handle_exception(e))
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
>>>>>
>>>>> reraise(exc_type, exc_value, tb)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
>>>>>
>>>>> response = self.full_dispatch_request()
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
>>>>>
>>>>> rv = self.handle_user_exception(e)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
>>>>>
>>>>> reraise(exc_type, exc_value, tb)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
>>>>>
>>>>> rv = self.dispatch_request()
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
>>>>>
>>>>> return self.view_functions[rule.endpoint](**req.view_args)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/base.py", line 69, in inner
>>>>>
>>>>> return self._run_view(f, *args, **kwargs)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/base.py", line 368, in _run_view
>>>>>
>>>>> return fn(self, *args, **kwargs)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/model/base.py", line 1920, in create_view
>>>>>
>>>>> model = self.create_model(form)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 1028, in create_model
>>>>>
>>>>> if not self.handle_view_exception(ex):
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 1011, in handle_view_exception
>>>>>
>>>>> return super(ModelView, self).handle_view_exception(exc)
>>>>>
>>>>> *
>>>>> File "/home/blaz/miniconda2/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 1022, in create_model
>>>>>
>>>>> model = self.model()
>>>>>
>>>>> TypeError: __init__() takes exactly 9 arguments (1 given)
>>>>>
>>>>>
>>>>>
>>>>> What am I doing wrong?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Blaz
>>>>> _______________________________________________
>>>>> MSNoise mailing list
>>>>> MSNoise at mailman-as.oma.be
>>>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>>> _______________________________________________
>>>> MSNoise mailing list
>>>> MSNoise at mailman-as.oma.be
>>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>>> _______________________________________________
>>>> MSNoise mailing list
>>>> MSNoise at mailman-as.oma.be
>>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>> _______________________________________________
>>> MSNoise mailing list
>>> MSNoise at mailman-as.oma.be
>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>> _______________________________________________
>>> MSNoise mailing list
>>> MSNoise at mailman-as.oma.be
>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>> _______________________________________________
>>> MSNoise mailing list
>>> MSNoise at mailman-as.oma.be
>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
> _______________________________________________
> MSNoise mailing list
> MSNoise at mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
> _______________________________________________
> MSNoise mailing list
> MSNoise at mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
>
>
> The University of Aberdeen is a charity registered in Scotland, No SC013683.
> Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.
> _______________________________________________
> MSNoise mailing list
> MSNoise at mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
More information about the MSNoise
mailing list