[MSNoise] Problem with database conection.

[Lukas Preiswerk]

Oscar,

I think “pip install msnoise -U” should do the trick (-U is for —upgrade).

Lukas







2017-06-02 18:55 GMT+02:00 Oscar Alberto Castro Artola
<oscar.cas.art at gmail.com>:
> Dear Christos and Lukas,
>
> first of all thanks for you help! I really apreciate it!
>
> Christos: I have a server with 40 cores, so I think the *-t 20* flag is
> working fine!
>
> Lukas: I tried to do a *pip install msnoise*, but I don't think it is
> actually upgrading MSNoise, the output is:
>
> *Requirement already satisfied: msnoise in
> /usr/local/anaconda/lib/python2.7/site-packages*
> *Requirement already satisfied: setuptools in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: numpy>=1.0 in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: scipy in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: pandas in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: matplotlib in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: statsmodels in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: sqlalchemy in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: obspy in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: click in
> /usr/local/anaconda/lib/python2.7/site-packages/click-4.0-py2.7.egg (from
> msnoise)*
> *Requirement already satisfied: pymysql in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: flask in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: flask-admin in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: multiprocessing_logging in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: markdown in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: folium in
> /usr/local/anaconda/lib/python2.7/site-packages (from msnoise)*
> *Requirement already satisfied: six>=1.10 in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: python-dateutil in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: functools32 in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: subprocess32 in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: pytz in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: cycler>=0.10 in
> /usr/local/anaconda/lib/python2.7/site-packages/cycler-0.10.0-py2.7.egg
> (from matplotlib->msnoise)*
> *Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in
> /usr/local/anaconda/lib/python2.7/site-packages (from matplotlib->msnoise)*
> *Requirement already satisfied: future>=0.12.4 in
> /usr/local/anaconda/lib/python2.7/site-packages (from obspy->msnoise)*
> *Requirement already satisfied: lxml in
> /usr/local/anaconda/lib/python2.7/site-packages (from obspy->msnoise)*
> *Requirement already satisfied: decorator in
> /usr/local/anaconda/lib/python2.7/site-packages (from obspy->msnoise)*
> *Requirement already satisfied: requests in
> /usr/local/anaconda/lib/python2.7/site-packages (from obspy->msnoise)*
> *Requirement already satisfied: wtforms in
> /usr/local/anaconda/lib/python2.7/site-packages (from flask-admin->msnoise)*
> *Requirement already satisfied: Jinja2 in
> /usr/local/anaconda/lib/python2.7/site-packages (from folium->msnoise)*
>
> This is the second time I do *msnoise -t 20 compute_mwcs*, last time it
> worked fine but I needed to change *dtt_lag* to "dynamic". So when I tried
> to rerun* msnoise -t 20 compute_mwcs* with the new parameter this error
> message appears. What I've done is run the process in the morning, all day
> it works fine, but the next morning I always find that error message.
>
> Thanks again!
>
> Oscar
>
> On Fri, Jun 2, 2017 at 8:39 AM, Lukas Preiswerk <preiswerk at vaw.baug.ethz.ch>
> wrote:
>
>> Hi Oscar
>>
>> The problem is that “MySQL server has gone away”, so it can’t update
>> the database after finishing the MWCS job. We fixed this in the newest
>> version, so you should indeed update.
>>
>> I think that updating is done by “pip install msnoise”, and then for
>> each project “msnoise upgrade_db” (see the release notes
>> http://msnoise.org/doc/releasenotes/msnoise-1.5.html ). Tom can
>> clarify if this method I just described is actually the way to go…
>>
>> Best, Lukas
>>
>>
>>
>> 2017-06-02 15:20 GMT+02:00 Christos Evangelidis <cevan at noa.gr>:
>> > Dear Oscar,
>> >
>> > Can you really run 20 instances of the code (-t 20) ? I find the -t 20
>> > rather high, but your machine might support it..
>> >
>> > Regards,
>> >
>> > Christos
>> >
>> > Στις 1 Ιουν 2017 6:01 μμ, ο χρήστης "Oscar Alberto Castro Artola" <
>> > oscar.cas.art at gmail.com> έγραψε:
>> >
>> > Hello,
>> >
>> > I have 10 years of data for 13 stations, I have computed the CCF and now
>> it
>> > is turn to compute the MWCS. When I do *msnoise -t 20 compute_mwcs *this
>> > error appears:
>> >
>> > *Process Process-4:*
>> > *Traceback (most recent call last):*
>> > *  File "/usr/local/anaconda/lib/python2.7/multiprocessing/process.py",
>> > line 258, in _bootstrap*
>> > *    self.run()*
>> > *  File "/usr/local/anaconda/lib/python2.7/multiprocessing/process.py",
>> > line 114, in run*
>> > *    self._target(*self._args, **self._kwargs)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/msnoise/
>> > s05compute_mwcs.py",
>> > line 150, in main*
>> > *    update_job(db, day, pair, jobtype='DTT', flag='D')*
>> > *  File "/usr/local/anaconda/lib/python2.7/site-packages/
>> msnoise/api.py",
>> > line 615, in update_job*
>> > *    job = session.query(Job).filter(Job.day == day).filter(Job.pair ==
>> > pair).filter(Job.jobtype == jobtype).first()*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/orm/query.py",
>> > line 2755, in first*
>> > *    ret = list(self[0:1])*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/orm/query.py",
>> > line 2547, in __getitem__*
>> > *    return list(res)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/orm/query.py",
>> > line 2855, in __iter__*
>> > *    return self._execute_and_instances(context)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/orm/query.py",
>> > line 2878, in _execute_and_instances*
>> > *    result = conn.execute(querycontext.statement, self._params)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/engine/base.py",
>> > line 945, in execute*
>> > *    return meth(self, multiparams, params)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/sqlalchemy/
>> > sql/elements.py",
>> > line 263, in _execute_on_connection*
>> > *    return connection._execute_clauseelement(self, multiparams,
>> params)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/engine/base.py",
>> > line 1053, in _execute_clauseelement*
>> > *    compiled_sql, distilled_params*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/engine/base.py",
>> > line 1189, in _execute_context*
>> > *    context)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/engine/base.py",
>> > line 1402, in _handle_dbapi_exception*
>> > *    exc_info*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/util/compat.py",
>> > line 203, in raise_from_cause*
>> > *    reraise(type(exception), exception, tb=exc_tb, cause=cause)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> sqlalchemy/engine/base.py",
>> > line 1182, in _execute_context*
>> > *    context)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/sqlalchemy/
>> > engine/default.py",
>> > line 470, in do_execute*
>> > *    cursor.execute(statement, parameters)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/pymysql/cursors.py",
>> line
>> > 158, in execute*
>> > *    result = self._query(query)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/pymysql/cursors.py",
>> line
>> > 308, in _query*
>> > *    conn.query(q)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> pymysql/connections.py",
>> > line 819, in query*
>> > *    self._execute_command(COMMAND.COM_QUERY, sql)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> pymysql/connections.py",
>> > line 1037, in _execute_command*
>> > *    self._write_bytes(packet)*
>> > *  File
>> > "/usr/local/anaconda/lib/python2.7/site-packages/
>> pymysql/connections.py",
>> > line 989, in _write_bytes*
>> > *    raise err.OperationalError(2006, "MySQL server has gone away (%r)" %
>> > (e,))*
>> > *OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server
>> has
>> > gone away (error(32, 'Broken pipe'))") [SQL: u'SELECT jobs.ref AS
>> jobs_ref,
>> > jobs.day AS jobs_day, jobs.pair AS jobs_pair, jobs.jobtype AS
>> jobs_jobtype,
>> > jobs.flag AS jobs_flag, jobs.lastmod AS jobs_lastmod \nFROM jobs \nWHERE
>> > jobs.day = %(day_1)s AND jobs.pair = %(pair_1)s AND jobs.jobtype =
>> > %(jobtype_1)s \n LIMIT %(param_1)s'] [parameters: {u'param_1': 1,
>> u'day_1':
>> > '2009-02-16', u'jobtype_1': 'DTT', u'pair_1': 'IG.ARIG:IG.MOIG'}]*
>> >
>> > At the beginning everything works fine, but because it takes a long time
>> to
>> > make the computation (around 5 days), every time  that I come back to see
>> > the results this happens. I am working with MSNoise 1.4 and I don't know
>> > how to upgrade to MSNoise 1.5.
>> >
>> > Does anyone know how to fix this?
>> >
>> > Thanks in advance!
>> >
>> > Oscar
>> > _______________________________________________
>> > MSNoise mailing list
>> > MSNoise at mailman-as.oma.be
>> > http://mailman-as.oma.be/mailman/listinfo/msnoise
>> > _______________________________________________
>> > MSNoise mailing list
>> > MSNoise at mailman-as.oma.be
>> > http://mailman-as.oma.be/mailman/listinfo/msnoise
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>
> _______________________________________________
> MSNoise mailing list
> MSNoise at mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise