[MSNoise] populate stations table with script

Sat Apr 30 11:04:17 UTC 2016

Hi Phil,

Glad it worked !

About the response field, don't worry about it, it isn't used. If you 
have a dataless or inventoryXML containing the information for the seed 
id (NET.STA.LOC.CHAN), the instrument response can be corrected. Again, 
this is only needed for tomo, not for dv/v if your stations didn't 
change vs time.

Cheers

Thomas

On 30/04/2016 12:56, Phil Cummins wrote:
> Hi Thomas,
>
> Yeah, that worked! Looks like I'm in business. I would like to know 
> how t do the instrument field also, but for the moment I will be 
> dealing with stations that all have the same response.
> Many thanks!
>
> - Phil
>
>> Thomas Lecocq <mailto:Thomas.Lecocq at seismology.be>
>> 30 April 2016 7:59 pm
>> Hi,
>>
>>
>> not sure without testing, but try passing float(st_inv[0].latitude) 
>> instead of the Latitude object...
>>
>>
>> Thomas
>>
>>
>>
>>
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>> Phil Cummins <mailto:phil.cummins at anu.edu.au>
>> 30 April 2016 11:05 am
>> Hi Thomas,
>>
>> Thanks for your reply. I am getting my station info from a seiscomp3 
>> fdsnws server, so the script I wrote to do this looks like:
>>
>> from msnoise.api import connect,get_stations,update_station
>> from obspy.clients.fdsn.client import Client
>>
>> client = Client('http://compute2.rses.anu.edu.au:8080')
>> stns_inv = client.get_stations(network='XU',station='JK*')[0]
>> msn=connect()
>> stns = get_stations(msn)
>> for st in stns:
>>     st_inv = stns_inv.select(station=st.sta)
>>     if len(st_inv) == 0:
>>         print 'WARNING! Station %s not found in fdsnws' % st.sta
>>         continue
>>     print 'updating %s:lon=%g, lat=%g' % 
>> (st.sta,st_inv[0].longitude,st_inv[0].latitude)
>> update_station(msn,'XU',st.sta,st_inv[0].longitude,st_inv[0].latitude,
>>                    st_inv[0].elevation,coordinates='DEG')
>> msn.commit()
>>
>> It produces the following output/error message, ending in 
>> "AttributeError: 'Latitude' object has no attribute 'translate' ". 
>> Any suggestions?
>>
>> - Phil
>>
>> updating JKA01:lon=106.872, lat=-6.1208
>> Traceback (most recent call last):
>>
>>   File "<ipython-input-93-fca1e24e504a>", line 1, in <module>
>> runfile('/home/seis/pcummins/Indonesia/Jakarta/msnoise/update_stns.py', 
>> wdir='/home/seis/pcummins/Indonesia/Jakarta/msnoise')
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", 
>> line 699, in runfile
>>     execfile(filename, namespace)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", 
>> line 81, in execfile
>>     builtins.execfile(filename, *where)
>>
>>   File 
>> "/home/seis/pcummins/Indonesia/Jakarta/msnoise/update_stns.py", line 
>> 22, in <module>
>>     st_inv[0].elevation,coordinates='DEG')
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/msnoise/api.py", 
>> line 380, in update_station
>>     session.commit()
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 801, in commit
>>     self.transaction.commit()
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 392, in commit
>>     self._prepare_impl()
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 372, in _prepare_impl
>>     self.session.flush()
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 2019, in flush
>>     self._flush(objects)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 2137, in _flush
>>     transaction.rollback(_capture_exception=True)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", 
>> line 60, in __exit__
>>     compat.reraise(exc_type, exc_value, exc_tb)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", 
>> line 2101, in _flush
>>     flush_context.execute()
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", 
>> line 373, in execute
>>     rec.execute(self)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", 
>> line 532, in execute
>>     uow
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", 
>> line 170, in save_obj
>>     mapper, table, update)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", 
>> line 706, in _emit_update_statements
>>     execute(statement, multiparams)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
>> line 914, in execute
>>     return meth(self, multiparams, params)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", 
>> line 323, in _execute_on_connection
>>     return connection._execute_clauseelement(self, multiparams, params)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
>> line 1010, in _execute_clauseelement
>>     compiled_sql, distilled_params
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
>> line 1146, in _execute_context
>>     context)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
>> line 1344, in _handle_dbapi_exception
>>     util.reraise(*exc_info)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", 
>> line 1139, in _execute_context
>>     context)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", 
>> line 450, in do_execute
>>     cursor.execute(statement, parameters)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", 
>> line 156, in execute
>>     query = self.mogrify(query, args)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", 
>> line 135, in mogrify
>>     query = query % self._escape_args(args, conn)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", 
>> line 115, in _escape_args
>>     return dict((key, conn.escape(val)) for (key, val) in args.items())
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", 
>> line 115, in <genexpr>
>>     return dict((key, conn.escape(val)) for (key, val) in args.items())
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", 
>> line 781, in escape
>>     return escape_item(obj, self.charset, mapping=mapping)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/converters.py", 
>> line 26, in escape_item
>>     val = encoder(val, mapping)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/converters.py", 
>> line 109, in escape_unicode
>>     return u"'%s'" % _escape_unicode(value)
>>
>>   File 
>> "/usr/local/anaconda2/lib/python2.7/site-packages/pymysql/converters.py", 
>> line 72, in _escape_unicode
>>     return value.translate(_escape_table)
>>
>> AttributeError: 'Latitude' object has no attribute 'translate'
>>
>>
>>
>>
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>> Thomas Lecocq <mailto:thomas.lecocq at oma.be>
>> 26 April 2016 5:28 pm
>> Hi Phil !
>>
>> Indeed, the populate step actually only lists the stations from the 
>> archive. No files are read at this point.
>>
>> Aurélien has written some updated populate code to read the metadata 
>> information from dataless or inventoryXML , but we still have to make 
>> it clean in order to make it available.
>>
>> Meanwhile:
>>
>> 1/ you can populate "normally"
>> 2/ select - deselect the stations you want in the Station table (if 
>> you are using 1.4: in the Admin).
>> 3/ write a small python script to interact with the API: use 
>> "connect", "get_stations" and "update_station" to update the existing 
>> fields.
>>
>> Let me know how it goes,
>>
>> Thomas
>>
>>
>>
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>> Phil Cummins <mailto:phil.cummins at anu.edu.au>
>> 26 April 2016 8:04 am
>> Hi,
>>
>> I'm new to msnoise, and was unable to find a searchable archive for 
>> this list(?), apologies if the question has been asked before.
>>
>> I want to populate my station table, and msnoise has scanned my sds 
>> database and identified all the stations. However, it has only filled 
>> out the name fields, and I need at a minimum the locations, and would 
>> like to do the instruments as well.
>>
>> So I would like to find a way to populate the stations table using a 
>> python script. Presumably one would use pymysql? I have never used 
>> pymysql, and was wondering if maybe someone had such a script. If you 
>> do, could you please send it to me?
>>
>> (actually, I want to do it this way in any case, since in future my 
>> sds database will have more stations than I want to use with msnoise).
>>
>> Thanks,
>>
>> - Phil
>>
>>
>