[MSNoise] advice on processing database subsets

[Thomas Lecocq]

Hi guys,

Yeah, I have been thinking about a "benchmark" mode for quite a number 
of weeks, i.e. since I tested a first run of PWS in order to compare the 
final dv/v ; to compare properly I have to test quite a number of 
parameters.

My current idea is to run a set of possible parameters, for different 
steps. This would lead to a large number of branches in a large tree, 
but it would definitively be quite interesting.

I am really not in favor of duplicating the database, rather to create  
a "config" file with an caller script, to set/change/ parameters... 
Theoretically, the API should let you do all the actions. The only thing 
that would be a little trickier is to store/reuse the results of each 
step in order to compare them. For info, using the "shutil" module you 
can move/copy files easily.

Let's keep brainstorming on that and see how it goes !

Cheers

Thomas

On 01/05/2016 16:52, Lukas Preiswerk wrote:
> Hi all
>
> I was in a similar situation as Phil, and I used (1). It’s not
> straightforward to copy the database and make msnoise work again in a new
> directory. But it’s definitely possible.
> I actually think it would be a nice addition to msnoise to not only make an
> option for multiple filters, but also for multiple other parameters (window
> lengths, overlaps, windsorizing, etc.). This would really help in the first
> “exploratory phase” to find out what is the best way to process your
> dataset.
> What do you think of this idea? Practically I would implement it by moving
> these parameters (window length etc.) to the filter parameters, and treat
> it in the same way as an additional filter. As far as I understand the
> code, this wouldn’t require many adaptions…
>
> Lukas
>
>
>
> 2016-05-01 11:35 GMT+02:00 Thomas Lecocq <Thomas.Lecocq at seismology.be>:
>
>> Hi Phil,
>>
>> I'd say (3) would be better indeed. You can script msnoise using the api.
>> If you need to change params in the config, you can alternatively use the
>> "msnoise config --set name=value" command.
>>
>> Please keep me updated of your progresses & tests !
>>
>> Thomas
>>
>>
>>
>> On 01/05/2016 10:34, Phil Cummins wrote:
>>
>>> Hi again,
>>>
>>> As some of you may recall, I'm just getting started with msnoise. I have
>>> a large database and have managed to get my station and data availability
>>> tables populated.
>>> At this point, rather than running through the whole database, processing
>>> it with parameters I hope might work, I'd rather process small subsets,
>>> e.g. 1 day at a time, to experiment with window lengths, overlaps, etc., to
>>> find what seems optimal. My question is, what's the best way to process
>>> subsets of my database?
>>> It seems to me I have several options:
>>>      (1) Make separate databases for each subset I want to test, and run
>>> through the workflow on each
>>>      (2) Set start and end times appropriate for my subset, re-scan and
>>> run through the workflow.
>>>      (3) Populate the jobs table, and write a script to activate only the
>>> jobs I want and not the others.
>>> I want to a fair bit of testing using different parameters before I run
>>> through the whole thing, so I think (3) may be best. But any advice would
>>> be appreciated.
>>> Regards,
>>>
>>> - Phil
>>> _______________________________________________
>>> MSNoise mailing list
>>> MSNoise at mailman-as.oma.be
>>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>>
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise at mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
>>
> _______________________________________________
> MSNoise mailing list
> MSNoise at mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise