# Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
7
• Sprint:
DB_S14_10, DB_W15_11
• Team:
Data Access and Database

#### Description

Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :

Making your "qserv data" directory independent of where qserv is installed

I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either:

• get all their information about various paths from a single file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or...
• refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS. The two are not mutually exclusive – e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them. Assuming you did that, your development loop may look something like this:   # assuming that qserv-configure.py has already been run  # in ../qserv-run    # do something with qserv-a clone  cd qserv-a  setup -r .  ... do some edits ...  scons    # now do the tests  cd ../qserv-run  ./bin/qserv-start.sh  ... do tests ...  ./bin/qserv-stop.sh    # now switch to qserv-b clone  cd ../qserv-b  setup -r .    # and do the tests again  cd ../qserv-run  ./bin/qserv-start.sh  ... do tests ...  ./bin/qserv-stop.sh   that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time. #### Attachments #### Issue Links #### Activity No builds found. Fabrice Jammes created issue - Fabrice Jammes made changes - Field Original Value New Value Epic Link DM-1047 [ 13839 ] Fabrice Jammes made changes -  Description * Making your "qserv data" directory independent of where qserv is installed I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either: * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if$QSERV_DIR is defined, use that qserv,     etc. This will let you switch binaries by simply setup-ing the new     one with EUPS.       The two are not mutually exclusive -- e.g., all of this logic     could be in etc/paths.cfg.sh, and depending on whether this is a     development build or a "non-EUPS" build, it can either pick up     the paths from the environment or hardcode them.     Assuming you did that, your development loop may look something like   this:       # assuming that qserv-configure.py has already been run     # in ../qserv-run       # do something with qserv-a clone     cd qserv-a     setup -r .     ... do some edits ...     scons       # now do the tests     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh       # now switch to qserv-b clone     cd ../qserv-b     setup -r .       # and do the tests again     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh     that is, as qserv-start picks up the relevant products from the   environment, there's no need to rebuild/reconfigure the qserv-run   directory each time. Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : {noformat} * Making your "qserv data" directory independent of where qserv is   installed     I think this is a big one, and largely independent of EUPS. You have   a problem where you want to use one set of test data potentially with   different qserv binaries (not at the same time, of course). I'd argue   you should refactor the scripts generated by qserv-configure to   either:     * get _all_ their information about various paths from a _single_     file, for example, from etc/paths.cfg.sh. Then you can easily     regenerate just that file when you need to switch to a different     qserv (or zookeper, or what not), or...     * refactor the generated scripts to learn from the environment which     binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS. The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them. Assuming you did that, your development loop may look something like this: # assuming that qserv-configure.py has already been run # in ../qserv-run # do something with qserv-a clone cd qserv-a setup -r . ... do some edits ... scons # now do the tests cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh # now switch to qserv-b clone cd ../qserv-b setup -r . # and do the tests again cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-run directory each time. Fabrice Jammes made changes -  Description Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : {noformat} * Making your "qserv data" directory independent of where qserv is installed I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either: * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if$QSERV_DIR is defined, use that qserv,     etc. This will let you switch binaries by simply setup-ing the new     one with EUPS.       The two are not mutually exclusive -- e.g., all of this logic     could be in etc/paths.cfg.sh, and depending on whether this is a     development build or a "non-EUPS" build, it can either pick up     the paths from the environment or hardcode them.     Assuming you did that, your development loop may look something like   this:       # assuming that qserv-configure.py has already been run     # in ../qserv-run       # do something with qserv-a clone     cd qserv-a     setup -r .     ... do some edits ...     scons       # now do the tests     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh       # now switch to qserv-b clone     cd ../qserv-b     setup -r .       # and do the tests again     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh     that is, as qserv-start picks up the relevant products from the   environment, there's no need to rebuild/reconfigure the qserv-run   directory each time. Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : {noformat} * Making your "qserv data" directory independent of where qserv is   installed     I think this is a big one, and largely independent of EUPS. You have   a problem where you want to use one set of test data potentially with   different qserv binaries (not at the same time, of course). I'd argue   you should refactor the scripts generated by qserv-configure to   either:     * get _all_ their information about various paths from a _single_     file, for example, from etc/paths.cfg.sh. Then you can easily     regenerate just that file when you need to switch to a different     qserv (or zookeper, or what not), or...     * refactor the generated scripts to learn from the environment which     binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS. The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them. Assuming you did that, your development loop may look something like this: # assuming that qserv-configure.py has already been run # in ../qserv-run # do something with qserv-a clone cd qserv-a setup -r . ... do some edits ... scons # now do the tests cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh # now switch to qserv-b clone cd ../qserv-b setup -r . # and do the tests again cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-run directory each time. {noformat} Fabrice Jammes made changes -  Watchers Andy Salnikov, Fabrice Jammes, Jacek Becla [ Andy Salnikov, Fabrice Jammes, Jacek Becla ] Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric [ Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric ] Hide Fabrice Jammes added a comment - At first glance, I think that making init.d scripts using eups environment isn't enough to complete this ticket. Indeed, eups products path are also used in xrootd, mysql-proxy, and client config file :  fjammes@clrlsst-dbmaster-vm:~/src/qserv/admin/templates/server/etc (u/fjammes/DM-1338)$ grep QSERV_DIR * lsp.cf:ssi.svclib {{QSERV_DIR}}/lib/libxrdsvc.so lsp.cf:ofs.osslib {{QSERV_DIR}}/lib/libxrdoss.so my-proxy.cnf:proxy-lua-script = {{QSERV_DIR}}/proxy/mysqlProxy.lua qserv-client.conf:base_dir = {{QSERV_DIR}}

Show
Fabrice Jammes added a comment - At first glance, I think that making init.d scripts using eups environment isn't enough to complete this ticket. Indeed, eups products path are also used in xrootd, mysql-proxy, and client config file : fjammes@clrlsst-dbmaster-vm:~ /src/qserv/admin/templates/server/etc (u /fjammes/DM-1338 ) $grep QSERV_DIR * lsp.cf:ssi.svclib {{QSERV_DIR}} /lib/libxrdsvc .so lsp.cf:ofs.osslib {{QSERV_DIR}} /lib/libxrdoss .so my-proxy.cnf:proxy-lua-script = {{QSERV_DIR}} /proxy/mysqlProxy .lua qserv-client.conf:base_dir = {{QSERV_DIR}} Fabrice Jammes made changes -  Status To Do [ 10001 ] In Progress [ 3 ] Jacek Becla made changes -  Team Database [ 10204 ] Fabrice Jammes made changes -  Story Points 2 5 Hide Fabrice Jammes added a comment - xrootd, mysql-proxy and qserv-czar can now use eups environment, but integration test fails. I may use git bisect to find the issue. Show Fabrice Jammes added a comment - xrootd, mysql-proxy and qserv-czar can now use eups environment, but integration test fails. I may use git bisect to find the issue. Fabrice Jammes made changes -  Summary Make "qserv data" directory independent of where qserv is installed Make QSERV_RUN_DIR able to detect qserv install paths using eups Fabrice Jammes made changes -  Summary Make QSERV_RUN_DIR able to detect qserv install paths using eups Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups Hide Fabrice Jammes added a comment - In init.d scripts: • paths management can be held by eups, • other services parameters, which are nor eups-managed or in services configuration files, are now sources from etc/sysconfig/service-name. Thanks in advance for your help, Fabrice Show Fabrice Jammes added a comment - Hi Andy Salnikov , In init.d scripts: paths management can be held by eups, other services parameters, which are nor eups-managed or in services configuration files, are now sources from etc/sysconfig/service-name. Thanks in advance for your help, Fabrice Fabrice Jammes made changes -  Status In Progress [ 3 ] In Review [ 10004 ] Hide Fabrice Jammes added a comment - Please note that this feature may allow to switch Qserv version using for example setup qserv -t qserv, and then restart Qserv with the same QSERV_RUN_DIR and different Qserv binaries. This should work also for Qserv dependencies (xrootd, mysql-proxy, mysql, scisql). But I think there's a few conditions for this to work (QSERV_RUN_DIR content related to a eups product has to remain compliant when product version change). That's why I propose to test it for a few time, and then document it, when we'll have more user feedback. Thanks, Fabrice Show Fabrice Jammes added a comment - Hi Andy Salnikov , Please note that this feature may allow to switch Qserv version using for example setup qserv -t qserv , and then restart Qserv with the same QSERV_RUN_DIR and different Qserv binaries. This should work also for Qserv dependencies (xrootd, mysql-proxy, mysql, scisql). But I think there's a few conditions for this to work (QSERV_RUN_DIR content related to a eups product has to remain compliant when product version change). That's why I propose to test it for a few time, and then document it, when we'll have more user feedback. Thanks, Fabrice Fabrice Jammes made changes -  Link This issue has to be done before DM-1426 [ DM-1426 ] Hide Andy Salnikov added a comment - I think we need to discuss this more before we could implement something that works. I can see several issues that are not addressed in a proposed change: • in general init.d scripts are not supposed to use the environment (this is why recommended way to run things is via service some-service start) • it looks like we want to satisfy requirement from both development side and production in a single solution, I'm not sure it's the best approach • scripts in run directory may not be even compatible with the qserv release, there is no way to test for that now. Maybe we can talk about this again at Wednesday meeting. Show Andy Salnikov added a comment - I think we need to discuss this more before we could implement something that works. I can see several issues that are not addressed in a proposed change: in general init.d scripts are not supposed to use the environment (this is why recommended way to run things is via service some-service start ) it looks like we want to satisfy requirement from both development side and production in a single solution, I'm not sure it's the best approach scripts in run directory may not be even compatible with the qserv release, there is no way to test for that now. Maybe we can talk about this again at Wednesday meeting. Hide Fabrice Jammes added a comment - Hi Andy, You're point of view is understandable and very reasonable, but I can't see how to to make point 1 and 2 compliant with previous Mario proposal (please refer to ticket description). Point 3 is related to DM-895, and this un-covered feature isn't adressed by current ticket, but it would be good to solve this point soon. As a reminder, the current Qserv configuration tool prototype was quickly re-written in order to enable developers to run integration tests with eups-installed Qserv. So I think it would be good, now that we have some user feedback, to define a better design, compliant with our use-cases. Furthermore, Daniel Wang [X] has also some good ideas to improve Qserv startup scripts. So I think that your proposal to discuss this at our Wednesday meeting is very interesting. Andy, maybe we could meet together before so that we can already propose some interesting proposals at the Wednesday meeting? Tomorrow around 10 a.m. or 2 p.m. would be fine for me. Cheers, Fabrice Show Fabrice Jammes added a comment - Hi Andy, You're point of view is understandable and very reasonable, but I can't see how to to make point 1 and 2 compliant with previous Mario proposal (please refer to ticket description). Point 3 is related to DM-895 , and this un-covered feature isn't adressed by current ticket, but it would be good to solve this point soon. As a reminder, the current Qserv configuration tool prototype was quickly re-written in order to enable developers to run integration tests with eups-installed Qserv. So I think it would be good, now that we have some user feedback, to define a better design, compliant with our use-cases. Furthermore, Daniel Wang [X] has also some good ideas to improve Qserv startup scripts. So I think that your proposal to discuss this at our Wednesday meeting is very interesting. Andy, maybe we could meet together before so that we can already propose some interesting proposals at the Wednesday meeting? Tomorrow around 10 a.m. or 2 p.m. would be fine for me. Cheers, Fabrice Hide Andy Salnikov added a comment - Hi Fabrice, I'm not sure yet I'm going to be at SLAC tomorrow, but if I will I'll let you know. Discussing that with everybody interested at Wed meeting may be even better. Andy. Show Andy Salnikov added a comment - Hi Fabrice, I'm not sure yet I'm going to be at SLAC tomorrow, but if I will I'll let you know. Discussing that with everybody interested at Wed meeting may be even better. Andy. Jacek Becla made changes -  Description Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : {noformat} * Making your "qserv data" directory independent of where qserv is installed I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either: * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if$QSERV_DIR is defined, use that qserv,     etc. This will let you switch binaries by simply setup-ing the new     one with EUPS.       The two are not mutually exclusive -- e.g., all of this logic     could be in etc/paths.cfg.sh, and depending on whether this is a     development build or a "non-EUPS" build, it can either pick up     the paths from the environment or hardcode them.     Assuming you did that, your development loop may look something like   this:       # assuming that qserv-configure.py has already been run     # in ../qserv-run       # do something with qserv-a clone     cd qserv-a     setup -r .     ... do some edits ...     scons       # now do the tests     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh       # now switch to qserv-b clone     cd ../qserv-b     setup -r .       # and do the tests again     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh     that is, as qserv-start picks up the relevant products from the   environment, there's no need to rebuild/reconfigure the qserv-run   directory each time.  {noformat} Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : Making your "qserv data" directory independent of where qserv is installed   I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either:   * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if \$QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS.   The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them.   Assuming you did that, your development loop may look something like this:   {code}     # assuming that qserv-configure.py has already been run     # in ../qserv-run       # do something with qserv-a clone     cd qserv-a     setup -r .     ... do some edits ...     scons       # now do the tests     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh       # now switch to qserv-b clone     cd ../qserv-b     setup -r .       # and do the tests again     cd ../qserv-run     ./bin/qserv-start.sh     ... do tests ...     ./bin/qserv-stop.sh  {code} that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time.
Hide
Fabrice Jammes added a comment -

I've removed the eups environment feature, but kept sysconfig and path.cfg.sh.
What do you think of it please?

Cheers,

Show
Fabrice Jammes added a comment - Hi Andy Salnikov , I've removed the eups environment feature, but kept sysconfig and path.cfg.sh. What do you think of it please? Cheers,
Hide
Andy Salnikov added a comment -

Hi Fabrice, I made few comments in a pull request: https://github.com/LSST/qserv/pull/7
My biggest issue now is that having multiple files in sysconfig directory does not seem to simplify things a lot. I think the idea that Mario suggested was to have all variables defined in one file (paths.cfg.sh) which is sourced by the init.scripts. If someone wants to change some paths then it's easy to find it in that file instead of searching many files for location to change. Can I propose to do this instead:

• merge all variable definitions from all files in /etc/sysconfig into one single file, it does not need to be in sysconfig, having it in /etc is OK, maybe name it differently (qserv-paths.cfg.sh?)
• that file does not need any conditional stuff or any verification that paths exist, it only defines all variables (again, the purpose is to have all variables in one place so that it is easy to change).
• all service-specific stuff and path validation should be in init.d scripts, those ideally should not have any hardcoded paths, only use whatever is defined in qserv-paths.cfg.sh

There is also one issue with lua version number hardcoded into scripts (and it was hardcoded in SConscript files before), we should find a way to determine that when we run qserv-configure (this could go into separate ticket).

Show
Andy Salnikov added a comment - Hi Fabrice, I made few comments in a pull request: https://github.com/LSST/qserv/pull/7 My biggest issue now is that having multiple files in sysconfig directory does not seem to simplify things a lot. I think the idea that Mario suggested was to have all variables defined in one file (paths.cfg.sh) which is sourced by the init.scripts. If someone wants to change some paths then it's easy to find it in that file instead of searching many files for location to change. Can I propose to do this instead: merge all variable definitions from all files in /etc/sysconfig into one single file, it does not need to be in sysconfig, having it in /etc is OK, maybe name it differently ( qserv-paths.cfg.sh ?) that file does not need any conditional stuff or any verification that paths exist, it only defines all variables (again, the purpose is to have all variables in one place so that it is easy to change). all service-specific stuff and path validation should be in init.d scripts, those ideally should not have any hardcoded paths, only use whatever is defined in qserv-paths.cfg.sh There is also one issue with lua version number hardcoded into scripts (and it was hardcoded in SConscript files before), we should find a way to determine that when we run qserv-configure (this could go into separate ticket).
Hide
Fabrice Jammes added a comment -

I tried to take in account all your remarks. Could you please review it again?

Cheers,

Show
Fabrice Jammes added a comment - Hi Andy Salnikov , I tried to take in account all your remarks. Could you please review it again? Cheers,
Jacek Becla made changes -
 Sprint DB_S14_10 [ 92 ] DB_S14_10, DB_S14_11 [ 92, 93 ]
Jacek Becla made changes -
 Rank Ranked higher
Jacek Becla made changes -
 Rank Ranked lower
Hide
Andy Salnikov added a comment -

Hi Fabrice,

I made few comments in the pull request. Basically I do not like what qserv-check-config does:

• having all checks in one place does not make much sense, they should be done where they are needed
• you are probably doing more checks than necessary, you try to avoid the cost of mis-configuration mistake (which I think is rather small) but you introduce potential code management cost instead. I believe some of those checks can be done by the services themselves, not init scripts
Show
Andy Salnikov added a comment - Hi Fabrice, I made few comments in the pull request. Basically I do not like what qserv-check-config does: having all checks in one place does not make much sense, they should be done where they are needed you are probably doing more checks than necessary, you try to avoid the cost of mis-configuration mistake (which I think is rather small) but you introduce potential code management cost instead. I believe some of those checks can be done by the services themselves, not init scripts
Fabrice Jammes made changes -
 Link This issue relates to DM-212 [ DM-212 ]
Fabrice Jammes made changes -
 Link This issue relates to DM-1470 [ DM-1470 ]
Fabrice Jammes made changes -
 Story Points 5 7
Hide
Fabrice Jammes added a comment - - edited

I tried to stick as much as possible to your recommendations. Nevertheless I kept 2 controls related to mysql-proxy, which could be removed in DM-1470.
Moreover, I don't like to much having such duplicated checks in each init.d scripts. Would it be possible to factorize the common ones?
If yes, would two functions be fine: one for checking qserv_run_dir, and one for both qserv_log_dir and qserv_pid_dir?

Thanks,

Fabrice

Show
Fabrice Jammes added a comment - - edited Hi Andy Salnikov , I tried to stick as much as possible to your recommendations. Nevertheless I kept 2 controls related to mysql-proxy, which could be removed in DM-1470 . Moreover, I don't like to much having such duplicated checks in each init.d scripts. Would it be possible to factorize the common ones? If yes, would two functions be fine: one for checking qserv_run_dir, and one for both qserv_log_dir and qserv_pid_dir? Thanks, Fabrice
Hide
Andy Salnikov added a comment -

Fabrice, I'm afraid that we are spending too much time on this simple issue, so please go ahead, implement it in a way that you think is reasonable.

Show
Andy Salnikov added a comment - Fabrice, I'm afraid that we are spending too much time on this simple issue, so please go ahead, implement it in a way that you think is reasonable.
Hide
Andy Salnikov added a comment -

I think it should be OK one way or another.

Show
Andy Salnikov added a comment - I think it should be OK one way or another.
Andy Salnikov made changes -
 Status In Review [ 10004 ] Review Complete [ 10101 ]
Hide
Fabrice Jammes added a comment -

Thanks Andy for your help in improving these scripts.

Show
Fabrice Jammes added a comment - Thanks Andy for your help in improving these scripts.
Fabrice Jammes made changes -
 Resolution Done [ 10000 ] Status Review Complete [ 10101 ] Done [ 10002 ]

#### People

Assignee:
Fabrice Jammes
Reporter:
Fabrice Jammes
Reviewers:
Andy Salnikov
Watchers:
Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric