Automatic Restart of Regions

Max Ping · **Joined:** Wed Nov 05, 2008 5:26 am **Posts:** 240

Has anybody figured out a script that restarts the regions automatically if it stops responding? Its not always possible to login and do it manually and they do seem to *coughs* crash occasionally

George Darwin · **Joined:** Thu Jul 31, 2008 2:48 am **Posts:** 394

Max Ping wrote:

Has anybody figured out a script that restarts the regions automatically if it stops responding? Its not always possible to login and do it manually and they do seem to *coughs* crash occasionally

I am still working that out...
Found http://opensimulator.org/wiki/User_talk:BlueWall#oswatchdog and it looked workable, but first cron install failed miserably.
However, running the script directly, lightly edited for my own setup, did manage to start the simulator leaving a screen session I could later attach to, so I hold out hope. Also I confirmed that when the simulator crashes/ends, the screen session closes.
Will post when I get a working model of auto-restart.
One thing I noticed was the grep for the screen title. At first glance I think you might throw off the "see if server is still running ..." bit by being logged in to the console w/screen. Further piping thru grep for mono should alleviate this problem. Just like the example given that gets rid of the grep process itself. Then again, if the server isn't running you certainly won't be in server console session. Geez! guess I gotta think these things thru a bit more b4 spouting off

OSwatchdog is looking like a really good starting point

Max Ping · **Joined:** Wed Nov 05, 2008 5:26 am **Posts:** 240

That looks like the kind of thing - the next step would be to rework this so you can run an external "ping" for want of a better word - maybe you have a third server simply monitoring if the regions are up & if it detects that they are stopped to then send an automatic restart request. Like an oswatchdog server

George Darwin · **Joined:** Thu Jul 31, 2008 2:48 am **Posts:** 394

Max Ping wrote:

That looks like the kind of thing - the next step would be to rework this so you can run an external "ping" for want of a better word - maybe you have a third server simply monitoring if the regions are up & if it detects that they are stopped to then send an automatic restart request. Like an oswatchdog server

More complicated than I need right now.
One can always telnet to port 9000 on the server, if there's a responce it's probably up 8-)

If there's no responce, it's time to mourn the poor server, then fire up ssh and go investigate
and reboot the durn thing :cry:

Happy Days!
I did manage to get around to installing OSwatchdog as a crontab in the user account that launches my OpenSim

My first miserable fail was of course my own fault for not {drum roll} rtfm
Yes my friends, even at this late date it still pays to man 5 crontab.
So without further ado, the working code examples.
Here is OSwatchdog -- pretty much as BlueWall published it at http://opensimulator.org/wiki/User_talk:BlueWall#oswatchdog

Code:

#!/bin/sh
#
# THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE CONTRIBUTORS BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
#
#
# Adjust To Your System:
#
# Sample Cron Entry: Runs Each Minute 24/7/365
# */1 * * * * exec /opt/opensim/scripts/oswatchdog
#
# my crontab.ex for vixie crond
# run every 5 mins 
# m h  dom mon dow   command
#*/5 *   *       *     *      /home/pocoloco/Scripts/OSwatchdog

# Where Does Your OpenSim Installation Live?
BASE="/var/opensim"

# Where Do You Run OpenSim.exe From?
RUN="$BASE/current/bin"

# Where Do You Want To Log Events To?
LOG="$BASE/logs"

# Where Is Your Screen Binary?
SCREEN="/usr/bin/screen"

# ... and where or which mono do you want to launch server
MONO="/usr/bin/mono"

#if the file exist, opensim wont be restarted
# Sacha Magne
# add locksimstart and unlocksim to create/delete $LOCKFILE
# PocoLoco Darwin
LOCKFILE="/tmp/norun.opensim"
if [ -f "$LOCKFILE" ]; then
echo $LOCKFILE "found. no restart"
exit
fi

# Check For For Our Screen Process, Named OSR
running=`ps ax|grep OSR|grep -v grep`

# If The Process Is Not There...
if [ -z "$running" ]; then
#...We Make A Log Entry And Start A New One

# The Log Entry
TIME="Simulator Down, Restarting: `/bin/date +"%A, %B %d - %r"`"
echo $TIME >>$LOG/OSGReport.txt

# Start new instance and bind it to screen so we can get to a console session
# "-S" names the session for easy re-attachment ; "-d -m" start new session but don't attach to it 
cd $RUN
$SCREEN -S OSR -d -m $MONO OpenSim.exe
fi
# All Done Till Next Time

The lock/unlock scripts

Code:

#!/bin/sh
#locksimstart
# skeleton for a script that writes "norun.opensim" to /tmp
# to stop the automatic restart of opensimulator by OSwatchdog
# need to add cases for cli args , right now it just touches to create
# norun.opensim
touch /tmp/norun.opensim

#!/bin/sh
# unlocksim
# works in tandem with locksimstart
# removes norun.opensim so OSwatchdog will restart on next CRON run
#
rm -f /tmp/norun.opensim

Don't forget to chmod [ugo]+x
or these scripts won't want to run

Install the crontab entry and watch mono appear in the process table :mrgreen:

Max Ping · **Joined:** Wed Nov 05, 2008 5:26 am **Posts:** 240

You are a clever bunny poco, even if you are a little crazy ;-)

I can't wait to set this up - except I have to spend the next 8 days lounging around in a luxury private beach resort so may not get the time. It's very important to devote as much time as possible to being lazy

George Darwin · **Joined:** Thu Jul 31, 2008 2:48 am **Posts:** 394

Max Ping wrote:

You are a clever bunny poco, even if you are a little crazy ;-)

I can't wait to set this up - except I have to spend the next 8 days lounging around in a luxury private beach resort so may not get the time. It's very important to devote as much time as possible to being lazy

It's BlueWall's code. I wish I could take credit.
Thought it deserved some exposure here. It can be tiresome finding the nuggets of gold at the wiki.
So far it's worked nicely when I remember to turn off the crontab entry when the simstart is locked else I get a mail
with every cron run ----> e.g. every 5 minutes I get a mail informing me the script found /tmp/norun.opensim so it's not
going to start the server.

Max Ping · **Joined:** Wed Nov 05, 2008 5:26 am **Posts:** 240

It's good to get stuff out of the wiki and describe in posts like this exactly how to implement them - especially for linux noobs like me - I'm pretty sure I';m not the only one out there

Max Ping · **Joined:** Wed Nov 05, 2008 5:26 am **Posts:** 240

Ok, here goes:

On Ubuntu for some reason I couldn't get crontab to work using "crontab -e" I wasted a couple of hours figuring this out, getting right back to the basics etc... in fact the key lies in /etc oddly... I haven't tried this yet on CentOS

Forget crontab, just go straight for the system wide crontab:

Code:

pico /etc/crontab

In there slap your entry after the other stuff, in my case:

Code:

*/1 * * * * root /root/opensim/oswatchdog.sh

Make sure you put a space after the entry or it wont work.
Meanwhile in your oswatchdog.sh file *change* the "#!/bin/sh" to "#!/bin/bash" - don't ask me why

I haven't tried the locksimstart bit yet, but I'm assuming that if you run that script it will prevent OS from restarting until you remove the lock - kind of a manual override if you are doing updates and stuff and don't want the scalliwag to keep trying to restart.

Now for the unrelated technical bit - the script looks for a screen session named OSR, which is great - and seems to work (so far)... however I have noticed that in some instances opensim will fail - but the screen session will continue, in whic case the watchdog will not detect that opensim has broken.

George Darwin · **Joined:** Thu Jul 31, 2008 2:48 am **Posts:** 394

Max Ping wrote:

On Ubuntu for some reason I couldn't get crontab to work using "crontab -e"

You have got to be kidding :shock:

When did Ubuntu change crontab functionality. I've got a box running Ubuntu for mail, ad-hoc web services, and testing things and man crontab clearly lists "crontab -e" and it works just fine for editing a user crontab with the editor specified in the $EDITOR environment variable. Perhaps you haven't an $EDITOR defined? I would think there would be a sane default value globally.
/me shrugs

Max Ping wrote:

Meanwhile in your oswatchdog.sh file *change* the "#!/bin/sh" to "#!/bin/bash" - don't ask me why

You've lost or misplaced the symbolic link /bin/sh ----> /bin/bash
Thought that was standard with most Linux distros since like ....forever

Max Ping wrote:

I haven't tried the locksimstart bit yet, but I'm assuming that if you run that script it will prevent OS from restarting until you remove the lock - kind of a manual override if you are doing updates and stuff and don't want the scalliwag to keep trying to restart.

Exactly

It places a marker file in /tmp that OSwatchdog looks for and if found it doesn't start anything else on that run.

Max Ping wrote:

Now for the unrelated technical bit - the script looks for a screen session named OSR, which is great - and seems to work (so far)... however I have noticed that in some instances opensim will fail - but the screen session will continue, in whic case the watchdog will not detect that opensim has broken.

I've not noticed that. ALL the command line switches need to be used. Screen needs to started in _Detached_ mode with a _Title_ you can look for in the process table and the "-m" option so the screen process doesn't get locked to any particular pseudo-terminal aka $STY
From the info file for screen

Quote:

`-m'
Tell `screen' to ignore the `$STY' environment variable. When
this option is used, a new session will always be created,
regardless of whether `screen' is being called from within another
`screen' session or not. This flag has a special meaning in
connection with the `-d' option:
`-d -m'
Start `screen' in _detached_ mode. This creates a new session
but doesn't attach to it. This is useful for system startup
scripts.

`-D -m'
This also starts `screen' in _detached_ mode, but doesn't fork
a new process. The command exits if the session terminates.

Using the switches shown in my OSwatchdog.sh the screen session that launches OS always terminates when "mono OpenSim.exe" terminates.

George Darwin · **Joined:** Thu Jul 31, 2008 2:48 am **Posts:** 394

Max Ping wrote:

Now for the unrelated technical bit - the script looks for a screen session named OSR, which is great - and seems to work (so far)... however I have noticed that in some instances opensim will fail - but the screen session will continue, in whic case the watchdog will not detect that opensim has broken.

Aha! Looks like core-devs are aware of this problem. Just came across this in irc log of #opensim-dev AM today 8Jan09
btw not a public log, I just leave a client attached to Freenode 24/7 to keep in the loop of daily chats for my own edification

Quote:

11:35 < sdague> I'm trying to figure out where to debug a deadlock startup race
11:36 < sdague> it turns out that if I start up opensim behind the scenes in screen, so there is no console to slow it down, I often just hang at loading
objects
11:36 < sdague> starting it manually, works fine
11:42 < melanie_t> see which process generates the most output
11:42 < melanie_t> put a sleep in places where output was generated
11:44 < melanie_t> then see which sleep works around it it
11:44 < melanie_t> then you can see an area of locking that might be critical
11:44 < melanie_t> i know of that deadlock
11:44 < melanie_t> i see it from time to tim
[...]
12:00 < sdague> ok, sothat was interesting
12:00 < sdague> mysql seemed to just be no longer doing things with connections
12:00 < sdague> mysql restart, and life is good
12:00 < sdague> I need to figure out why mysql went stupid though
12:04 < SachaMagne> no more threads maybe ?

Looks like sdague's problem was broken connections to MySQL...
Does appear dev team is aware of this and it looks like it is related to your OS hang and why screen doesn't terminate. OS hasn't terminated, just hung, so screen won't terminate. Or is mono absent from process table when the problem you describe occurs? I hate getting caught making assumptions, especially when they are wrong

Forum rules

Automatic Restart of Regions

Who is online