This years JOTI started for me with driving 4 hours by car to the the southern german JOTI Headquarters in Amberg. We are here at an Amateur Radio Club Station somewhere in the nowhere but with fast internet.

While i was fixing some smaller stuff in the morning there were some bad news when I arrived at about 20:00 CEST at the HQ. The services software we use (you may know NickServ, ChanServ and so on) stopped working from time to time.
This would not have been an too big issue during the year but this weekend it is critical! It turned out very soon that there was no configuration error so the reason had to be somewhere else. Even the most verbose debug logs did not show the exact error but they pointed us into the right direction.
Long story short: After some core dumps, debugging with gdb and primary due to the help of acoster and the atheme developers we were able to fix two bugs in the software that caused the problems.

On saturday morning we hit another bug, but this time in the ircd server software which caused our network (consisting of 19 servers atm) to restart itself. After about 10 minutes the complete network started up and configured itself properly again. 

For those who are wondering: Yes we are using alway the latest stable release for our server software. We are happy to profit from open source software and als happy that we could give back at least bug fixes and testing in high load situations.

By the way: Here's a graph of how many users are concurrent connected to our network. I hope we will reach the 5k limit this year.

Screenshot