We’re very happy to welcome Philipp Kaiser as our first full-time employee! As an experienced system administrator, Philipp will help us get our IT operations backlog shortened to a more human size, and to get new, innovative changes out of the door more quickly.
Did you know this is actually the second time I’ve hired Philipp? Back in 2004, as the new IT team lead “CRM, Billing and WEB.Cent” at WEB.DE, I got him to join my team as its second member. During his interview, he had impressed me by how he handled the stress at the place where he did his apprenticeship as a Qualified IT Specialist. I was responsible for a bunch of business-critical systems and thought “If he’s lived through this, he’ll be able to survive here, too.” He did much more than just survive and I enjoyed working with him during several years.
As sysadmin #1 at freistil IT, Philipp will be responsible for business-critical systems, too — business-critical not only for us but especially for our customers. And I’m 100% sure he’ll be doing a great job.
The day-to-day in our distributed team at freistil IT will be quite different from what he’s used to and we’re well aware that there’s a learning curve waiting for all of us. That’s exactly what makes running freistil IT so exciting for me. Learning is one of our core values, and every new employee will notice this quickly. From his time back at WEB.DE, Philipp is already used to the concept of the Ops Bootcamp which we’ve implemented at freistil IT: In this programme spanning several weeks, we’re going to introduce him to all important aspects of his job from Asana to zsh. At the same time, he’ll already be working on internal projects and support requests.
So, welcome, Philipp, and thanks for joining our team! Cheers — to a lot of fun!
05 May 2014
The Open Source Datacenter Conference has been an established DevOps conference even before the term was widely used. For 2014, the Netways team left Nürnberg as their traditional location behind and moved the event to Berlin.
I’m happy that they kept the “conference hotel” concept; nothing beats being able to walk from bed to breakfast to conference session under the same roof. For me, the Hotel MOA in Berlin-Moabit ticked all the boxes. My room was comfortable and spacious, the catering was excellent and there were more than enough seats in the conference rooms for all the attendees.
There is an essential aspect of tech conferences where OSDC 2014 unfortunately left a lot to be desired: reliable internet connectivity. The hotel WiFi clearly was not built to accomodate a crowd of techies who each brought 2 or more WiFi clients. Unfortunately, Netways had not prepared a dedicated network in advance so I had to fall back to my 3G hotspot when my WiFi connection broke down only a few minutes into the introductory talk.
As regular OSDC participants have come to expect, the talks were diverse, interesting and on a professional level. Here is a short excerpt of the session programme:
- Lennart Koopmann & Jordan Sissel: Intro to Log Management
- Mike Adolphs: How we run Support at GitHub
- Martin Gerhard Loschwitz: What’s next for Ceph?
- Thomas Schend: Introduction to the Synnefo open source cloud stack
- Michael Renner: Secure encryption in a wiretapped future
- Christopher Kunz: Software defined networking in an open-source compute cloud
- Andreas Schmidt: Testing server infrastructure with serverspec
Netways generously published their recordings from OSDC 2014 on Youtube, including my own talk, “Dynamic Infrastructure Orchestration”.
Making our configuration management more dynamic is one of our main concerns at freistil IT. Our infrastructure is growing steadily and we have to make sure that changes are executed in a quick and reliable manner. Conventional configuration management tools like our trusted Chef are limited in terms of change rollout speed because they are based on regular convergence runs. That’s why I took a closer look at serf and etcd. I’ve then summarised my findings in my OSDC talk:
After I watched the recording, I realised sadly that I missed the bar at least on the aspect of “professional presentation”. Judging from the many interested questions during and after the talk, I managed to convey everything I had intended to. But I had not rehearsed the talk enough and that showed clearly in far too many errs and uhms. I apologise to all attendees for not delivering the presentation quality that you can expect at such a conference. I’ll push myself harder in order to always arrive fully prepared when I’m invited as a speaker.
With the move to Berlin, the Open Source Datacenter Conference is on the way from a German IT event to an international DevOps conference. And I’ll be happy to return next year!
04 May 2014
Tomorrow morning, I’m going to jump on the train to the first DrupalCamp Frankfurt. I’m looking forward to meet my friends of the German Drupal community and also to get to know some new folks, so say hi!
If you’re interested in how to build a Drupal development environment with Vagrant and automation tools, come by my session that I’ll be doing on Sunday morning!
See you in Frankfurt!
11 Apr 2014
And just so, the first quarter of 2014 has passed. Wow, time flies when you’re busy. And busy we were!
Our conference schedule for this year is so packed that we need to distribute events within the team. In March, while I visited Szeged, Hungary, and enjoyed the community feeling at the Drupal Developer Days, Markus attended DevOpsCamp Nuremberg. DevOps is at the core of our business, so DevOps Days and DevOps Camps are ideal opportunities for us to exchange experiences with other IT specialists and also gain new inspiration.
Our managed hosting platform freistilbox keeps growing and now hosts 282 websites (Feb: 266, +6%). Our total uptime was 99.93% (Feb: 99,97%) and with 11.96 TB, our website traffic came up to about the same level as in the previous month (Feb: 12.5 TB).
We’re working hard every day to keep our promise, “Work efficiently, sleep peacefully.” And it’s feedback like this customer’s that always makes our day: “Pingdom reports 99.99% availability of our cluster for March. Thank you.”
In March, we achieved a great improvement in tech support quality. While ticket traffic resembled the one in February, our reaction times were substantially better.
We got 150 support requests (Feb: 149) and were able to resolve 126 tickets (Feb: 127), leaving a backlog of 48 (Feb: 35). Our average first reply time came in at 16h, a whopping 31% less than last month (Feb: 23h). You can see the cause in the time breakdown: We shifted a significant part of “tickets we didn’t answer within 24h” to the 8h bracket. This month, we tackled 42% of all tickets within an hour (Feb: 43%) and 31% within 8h (Feb: 23%); together, this makes 73% of all incoming requests. The number of tickets answered within 24h stayed the same (Feb, Mar: 15%), and only 12% took us longer than 24h (Feb: 20%).
With these results, it’s not surprising that our customer’s satisfaction feedback stayed at 100% “good”. These are some of the comments we got from solved support requests:
- “VERY fast and great support!”
- “CSI Varnish ^^”
- “Super fast response, thanks!”
- “A kind of magique”
Customer happiness is our primary goal and we’re glad that we’re well on target here.
Since we’re able put existing capacity to more efficient use, we only added 2 new hosts to our infrastructure (Feb: 282). From all these hosts, we’re collecting 108,880 distinct health and performance metrics (Feb: 107,706) — every 10 seconds.
The number of on-call notifications decreased further to 1122 (Feb: 1397, –20%). This number doesn’t mean that we experience an outage every 40 minutes on average. We’re quite happy that the biggest part of these alerts get automatically resolved within minutes because the issue (for example a high system load) was only short-lived. As soon as we get to fine-tune our alert thresholds, the number of notifications will go down significantly.
All in all, it’s been a great month and a successful first quarter. We’re looking forward to what’s coming next!
07 Apr 2014
Next week, I’m going to attend the Open Source Datacenter Conference (OSDC). Conviently for me, the event will take place in Berlin now after Netways, the company behind the conference, decided to move it there from its traditional location in Nuremberg.
The OSDC takes place on Wednesday and Thursday and is packed with sessions about the newest developments in Open Source technology. Some of my personal highlights:
- Jordan Sissel: “Find Happiness in your Logs”
- Andreas Schmidt: “Testing server infrastructure with serverspec”
- Martin Gerhard Loschwitz: “What’s next for Ceph?”
- Mike Adolphs: “How we run Support at GitHub”
On Thursday, I’ll give a talk myself, titled “Dynamic Infrastructure Orchestration”. My goal is to point out possible next steps after getting configuration management solutions like Puppet or Chef in place. I’ll cover two different approaches:
- using central key-value stores like etcd for service discovery and
- configuration, and
- decentralising system automation with tools like serf.
OSDC always is a fun event and I’m excited to meet old colleagues and talk to fellow IT professionals.
Hm. I wonder if the city has changed much since I last visited Berlin in 1987…
04 Apr 2014
Three findings on my flight home:
- Compared with Hungarian, German with its puny three umlauts can pack up and go home.
- Compared with Ireland, Hungary has much nicer weather.
- Compared with the pubs at home, in Hungary you can treat four or five times as many people to a beer for the same amount of money.
As you can see, Hungary has the advantage in many regards and I had a great time here at the Drupal Developer Days this past week.
When I arrived in Szeged on Wednesday evening, Drupal 8 coding sprints had already been running for a few days and they’d continue all week. During this time, up to 150 Drupal developers were working to make progress on code and documentation issues. There were 115 commits to Drupal core and after removing 19 blocker issues, we’re now 40% closer to Drupal 8 Beta. Our Git repository traffic was so high that it even triggered drupal.org’s DDoS defenses!
At the event location only, we consumed 5500 sandwiches, 1000 servings of coffee, 240l of beer, 300kg of sweet snacks and 120kg of bananas. The venue was a really good choice. It had all the space we needed, good catering and the WiFi worked well throughout the event. And having the Novotel (with its amazing value for money) right next door is unbeatable convenience.
From Thursday to Saturday, there were a lot of interesting presentations as well as a reprise of the “Caching Deep Dive” multi-hour workshop that had a lot of success at DrupalCon Prague. Almost every talk referenced the upcoming Drupal version. And although many things are still in flux, it feels to me like Drupal 8 is taking shape.
What drove engagement most was the great community spirit. Everyone was welcome, from the Drupal novice to the long-time core contributor. There were smiling faces all around and you could simply walk up to anyone to have a chat or ask a question. People got together spontaneously, be it to code or to go have dinner. These personal experiences are what I love most about the Drupal community. If you haven’t been to a DrupalCamp yet, go to DrupiCal now and see what’s happening near your place! Go on, I’ll wait here.
My personal highlight was the #AberdeenFreistilCloudBox party on Friday night which I had the pleasure to co-organize. I had contacted Aaron Porter before the event, suggesting we join forces and do something about the community’s lack of awareness of our European Drupal hosting companies that we both had perceived at DrupalCamp London a few weeks earlier. When I met Aaron on Thursday morning, he invited me to join him in scouting for a party venue for Friday night. We looked at two bars in the center of Szeged and decided on hosting (that’s what we’re good at, after all!) the party at the CoolTour Cafe. We were able to make a deal that secured our guests 100 free beers as well as free admission to the concert room where one of Hungary’s best-known Jazz singers was going to be on stage. And when on Friday afternoon the number of sign-ups for the party crossed the 100 mark, Aaron and I decided to throw in another 100 beer vouchers. The party was a success and fun was had by everyone. The title of “drink distributor extraordinaire” goes to Dave Hall who’s apparently related to a huge Dutch family since he returned every few minutes to grab a voucher for another member of the Needabeer clan.
In the end, I’d like to congratulate the organization team around Kristof van Tomme to a great Drupal community event! I’m thankful for a lot of good conversations, and happy that I overcame my initial reluctance to register and got to be a part of Drupal Developer Days Szeged 2014.
31 Mar 2014
2014 is the year of Drupal 8 and we’re getting our freistilbox hosting platform ready to host its first Drupal 8 websites. Of course, Drupal 8 will be the central topic of the Drupal Developer Days in Szeged from 24 to 30 March. We can’t miss such an important event, so Jochen will be following DrupalMarvin’s invitiation from DrupalCon Prague last year.
Jochen is going to be in Szeged from Wednesday the 26th to Sunday the 30th, so if you’re there, make sure to say Hi! He’ll be more than happy to get you a beer or coffee, and you can ask him whatever you’d like to know about freistilbox.
And don’t forget to pack your towel!
22 Mar 2014
February put an end to the quiet time at the beginning of the year. We went into our final preparations for moving our business to our Ireland-based company freistil IT Ltd. From April on, all our services will be provided – without any changes in quality or technology – by our new company. We’re very excited about this new phase in our company’s development!
We kickstarted our community engagement by visiting DrupalCamp London. With about 600 attendees, it was an impressively big event for a local DrupalCamp. Discussions at the CxO meet-up and conversations during the whole event made us realise that we need to ramp up our marketing efforts because most people seem to know only the two US providers for Drupal hosting. We think this needs to change. ;-)
Taking the opportunity of Markus and me being at the same place, we stayed for another two days after DrupalCamp for our Q1 board meeting. Inspired by Marc Benioff’s book “Behind the Cloud”, we worked on defining our company’s key vision, values, methods, obstacles and metrics. I’m going to write some more about that at a later time. Culture is an important part of business and these basic aspects will be the foundation on which we’re going to grow our company.###freistilbox
Our hosting platform keeps growing steadily, with now 266 websites running on freistilbox (January: 233, +14%). We’re happy to see our availability improving as Pingdom reports a total uptime of 99,97% (January: 99,94%). Last month, we expected our traffic to cross the 10 TB mark, and it did so with an impressive 30% jump from 9.6 TB to 12.5 TB!
Probably mostly because of the short month, our technical support numbers are a bit lower than in January. We received 149 new support requests (January: 181) and resolved 127 (January: 140). The ticket backlog was 35 (January: 41).
This month, our average resolution time came in at 22.6h, nowhere near the insane 250h in January. We managed to answer 43% of new tickets within an hour (Jan: 45%) and another 23% within 8h (Jan: 28%). 15% of new requests were answered within 24h (Jan: 8%) and the remaining 20% took us longer.
We’re glad to report that our customers’ support satisfaction remained at 100%.
In February, we grew the number of servers to 282 (Jan: 264, +7%). We’re now collecting 107,706 metrics in our monitoring system (Jan: 99,518). The number of on-call alerts went down to 1397 (Jan: 1781, –27%) and we’re working on improvements that will lower this number even more significantly.
###Other notable stuff
In February, I celebrated my 44th birthday. It was my first birthday party in Ireland. It feels good to get settled in, and having freistil IT based in Ireland now, too, will make me feel at home here even more. I’m very excited to see how my life and work will develop over the coming months!
20 Mar 2014