February put an end to the quiet time at the beginning of the year. We went into our final preparations for moving our business to our Ireland-based company freistil IT Ltd. From April on, all our services will be provided – without any changes in quality or technology – by our new company. We’re very excited about this new phase in our company’s development!
We kickstarted our community engagement by visiting DrupalCamp London. With about 600 attendees, it was an impressively big event for a local DrupalCamp. Discussions at the CxO meet-up and conversations during the whole event made us realise that we need to ramp up our marketing efforts because most people seem to know only the two US providers for Drupal hosting. We think this needs to change. ;-)
Taking the opportunity of Markus and me being at the same place, we stayed for another two days after DrupalCamp for our Q1 board meeting. Inspired by Marc Benioff’s book “Behind the Cloud”, we worked on defining our company’s key vision, values, methods, obstacles and metrics. I’m going to write some more about that at a later time. Culture is an important part of business and these basic aspects will be the foundation on which we’re going to grow our company.###freistilbox
Our hosting platform keeps growing steadily, with now 266 websites running on freistilbox (January: 233, +14%). We’re happy to see our availability improving as Pingdom reports a total uptime of 99,97% (January: 99,94%). Last month, we expected our traffic to cross the 10 TB mark, and it did so with an impressive 30% jump from 9.6 TB to 12.5 TB!
Probably mostly because of the short month, our technical support numbers are a bit lower than in January. We received 149 new support requests (January: 181) and resolved 127 (January: 140). The ticket backlog was 35 (January: 41).
This month, our average resolution time came in at 22.6h, nowhere near the insane 250h in January. We managed to answer 43% of new tickets within an hour (Jan: 45%) and another 23% within 8h (Jan: 28%). 15% of new requests were answered within 24h (Jan: 8%) and the remaining 20% took us longer.
We’re glad to report that our customers’ support satisfaction remained at 100%.
In February, we grew the number of servers to 282 (Jan: 264, +7%). We’re now collecting 107,706 metrics in our monitoring system (Jan: 99,518). The number of on-call alerts went down to 1397 (Jan: 1781, –27%) and we’re working on improvements that will lower this number even more significantly.
###Other notable stuff
In February, I celebrated my 44th birthday. It was my first birthday party in Ireland. It feels good to get settled in, and having freistil IT based in Ireland now, too, will make me feel at home here even more. I’m very excited to see how my life and work will develop over the coming months!
20 Mar 2014
Tomorrow, I’ll fly to our neighbor island for DrupalCamp London and Markus is going to join me on Friday. Together, we’re going to breathe some community air again and get a feel for what British Drupal shops need in terms of hosting.
With 600 attendees, DrupalCamp London is going to be an impressive event! There will be 30 community sessions as well as BoFs and sprints across the weekend. I’ll certainly try to at least attend the “Next Generation DevOps” and “Concurrent Programming” talks. I’m also looking forward to the Drupal CxO meet-up on Friday before the actual conference.
For Markus and me, it’s a valuable opportunity that we’ll be at the same place at the same time for a change. That’s why we’ll stay a few more days more after the weekend to do important strategy work for 2014 together.
We’re very excited to meet a lot of enthusiastic Drupal developers In London! So, if you’d like to join us for a pint and talk about your Drupal hosting needs, simply drop us a line via email or on Twitter!
26 Feb 2014
Do you still remember these text messages you should not have sent last night? SMS von gestern Nacht (“last night’s text message”) does! The entertainment website publishes texts from its users that are simply too good, funny or rad to be read only by their original recipient.
SMSvonGesternNacht.de has been running on freistilbox since 2013. We thank the founding team for their following feedback:
Why we chose freistilbox
Our first contact with freistil IT was already in 2010 when they still offered their hosting under the name “DrupalCONCEPT”. We were looking for good hosting solutions for Drupal and had found freistil. Back then, we decided on a single managed server from another provider. Over the years, we extended it to a group of three servers. When we planned a relaunch in 2013, the Drupal experts from Palasthotel recommended a switch to the freistilbox hosting platform that they praised enthusiasticly.
Out of experience, we wanted to build on a proven and reliable solution. To learn the detail aspects of freistilbox, we invited Markus from freistil IT to our office in Berlin. In this meeting, we worked out that freistilbox would cover our needs very well. Its caching infrastructure, optimised for Drupal, promised high delivery performance for our many anonymous visitors. At the same time, the price was very attractive; we don’t pay significantly more for freistilbox today than we did previously for our managed servers. freistilbox brings additional advantages, too, for example the developer-friendly design and the fast technical support. The easy scalability is another important factor because our traffic grows steadily.
Our hosting setup
At launch, we started with a cluster of two freistilbox S. Since performance is more important to us than availability, we decided after a short time to switch to a single freistilbox M.
Our start with freistilbox was a very good experience. With one exception, everything worked out of the box. And the problem that our voting feature at first didn’t work because of the necessary extra cookie was quickly resolved by the freistil team with a custom cache configuration. The target-oriented way we discussed and solved this problem together with Palasthotel in a “war room” chat conducted by freistil IT significantly increased our trust in the team’s competence.
Hosting on freistilbox is fun because we don’t have to take care of anything. freistilbox simply works and we notice little of daily operations. Our questions usually are answered in much less than 24h and contact with the freistil IT technical support always is a pleasant experience.
If there is a problem eventually, someone immediately deals with it. At the end of 2013, our website suffered from network issues, today our availability is 99,9% even without redundant servers.
Since the launch on freistilbox, our traffic has more than doubled from about 18 million to 40 million page impressions per month. The highest traffic spike has been 1.5 million PI on a single day. The website still runs with full performance without us having to grow our configuration. We’re now considering allowing user logins which would probably require an expansion.
freistilbox leaves little to be desired. Comprehensive statistics that show how well our cluster is working would be a great additional feature, though.
freistilbox clearly holds up to its promise “work efficiently, sleep peacefully”. When the next vacation comes up, we’ll be able to go on holidays without worrying about our hosting.
13 Feb 2014
Starting with January, we’re going to post a monthly summary that will give you a bit of insight into what’s happened at freistil IT.
On our managed hosting platform, we’re now hosting 233 websites for which Pingdom reported an average availability of 99.94%. When we take into account that some of our customers chose to run their sites on a single box and/or near the maximum capacity of their cluster, this is a good result.
Our edge routers delivered a total traffic volume of 9.611 TB in January. Let’s see if we’ll crack the 10 TB threshold in February!
In January, we received 181 support requests. We solved 140, leaving a backlog of 41 tickets (most of these are usually pending customer response). Our average reaction time in this month is… catastrophic. A single request that went unanswered for half a year because it was covered elsewhere launched our average response time to a whopping 250 hours. When we break reaction time down into categories, you can see that this is an absolute outlier. Of all support requests in January, we answered:
- 45% within the first hour,
- 28% in 1–8h,
- 8% in 8–24h, and
- for only 19%, it took us longer than 24h to react.
We’re very happy that the support feedback we got was 100% positive. Here are some notable comments:
- “Competent feedback as always.”
- “Fast and reliable!”
- “freistil is always there when you need them.”
Our team is taking care of 264 servers, resulting in an average of 132 servers per system administrator. We think that’s quite respectable.
Each minute, our monitoring system processes 99518 separate metrics. In January, it sent 1781 alerts to the on-call engineer. This is an enormous amount, 54% more than in December. There were no significant outages during this month, though. Like with our support request reaction time, this value is so high because of an anomaly for which we needed to adapt our alert thresholds first.
Jochen attended the first Drupal Dublin meet-up in the new year. The group was treated to some detailed peeks behind the curtains of a huge museum website.
Other notable things
We signed the contract with our first employee! It took us quite some time to work out the right way to put our Results-Only Work Environment into legal writing, but now we’re excited for the first system administrator to join our team.
Overall, it’s been a good start into 2014 and we’re highly motivated to make it a great year!
Any questions? Please post them in the comments!
07 Feb 2014
2014 has gotten off to a good start but it’s still time to do a review of the last year. Learning is a big part of what we do at freistil IT and 2013 did teach us quite a lot of things!
I don’t exaggerate when I say that 2013 started with the most important events of the whole year. My son Richard was born in January and Markus became a #newdad with the arrival of his wonderful daughter Marlene in February. First there had been a comfortable span of time between the two estimated birth dates, but Richard decided to take his time while Marlene wasn’t patient at all. This lead to the first challenge in 2013: How can we both at the same time put our “Family first” principle to practice by taking care of mothers and children without losing our productivity (or even actual business)? The serious product problems we had at that time made things even more difficult, but I’ll come back to that later. Well, we managed somehow and two beautiful children are celebrating their first birthdays these days.
In June, we went to the Emerald Isle for Drupal Dev Days Dublin. freistil IT was Gold Sponsor and we almost didn’t get our freistilbox banners made in time because of a botched job by the printer. At the event, we launched freistilbox Solo, our virtual freistilbox environment for development and testing. I had arrived a few days early to start scouting for houses because this would be the year my family would finally make a long-held dream come true and move to Ireland.
In August, I flew over with my family and, as luck would have it, we found a house on the very first day of hunting. freistil IT is a distributed team and completely location-independent, so my move didn’t cause any major disruption. I only had to make do with a 3G modem until they switched on our broadband. In Freiburg, I had worked from a shared office in town, now my office is only a few steps from my bedroom. A home office has its own challenges, but if things gets too distracting, I simply go to a coffee shop like the one I’m typing this right now.
Drupal Ireland is a lovely bunch of people and I enjoy the monthly meetups in Dublin. In September, I met a few of the folks at the Prague airport where I had arrived for DrupalCon Europe and not much later, we found ourselves at an Irish Pub watching the GAA final. Prague actually was my first trip to eastern Europe and I enjoyed it very much. DrupalCon was a good opportunity to catch up with some of our customers. Not all of the feedback we got in these talks was positive but that’s not why we do them anyway: Improving first needs learning what we can do better. That’s why we appreciate honest feedback , and in the long run, constructive criticism pays for both sides.
We attended not only Drupal conferences, though. Since we’re doing cutting-edge IT work, we spent time exchanging knowledge and experience at DevOps Days in London and Berlin as well as at the Open Source Datacenter Conference in Nuremberg. We found these events great to gain new impulses and to share what we’ve learned.
During the year, especially in the second half, the reliability of our freistilbox hosting platform suffered a lot from network issues. In order to be more resilient against power or network outages, we had spread our servers over several datacenters. We didn’t experience a single outage on the datacenter level. What happened a lot, though, was the “noisy neighbor” problem: servers on the same network segment that were either origin or target of a DDoS attack which then impacted the whole segment. The only way to solve these problems back then was to notify datacenter staff who then identified and isolated the server involved in the attack; a procedure that usually takes 5 to 10 minutes. We realised that the distribution of our servers exposed our infrastructure too much to these problems and decided to move into dedicated racks that housed only our own servers. With that change, things got much quieter. This is a big advantage our bare-metal infrastructure has over cloud-based solutions: we have full control over where our IT resources are located and how much or how little they share with others. In November, our datacenter provider experienced a massive DDoS attack with no clear target and a traffic volume that caused problems even at their uplink carrier. It was a rough weekend for us and our customers. Interestingly, after that event, we’ve not experienced any other serious network problem at all. Looks like our datacenter provider made some effective changes. Additionally, they officially announced plans to overhaul their network infrastructure and to put anti-DDoS systems in place.
We’re proud to say that we achieved quite a bit in 2013, especially with freistilbox. To be honest, we had started the year with a massive low because we launched the platform before it was ready for prime time. The negative feedback from our customers made it clear that by rushing the launch, we had caused big disappointment and lost a lot of our customer’s trust. Some customers also questioned the decision to divert manpower from the production platform to freistilbox Solo. In consequence, making the system reliable and efficient was our topmost priority during the year and we put a lot of hard work and many hours into it. I’m very happy to say that we now managed to give freistilbox the high level of quality it should have had from the start. In a round of customer calls I did in December, we got a lot of praise for how well the platform is working and how good a job we do with improving and supporting it. We are incredibly grateful to our customers for all their feedback.
We received 1,693 support requests last year, down 28% from 2,367 in 2012. The first quarter had the most new tickets (479) and Q4 ranked lowest (367). This is a good sign that we’re making progress in improving both the quality and the ease of use of freistilbox.
For a team with only two DevOps engineers, we’ve certainly created a great product for hosting Drupal and WordPress sites. And we’re only able to keep up doing the daily business as well as the tech support and 24/7 on-call because we’ve learned to collaborate effectively. The key to effective teamwork is communication, especially in a distributed team. Judging from the fact that we replaced Campfire as our communication backbone with HipChat, and heavily use Confluence for internal documentation, JIRA for task and project management and BitBucket as our code repository, Atlassian products seem to fit our needs quite well. Another communication product that I first didn’t expect to catch on actually became one of our most important channels: Sqwiggle. It took a bit getting used to a slight “Big Brother” feeling but we highly appreciate being able to tell at a glance who’s currently at their desk and to start a video call with a single click.
Over the recent months, we also reshuffled our areas of responsibility in order to put our strengths to use more effectively. Markus took over some of my technical tasks and now takes care of daily operations. I’ve assumed a more strategic position in business development.
Laying stronger foundations: We’ve chosen Ireland as our new base of operations because of its growing importance as a tech hub for Europe. Our new company “freistil IT Ltd” will soon take over all business we’ve done so far as the partnership “freistil IT GbR”. The first benefit this change brings for our existing customers is that we’ll now be able to accept credit card payments.
Growing our team: We’re great in automating IT processes but we can’t automate innovation, that needs pure brain power. And for better “load distribution”, we need more “nodes”. We’ve been looking for quite some time and finally found two talented system administrators that are going to join our team over the coming weeks. This means we need to learn quickly about hiring and being a great employer.
Expanding our business: So far, we’ve mostly gained new customers by word of mouth. We love the fact — and we can’t be thankful enough for it — that our customers are so happy with freistilbox that they recommend our hosting platform to their friends and clients. To really expand our customer base internationally, though, we need to increase our sales and marketing activities. Since neither Markus nor I have a strong background in these areas, we decided to get external help. Thanks to an Enterprise Ireland Mentor Grant, a consultant experienced in international business will work with us on our growth strategy over the coming months.
It’s obvious that we’ve learned a lot of things in 2013, sometimes the hard way. We’re thankful for all the good will, feedback, advice and encouragement we’ve received from our families, friends and our customers. Every day, we’re getting better in fulfilling our mission: Making sure that our customers can work efficiently and sleep peacefully. To a great 2014!
 I recently found out that we were destined to go to this conference: Who would have thought that “Markus & Jochen” is an anagram for “Shamrock & June”!
29 Jan 2014
The important aspects for working in a remote team like here at freistil IT are very well explained in Mark Campbell’s blog entry “How to work remotely as a software developer”:
- Setting limits
- Time boxing
I’ve found the communication aspect the most essential one, because a remote team needs to compensate for the distance between coworkers. In an office, there are countless opportunities to have short exchanges to bounce ideas off each other, discuss findings and talk about other things. So, to keep round-trip times short in a virtual office, it’s important to use fast communication channels like HipChat or Sqwiggle. (We’ll expand on how we actually use these tools in later posts.)
There’s another great insight in Mark’s post I can wholeheartedly agree to:
“Working from a coffee shop is great. People will ignore you, you get hopped up on caffeine, and there’s a constant noise level about you.”
Oh, and by the way, Mark’s points are fully valid also for system administrators.
(Picture credit: Tracy Ruggles)
06 Jan 2014
From this Saturday (2013-12-21) on, our team will be off recharging.
During the holiday time, we’ll only do emergency support. That means we’ll only handle outages and other incidents that impact the delivery of existing websites.
We’ll resume working on tasks that aren’t connected to such incidents on Monday, January 6th 2014.
17 Dec 2013
We’re now providing web applications with two variants of temporary file directories: one that is shared between boxes and a faster one that is stored locally on each node of a freistilbox cluster.
Our main goal with freistilbox is giving website developers maximum performance at minimum effort. Storage access is an important aspect in web performance tuning and it’s a good idea to avoid expensive disk operations whenever possible. That’s why we decided to store temporary files created by web applications locally. It’s obvious that writing to a local disk is by far faster than shipping them over a network connection to a shared storage.
We chose that approach under the premise that temporary files are only created and used during a single content request, for example for uploading a file, aggregating CSS code or compressing data. Under this condition, it doesn’t matter when the next request is handled by another cluster box that has its own separate temporary file directory.
Support requests we got over the recent weeks were a clear indication that this premise was wrong. As it turns out, there are situations where temporary files are expected to persist beyond the lifetime of a single content request.
As an example, there was the customer who noticed that batch operations of the “views_export” Drupal module delivered incomplete data. We found out that the batch process saves intermediate results to the temp space. Since the batch runs were distributed over their boxes, so were the result files. At the end of the batch process, the box doing the final run only found the files that were created on this particular box and so returned corrupted data.
In order to make sure that even temporary files are visible to all boxes in a consistent way, we decided to relocate them to the shared file storage. The default temporary file directory available at ../tmp, relative to the document root, is now shared.
Obviously this has a significant impact on performance: the data still has to be written to and read from disk, but now on multiple separate storage servers; and the data transfer over the network comes on top. For those customers that don’t need a shared temporary file directory but depend on speedy file handling, we now also provide an alternative in the form of ../tmp_local.
Oh, and we also put a cleanup process in place that makes sure that temporary files stay true to their name: Files that haven’t been touched for a week are removed automatically.
05 Dec 2013