Websites can fail for several reasons. Human error is a common one. In the hectic environment of the server room, pulling the wrong cable can cause servers to fail—not always in obvious ways. Hardware failure is another one—power supplies and hard disks. Software bugs and issues are also another one, or configuration errors—changing a configuration file but not saving it to disk has caused many problems. Then there are network problems when parts of a network or the larger internet can longer talk—often from one of the previously mentioned issues. Finally, hacking and other malicious causes play a small part as well.
The systems that host a web server are just as complex as ever, and the underlying causes are all still present. So, unfortunately, server downtime is still a common issue for websites. There are many ways you can host a website ranging from self-hosting up to and including major shared hosting sites. Even if your website is hosted on managed service, their 99.99% uptime promises can mean little in the real world, often lost in the small print. Some hosts will undoubtedly be more reliable than others in such a cut-throat business, while hosting yourself makes it your responsibility. Ultimately, if a website is important to you, it is your responsibility to ensure its reliability.
If an SSL certificate expires, most modern browsers will refuse to open it, showing an error that this site is insecure. Depending on the browser and security settings, you may have the option to open the page then, but only after acknowledging the risk. Effectively, the site is down to the average user, blocked by the web browser. The most common SSL certificate from Lets Encrypt lasts only three months, at which point you must renew it. If correctly configured, this should be an automatic process but still needs to be verified.
Depending on who you purchased it from, a domain expiring is a severe issue. Upon expiry, most domains will very quickly end up in the hands of bulk name squatters eager to sell it back for a profit (generally in the thousands of USD). Sometimes you will even hear of domains expiring for large companies - leading to embarrassment for those involved as they claim it back. On the other hand, it can be costly for smaller companies, losing money or years of hard work to build up their SEO.
If a website is of value, then it should be monitored. Because when it fails, then unless you have a monopoly, expect your customers to go elsewhere. Fault checking to determine the cause can take some time. For example, if another company hosts your site, you will need to contact someone and convince them that this is their problem. Time becomes precious in such a situation unless, of course, you have no idea there is even a problem. Then it will be your customers who may eventually tell you, assuming they have not just gone somewhere else. Certificate or domain expiry can lead you their own set of issues if you are not on the ball. So if a website is essential to you, it should be monitored.
We use the same methods as a standard web browser to test the uptime of a website. We make a standard web connection to the tested site and attempt to download the web page only, without content such as images. We do this every minute (longer if requested) and quickly determine if a website is down. Thus we use standard and reliable means to test a website the same way a browser would.
As part of our uptime monitoring, we keep track of the timings involved. Over time, this allows a picture of the website's performance, especially with our charts.
We monitor the domain by checking the detailed DNS records. In addition, we can check the certificate health by preventing the associated record of the website.
If there is a problem, we will send you a message by one of our provided methods: email or Slack.
We place a lighter load than a typical customer accessing your website in practice. The method we employ only access the page itself and none of the content, minimising the load on your server. If you are worried, it is possible to monitor at a reduced rate, say once every 5 minutes or even less.
We check every minute. This speed should be ok for most sites and only produce a little extra load. Still, if you would like to reduce it more, you may consider a more extended period (5, 15, or 60 minutes are available). But be aware this will slow our response to any problems that occur.
Unfortunately, we do not use fixed IP addresses. So it is not possible to identify our service accessing your site in this way. However, see the question about how to detect our crawlers below.
Many analytics use a small piece of Javascript to detect user traffic. However, our traffic will generally not appear as customer traffic because we do not run a browser and only access the actual page itself. If you monitor your logs, see the next question on detecting our crawler.
Our crawler will identify itself via the browser user-agent string; we do not use a fixed IP, so this is the best way to detect them:
Our crawlers respect your 'robots.txt' file and will check this regularly. Simply include the following line to stop our crawlers from accessing your website. We require our customers to own or have appropriate permission to access a website. Generally, this should not be a problem.
Webdar is under full-time development. Currently, we are working on completing the beta and making final preparations for going live. We will make a roadmap page available after we have gone live.
We are running a beta to ensure that everything is working as intended once we go live. We also provide a means for our potential customers to gain early access.
We are offering a fully functional service during the beta, but with a slightly reduced allowance for our customers. However, the service will be free during this time to compensate. So there will be further benefits for our early customers once we go live.
If you need more than the allotted beta service, then contact us with your use case, and we will attempt to provide greater access to your account.
We will give approximately two weeks notice, and all of our beta users will have the option to continue with one of our paid accounts with the first month free. Furthermore, they will receive a discount on their first subscription payment (1, 3 or 12 months) as a thank-you for helping us test the service.
A website is a fundamental entity that we are testing. It refers to a single domain and may have multiple tests. We will also test the SSL certificate and domain's health as part of our service by default.
A test is a check made on a specific page of a website. Multiple such tests can be added to each website. We also offer more specialized tests such as ping & port connection tests.
Integration refers to a means of communication when there is a problem. For example, email and Slack are our two main methods of contact.
An alert team is a group that will be contacted when there is a problem with a Website. It comprises one or more integration contacts (i.e. an email address).
The website's health are its secondary properties such as SSL certificate or domain name. How close they are to expiry or other issues is considered the website's health. They do not have to be checked as often, but their failure will still result in the site failing to work.