rDNS Epic Failures

Over the past week we've been helping a client get their systems properly configured. One of the issues we needed to resolve was reverse DNS entries (rDNS). Pretty simple task huh? Especially for the "network engineers" at a data centre. Surely they should know how to do this.

For more than eight days we had to work with this colo-provider. We asked for rDNS entries. After two days they responded, asking for clarification. Um? REVERSE DNS, add it to your NS!!.

A day later they responded that it was complete. We checked. No entry was found. SERVFAIL was the response from the `host` command.

Using `dig` directly to their NS we were able to see that their NS (the authoritative one for this block of IPs according to ARIN) was responding to these was REFUSED!!.

Usually `dig` responds with something like this:

;; HEADER opcode: QUERY, status: NOERROR, id: 64327
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

Their epic-failure response looks like this:

;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 34593
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

They are the authoritative NS for this block of IPs and they refused PTR queries from the public internet, which they are supposed to be serving. Apparently this had been an issue at their facility for weeks if not months. It took us three days to explain to their "engineers" why this was an issue! Read the fucking RFCs idiot!

Of particular humour was when they demonstrated to us their NS was working, by testing the NS from it-self (epic fail!). Our point was that the NS wasn't working from outside (London, Dallas, Seattle). What did they do? Test from localhost. Um? Are you kidding me?

It's unfortunate that there are so many amateurs. If you're sick of working with them call us. We follow standards and persistently push others to do the same.

Site Slowness? Too many Queries Maybe?

We are currently migrating a clients site to a new platform. One of the original complaints with regard to the existing site was slowness. Think I found the answer today.

It takes 119 queries to display a product page. There was one query (exactly the same, no updates in between) executed 16 times! And another one run 30+ time because the author of the code didn't want to use the SQL operator IN or do a JOIN.

When you're writing a store front, ask your self. How often do you need to run

SELECT COUNT(*) FROM product_option where product_id = 345

We think 20 is too many.

Come on guys, trace your code and monitor SQL query logs when you're developing. These kinds of things are easy to catch and fix. Computer are fast but, lets not waste resources.

Keep your Systems Simple (kyss Method)

We are helping a client assume ownership of a recent business they purchased.
We got all the code, domain, etc.
But! Like many IT projects the documentation was insufficient.

Our first step was to then audit the existing system. We discovered that multiple web-servers were involved, not just physical-servers, but Apache, Lighttpd, Tomcat, etc) all on the same box. Some Apache proxy to the Tomcat, etc. A complicated configuration with a wide set of dependencies.

It was clear there was some scrambling in the old project, code forks all over, duplicated libraries and a wide varsity of languages used (PHP, PERL, Java, JavaScript, BASH). There was duplicated logic-libraries in each language! More complications that introduce inconsistencies (ie: fix the PHP lib but forget the PERL - OOPS!). This is also clearly too complicated.

There are many reasons why projects get to this state. Maybe one engineer likes Java more than PERL or historically it started with a shell-script. History, politics and engineer preferences can all be a factor. At the end however, we don't care why.

Complexity Kills. Computers already have 1000s of layers of software between the Human and the Hardware. Putting a Rube Goldberg application on top does nothing to help.

Our KYSS Method

So what did we do to fix it? Our policy was to pick the low hanging fruit first. About 80% of the project was PHP we decided to push it all there. We simplified the system by replacing the Tomcat/Java piece with a PHP component. Reduced by one language and Tomcat dependency. Then we focused on performance of that component to reduce resource (network/memory) consumption by over 60%) Added a bit of caching and it was done. Then the back-end libs in PERL and Bash were streamlined to PHP as well.

The end result was taking a project with three web-server-software dependencies and reducing that to two (Apache/Lighttpd). Reduced the language count from five to two. A result of those to is a smaller, faster and simpler configuration that is much easier to maintain.

It will take roughly eight months to return on our investment of cleaning and simplifying this project. Awesome.

There are many cases when multiple languages need to be used, too numerous to list. In the end a seasoned architect & business analyst need to evaluate this and make a plan. Otherwise, like in this example, developers/engineers go their own fractured way and entropy reigns.

PHP Timezone Handling

Today we saw another abomination of Timezone handling in PHP.
100s of lines of code were used with many hard coded values and the system was still incomplete.
So, in efforts to put a stop to this we've posted our implementation of php timezone handling which we think is easy, fast and portable.
We've included a few other examples too, but just to show why we don't use them.

javascript:void(0)

E-Commerce Migration from Django/Satchmo to PHP/Radix

One of our clients is using an abomination of e-commerce called Satchmo.
The client has described the solution as "cumbersome", "difficult", "slow" and a few other words not suitable for print.
We've been assigned the task of re-vamping their solution.

Despite the odd database schema and lack of documentation we're confident that this can be re-worked into our PHP/Radix based solution in short order.
Check back in three weeks for details.

HP Printers & Softare are the WORST!

Edoceo has been recommending against HP products (printers specifically) since at least 2002.
Why? Their hardware quality is very low and their software is bloated and buggy.

Today a client's HP PhotoSmart C4780 WiFi printer which was only a few weeks old stopped working.
The WiFi had died a few days ago and today the touch-panel screen was non-functional.
We decided to return the printer and get a better quality Brother.

When un-installing the nine (yes, really nine) software packages that HP requires to use the printer (totalling 580MiB!) their buggy software then wiped out the "All Users" settings directory.
Desktop icons lost, Start Menu items gone.

As if a low quality printer with bloated software wasn't bad enough when you try to extricate yourself from this HP Printer hell it destroys software & settings it has no business touching.

Edoceo's new policy regarding HP hardware/software?
Do not acquire, do not use, remove promptly (manually).
These products are not suitable for use in Business, Home or School environments.

First Weeks with the HTC Evo

Have been using the HTC Evo since it's launch. The experience buying that day was the same for us as it was for everyone. Sprint's site was slow, the phone took a long time to activate and only came with about 40% battery (weak).

After a week of using this phone here's what we think, first the positive, then the negative.

• Fast, Andriod
• Cannot Un-Install Sprint Applications
• Cannot Un-Install Facebook Application!
• VoiceMail is an epic failure, no notifications
• Sprint Support hardly knows this device, very unhelpful
• VoiceMail Service is not flexible and forces users to un-desired options.
• Phone Battery is not sufficient - barely lasts a full day.

And to top this all off the charger stopped charging. Had to return to Radio Shack for a replacement and they did not have any. Sent me to another Radio Shack, who could only swap for a non-OEM type adapter. Pretty lame.