Rojo

I’ve been looking at replacing Munin with our own higher level proprietary monitoring system for keeping track of cluster-wide statistics.

This is needed for a new feature we’re trying to ship with Spinn3r so that we can expose some of our internal statistics to our customers.

We weren’t able to do this before because our internal monitoring was a bit of a hack due to the lack of quality in the open source monitoring tool chain.

About 80% of the work in a performance monitoring tool is in the charting component and I’m glad to see that the Google chart API basically rocks.

It’s very well done. The only real complaint I have is that you can’t submit a URL longer than 2000 bytes. It seems like a limitation in their own internal tool as Firefox, Safari, etc can support URLs up to 65k.

This limitation is more pathological than you would think because you can’t place two metrics on top of each other in the same graph. They need to move into two separate graphs or you’ll generate a long URL and Google will return an HTTP 40x response.

The other cool thing is the API is FAST. Check out the render time of this chart which is 300k pixels….

I can now render graphs in about .09 seconds which is an order of magnitude faster than Munin with rrdgraph on a quad core Opteron with a 2.6Ghz processor.

I hope they do something about exposing the component used with Google Finance. It is pretty sweet as well.

This has me thinking:

When Amazon S3 receives a PUT request with the Content-MD5 header, Amazon S3 computes the MD5 of the object received and returns a 400 error if it doesn’t match the MD5 sent in the header. Looking at our service logs from the period between 6/20 11:54pm PDT and 6/22 5:12am PDT, we do see a modest increase in the number of 400 errors. This may indicate that there were elevated network transmission errors somewhere between the customer and Amazon S3. We are continuing to investigate and will post an update when we have further information.

The MySQL protocol doesn’t seem to have a checksum or hashcode:

If a frame were corrupted in a way that didn’t break the SQL, then it would be possible to INSERT/UPDATE/DELETE incorrect data.

Granted, adding MD5 or SHA1 would burn a bit of CPU but it might be worth it in a lot of situations.

… of course the TCP checksum can come into play here but it’s just a weak checksum.

When you’re pushing LOTS of data this could be seen in production environments (not just in theory).

There doesn’t appear to be any documentation on how to figure XFS to stride across a RAID array.

After about two hours of google searches I finally figured it out.

There are two variables - sunit and swidth:

sunit=value

This is used to specify the stripe unit for a RAID device or a logical volume. The value has to be specified in 512-byte block units. Use the su suboption to specify the stripe unit size in bytes. This suboption ensures that data allocations will be stripe unit aligned when the current end of file is being extended and the file size is larger than 512KiB. Also inode allocations and the internal log will be stripe unit aligned.

swidth=value

This is used to specify the stripe width for a RAID device or a striped logical volume. The value has to be specified in 512-byte block units. Use the sw suboption to specify the stripe width size in bytes. This suboption is required if -d sunit has been specified and it has to be a multiple of the -d sunit suboption.

So based on this here’s how to configure both values:

sunit = strip_size / 512
swidth = sunit * nr_raid_disks;

In my situation I’m using RAID 0 across 5 disks so my sunit is 128 and my swidth is 640.

In hindsight I think the su and sw options (expressed in bytes) would have been easier to compute.

su would just be 65536 and sw would just be 5 * su.

In our tests this yielded a 30-40% performance boost so certainly worth the trouble.

XFS is supposed to do this internally by calling an ioctl on the device to obtain the stripe settings. It doesn’t appear that the megaraid driver that we’re using supports this and it came back as zero/zero for both sunit and swidth. I imagine if you’re using Linux software RAID that it will work just fine.

By my math this is 15TB of storage as the original xbox had an internal 10GB disk.

Well I’m up in Seattle for another week. I’m going to be here until the Google Scalability Conference.

I’m trying to compile a list of decent places to work out of until I head back to Seattle. I’ve become spoiled by my 30″ Apple cinema display setup.

I’m going to head over to Victrola Coffee tomorrow morning.

Any other suggestions?

I will often use a TreeMap in place of a HashMap because while TreeMap is theoretically slower (O(logN) vs O(1)) it is often much more memory efficient, especially for larger maps.

The problem is that you can’t just swap your map for TreeMap without modifying code because you can’t store null keys in TreeMap.

This code:

map = new HashMap();
map.put( null, “bar” );
System.out.printf( “%s\n”, map.get( null ) );


works fine and prints ‘bar’.

This code:

map = new TreeMap();
map.put( null, “bar” );
System.out.printf( “%s\n”, map.get( null ) );

… will throw a null pointer exception.

This is true because the TreeMap Comparator doesn’t support comparing null values. Why not?

I can pass in my own Comparator but now I need to remember this for every instance of TreeMap.

Come to think of it. I’ve often seen Map implementations slowing down somewhat decent code. The 2N rehash functionality of HashMap often means that it can blow your memory footprint out of the water and crash your JVM.

If one were to use Guice to inject the dependency on the right Map implementation 3rd party libraries would be able to swap their Map implementations out at runtime.

Further, if memory is your biggest problem, I think you might want to punt on using the Java internal Map implementations and use FlatMap (even though it has some limitations).

Check this out….. Imation is OEMing Mtron drives for 20% cheaper then the normal/stock Mtron drive.

Same hardware but 20% cheaper.

Spinn3r is hiring for an experienced Senior Systems Administrator with solid Linux and MySQL skills and a passion for building scalable and high performance infrastructure.

About Spinn3r:

Spinn3r is a licensed weblog crawler used by search engines, weblog analytic companies, and generally anyone who needs access to high quality weblog data.

We crawl the entire blogosphere in realtime, remove spam, rank, and classifying blogs, and provide this information to our customers.

Spinn3r is rare in the startup world in that we’re actually profitable. We’ve proven our business model which gives us a significant advantage in future product design and expanding our current customer base and feature set.

We’ve also been smart and haven’t raised a dime of external VC funding which gives us a lot more flexibility in terms how how we want to grow the company moving forward.

Overview:

In this role you’ll be responsible for maintaining performance and availability of our cluster as well as future architecture design.

You’re going to need to have a high level overview of our architecture but shouldn’t be shy about diving into MySQL and/or Linux internals.

This is a great opportunity for the right candidate. You’re going to be working in a very challenging environment with a lot of fun toys.

You’re also going to be a core member of the team and will be given a great deal of responsibility.

We have a number of unique scalability challenges including high write throughput and massive backend database requirements.

We’re also testing some cutting edge technology including SSD storage, distributed database technology and distributed crawler design.

Responsibilities:

• Maintaining 24 x 7 x 365 operation of our cluster
• Tuning our MySQL/InnoDB database environment
• Maintaining our current crawler operations
• Monitoring application availability and historical performance tracking
• Maintaining our hardware and linux environment
• Maintaining backups, testing failure scenarios, suggesting database changes

Requirements:

• Experience in managing servers in large scale environments
• Advanced understandling of Linux (preferably Debian). You need to grok the kernel, filesystem layout, memory model, swap, tuning, etc.
• Advanced understanding of MySQL including replication and the InnoDB storage engine
• Knowledge of scripting languages (Bash and PHP are desirable)
• Maintaining software configuration within a large cluster of servers.
• Network protocols including HTTP, SSH, and DNS
• BS in Computer Science (or comparable experience)

Further Reading:

• Spinn3r
• Spinn3r Architecture Talk from 2008 MySQL Users Conference
• Spinn3r Corporate Blog
• Kevin Burton’s Feedblog (CEO of Spinn3r)

It dawned on me that if I were working for Twitter that I would just assume the service is down unless told otherwise.

This lead to the conclusion that one should invert monitoring to send off a notification when Twitter is online …

Seriously. I like those guys but this is getting kind of embarrassing.

As someone interested in distributed, scalable, and reliable web services, I think I might stop using it out of protest.

Things could be worse though - they could be using Hadoop!
You can see a picture of Twitter’s main database server below:

Ha….. this is what I get if I try to share an item.

I guess they’re using AHAH?