Considering there's less than a week left in this one I'm pretty sure this is my movie of the year.
My movie of the year is actually from 2002. And, surprisingly, it's a tv movie. It's called Bang Bang You're Dead. I considered writing a short review/plot synopsis but I won't. All I'm going to say is watch it. It's great.
If you're reading this, I'm sorry. This post isn't for you. It's for me. I've decided to write my hockey app once again (version what? 4? 5? I'm not even counting anymore), this time in Silverlight and this time much more sophisticated.
Planned features:
I'm writing this here in the hope that the fact I've decided to do this is out there will help me actually go through with it. I obviously don't plan on being done this soon, I just hope it'll all be finished before the start of the next season so that I can actually use it!
The SANS Top 20 list is out and check out this one:
* Operating systems have fewer vulnerabilities that can lead to massive Internet worms. For instance, during 2002-2005, Microsoft Windows worms like Blaster, Nachi, Sasser and Zotob infected a large number of systems on the Internet. There have not been any new large-scale worms targeting Windows services since 2005. On the other hand, vulnerabilities found anti-virus, backup or other application software, can result in worms. Most notable was the worm exploiting the Symantec anti-virus buffer overflow flaw last year.
So now it's not your OS's fault you have a virus it's the anti-virus software's.
I'm only surprised it was Symantec and not McAfee. But then again Symantec would be my second pick.
(via Bruce Schneier's blog)
I read Brad Abram's blog and Silverlight is something I want to try. He recently blogged about a webcast introduction of Silverlight. I figured I'd watch it before I started playing around with it, at least it would give me some basis to work from.
Too bad the webcast is hosted by Microsoft.
First step forces you to log in with a passport account (the account you use for hotmail or msn messenger). The second step presents you with a form to register for the "event". However on the top of that screen it says if you don't want to submit this info you can press on cancel. I read that assuming the registration was optional and clicked cancel. I ended up at a screen giving details about the webcast and a button saying register online. I click the button and get the same form as before. I figured they were just going to spam me to fill it out every time I tried to go somewhere but let me through anyway so I clicked cancel again and ended up at the same page I was on before. So I clicked on the button again and filled in the form with bogus info like I usually do. (I live on Capitol Hill!) Then it gives me information about when it was recorded and basically all the same stuff as the page before. So I click lauch webcast or something of the sort. Then I end up at a livemeeting form that wants me to fill out a profile again. Great. More bogus info. Finally get through and what happens?
View And Download Recordings
Microsoft Internet Explorer 6.0 with SP1 and above is required to view the recording in Live Meeting replay format.
All that for nothing. I'm not going to boot into windows and launch IE just to watch a 30 min video. Google lets me stream any of their TechEd videos immediately without a bazillion forms and it works on everything automatically. Sorry Microsoft but you fail at the web.
Epic fail.
This is amazing:
def wazup?(a) puts "#{a} dat!" end
wazup?!!!?!??1:!!??
Happened upon it here. Tim now hates ruby even more. =)
A while back I posted about an error in the static page generation that took down our database server and what were were doing to make sure it wouldn't happen again.
Well, it worked.
One of the sites seemed like it wasn't generating pages (it was actually a cron job stuck in a loop deleting all the static pages non-stop as soon as they were generated instead of every 20 minutes) and it was serving all dynamic content. But memcached saved us. We were getting over a million memcache hits in less than 4 minutes. So yeah, a million db hits from ONE site (there are many) to a single db server in 4 minutes would definitely have taken it out.
Thank you memcached.
I really like Larry O'Brien's blog. It has the perfect (for me) mix of interesting technical stories and random personal posts to lighten the mood. This recent post reminds me of what sometimes happens at work.
Right when I got there I helped design an entirely new system that makes code re-use much easier. Basically we develop "plug-ins" that do some discrete bit of functionality (like output a list of elements in a table-ish form or poll functionality or rating functionality or something like that). However the setup makes some of these independent sites highly dependent on each other. We have all our plugins in a single repository that we check out to our dev drives. When we start a new site or a new site section we typically make a new repo, check it out and, inside, make a symlink to the plugin repo checkout. This way all sites use the same plugins... which sometimes just isn't understood.
I have no idea why someone would be making a new site and, instead of making a new template for the plugin and using that one, leave all functionality the same and edit the default template! Then they put up the site, everything works (well more or less anyway) and people are happy. Until one of the old sites gets changed a little, someone makes a change and then updates (site deployment is typically just an svn checkout on the right server). Then the whole thing comes crashing down because the class names of all divs for some plugin were changed and none of the css is being applied to them so the whole thing looks like shit. Then we run around going wtf, check subversion for the commit logs and notice someone edited the default template and therefore have to revert back to the old, make a new one and fix BOTH sites (fix the new one too so that it uses its own specific template, the way it should have been done). It can be really really frustrating when something so basic is done completely wrong.
I don't think I've sworn at anybody yet though. =)
Monday I made a big mistake.
We deployed the new member's area and had messed up the static page generation so that only a few of the pages were actually being generated and properly served back while everything else, including all the heavy hitting pages such as the home page were all dynamic. The mysql server couldn't handle the load and crashed. All our systems run on the same mysql box so while it was rebooting everything was down and we swapped the old member's area back in so that everything wouldn't go down again. We then spent monday fixing up the static page generation and it worked. Everything was ready to be deployed for tuesday morning but then we decided we might as well go all out and really start scaling our systems.
This whole problem could've been avoided had everything not been running on a single mysql box. That's a single point of failure and those are very very bad. Luckily we have an ORM layer so it's easy to make database access changes. Our apps only use objects and make calls to Mappers to get these objects. The Mappers call TDGs to get the information they need from the database. All sql is contained in the TDGs. (So Controllers say can has object? to Mappers and Mappers say can has sql result set? to TDGs.) We used to use MDB2 to access the db but not anymore. Yesterday we wrote a wrapper class for memached and overall db access. I'm going to describe our API but first, a bit about memcached.
Memcached was thought up by the LiveJournal guys as a way of saving your database. It seems to basically be a big cache of db results (probably a bit more complex than that but overall it gives a good picture). You have to build your own key to store results with (we picked an md5 hash of the querystring itself) and then you just look in the cache to see if you have anything fresh for that key (there's a timeout that you can specify for when a cache entry goes stale that we set at 3 minutes). If you find something then you can just use that, otherwise you do the db lookup, store the results in the cache and then return them. We now have memcached daemons running on all servers.
The way we used to access the db was that we would request a new db connection through MDB2 (a PHP database abstraction package) and use that. Something like:
$db = MDB2::singleton(AFramework::getConfig("dsn"));
$query = "Something though probably not built as a string, probably use something like autoPrepare and autoExecute instead";
$result = $db->exec($query);
if (PEAR::isError($result))
throw new FrameworkTDGSQLErrorException($result->getMessage()." - ".$result->getUserInfo());
Now, we instead do:
$query = "Something";
$result = DBConn::exec($query);
DBConn has 3 methods we're interested in: insert, select and exec.
Calling insert will return the last auto_increment id and calling exec will return the number of affected rows but all methods are quite flexible. If you give it just a query it will execute it on the default db. (Which, in the case of insert or exec is the default write db but, in the case of select looks up in memcached and then selects one of the read dbs through a balancing method.) However, you can give it an array with one or more options.
Array options:
'query' => $query : The query to be executed
'old' => true|false : Whether to access the new database or the legacy database for postbacks (default: false)
(This next one for select only)
'type' => "all"|"row"|"one"|"query" : Whether to return the results of a queryAll, queryRow, queryOne or query (default: "all")
Also, all methods do the error checking and throw exceptions as in the first example if there's an sql error. Instead of being copy/pasted all over the codebase like before it's now just in one spot. Copy/paste is the devil.
So now it's not only much more flexible but a whole lot more reliable and scalable. This would've been a nightmare to implement if the sql was just scattered randomly across the application though. Yay for good design. Now we can just keep executing our queries nearly the same way as before and it's all balanced across multiple reads with a write that replicates to them along with memcached access to save db hits. In the end we've wrapped MDB2 database abstraction, memcached lookup and storage along with balanced read db access and selective write db access into a single abstraction. Pretty sweet.
Ok, some of you know that when I started my new job back in may, I was basically told we want new everything from scratch.
So we hunkered down and came up with a decent way of doing web development in PHP (there are many others but many more are horrible). We basically made a cross between Rails and J2EE, taking the parts we liked from each and making some up on the fly.
I obviously didn't re-implement everything we did because a) I'm too lazy and b) I didn't need it all. So the following is a description of all the stuff I didn't implement and at the end of the post there's a link to a zip containing the source code to this year's hockey pool app I wrote based largely off the ideas we use at work. The two second summary of the zip contents are: extremely simple MVC with a very basic ORM layer.
We have a "Framework" that automatically loads "Modules" if they're properly configured. There are only 3 configurations you need to do. You need a config.ini which tells the framework what mode to run in (local for development off our dev drives or live when deployed so that it negotiates connections with Strongbox, our security software) and gives the database connection string, a module.ini which gives 3 or 4 parameters and a navigation.xml which builds the menu on the left of the cms and also manages permissions.
The navigation.xml looks something like:
<action>
<name>polls</name>
<action>
<name>add</name>
</action>
<action>
<name>manage</name>
</action>
<name>edit</name>
</action>
</action>
<action>
<name>view</name>
</action>
</action>
It's not quite like that but this should give a good idea of how it works. The navigation menu will look something like:
POLLS (a header)
add (a link to action=polls/add)
manage (a link to action=polls/manage)
view (a link to action=polls/view)
Only the top-level of each category is drawn in the navigation. To access edit we need to go to action=polls/manage/edit and generally the manage page will show a list of all polls and they will have links to edit. We also manage permissions like this. We can give permissions to polls/add or polls/view without giving permissions to polls/manage/edit. It also gives us very basic logging capability. (And we can add more fine-grained logging through calls to the framework's logger.)
We also use Savant3 as our template engine which I'm not using for the hockey app. We have special functionality for ajax requests (if you pass guimode=ajax in the querystring it'll handle it as an ajax request) and we also get static page generation pretty much for free. We just need to add generate=true to the querystring and it'll generate a new page. Otherwise it'll serve a static page if one exists on any page defined as being static. Of course there are some parts of pages you want to stay dynamic (such as choosing a random poll on each page load instead of always the same one and ajax requests in general although that's a trickier issue) so you can put a special html comment we defined around any part of a template you want to keep as dynamic and it'll keep the php in the static page instead of the generated html.
So, as promised, here's a link to the source of the hockey app. None of it is commented but it's extremely straight-forward (I think). At work everything is commented with a specific style @description blah blah etc. It's useful because our editor (PhpED, a really great php editor) will give you the information in the comments if things are properly commented when you're looking for which function to call. Quite useful sometimes. So while the commenting is an exception, the rest of the coding style is pretty much what our entire codebase at work looks like.
One of the things you might notice when looking at the source code is that the ORM layer is nearly useless since I pretty much use the database field names directly in the classes. We solved this problem by having a method mapFields in the mappers that map db field names to the names we want. So when we first designed the db we also used the names in the classes as private members (since we liked the names and find keeping very short strings as names for db columns useless when you have an ORM layer, it just makes things harder to understand when looking at only the db without saving typing) but whenever we make a change to the db structure, we just need to add an entry to mapFields to keep everything working.
For example, let's say you have a db column called site_short_name in the sites table. It really pisses someone off because he finds the site_ prefix redundant. So he does into the db and renames the field. He also goes through the entire application and changes site_short_name to short_name. (He's very pedantic.) But he only did it for one project! He forgot all the others that also use the same classes. Now normally everything would crash in on itself (or at least the short names of all sites would be blank everywhere) but by adding an entry to the hash we store, in mapFields ('site_short_name' => 'short_name'), everything kept working because the fields are passed through the map and either can be used. It keeps things a little safer when multiple teams working on multiple projects use the same classes/mappers/tdgs.
I didn't implement the mapFields method because I'm alone working on the app and because the db structure isn't going to change but basically you just call mapFields with your result set as the first action in the load method, passing it your row and it passes it back properly mapped.
The ORM layer I implemented isn't exactly like what we use at work but it's fairly close.
The whole picture worth 1000 words thing is an annoying cliche but I was struck by the image on this post by Roy Osherove.
It's the perfect metaphor for software. We build it as a nice series of links, have good abstractions and we think it's generally solid. Then we notice a bug. We go in and fix it so well and make so sure it's fixed that it's the most solid part of the system. Then, as deadlines start looming and we start getting tired of the project or just all-around burnt out, our fixes get less rigorous and new feature requests or feature changes aren't implemented as well as everything else and our software starts looking like that image.
