rake-dotnet talk at Skillsmatter notes

The presentation I gave yesterday at Skillsmatter seemed to go well; I was really enthused when a few people came up to me afterwards and said they’d start using it :-)

Notes & links:

All quiet? No. Stand-up comedy imminent…

I haven’t posted for a while, clearly.  Why?  Real-life happenings – new job, new girlfriend, both at the same time.  Life is currently rather good.

The new job is as a developer at Onalytica.  I’m still figuring out precisely what it is I’m working on, though, and exactly what I can blog about in reference to it.  I can tell you it’s a great change, however; just the fact that I’m now working in a team where everyone is in the same room is a massive improvement!  Suffice to say I’m enjoying myself immensely.  We’re hiring, by the way; currently looking for a quantitative models researcher and an internet-scale data-centre Windows sysadmin.

I’m putting in some effort on IncludeCombiner, so I can get to the stage where I can contribute it to MVCContrib; the idea is that it’s a drop-in, turn-on kind of toy for people doing ASP.NET MVC websites.  “Oh, I need to improve my site’s loading times; what’s the easiest thing I can do?  Ah!  Turn on CSS/script minification/compression/caching!” kinda thing.

I’m also gradually working on rake-dotnet, though here I’m incrementally fixing issues that come up during real-life use at work and on IncludeCombiner.  Kyle Baley wired it up to Codebetter.com’s TeamCity instance recently, and once the latest rake-dotnet gem is installed there, I should see the first green build (“It works on my machine!” dammit! ;-) ).  Next on the list to-do is Gallio support, NDepend support, more NCover reporting support (since the nice people at NCover granted me a license for this, I kinda feel obligated!) and a few bits of friction-elimination like detecting where 3rd-party tools are installed when they’re not present in source-control.

I’m also preparing for my first ever round of stand-up comedy; I’ll be presenting rake-dotnet at the forthcoming Skillsmatter event on 16th June in London.  I’m quite excited about it.  It won’t just touch rake-dotnet (since the point of the library is “working build script in 30ish lines of code” I don’t think that will take awfully long!); I’ll see if I can work in a demo of wiring a project up to TeamCity continuous integration, since that’s (CI, not specifically TeamCity) most of the point of having a one-click build in the first place.

Improving front-end website performance in ASP.NET MVC

Load times, according to people like google, are important.  I kinda see it as basic politeness; wasting a user’s (or worse, a customer’s) time making him wait for my site to become active is a bit rude.  According to YSlow! (a Firebug plugin within Firefox), one should:

  • Reduce the number of external requests (or; make one big Javascript/CSS request instead of many little ones) – browsers typically will only hold two open connections per domain, and scripts load sequentially (I gather because of the possibility of document.write calls modifying the DOM, which would be a shared resource if they were allowed to load and execute in parallel).
  • Reduce the size of external requests (less on-the-wire weight is better; remove comments and white-space).
  • Compress external requests where possible (gzip/deflate compression will trade CPU cycles per request to gain better on-the-wire weight).
  • Use a far-future cache-header for assets that won’t change often (so that clients don’t need to receive a new copy on each visit).
  • Move scripts to the end of the document so the content loads first – the user has stuff to look at while scripts arrive.

Happily, all achievable with only a modicum of effort.  Yahoo has released its YUICompressor component to open-source, and there’s a .NET version available from codeplex – this takes care of minifying Javascript and CSS.  So, all we need to do is glue the files together and push that glob to the compressor.  Right?  Well, we also have to tell our component which things to work on.  This set will vary, usually, per page (ideally, we will offload all script/style to separately-loaded assets, since we can cache those and slim down our actual document at the same time by doing so).  It’s more pleasant to develop within individual files; one file per class/feature/whatever; given that we’re not perfect, it would also be invaluable to be able to turn on a debug-mode so we can debug through readable code.  

Lastly, do this stuff at run-time, because maintaining config or build-automation to do it would save some CPU cycles at the expense of developer time (can you spot someone that’s been burned by that kind of thing…? :-P ).

I’m going to be doing this within the context of ASP.NET MVC (and maybe fiddle with Spark’s modularity feature once it’s done).  My list of features:

  • Register an include (from both the master-page and the views).
  • Write out the include-combination (or, in debug mode, write out each include separately, raw and untouched).
  • Combine the includes within the set that is registered (glue the files together – and, necessarily, read the files from where-ever they live!).
  • Make it possible to link to the combination (a controller with actions to respond with the combination when tickled via an appropriate GET)
  • Minify the combination.
  • Compress the combination (gzip/deflate – and detect which browsers we shouldn’t do this for – I’m looking at you, IE<=6, you lying so-and-so…).
  • Cache the combination with a far-future expiry (and therefore also use a URL amenable to browsers caching it).
  • Generate an ETag.
  • Support a debug-mode where the includes are written out one-by-one and raw.
  • Be trivial to understand and add to your own application :-)

I’ll host the source in github, and blog as I go.

Some more references for front-end performance:

rake-dotnet release v0.0.8

Just a smattering of small changes and bug-fixes; here’s the change-log:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
=== 0.0.8 / 2009-?
 
* CHANGE: Eliminate the need to pass stuff to AssemblyInfoTask when it can just use the defaults ... by default.
* FIX: Bug with harvesting when output contains directories
* FIX: Test-runner now executes from within the same directory as the DLL(s) containing tests that it's running through.
 
=== 0.0.7 / 2009-04-15
 
* CHANGE: Adjust TOOLS_DIR initialisation so it looks for a shared 3rdparty folder at the same level as the product folder first, then it looks for a 3rdparty folder inside of the product folder.
* CHANGE: Adjust the Demo's Rakefile to make FxCop ignore assemblies with "Tests" in their filename.
 
=== 0.0.6 / 2009-04-06
 
* Move source control to github at http://github.com/petemounce/rake-dotnet
* Change format of readme to markdown for prettiness on the repo homepage
* FIX: YYYYMMDD is invalid for file/assembly-version attribute; use '0' instead in non-svn source-control case.
* FIX: Fix harvesting a web-application when it's not under svn source-control

What’s next? The nice people at NCover have granted me a free license to their code-coverage tool; I’ll be wrapping that in the next major release. I also plan to implement tasks for NUnit and Gallio. That should do for planning, for the time being…

Oh – it occurs to me that I’ve talked about it, but not really shown what it allows (though there is a demo inside of source-control -> DemoRoot). So, here’s the ~35 line Rakefile that gets you:

  • AssemblyInfo watermarking
  • compilation
  • XUnit.NET testing/reporting
  • FxCop reporting
  • simple NCover reporting (as in, only one report choice at the moment)
  • harvesting-to-one-place for publishing to CI artifacts
  • packaging output into zip files for xcopy-deployment (even if xcopy is so … DOS… ;-) )
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Documentation: http://rake.rubyforge.org/ -> Files/doc/rakefile.rdoc
require 'rake'
require 'rake/tasklib'
require 'rake_dotnet'
 
PRODUCT_NAME = ENV['PRODUCT_NAME'] ? ENV['PRODUCT_NAME'] : 'IncludeCombiner'
COMPANY = ENV['COMPANY'] ? ENV['COMPANY'] : 'NeverRunWithScissors.com'
RDNVERSION = Versioner.new.get
 
Rake::AssemblyInfoTask.new
bin_out = File.join(OUT_DIR, 'bin')
Rake::MsBuildTask.new({:verbosity=>MSBUILD_VERBOSITY, :deps=>[bin_out, :assembly_info]})
 
Rake::HarvestOutputTask.new({:deps => [:compile]})
 
Rake::XUnitTask.new({:options=>{:html=>true,:xml=>true}, :deps=>[:compile, :harvest_output]})
Rake::FxCopTask.new({:deps=>[:compile, :harvest_output]}) do |fxc|
	fxc.dll_list.exclude("#{@suites_dir}/**/*Tests*.dll")
end
Rake::NCoverTask.new({:deps=>[:compile, :harvest_output], :ncover_options=>{:arch=>'amd64'}, :ncover_reporting_options=>{:arch=>'amd64'}})
 
demo_site = File.join(OUT_DIR, 'Demo.Site')
Rake::HarvestWebApplicationTask.new({:deps=>[:compile]})
 
Rake::RDNPackageTask.new(name='bin', version=RDNVERSION, {:deps=>[:harvest_output, :xunit]}) do |p|
	p.targets.include("#{bin_out}/*")
end
Rake::RDNPackageTask.new(name='Demo.Site', version=RDNVERSION, {:deps=>[:harvest_webapps, :xunit]}) do |p|
	p.targets.include("#{demo_site}/*")
end
 
task :default => [:compile, :harvest_output, :xunit, :package]

The Rakefile is from the other thing I’m working on at the moment while I’m at a loose end. Watch this space…

TeamCity 4.0.2 + rake-dotnet == parallelisation in the build cloud

I’ve said before; I’m kinda the tools-guy at work. Never get a human to do a computer’s job, you know? With that in mind, I’m working on a library of custom rake tasks for making a build automation dev’s life easier. That’s only half the story, though – the ability to make a build in one click is useful, and all, but it’s far more useful when there’s a continuous integration process doing it for you. Checking out the latest code, building it, running tests against it, packaging it, deploying it (I was particularly inspired when I read IMVU’s doing the impossible 50 times a day…!) – all of that good stuff that, without computers, would fail miserably because humans just aren’t good at doing mind-numbingly repetitive tasks from checklists consistently well.

So I’ll explain a few of the design decisions I made when working on rake-dotnet.

Firstly, hopefully the aim to keep the Rakefile as short as possible is understood and well-received. I wanted it to be as by-convention as possible, because honestly, writing a build-script for the 80% case is surely pretty much a solved problem by now; conceptually, at least. It’s often making time to do it that’s the hard part, right?

Secondly: Why isn’t the Rakefile in the root, next to the solution file, so I can just type rake and have stuff happen? I need to cd build; rake before I see stuff! I did this because of the way TeamCity (TC) and subversion (svn) work together. In TeamCity you set up a Version Control System (VCS) root. The simplest thing is one pointed at the trunk of your codebase, so it gets everything. You can customise the VCS roots by telling TC to only check out paths matching some rules you tell it, or omit externals, and some other tricks. However, it doesn’t check out lazily, it checks out greedily (as at 4.0.2 anyhow). Say you set up the VCS root {svn}/trunk and a checkout rule to skip {svn}/trunk/foo – it’ll check out everything from {svn}/trunk and then, afterwards, delete the foo directory. This takes time – at my place, we’ve got a modest project, but sadly a slow svn server – the checkout takes more time than the do-work part of the build.

A bit of background. For this project, we have one svn repo for the code that we write, and one for the 3rd-party libraries, tools, code, and other artifacts that we depend on to produce a build. This keeps the size of checkouts smaller for remote staff over the VPN (which is also not the most robust), and backups quicker (since we make fewer changes to the 3rdparty repo, we back it up less frequently, especially since it’s not complicated to reconstruct because we’ve got commit emails of what changes happened to it). We also only have the one NCover license, which is installed on just one TC build-agent – this means our setup can’t just run all the build-tasks from beginning to end, since the other two agents don’t have an NCover license. So, TeamCity isn’t set up in the simplest-way-that-works. A build project has a number of build configs:

  • Continuous (in Debug configuration) (this will generate AssemblyInfo.cs files, compile, run unit-tests, harvest the artifacts, and publish them to the CI server for later configs’ use)
  • Coverage
  • FxCop
  • TeamCity Duplicate Finder
  • Continuous (in Release configuration)

It would be easy to see that later we’d also want:

  • Integration Tests
  • NDepend
  • Deploy to test environment
  • UI Test
  • Deploy to production
  • Notify clients of new functionality drop :-)
  • etc

As I say, running end-to-end in each build-configuration on each agent is not only not possible, it’s wasteful. It’s also wasteful to hammer our feeble little svn server for a full checkout of stuff we don’t need during subsequent stages (we don’t need the source-code if we’ve got a convenient zip file of all the build output already, right?). Fail fast. So, how? VCS roots. The table below describes how I’d set up the rake-dotnet Demo product in TeamCity (assuming the rdn FxCop task didn’t have a bug in it that I need to fix ;-) ).

Demo TeamCity build-configs

(or as HTML: [download id=”3″])

What all of this effort lets us have is a config up front that does the basics (compile, unit-tests-that-are-fast, package, etc), then pushes that up to the CI server; the process then branches out into parallel-building by taking advantage of the fact that TeamCity gives you 3 agents for free.  Each config pulls down the DLLs etc that the first one generated, and runs off them.  The subsequent configs will run in parallel, then the Continuous-Release one will run if all three of the Order-2 ones pass.

Now, if only I could get the git plugin to work.  No joy, so far.  Git renders some of this irrelevant, because its checkout speed is so much faster that svn’s (in our environment, at any rate).  Another day…

rake-dotnet – source moved to github.com

I’ve jumped onto the git bandwagon and discovered github.com – I’ve moved the rake-dotnet source-code to a new git repository there.  Now if only I could break the habit of pushing on every commit…  Thanks to Aaron for a helpful guide on migrating, and myriad others for other guides and tools (I was particularly enthused when I saw how quickly TortoiseGit has become featureful!).

Raking it in; rake-dotnet “Hello World!” release v0.0.5

It’s been quite a while since I said I was going to publish rake-dotnet.  In that time, I’ve learned more about making custom tasks for rake, and how to publish a ruby gem (which is ruby’s packaging mechanism).  I’ve also implemented some features:

  • AssemblyInfo generation – generate assembly-info files that pull in:
    • product name
    • company name
    • build configuration
    • build date
    • version number (major.minor from a text file; build from an environment variable (or 0 if not build with CI), revision from svn working copy).

    This task will overwrite Properties/AssemblyInfo.cs in each project within the source directory, either from a template side-by-side to it, or from the common template in the source directory.  If source-control is other than Subversion, revision number is replaced as YYYYMMDD of the current date.  If the script is run locally, build-number will not be set (since this environment variable comes from TeamCity, or another build-server).

  • MsBuild compilation – compile projects using MsBuild.  Vary the output verbosity and the configuration.
  • Harvest the output to an all-in-one directory, for later packaging for convenience of artifact-publishing to a build server like TeamCity.  This enables parallelisation, because we can push out the artifacts to other agents to process their own tasks against (farming out separate test suites or coverage processing, for example).
  • Harvest web-applications to a directory each for later packaging for deployment convenience.
  • Run XUnit.NET fixtures and generate reports
  • Run FxCop and generate a report
  • Run NCover to generate coverage data, and run NCover.Reporting to generate reports off that data (currently, just one report type of many in v3)
  • Package up output for publishing to build-server artifacts.  The conventions are set up with a build-server distributed build in mind.
  • All of that in a Rakefile that is currently less than 40 lines long :-)

So; I’ve wrapped up what I’ve done in a ruby gem; you can install it:

  • First get ruby; running the installer with defaults is fine
  • At a command-line: gem install rake-dotnet --include-dependencies (prepend ‘sudo’ if you’re not on Windows).
  • Get a whole bunch of tools – see the gem’s README.txt for details (gems are installed under {ruby}/lib/ruby/gems/{ruby version}/gems/.
  • svn checkout http://my-svn.assembla.com/svn/nrws/trunk/rake_dotnet/lib/DemoRoot to see the demo project.
  • Populate the 3rdparty tools directory (inside DemoRoot) according to the readme.txt within (I plan to make this easier…).
  • Try out the demo by changing into its build directory and running rake from the command-line.  rake -T will give you a breakdown of available tasks.

The tasks assume conventions for things like tools-location and solution layout.  So until I document those properly with rdoc (on my list of things to learn, right after unit-tests in ruby!), here they are:

  • PRODUCT_ROOT defaults to .. – rake’s working directory is build/, where Rakefile.rb is used.  All paths are relative to the rake working directory.
  • OUT_DIR defaults to out, hence equivalent to #{PRODUCT_ROOT}/build/out – build-output gets squirted here.
  • SRC_DIR defaults to #{PRODUCT_ROOT}/src –  buildable projects should live here (this includes test projects).  One directory per project, directory-name matches project-filename matches generated assembly-name.
  • TOOLS_DIR defaults to #{PRODUCT_ROOT}/../3rdparty – intentionally, tools are kept outside of the source tree.  This allows for efficient xcopy deployment, or a source-control symbolic link arrangement (svn:externals works well).
  • test projects should have “Tests” somewhere in their project-name.
  • web-application projects should have “Site” somewhere in their project-name.
  • msbuild is assumed to follow its defaults and output into {project-being-built}/bin/{configuration} for class libraries and so forth.
  • Each project directory within SRC_DIR is assumed to have a Properties directory where AssemblyInfo.cs in generated. The template for this file is taken either from a side-by-side AssemblyInfo.cs.template file, or #{SRC_DIR}/AssemblyInfo.cs.template

The Demo solution is set up with these in mind, obviously.  They are possible to override to some extent by passing constants via the command-line like rake MSBUILD_VERBOSITY=n CONFIGURATION=Release TOOLS_DIR=../tools.

Source code is available, of course.  I have yet to figure out rubyforge for a project home-page and announcements, rdoc for documentation, and ruby unit-tests for ensuring quality.  There’s a roadmap in the readme; next major release will be 0.1.0 and will see more in the way of test coverage, nunit support, and more ncover reports (I have to do these within the 21-day trial license, I think…).  I am also likely to move the source control to github, since I’ve been wanting to learn git, too…

Successfully migrated

Well, distractions abounded over the last couple of weeks, but if you’re reading this, welcome to the new host.

Migrating to another host

The web-host this is currently on has had some of its IP-range spam-blacklisted; my domains appear to have been tarred with the same brush as the spamming customer.  It seems like it’s not possible for support to get me (and I guess others) un-blacklisted, so I’m jumping ship to webfaction.com.  This will undoubtedly involve a bit of downtime/DNS-mismatchery, but hopefully this will be confined to the weekend…

If you need to reach me in the meantime, use my firstnamenotabbreviated.lastname [at] gmail.com address and defeat my anti-spambot encryption.

Building a web-crawler for fun and … the hell of it

So I’m reading Programming Collective Intelligence at the moment, by Toby Segaran.  It’s about things like search-engines, recommendations-engines, filtering/sorting results, data-mining and generally Web 2.0 social interactions.  This is interesting in (at least!) three ways:

  1. The examples are in Python, which is new to me, and outside my comfort-zone (I’m writing this as I wait for XCode to come down, so I can get gcc with which to build pysqlite – apparently).
  2. It’s making me dust off my maths/stats from way back, which is painful but probably ultimately healthy.
  3. C’mon; teaching computers how to be smart?  It just is, ok?  Figuring out why people like things is both interesting and incredibly sellable; developing insight via some automated process is clearly a hugely useful tool!

I’ll be interviewing at a place that cites “experience building web-crawlers” on its job-spec, so I thought I’d have a go, while I’m recuperating at home.  Chapter 4 covers this.  It turns out that it’s possible to create a web-crawler that will go off from a starting point that you give it, pull down that page, parse it (BeautifulSoup appears to provide jQuery-selection-engine-like functionality), pull out the links, and start following them to subsequent pages in … ~30 lines of code (leveraging two libraries).  This is … cool.  I think the same kind of thing in C# would have needed … rather more work!  I wonder whether I can get IronPython to play…?

Anyhow.  Xcode is down now, and gcc with it, so I’m off back to play…  I’ll review the book once I’ve finished it.