From a TeamCity install, here’s your sad build agent status:

“Agent will upgrade”…hm….

After giving the account that runs the TeamCity build agent service local administrator privileges, the status magically changes:

Gah!!

So, really, the agent WON’T upgrade, because it lacks some (as yet unknown) permission.  Once I give it permissions, it apparently will upgrade (or do something; I’m not convinced “upgrading”, whatever that means in the TeamCity context, actually happened).

Tell me what the real problem is, and don’t give me false/misleading info.

In the process of getting a system of build scripts and CI server in place for a codebase living in Kiln, I blithely ran hg from a PS command line and was perplexed for a moment at what happened:

Huh?

Turns out that when I installed Cygwin on the new laptop, I had chosen the Mercurial application that Cygwin makes available.

which hg

told me that I was indeed trying to fire up the Cygwin version,  because Cygwin was placed ahead of the Kiln/Mercurial directories in my PATH environment variable:

$env:path | split-string ";" |% {
    $d = @{};
    if (($_ -match "cygwin") -or ($_ -match "Mercurial")) {
        $d.Add($i, $_);
    }
    $i++
    $d
} | out-host

So just by moving the following two items before c:\cygwin\bin in my PATH, I was able to get the correct (for me) version of Hg to run:

C:\Program Files\Mercurial\;
C:\Program Files\TortoiseHg\;

The title of this post is exactly what I told my students in Cloud 302 the other day, and here’s a good example.

Scenario: setting up Cygwin on an EC2 instance to allow me to SSH in to the box.  After running ssh-host-config, I wanted to run:

Anthony@Anthony-14Z ~
$ mkpasswd -l > /etc/passwd

But in my haste I omitted the > (redirection operator), and instead ran:

Anthony@Anthony-14Z ~
$ mkpasswd -l /etc/passwd

Here’s the error I got:

Huh?  What does the RPC server have to do with anything?  I scratched my head for a second before I realized my mistake and retyped the command.

Had I been a different person, I may have gone hunting around for some RPC server service to turn on, not knowing that the problem had nothing to do with RPC or networking at all.

How would you write an automated test to capture this problem?

I’ve seen this a hundred times – the site itself is set up behind an SSL cert, but there are some elements on the page which are being served up via plain old HTTP, so the browser (in this case Chrome) throws a certificate warning.

Usually it’s either images or CDN resources or 3rd-party client-side components like GA that are the problem, but: how would you write a test?

I don’t have time right now, but I’ll bet the answer is to script out a call to curl or wget and capture any cert errors, or perhaps verify against the site’s self-claimed public cert.  I’ll try that another day.,

Bryan Starbuck, a pretty smart guy as far as I can tell, is wading into a flame war.   He’s giving a talk next week at the Eastside Incubator meetup on “Understanding the OPEN SOURCE STACK very commonly used by startups” (longer title: “Understanding the OPEN SOURCE STACK very commonly used by startups and contrasted to .net for Microsoft employees to understand the comparison”).

Key inferences:

1) Bryan doesn’t think that the “MSFT Stack” is the best choice for startups.

2) See (1)

He asked for input on the topic on the STS list today, so, rather than just e-mail him, I thought I’d pontificate a bit here.

First, my background: I’m a MSFT guy.  Never blue-badged it, but was orange for a while and have worked for startups, professional services firms, and on my own projects using the MSFT stack for a long, long time.

Along the way I’ve incorporated lots of open-source tools and technologies, but I’m near as far away as you can be from an open-source zealot and still be part of the startup community.  I’m not a MSFT zealot, either, but I do tend to evangelize a lot of their platform technologies, and for good reason.

They work.  They work really well.

I’m teaching a course this quarter at the UW PCE on developing applications for the cloud.  At the top of my list of esteemed platforms is AWS, which is essentially platform-agnostic.  EC2 instances can run Windows or they can run Ubuntu or they can run RedHat or what have you.  Their supporting data and PaaS services, like S3, SimpleDB, SNS, SES, ELB, etc. are all essentially platform-ignorant for the consumer.

Lower, but still viable, are consumer-grade services from Microsoft’s Windows Azure program.  It’s essentially got a V1 feel to it, or perhaps a V1.1 feel, but the handcuffs you put on in order to get the simplicity are (IMHO) not worth it for a technically capable startup engineering team.

Much much lower are a large handful of independent open-source platforms that appear to be running on duct tape and baling wire, and which expose gnarly open wounds around every third press of the ENTER key. I won’t name names, but in general, and in my opinion, pure open-source suffers from a key philosophical problem:

1) What’s the incentive to make a polished, quality, bug-free product?

You know, the kind that startups can rely on and not be up at night worrying that their newfangled distributed database is going to lose their data.  The kind that let the startup guys focus on how they’re going to make money, not how to get that RPC call working just so.

Companies who make money have an incentive: they get paid.  If they do it well, they get paid A LOT.  As much as I criticize unchecked, predatory capitalism over coffee with friends, essentialist free-market capitalism provides really clear and measurable incentives.

Open-sourcers, particularly the wacky, lovable “free as in beer” eccentrics, don’t have those same incentives.  What then?

Passion?

Garnering the esteem of their peer group?

Making a living off of services and offerings directly related to the open-source project (the RedHat / Cloudera model, if you will?)

Hm.  All three reasons have their adherents, but *common* examples of each are the exception, not the rule.  For every successful open-source project that succeeds despite zero (as in zero beer) income, there are dozens (hundreds? thousands?) that end up as abandonware.  I’m trying to remember the citation, and can’t, but recently somebody quoted the number of projects or KLOC on SourceForge or CodePlex or somewhere that had been abandoned.  It was huge.

Getting back to the central point: Bryan asked some questions in his email to STS.  Here are my answers:

Strengths of the .NET & C# stack:

• Best-in-class development environment
• Robust, mature, yet still fast-evolving language
• Productivity
• BizSpark gets you off the ground for 3 years for essentially free
• Excellent corporate support and longevity

Issues with the .NET & C# stack:

• You’ll be recruiting from a smaller pool of candidates
• No cross-platform (even with *ye gods* Mono).  Note that I consider this a good thing.
• You’ll have to endure the implied ridicule of people who don’t necessarily know what they’re talking about

Strengths of open stack stack:

• Free
• Lots to choose from
• Lots of high-energy young burners know LAMP or RoR or Python or Java from school or hacking
• Hey, if it doesn’t work, you can go fix it!  Note that I consider this a bad thing, for startups. (see: Loss of Focus)
• Some platforms just can’t be easily targeted with current MSFT tools (ex: iOS native apps)

Issues with open stack stack:

• Abandonware
• Buggy, partially complete releases
• Lack of support
• Fragmentation (for most open-source platforms and tools, the size of the community becomes a key real-world issue)

I don’t think it’s a wash, as in “choose whatever you want, each has their pros and cons.”  I’m firmly of the opinion that a MSFT stack results in higher productivity for *most* startups where the skill levels of the developers is the same.  That’s partly because Microsoft bootstraps you into a certain minimum set of capabilities just by virtue of using the platform – the “Hello, World” examples on steroids; partly because you have to spend less time debugging and more time working on features customers asked for; and partly because the IDE – Visual Studio – is so ridiculously superpowered (in relative terms) for the average developer, it makes a lot of the day-to-day pain of developing software recede.

Now, like I said, I’m not a zealot.  Here are some tools I would *not* recommend for the average startup.

TFS – avoid this like the plague.

VB.NET – avoid this like the plague’s slightly less virulent cousin.  Ebola?

Office 365 – Microsoft still hasn’t gotten the hosted business productivity thing down yet.  Again, it’s got a V1 or V1.1 feel.

Active Directory – too heavyweight for most startups, whose headcount (and logical POPs within the infrastructure) will number in the ones or tens, not the hundreds or thousands.

What about Azure? I like hosting Windows apps and services on EC2 way better, and having gotten production quality apps out the door on EC2 the advantage Azure brings in terms of simplicity is moot to me, as it should be for most startups after an initial ramp-up period.  You’re going to need/want to do things that Azure just can’t provide, or you’re going to want to tweak a knob that Azure doesn’t expose.

What about SQL Server?  I like the product, and have developed on it for years, and can sling joins and constraints and correlated subqueries with the best of them.  However, I’m sort of ambivalent about SQL Server at the moment; my heart has sort of been stolen by various No-SQL options that force you to give up a lot of sexy, tested, robust things that SQL Server provides in favor of a stripped-down, we-do-one-thing-and-we-do-it-well approach.  SimpleDB and S3 come to mind.

One thing I haven’t mentioned is queuing.  Queues are a must for cloud-hosted startups, and here’s why: you reduce your need for compute resources at any given point in time by explicitly acknowledging that not everything has to be done RIGHT NOW.  Let a dinky little job pull stuff lazily off the queue in seconds, not milliseconds, and for a lot of data sets that’s just fine.  For human-scale interactions (i.e. click a page, fill out a form, click a button, go to the next page…) async times are just fine and can save you a boatload of steep OLTP costs.  And it sort of forces you to think (or rethink) your architecture in a more detached way, which is a good thing.  For non-human-scale interactions, i.e Very Large Sets of Highly Heterogeneous Data, or VLSHHD, queuing also works, because who cares if your carefully constructed report showing Pizza Pocket Sales By Zip Code For February 2012 takes a few minutes longer or a few minutes less. As far as cloud queuing solutions, SQS and Azure Queue Storage both seem about equivalent from a developer’s point of view, and I’ve used both in production code.  There are a slew of other upstart candidates that I haven’t looked in to, but redundancy and reliability are key here.

There’s one other thing that I want to mention, and that’s mindset.  If you’re a corporate developer used to a Herman Miller chair and a steady paycheck and dental insurance, you may (will probably?) feel anxious and lost when you get into the startup world.  That’s not a problem with the MSFT stack, per se.  It’s a problem with any corporate devs leave their preferred environment and cast out to sea.  I’m sure 20-year IBM vets who have done nothing but REXX or RPG or Java would have the same sorts of adjustment issues.  I’m curious to hear how that part of the talk unfolds.

Best of luck to Bryan in his talk next week.  I would love to hear your comments.

I had to get a new install of PowerShell 2.0 to work with the .NET 4.0 framework.  I’ve done this before – it’s merely a matter of adding the right *.exe.config files to the PowerShell home directory.

This time, on Windows 7 64-bit, it didn’t work.  Ruh roh.  I kept getting this error in the event viewer:

Activation context generation failed for “C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe”.Error in manifest or policy file “C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe.Config” on line 1. Invalid Xml syntax.

Huh?”

Here’s the (boringly standard) XML I was using:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<!-- http://msdn.microsoft.com/en-us/library/w4atty68.aspx -->
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0" />
<supportedRuntime version="v2.0.50727" />
</startup>
</configuration>

So, what gives?  Well, here’s a good rule of thumb for diagnosing file errors: when you see “line 1” or “line 1, character 1” or something like that:

CHECK THE FILE FORMAT

Turns out by default my Notepad++ editor was saving the files as ASCII.  Changing the encoding to UTF8 solved the problem.

For reference, here’s a PowerShell script that sets the files up with the right encoding:

param
(
[string]$path="c:\windows\system32\WindowsPowerShell\v1.0"
)
$x = @"
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<!-- http://msdn.microsoft.com/en-us/library/w4atty68.aspx -->
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0" />
<supportedRuntime version="v2.0.50727" />
</startup>
</configuration>
"@
$psPath = join-path $path "powershell.exe.config"
$psisePath = join-path $path "powershell_ise.exe.config"
 
if (-not (test-path $psPath)) {
(new-item -itemtype file -path $path -name "powershell.exe.config") | out-null
}
 
if (-not (test-path $psisePath)) {
(new-item -itemtype file -path $path -name "powershell_ise.exe.config") | out-null
}
 
$x | out-file -FilePath (resolve-path $psPath) -Encoding UTF8
$x | out-file -FilePath (resolve-path $psisePath) -Encoding UTF8
 
<#
cat $psPath
cat $psisePath
#>

Happy camping!

Open SSMS.

Connect to your SQL Server instance.

Look in the Object Explorer (typically the left pane; if you don’t see it press F8 or click on View | Object Explorer). The instance is represented as the topmost item in the tree in the Object Explorer:

The instance will be named something like:

localhost\sql2008dev (SQL Server 10.0.2573 – anthonys-m1330\Anthony)

That’s

server address\instance name (SQL Server product version – machine name\current user)

So our instance version is 10.0.2573.

If you right-click on the instance, and click Properties, the Server Properties dialog will open. In the General tab, you’ll also see the product version, this time with one extra level of precision:

To find out what the number represents, you can always LMGTFY, or consult this chart from Microsoft, which is current for SQL Server 2008 as of February 2012.

In my case, 10.0.2573 yields the product version SQL Server 2008, GDR Security update (2494096), June 14, 2011.

How do I tell the edition? The edition – i.e. Express, Developer, Standard, Enterprise – is important to know when you’re trying to determine if your SQL Server supports a particular feature (for example, database mirroring). To determine this in the GUI, again right-click on your instance in the Object Explorer, click Properties, and in the General Tab, you’ll see a Product attribute:

This clearly shows that I’m running the Developer Edition.

via @GreatDismal, this article on Coding Horror, talking about Netflix:

One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances and services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage.

I love this.  Too many architectures are fragile and untested.  I say test the hell out of it.  Break it.  Pull power plugs.  Shut off the network.  See how your systems respond.  You’ll learn a ton, and develop a confidence in your countermeasures (what I like to call your avoidance strategy and your contingency strategy) that will help you sleep much, MUCH better at night.

Well worth a read.

What would you add to this list?  What would you take away as non-essential?  Feedback wanted!

1. Should get you as close to 1-click deployment as possible
2. Should be easy to understand and modify by the average developer
3. Should be committed to source control, and versioned, just as if they were application components
4. Should allow you to roll out all or part of our application and/or database environment, as necessary, without a lot of (or any?) manual tinkering
5. Should run relatively quickly
6. Should provide variable output logging, so that you can turn up the diagnostic level when you need to debug something, and then turn it back down when you just need a straightforward deploy
7. Should embed a backup and rollback mechanism, in the event that something fatal happens during the middle of a script run (i.e. network interruption)
8. Should allow for interactive development and debugging (i.e. no compiled binaries, or as few as possible and then only for valid reasons, such as performance)
9. Should NEVER allow for accidental production deployments by well-meaning but uninformed developers
10. Should guard against the exposure of secrets (such as production passwords) as best as possible. This is a classic risk vs. cost tradeoff decision
11. Should embed steps that verify that you have a successful deploy after-the-fact, using a mechanism separate from the actual deploy (i.e. don’t just rely on an exit code of “0” to infer success; go check the website and make sure it doesn’t return a 500)
12. Should require as few third-party plugins/add-ons/tools as possible, given the requirements of the deployment
13. Should have some sort of help (e.g. –help or /?) to give the new user some guideposts on proper use

As part of the Cygwin SSH Server setup, there’s a step that prompts you to enter a value for CYGWIN, aka the CYGWIN environment variable.  Actually that last part is not really correct.  It’s not an “environment variable” in the strict Windows sense – manageable via the Windows Advanced Systems Settings applet.  What it is is a little-e environment variable that is used when the SSHD service starts.

I used some old documentation that asserted that the correct value for this item was ntsec bin tty, but ran into the following when I sshed in to the server from a remote client:

So TTY is no longer supported. Duh – the official docs show that TTY is obsolete.

So how to change it?

It turns out that there is no configuration file in your Cygwin install directory that stores this information.  It’s in the Windows Registry, in a few places:

HKLM\System\CurrentControlSet\Control\SessionManager\Environment
HKLM\System\ControlSet001\services\sshd\Parameters\Environment
HKLM\System\ControlSet002\services\sshd\Parameters\Environment

Without digging too deeply, I’m guessing that those last two are implied from the first – but I went ahead and changed them all, just to be safe.  Then I restarted the SSHD service and voila! No more TTY nag message upon SSH login.