Marquee de Sells: Chris's insight outlet via ATOM 1.0 csells on twitter

You've reached the internet home of Chris Sells, who has a long history as a contributing member of the Windows developer community. He enjoys long walks on the beach and various computer technologies.



Tuesday, Dec 27, 2011, 12:20 PM in The Spout

Moving to the Cloud Part 2: Mostly Sunny

In part 1 of this now multi-part series (who knew?), I discussed my initial attempts at moving my digital life into the cloud, including files, music, photos, notes, task lists, mail, contacts, calendar and PC games.There were some issues, however, and some things that I forgot, so we have part 2.

Before we get to that, however, it’s interesting (for me, at least) to think about why it’s important to be able to move things into the cloud. Lots of vendors are busy making this possible, but why? There are backup reasons, of course, so that a fire or other natural disaster doesn’t wipe out all of the family pictures. There are also the ease of sharing, since email makes a very poor file sharing system. Also, multi-device access is certainly useful, since the world has moved into a heterogeneous OS world again as smartphones and tablets take their place at the table with PCs.

For me, however, moving my data into the cloud is about freedom.

The cloud enables me to get myself bootstrapped with data associated with my personal or business life, using whatever device or OS I feel like using that day. It provides me freedom of location or vendor.

The cloud is still forming, however, so hasn’t really been able to make this a seamless experience, which is why I’m onto part 2 of this series.

Mail, Contacts and Calendar

Hotmail is a fine system for online access to mail, contacts and calendar that integrates well with Windows Phone 7. However, the integration with desktop Outlook and my custom domain isn’t good enough yet to rely on. The primary problem was the Hotmail Outlook Connector, which isn’t ready yet for prime time. It worked great with calendar and contacts, but fell down badly when it came to large email folders that I moved from my PST file. It never showed the sync’ing progress as complete, which made me uncomfortable that it never actually completed sync’ing and therefore my data wasn’t safe. Also, when I sent an email from Hotmail, either via the web or via Outlook, it showed the reply address as hotmail_44fe54cff788bdde@live.com. I assume the latter would’ve been fixed with Windows Live custom domains, but the former was the real deal-killer for me.

Also, I heard that Google Apps is the way to go, but that also requires some special software to enable sync’ing with desktop Outlook – I wanted something that was native to both Outlook 2010 and Windows Phone 7. Further, it cost money, so if I was going to pay, I wanted something that Microsoft was going to integrate well with.

So, I bit the bullet and hooked myself with the latest in hosted Exchange – Microsoft Office 365. That’s what I’m using now and just like the on-premise Exchange that worked great for me as a Microsoft employee, I’ve been very happy with it. However, because of the way I was using it, it was a pain to configure properly for use in hosting my csells@sellsbrothers.com email.

The easy way to configure Office 365 is to let it be the DNS name manager, which lets it manage everything for you, including your web site (via SharePoint), your mail, your Lync settings and any future service they care to tack on. However, that doesn’t work for me, since I didn’t want to move my 16-year-old web site into SharePoint (duh). Instead, I wanted to leave my DNS name manager at securewebs.com, which has been a fabulous web hosting ISP for me.

A slightly harder way to configure Office 365 for use with your domain is to only be used for selective services, e.g. set the MX record for mail, but don’t mess with the CNAME record for your web site. This would’ve been nice, too, except I don’t want to move all of the email accounts on sellsbrothers.com – only csells. Why? Well, that’s a family matter.

Over the years at family gatherings, to seem geek cool, I’ve offered free email boxes to my relatives. “Oh? You’re moving to another ISP again? Why don’t you move your email to sellsbrothers.com and then you can keep the same email address forever! And the best part is that it’s free!”

Now, of course, I’d recommend folks get an email address on hotmail or gmail, but this all started before the email storage wars back when you needed an actual invitation to set up a gmail.com account. Now I’ve got half a dozen family members with “permanent” and “free” email boxes and I don’t want to a) move them, b) charge them or c) pay for them myself on Office 365.

As cheap as you might think I am, it’s really migration that I worry most about – having successfully gotten them set up on their phones and PCs with their current email host, I don’t want to do that again for Outlook or migrate their email. Maybe it’s easy, maybe it’s hard. We’ll never know ‘cuz I’m not doing it!

So now, I have to make csells@sellsbrothers.com sync with Office 365 and leave everyone else alone. This is the hardest way to use Office 365 and involved the following:

Obviously, this is a crappy configuration experience, but no amount of manual updates to Outlook provided by the Office 365 site seemed to help. It was nice that the WP7 Outlook was much easier, although I’d really loved to have just told desktop Outlook that I was an Office 365 user and had it figure out all the touchy config settings.

Everything seems solid except one minor annoyance: when I do a Reply All, csells@sellsbrothers.com stays in the list because my mail programs don’t know that my csells@sellsbrothers.com and csells@sellsbrothers.onmicrosoft.com email addresses are logically the same. I assume if I was hosting my MX records at Office 365, this problem, along with the crappy config experience, would go away.

The good news is that I’ve got access to my full range of Mail, Contacts and Calendar from the web, my phone and my desktop, including multi-GB email folders I’ve copied over from my PST file, all for $6/month. Had I to do it over again, I’d have long ago moved my family to hotmail and avoided the config nightmare. I may yet do just that.

Encrypted Files

With my mail et al sorted, my next fix from last time was the lack of confidence in my most sensitive files with Dropbox. Dropbox can be hacked or subpoenaed like anyone else, so I want a client-side encryption solution. Dropbox may someday provide this themselves, but currently they gain a great deal of storage savings by detecting duplicate blocks amongst their users, saving significantly on uploads and storage, which client-side encryption disrupts. In the meantime, I really want an app that encrypts on the client and drops my data into Dropbox, which BoxCryptor does nicely.

In addition to supporting Windows, BoxCryptor also supports MacOS, iOS and Android, although not WP7 yet. Further, it’s free for 2GB and only a $40 one-time fee for unlimited data, so it’s cheap, too.

I also looked at SecretSync, which has a similar cross-platform story and pricing model (although it’s $40/year instead of $40/once), but it requires Java and I don’t put that on my box. For an open source solution, you may be interested in TrueCrypt.

imageFinancial Data

I’m a mint.com user. I like the idea of an all-up look at my finances across 29 online financial accounts. However, as a backup of that data, I wrote a mint.com scraping tool that downloads the CSV transactions export file and digs current snapshot data out of the homepage HTML. The format on the web site is constantly changing of course, so it’s a support problem, but having that data even a little messed up over time is way better than not having it at all, so I’m happy. The data itself goes into CSV files that I can query with LINQPad and that are stored in my Dropbox folder, which keeps them sync’d.

Books and Bookmarks

I can’t believe I missed this last time, but one of the big things I keep in the in the cloud is my set of Amazon Kindle books. I think that the proprietary format and DRM of Kindle materials will eventually open up because of competition, but until then, Amazon has been a great steward of my online books and bookmarks, providing me clients for all new platforms as well as their own e-ink-based hardware. I have an extensive book collection (this is just part of it), but am adding to the physical part of it no more.

Further, in the case that I have the “what the hell was that book I used to have?” moment after I finally truck all of my books off to Powell’s, the Brothers Sells have scanned all of the ISBN numbers from my 500+ books into LibraryThing. I won’t have the books anymore, but at least I’ll be able to browse, refresh my memory and add the books to Kindle on demand. The reason I picked LibraryThing is because it was easy to get all of the book metadata from just an ISBN (so it’s easy to spot data entry errors early), it’s easy to export from in a CSV file and, should I decide, easy to user their API.

App Specifics

In addition to the big categories, several apps keep data important to me:

Things Left On The Ground

As you may have surmised, I don’t put a lot of sentimental value in physical things. They’re not nearly as important to me as people, experiences or data. However, there are some things that I’d want to rescue in case of disaster given the chance:

As hard as I try, I can’t think of anything else. Should I have to jam, my plan is to place these few items into safe keeping and sell, donate and/or toss the rest.

Where Are We?

As I write this, I’m sitting in a Starbucks in Sandy, OR, 20 minutes from a cabin I’m renting for a few days. When I’m done here, I’ll explore the town, see a movie and make myself some dinner. I won’t worry about my phone, my laptop or my home being lost or destroyed, since 98% of the possessions I deem most valuable are being managed by cloud vendors I trust.

The cloud doesn’t just represent a place to backup or store data – it represents a way of life.

My data stores a lifetime of experiences, people and knowledge. By keeping it safe and available no matter where I go, I gain the freedom to wander, to experience new physical places and new hardware and software solutions, all without being unduly burdened.

Creative work requires a comfortable place to labor filled with the tools and the materials the worker needs to be creative. Today my tools are an Apple MacBook, Windows 7, Office 2010, Visual Studio 2010 and a Samsung Focus. Yesterday those tools were different and I’m sure they’ll be different again tomorrow. However, while other people build up their place with comfortable things around them – a bookshelf for reference, a comfy chair, knick-knack reminders of events or trips – my place is a lifetime of data and anywhere that provides access to electrons and bits.

Having my data safe, secure and available makes me feel comfortable, creative and free.

0 comments



Friday, Dec 16, 2011, 2:26 PM in The Spout

Sells Manor: Running 64-bit Win8 on My MacBook Air

With the exception of //build/, I haven’t really been a public part of the Microsoft developer community for about a year. So, to make up for some lost time, I’m giving a talk about some of the //build/ bits at the Portland Area .NET User Group first thing in the new year. This means that I need a running installation of the Windows 8 Developer Preview on my new laptop, ‘cuz THE MAN took my old laptop back when I handed in my badge (although, to be fair, they paid for it in the first place : ).

My constraints were as follows:

So, with all of that in mind, of course I started with Hanselman’s Guide to Installing and Booting Windows 8 Developer Preview off a VHD post. If you’re willing to build a new VHD, that’s the way to go. However, I was able to use the techniques I learned from that post, especially the comment section and a couple tips from my friend Brian Randall to make my existing Win8 VHD work. Some of this may work for you even if you don’t have a MacBook Air.

Getting Windows 7 Running on my MacBook Air

I started with a virginal MacBook and used the built in Boot Camp to create a Win7 partition, point to a Win7 Ultimate ISO I have on a network share for just these kinds of emergencies and get it installed and running. It wasn’t seamless, but Bing was helpful here to straighten out the curves.

Replacing the Boot Manager

The way I like to create VHDs is via Windows Server 2008 and Hyper-V. Once I have the VHD, I drop it onto the c:\vhd folder on my computer, do a little bcdedit magic and boom: when I reboot, I’ve got a new entry from which to choose my OS of the moment.

However, Win8 doesn’t boot from the Win7 boot manager, so the first thing I needed to do (as implied by the comments in Scott’s post) was use bcdboot to replace the Win7 book manager with the Win8 boot manager. To do that, boot into Win7 and fire up the Disk Management tool (Start | Run: diskmgmt.msc). Select your BOOTCAMP drive and choose Action | Attach VHD. Choose the path to your VHD and you’ll get another virtual disk:

image

In my case, C was my Win7 Boot Camp HD and F was my Win8 VHD. Now, start an elevated command prompt and use bcdboot to replace the Win7 boot manager with the Win8 book manager.

DISCLAIMER: I’m stealing the “works on my machine” graphic from Hanselman’s site because this action replaces a shipping, maintained, supported boot manager with one that is still in “developer preview” mode. Make sure you have your computer backed up before you do this. I am a trained professional. Do not attempt this at home. All stunts performed on a closed course. Some assembly required. Void where prohibited. I’m just sayin’.

image

Now that you’ve got the right boot manager in place, getting Win8 to boot requires bcdedit.

Getting Windows 8 to Boot

Scott’s post on booting to VHD involves bcdedit, which describes adding a new boot option in the “Setting up your Windows Boot Menu to boot to an Existing VHD” section:

image

Use bcdedit to point to the Win8 VHD.

Logging into Windows 8 on a MacBook

Now when you boot your MacBook, you’ll choose to boot to your Windows partition as you always have (which should just happen automatically), but then the Win8 book manager will kick in and you choose your Windows 7 install or your new Windows 8 install. Booting into Windows 8 shows you the login screen as normal, but now you have another problem.

The MacBook keyboard comes without a Windows Delete button. Oh sure, it’s labeled “delete” in trendy lowercase letters, but it’s really the equivalent of the Windows Backspace button. And that’s a problem, because you need to press Ctrl+Alt+Del to log into Win8.

Of course, Apple thought of that, so they created the Boot Camp drivers for Windows that maps fn+delete to Delete, but you can only install them after you’ve logged in.

So how do you log into a MacBook without a Delete button? Easy. You attach an external USB keyboard, press that three-fingered salute and login as normal.

Once you’re in that first time, you can install the Boot Camp drivers and never have to use the external keyboard again.

Installing the Boot Camp Drivers on Win8

When I created the Boot Camp USB to install Win7, it came with a set of drivers in the WindowsSupport folder with a wonderful setup.exe that makes Windows run great on the MacBook. Unfortunately, when you try to run it, you get a message that says you can’t:

Untitled

If you search the internet, you can find folks that have gotten past this by tricking setup.exe into thinking it’s running on Win7, but you’ll also find that those tricks don’t seem to work for 64-bit installs on MacBook Air, i.e. the one I was doing. However, this is where Brian had another suggestion: you can edit the Boot Camp MSI itself.

DISCLAIMER: This is something that I made work surprising well on my own personal MacBook Air, but I provide no guarantee that it won’t cause your computer to burst into flames on an international flight causing your body to be lost at sea. These techniques are not supported by Microsoft, Apple or the American Dental Association. You’ve been warned.

You may wonder, “To what MSI is Mr. Sells referring?” And I answer: WindowsSupport\Drivers\Apple\BootCamp64.msi. This is the 64-bit MSI with the check in it for Windows 7. To make it work for Windows 8, you need to edit the MSI and change the version number. And to do that, the easiest tool I know of is the unsupported, discontinued Orca MSI editor from Microsoft, now hosted on technipages.com. Running Orca allows you to edit BootCamp64.msi and change the Windows version part of the LaunchCondition from 601 (Windows 7) to 602 (Windows 8):

orca

Once you’ve changed this version, WindowsSupport\setup.exe seems to run just fine, installing the keyboard entries that allow you to login and the control panel that allows you to customize everything.

Where Are We?

Starting from a Boot Camp installation of Windows 7 on my MacBook Air, I showed you how I was able to get Windows 8 booting from a VHD. It wasn’t pretty and it required tips from all over the internet. I gather them here today so that future anthropologists will know how hard we worked to enable the coming of our robotic overlords. If you’re able to use these instructions to expedite their arrival, I’m sure they’ll take that into consideration when they’re sorting us into work details.

P.S. This post is dedicated to Jerry Pournelle. I used to pour over his Byte magazine column every month like he was the computer Sherlock Holmes.

0 comments



Wednesday, Dec 14, 2011, 11:35 AM in The Spout Tools Data

Moving My Data To The Cloud: Stormy Weather

For years, I’ve maintained a single “main” computer. It was the computer that was the central authority of all of the personal data I’d accumulated over the years and from which it made me uncomfortable to be separated. Because I needed a single computer for everything, it had to work on my couch, on a plane, on a desk and everywhere else I ever needed to go. Also, it couldn’t have a giant monitor or multiple monitors, because it had to go everywhere. All of this was because I needed all of my data with me all of the time.

My process for moving to a new computer used to include a lot of manual copying of files from the old D hard drive (D is for Data) to my new hard drive, which was also carefully partitioned into C for Windows, Office, Visual Studio, etc. and D for a lifetime of books and articles, coding projects and utilities I’ve collected over the years, e.g. LinqPad, Reflector, WinMerge, etc. This is 30GB of stuff I wanted access to at all times. I was also backing up via Windows Home Server, keeping photos and music on the WHS box (another 30GB), then backing that up to the cloud via KeepVault. And finally, as I upgraded HDs to go bigger or go to solid state, I kept each old HD around as another redundant backup.

All of that gave me some confidence that I was actually keeping my data safe right up until my Windows Home Server crashed the system HD and I found out that the redundancy of WHS doesn’t quite work the way you’d like (this was before I installed KeepVault). This was a first generation HP Home Server box and when it went down, I took it apart so I could attach a monitor, keyboard and mouse to diagnose it, pulled the HDs out so I could read what files I could and ultimately had to drop it off in Redmond with the WHS team so I could get it up and running again.

There are some files I never got back.

KeepVault gave me back some of the confidence I’d had before WHS crashed, but they didn’t provide me a way to see what files they were backing up, so I didn’t have the transparency I wanted to be confident. Further, they don’t have clients on every kind of platform like Dropbox does.

Of course, simply sync’ing files isn’t enough – sync’ing my 10GB Outlook PST file every time I got a new email was not a good way to share 20 years of contacts, email and calendar items.

The trick is to sync each kind of data in the right way, be confident that it’s safe and have access to it across the various platforms I use: Windows, Windows Phone 7, iOS and possibly Android (you know, if I feel like walking on the wild side!). And since I’m currently under employed (my new gig doesn’t start till the new year), I figured I’d do it once and do it right. I almost got there.

Files

Let’s start easy: files. Dropbox has made this a no-brainer. You install the software on any platform you care to use, drop everything you want into the folder and it just works, keeping files in sync on the cloud and across platforms, giving you adequate (although not great) status as it does so. Most platforms are supported natively, but even on platforms that aren’t, there are often alternative clients, e.g. I’m using Boxfiles for Windows Phone 7. When I gave up my Microsoft laptop, instead of doing the dance of the copy fairy to my new Mac Book Air, I installed Dropbox on both computers, dropped everything I want backed up and sync’d between computers into the Dropbox folder. 36 hours and 30GB later, all of it was copied into the cloud and onto my new laptop, at which point I reformatted my Microsoft laptop and handed it into my boss.

Further, as a replacement for WHS and KeepVault, I now keep all of the files that I was keeping just on my WHS server – photos and music primarily – into Dropbox.

image

This keeps me the confidence I need to know that my files are safe and backed up to the cloud, while making it very easy to keep it backed up locally by simply running Dropbox on more than one computer at my house. If at any time, I don’t want those files on any one computer, I tell Dropbox to stop sync’ing those folders, delete the local cache and I’m all done.

There are two tricks that I used to really make Dropbox sing for me. The first is to change my life: I no longer partition my HDs into C and D. The reason I’d always done that was so that I could repave my C with a fresh Windows, Office and VS install every six months w/o having to recopy all my data. Windows 7 makes this largely unnecessary anyway (bit rot is way down on Win7), but now it doesn’t matter – I can blow any computer away at will now, knowing that Dropbox has my back. In fact, Dropbox is my new D drive, but it’s better than that because it’s dynamic. The C drive is my one pool of space instead of having to guess ahead of time how to split the space between C and D.

The other thing I did was embrace my previous life: I wanted to keep D:\ at my fingertips as my logical “Data” drive. Luckily, Windows provides the “subst” command to do just that. Further, ntwind software provides the fabulous VSubst utility to do the mapping and keep it between reboots:

image

Now, I’ve got all the convenience of a dedicated “data” drive backed up to the cloud and sync’d between computers. Because I needed 60GB to start, I’m paying $200/year to Dropbox for their 100GB plan. This is more expensive than I’d like, but worth it to me for the data I’m storing.

There is a hitch in this story, however. Right now on Dropbox, data and metadata is available to Dropbox employees and therefore to anyone that hacks Dropbox (like the government). I don’t like that and for my very most sensitive data, I keep it off of Dropbox. When Dropbox employees themselves aren’t able to read Dropbox data or metadata, then I’ll move the sensitive data there, too.

Music

I’m not actually very happy with how I’m storing music. I can play all my music on any PC, but I can only play it one song at a time on my WP7 because there’s no Dropbox music client. I could use the Amazon cloud drive that provides unlimited music storage for $20/year, but there’s no WP7 client for that, either. Or I could spend $100/year on Amazon and get my 100GB of storage, but their client isn’t as widely available as Dropbox. Ironically, Dropbox is using Amazon as their backend, so hopefully increased pressure in this space will drop Dropbox’s prices over time.

Photos

I’m not using Facebook or Flicr for my photos simply because I’m lazy. It’s very easy to copy a bunch of files into Dropbox and have the sync’ing just happen. I don’t want to futz with the Facebook and Flickr web interfaces for 15GB worth of photos. Right now, this is the digital equivalent of a shoebox full of 8x10s, but at least I’ve got it all if the house burns down.

imageNotes and Tasklist

For general, freeform notes, I moved away from Evernote when they took the search hotkey away on the Windows client (no Ctrl+F? really?) and went to OneNote. The web client sucks, but it’s better than nothing and the Windows and WP7 clients rock. I have a few notes pinned to my WP7 home screen that I use for groceries, tasks, etc., and I have all of my favorite recipes in there, too, along with my relatives’ wi-fi passwords that they don’t remember themselves, a recording of my son snoring, etc. It’s a fabulous way to keep track of random data across platforms.

On the task list side, I only sorta use OneNote for that. I also send myself emails and write little TODO.txt files every time I get a little bee in my bonnet. I’ve never found that the Exchange tasks sync well enough between platforms to invest in them. Maybe someday.

imageMail, Contacts and Calendar

And speaking of Exchange, that’s a piece of software that Microsoft spoiled me on thoroughly. This is sync that works very well for contacts, emails and calendar items. IMAP does email folders, but server implementations are spotty. For years, I used Exchange for my personal contacts and calendar, only keeping my personal email separate in a giant PST file, pulling it down via POP3. This can sorta be made to work, but what I really wanted was hosted Exchange.

However, what I found cost between $5 and $11 a month per user. I’d probably have gone with Office 365 for sellsbrothers.com mail, even at $5/month except for two reasons. The first is that Microsoft requires you to move your entire DNS record to them, not just the MX record, which means there is all kinds of hassle getting sellsbrothers.com working again. They do this so that they can get all of the DNS records working easily for Lync, Sharepoint, etc., but I don’t want those things, so it’s just a PITA for me. If they change this, I’d probably move except for the other problem: I’m not the only user on sellsbrothers.com.

For years to be the big shot at family gatherings, I’ve been offering up permanent, free email addresses on my domain. That’s all well and good, but now to maintain my geek cred, I need to keep my mom, my step-mom, my brother, my sons, etc., in an email server that works and one that they don’t have to pay for. So, while I was willing to pay $5/month for hosted exchange for me, I wasn’t willing to pay it for my relatives, too!

One option I tried was asking securewebs.com (my rocking ISP!) to upgrade to SmarterMail 8.x, but that didn’t work. I even footed the one-time fee of $200 for the ActiveSync support for SmarterMail, but I couldn’t make that sync from Outlook on the desktop or the phone either.

Eventually I made an imperfect solution work: Hotmail. The nice thing about Hotmail is that it’s free for 25GB (yay webmail storage wars!) and it syncs contacts, mail and calendar items just like I want. Further, with some effort (vague error messages are not useful!), I was able to get Hotmail to pull in my personal email. And, after installing the Outlook Hotmail Connector (explicitly necessary because my Windows Live ID is not a @live.com or an @hotmail.com email address), I was able to sync almost everything, including the folders I copied from my giant PST file, via hotmail to both my desktop and phone Outlook. However, there are a few downsides:

The good news is that this all works for free and my relatives continue to have working email. The bad news is that it doesn’t work nearly as well as the Exchange server I’m used to. Hopefully I will be able to revisit this in the future and get it working correctly.

imagePC Games

I purchase all of my games via Steam now and install them as the mood strikes me. I love being able to reinstall Half-Life 2 or Portal on demand, then blow it away again when I need the hard drive space. Steam is the only viable app store for Windows right now, although I am looking forward to have the Microsoft app store in Windows 8.

imageBackups

I no longer maintain “backups” in the sense that I can slap in a new HD, boot from a USB stick and have my computer restored in 30 minutes or less (that never worked between WHS and Dell laptops anyway). I’ve had HD problems, of course, but they’re so rare that I no longer care about that scenario. Instead, what I do is keep all of the software that I normally install on a file server (the new job of my WHS box). If the file server goes down, then most of the software I install, i.e. Windows 7, Office and Visual Studio, is available for download via an MSDN Subscription. The rest is easily available from the internet (including Telerik tools and controls!) and I just install it as I need it.

Where Are We?

In order to free myself from any specific PC, I needed to pick a new centralized authority for my data: the cloud. The experience I was after for my PCs was the same one I already have on my phone – if I lose it, I can easily buy a new one, install the apps on demand and connect to the data I already had in Exchange, Hotmail, Skydrive, etc. Now that I’ve moved the rest of my world to Dropbox, I can treat my PCs and tablets like phones, i.e. easily replaceable. It’s not a perfect experience yet, but it’s leaps and bounds ahead of where it was even a few years ago.

Hardware and software comes and goes; data is forever.

0 comments



Tuesday, Dec 13, 2011, 4:50 PM in The Spout

Goodbye Microsoft, Hello Telerik!

I have gotten to do a ton of really great things at Microsoft:

Those and dozens more have all been extraordinary experiences that have made my time at Microsoft extremely valuable. But, like all good things, that time has come to an end.

telerikLogo-web-450x180pxAnd now I’m very much looking forward to my new job at Telerik!

Telerik is an award-winning developer tools, UI controls and content management tools company. They’re well-known in the community not only for their top-notch tools and controls, but also for their sponsorship of community events and their free and open source projects. Telerik is a company that cares about making developer’s lives better and I’m honored that they chose me as part of their management overhead. : )

My division will be responsible for a number of UI control sets – including WinForms, WPF, Silverlight and ASP.NET – as well as a number of tools – including the Just line, OpenAccess ORM and Telerik Reporting. I’m already familiar with Telerik’s famous controls and am now ramping up on the tools (I have been coding with JustCode recently and I like it). My team is responsible for making sure that developers can make the most of existing platforms, knowing that when you’re ready for the next platform, we’ll be there ready for you.

These controls are already great (as is the customer support – holy cow!), so it’ll be my job to help figure out how we should think about new platforms (like Windows 8) and about new directions.

And if you’ve read this far, I’m going to ask for your help.

I’m going to be speaking at user groups and conferences and blogging and in general interacting with the community at lot more than I’ve gotten to do over the last 12 months. As I do that, please let me know what you like about Telerik’s products and what you don’t like, what we should do more of and what new things we should be doing. Telerik already has forums, online customer support, blog posts and voting – you should keep using those. In addition:

Feel free to reach out to me directly about Telerik products.

Of course, I can’t guarantee that I’ll take every idea, but I can guarantee that I’ll consider every one of them that I think will improve the developer experience. I got some really good advice when I first arrived at Microsoft: “Make sure that you have an agenda.” The idea is that it’s very easy to get sucked into Microsoft and forget why you’re there or what you care about. My agenda then and now is the same:

Make developers’ lives better.

That’s what I tried to do at Intel, DevelopMentor and Microsoft and that’s what I’m going to try to do at Telerik. Thanks, Telerik for giving me a new home; I can’t wait to be there.

0 comments



Sunday, Dec 4, 2011, 8:13 PM

Roslyn Syntax Visualizer Tools

As I do more with Roslyn, I find I want more information about what I’m parsing and how it’s represented in the Roslyn object model. I could, of course, have built myself a little OM dumper for Roslyn, but instead I dug through the samples and found two cool ones built right in, both provided in the Documents\Microsoft Codename Roslyn CTP - October 2011\Shared folder.

Both Roslyn visualizer samples show a set of objects from the syntax part of the Roslyn API and the associated properties for the currently selected node as well as the associated text. The difference is only where the text comes from, a syntax tree or a text file.

Syntax Debugger Visualizer

The SyntaxDebuggerVisualizer sample (fully described in the associated Readme.html) allows you to create a new Visual Studio visualizer such that you can hover over a node from the syntax tree, click on the magnifying glass in the data tip and get this:

SyntaxDebuggerVisualizer

The text comes from the syntax tree parsed in the program. As each node in the syntax tree in the visualizer is selected, the associated properties are showed below and the range of text is shown on the right.

Syntax Visualizer Extension

The SyntaxVisualizerExtension sample (also described in its own associated Readme.html) shows the syntax tree from the current C# or VB file that’s open in Visual Studio. You get to the visualizer by loading the SyntaxVisualizerExtension sample project, starting the app (under the debugger [F5] or not [Shift+F5] as you choose), which starts another copy of Visual Studio. In this instance of VS, open a C# or VB source file and choose View | Other Windows | Roslyn Syntax Visualizer, which shows the following:

SyntaxVisualizerExtension

This version of the visualizer works just like the other one except that it gets the text from the current source code file. As you open different files, the visualizer window updates itself. As you change the selected node in the visualizer’s syntax tree, the associated code in the file is selected.

Both of these tools are very helpful for understand what’s been parsed by Roslyn. I personally like the debugger visualizer, as it’s always available without starting up a new instance of VS, but honestly I’m happy to have either, let alone both!

Update: Plus, there’s a cool tree view, too. Check it out!

0 comments



Saturday, Nov 26, 2011, 11:16 AM in Tools .NET

REPL for the Rosyln CTP 10/2011

I don’t know what it is, but I’ve long been fascinated with using the C# syntax as a command line execution environment. It could be that PowerShell doesn’t do it for me (I’ve seriously tried half a dozen times or more). It could be that while LINQPad comes really close, I still don’t have enough control over the parsing to really make it work for my day-to-day command line activities. Or it may be that my friend Tim Ewald has always challenged csells to sell C shells by the sea shore.

Roslyn REPL

Whatever it is, I decided to spend my holiday time futzing with the Roslyn 2011 CTP, which is a set of technologies from Microsoft that gives you an API over your C# and VB.NET code.

Why do I care? Well, there are all kinds of cool code analysis and refactoring tools I could build with it and I know some folks are doing just that. In fact, at the BUILD conference, Anders showed off a “Paste as VB” command built with Roslyn that would translate C# to VB slick as you please.

For me, however, the first thing I wanted was a C# REPL environment (Read-Evaluate-Print-Loop). Of course, Roslyn ships out of the box with a REPL tool that you can get to with the View | Other Windows | C# Interactive Window inside Visual Studio 2010. In that code, you can evaluate code like the following:

> 1+1
2

> void SayHi() { Console.WriteLine("hi"); }

> SayHi();

hi

Just like modern dynamic languages, as you type your C# and press Enter, it’s executed immediately, even allowing you to drop things like semi-colons or even calls to WriteLine to get output (notice the first “1+1” expression). This is a wonderful environment in which to experiment with C# interactively, but just like LINQPad, it was a closed environment; the source was not provided!

The Roslyn team does provide a great number of wonderful samples (check the “Microsoft Codename Roslyn CTP - October 2011” folder in your Documents folder after installation). One in particular, called BadPainting, provides a text box for inputting C# that’s executed to add elements to a painting.

But that wasn’t enough for me; I wanted at least a Console-based command line REPL like the cool Python, JavaScript and Ruby kids have. And so, with the help of the Roslyn team (it pays to have friends in low places), I built one:

RoslynRepl Sample Download

Building it (after installing Visual Studio 2010, Visual Studio 2010 SP1, the Visual Studio 2010 SDK and the Roslyn CTP) and running it lets you do the same things that the VS REPL gives you:

RoslynRepl

In implementing my little RoslynRepl tool, I tried to stay as faithful to the VS REPL as possible, including the help implementation:

replhelp

If you’re familiar with the VS REPL commands, you’ll notice that I’ve trimmed the Console version a little as appropriate, most notably the #prompt command, which only has “inline” mode (there is no “margin” in a Console window). Other than that, I’ve built the Console version of REPL for Roslyn such that it works just exactly like the one documented in the Roslyn Walkthrough: Executing Code in the Interactive Window.

Building a REPL for any language is, at you might imagine, a 4-step process:

  1. Read input from the user
  2. Evaluate the input
  3. Print the results
  4. Loop around to do it again until told otherwise

Read

Step 1 is a simple Console.ReadLine. Further, the wonder and beauty of a Windows Console application is that you get complete Up/Down Arrow history, line editing and even obscure commands like F7, which brings up a list of commands in the history:

replbeauty

The reading part of our REPL is easy and has nothing to do with Roslyn. It’s evaluation where things get interesting.

Eval

Before we can start evaluating commands, we have to initialize the scripting engine and set up a session so that as we build up context over time, e.g. defining variables and functions, that context is available to future lines of script:

using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
using Roslyn.Compilers.Common;
using Roslyn.Scripting;
using Roslyn.Scripting.CSharp;
...
// Initialize the engine
string[] defaultReferences = new string[] { "System", ... };
string[] defaultNamespaces = new string[] { "System", ... };
CommonScriptEngine engine = new ScriptEngine(defaultReferences, defaultNamespaces);
// HACK: work around a known issue where namespaces aren't visible inside functions
foreach (string nm in defaultNamespaces) {
  engine.Execute("using " + nm + ";", session);
}

Session session = Session.Create();

Here we’re creating a ScriptEngine object from the Roslyn.Scripting.CSharp namespace, although I’m assigning it to the base CommonScriptEngine class which can hold a script engine of any language. As part of construction, I pass in the same set of assembly references and namespaces that a default Console application has out of the box and that the VS REPL uses as well. There’s also a small hack to fix a known issue where namespaces aren’t visible during function definitions, but I expect that will be unnecessary in future drops of Roslyn.

Once I’ve got the engine to do the parsing and executing, I creating a Session object to keep context. Now we’re all set to read a line of input and evaluate it:

ParseOptions interactiveOptions =
  new ParseOptions(kind: SourceCodeKind.Interactive,
                   languageVersion: LanguageVersion.CSharp6);
...
while (true) {
  Console.Write("> ");
  var input = new StringBuilder();

  while (true) {
    string line = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(line)) { continue; }

    // Handle #commands
    ...

    // Handle C# (include #define and other directives)
    input.AppendLine(line);

    // Check for complete submission
    if (Syntax.IsCompleteSubmission(
          SyntaxTree.ParseCompilationUnit(
            input.ToString(), options: interactiveOptions))) {
      break;
    }
    Console.Write(". ");
  }

  Execute(input.ToString());
}

The only thing we’re doing that’s at all fancy here is collecting input over multiple lines. This allows you to enter commands over multiple lines:

replmultiline

The IsCompleteSubmission function is the thing that checks whether the script engine will have enough to figure out what the user meant or whether you need to collect more. We do this with a ParseOptions object optimized for “interactive” mode, as opposed to “script” mode (reading scripts from files) or “regular” mode (reading fully formed source code from files). The “interactive” mode lets us do things like “1+1” or “x” where “x” is some known identifier without requiring a call to Console.WriteLine or even a trailing semi-colon, which seems like the right thing to do in a REPL program.

Once we have a complete command, single or multi-line, we can execute it:

public void Execute(string s) {
  try {
    Submission<object> submission = engine.CompileSubmission<object>(s, session);
    object result = submission.Execute();
    bool hasValue;
    ITypeSymbol resultType = submission.Compilation.GetSubmissionResultType(out hasValue);

    // Print the results
    ...
  }
  catch (CompilationErrorException e) {
    Error(e.Diagnostics.Select(d => d.ToString()).ToArray());
  }
  catch (Exception e) {
    Error(e.ToString());
  }
}

Execution is a matter of creating a “submission,” which is a unit of work done by the engine against the session. There are helper methods that make this easier, but we care about the output details so that we can implement our REPL session.

Print

Printing the output depends on the type of a result we get back:

ObjectFormatter formatter =
  new ObjectFormatter(maxLineLength: Console.BufferWidth, memberIndentation: "  ");
...
Submission<object> submission = engine.CompileSubmission<object>(s, session);
object result = submission.Execute();
bool hasValue;
ITypeSymbol resultType =
  submission.Compilation.GetSubmissionResultType(out hasValue);

// Print the results
if (hasValue) {
  if (resultType != null && resultType.SpecialType == SpecialType.System_Void) {
    Console.WriteLine(formatter.VoidDisplayString);
  }
  else {
    Console.WriteLine(formatter.FormatObject(result));
  }
}

As part of the result output, we’re leaning on an instance of an “object formatter” which can trim things for us to the appropriate length and, if necessary, indent multi-line object output.

In the case that there’s an error, we grab the exception information and turn it red:

void Error(params string[] errors) {
  var oldColor = Console.ForegroundColor;
  Console.ForegroundColor = ConsoleColor.Red;
  WriteLine(errors);
  Console.ForegroundColor = oldColor;
}
public void Write(params object[] objects) {
  foreach (var o in objects) { Console.Write(o.ToString()); }
}

void WriteLine(params object[] objects) {
  Write(objects);
  Write("\r\n");
}

replerror

Loop

And then we do it all over again until the program is stopped with the #exit command (Ctrl+Z, Enter works, too).

Where Are We?

Executing lines of C# code, the hardest part of building a C# REPL, has become incredibly easy with Roslyn. The engine does the parsing, the session keeps the context and the submission gives you extra information about the results. To learn more about scripting in Roslyn, I recommend the following resources:

Now I’m off to add Intellisense support. Wish me luck!

0 comments



Saturday, Nov 19, 2011, 8:03 AM

Telerik: Best tech support response ever

I was playing around with the Telerik WPF controls the other day and I ran into an “issue.” It wasn’t a bug, just a bet peeve of mine, so knowing that two friends of mine, Stephen Forte and Doug Seven, both work at Telerik, I thought I’d report it. I created an account on their support web site and dropped in the following message:

From: Chris
Date: 11/17/2011 10:59:38 AM

when I'm choosing setup options in the Telerik installer, I can select options by clicking on the little, tiny box but I cannot select options by clicking on the wide, giant checkbox label. I'd really love to be able to do the latter as well as the former. thanks!

Within 24 hours, I got the best developer tech support response I’ve gotten in 30 years in this industry (damn, I’m old):

From: Telerik Admin
Date: 11/18/2011 8:09:42 AM

Hi Chris,
A very valid point indeed, thanks for sharing your opinion!
The thing is a bit tricky in terms of UX actually and I'd love your input here. Let me add some details:
The idea is that the click on the label is used for highlighting the item (thus change the displayed images) and the click on the checkbox is used for checking it.
Now, there are three approaches (different than the current one) of which I like none (maybe prefer the third actually):

1. The label click both highlights and checks/unchecks the checkbox. I don't like this one for two reasons: 1) if Telerik've set an item to be checked by default, we don't want the customer to uncheck it mistakenly and 2) if the item is unchecked by default, the customer might just want the original stuff and he'd need a second click to uncheck it back.

2. Use double-click on the label to check/uncheck the checkbox. Don't like it for it's not intuitive and noone would use it. As an example, the Windows Platform Installer has such a feature and we discovered it a year after its initial release - when we started checking it deeper.

3. Only check/uncheck a checkbox on label click if the item has already been highlighted. The drawback of this approach is that you would need two clicks to have the item state changed the first time you're on it. But still, this one seems kinda reasonable.

How do you find these?
Thanks,
Erjan Gavalji
the Telerik team

Explore the entire Telerik portfolio by downloading the Ultimate Collection trial package. Get it now >>

I have since learned that the Telerik developers support their own software and the benefits are obvious:

Erjan from Telerik – you’re my new hero. Thanks for answering my question so thoroughly. I have confidence that you’ll take my initial feedback and my reply to this email (show selection when you click on the checkbox label like the VS2010 installer does) and make the product even better.

It’s no wonder Telerik is an award winning software vendor. You have to love a company that’s willing to be open with their developers.

Update: as of a few days later, a new installer was posted that included my fix suggestion. Wow!

0 comments



Tuesday, Mar 1, 2011, 11:27 AM

Mary Sells, 1921-2011, Rest In Peace

Mary Hohnke Sells died on a Monday afternoon on the last day of February, 2010 in her home in Fargo, ND. She had just turned 89 years old on the 18th of February. She passed away peacefully in her sleep during an afternoon nap, having been tucked in by her daughter-in-law earlier that day. She’s survived by her son J. Michael Sells, her daughter-in-law Charlene Schreiber, her grandson Chris Sells and granddaughter-in-law Melissa Plummer and her two teenaged grandsons, John Michael Sells and Thomsen Frederick Sells. She was the last of four siblings; John, Shirley and Jim have all gone ahead of her to prepare the way.

Mary died a much beloved mother, grandmother and great grandmother as well as a dear friend to most everyone she met. She was generous of spirit, baking and cooking for her friends and family almost right up until the day she died, making sure her loved ones stayed plump in her love. She was talented in the kitchen, keeping her family recipes close to her heart for only those most special in her life. She was also a mischievous soul, taking advantage of her quick mind and her family’s sympathy for her ailments in later years to cheat outrageously at games of all kinds.

Mary was born in 1921, making her a child of the Great Depression. She graduated from Fargo’s Central High School in 1940, after which she pursued a course of study in Radiology Technology. She attained her national certification in 1946 and held it for 60 years. She married her late husband John Dickenson Sells in 1946, being secretly thrilled but outwardly scandalized when he insisted on public displays of dancing and other such tom-foolery. John worked for the Northern Pacific Railway and Mary worked in multiple clinical locations as she moved with her husband to sites ranging from North Dakota to Washington state. Her first child, Mike, is 61 and a successful draftsman at a local civil engineering firm. Her second child, Gretchen, died when she was only 15 in 1969, taking some of the light from Mary’s eyes. Her husband John was soon to follow, dying in 1971 of complications following gall bladder surgery.

None of this stopped Mary from living her life, however, having gone to Seattle in 1978 to be closer to her sister Shirley and to follow her career, then moving back Fargo in 1983 to be with her son and grandson. Before her move to Seattle, Mary helped take care of her grandson during the summers when he would visit. She was a second mother to him, doting on him and spoiling him thoroughly his whole life with food and attention. Till the day she died, Chris was her “baby boy,” in spite of his age of 41 and his height of 6’5”.

Mary was a member of the Mecca Chapter of the Order of the Eastern Star and received a 50-year membership acknowledgement for her many years of service. She was a life-long member of St. Mark’s Lutheran Church, an active member in the Women of the Evangelical Lutheran Church in America and a church quilter for most of those years. Mary was also an active member of PLS and enjoyed those friendships immensely.

Later in life, when her sister Shirley was diagnosed with cancer, Mary and she, both in their 70s, made sure that Shirley’s “bucket list” was fulfilled, which included wine tours, roller coasters in Las Vegas and even, for Mary, an incident on a tipped raft in the Rouge River in Oregon. She lived a full, rich life on her own terms, never shy about what she wanted for herself and others, and always ready with advice, wanted or not.

Mary died in her own home while she still had the faculties to interact with the ones she loved, as she wanted. She will be deeply missed and felt daily in the hearts and minds of those she left behind.

0 comments



Saturday, Jan 15, 2011, 11:00 PM

The Basics of EF Validation: IDataErrorInfo

When you’re adding or updating data in your database, you really want to make sure that the data being sent to the database is good and true. Often, that’s something that can be checked in the database itself. The first thing you’ll want to do is make sure that the database has validation constraints set on the columns, like nullability or max data sizes. If you’re going EF model-first, you can set these properties on the properties of your entities. If you’re not, you can set these properties in the database or get even fancier and write triggers that check the validity of the data. Finally, you can disable insert, update and delete altogether in favor of stored procedures, changing your EF mapping to generate calls to those instead. The nice thing about checks in the database is that no matter how the data gets there, whether it’s via your EF-based app or not, the checks happen.

However, if you’d like to also put checks into your EF code, perhaps because you’d like to avoid a round-trip to the database for bad data, you can do so in your EF-enabled language of choice, e.g. C#.

Imagine a very simple EDM to describes web advertisements:

image_thumb4

All properties on our entity type get a generated On<<PropertyName>>Changing and On<<PropertyName>>Changed method. If you want to check one property in isolation, it’s easy to provide an implementation of your partial method of choice, e.g.

namespace EdmTest {
  partial class Ad {
    partial void OnLinkChanging(string value) {
      if (!value.StartsWith("http://",
                   StringComparison.InvariantCultureIgnoreCase)) {
        throw new ArgumentOutOfRangeException("Link must start with 'http://'");
      }
    }
  }
}
Here we’ve made sure that whenever we set the Link property, it must be of a certain format. If we were to violate that restriction, things go boom:

clip_image002

As MVC translates the form fields into values on the Ad object that is passed to the controller’s Create method, setting the Link property with a bad value triggers the exception:

// POST: /Ad/Create
[HttpPost]
public ActionResult Create(Ad ad) {...}

MVC catches the exception before the Create method is even called and the view shows the error message. Notice that the error message is not what we provided, however.

Further, sometimes there are problems on an object’s state that span more than one property. Unfortunately, because such a constraint can’t be checked on any single property change, we need to at least check it at the object level, not just at the property level.

For both of these issues, we have IDataErrorInfo.

IDataErrorInfo

The IDataErrorInfo interface was introduced back in the mists of time with .NET 1.x for use specifically with data binding in Windows Forms. I wouldn’t recommend that anyone invest in anything but maintenance on their WinForms apps, but ASP.NET, the Windows Presentation Foundation (WPF) and Silverlight all support IDataErrorInfo . Data binding is involved enough and GUI-framework-specific enough that you’ll need to read up on it in your favorite GUI-framework-specific book, but the IDataErrorInfo interface is exactly what we need even without data binding:

namespace System.ComponentModel {
  public interface IDataErrorInfo {
    string Error { get; }
    string this[string columnName] { get; }
  }
}

Notice that IDataErrorInfo exposes error descriptions at both the object and the property/column level. It’s easy to implement this standard interface on our example Ad class:

partial class Ad : IDataErrorInfo {
  public string Error {
    get {
      // Check the ad for errors
      if (string.IsNullOrEmpty(Title) && string.IsNullOrEmpty(ImagePath)) {
        return "Must set Title or ImagePath";
      }
      return null;
    }
  }





  public string this[string columnName] {
    get {
      // Check any specific property for errors
      switch (columnName) {
        case "Link":
          if (Link != null &&
              !Link.StartsWith("http://",
                      StringComparison.InvariantCultureIgnoreCase)) {
            return "Link must start with 'http://'";
          }
          break;
      }
      return null;
    }
  }
}

Because each of the generated entity classes is partial, you can provide your own implementation to be merged with the generated implementation, in our case the IDataErrorInfo interface implementation. Now, when MVC hydrates an object that implements IDataErrorInfo, it’ll check to see if there are problems. To check, our controller provides the ModelState property, which itself provides the IsValid flag:

// POST: /Ad/Create
[HttpPost]
public ActionResult Create(Ad ad) {
  try {
    if (!ModelState.IsValid) { return View(); }
    ...
  }
  catch {
    return View();
  }
}


// POST: /Ad/Edit/
[HttpPost]
public ActionResult Edit(Ad ad) {
  try {
    if (!ModelState.IsValid) { return View(); }
    ...
  }
  catch {
    return View();
  }
}

In addition to the IsValid flag, the ModelState provides a list of property name/error message pairs that show in the code that the view generator spits out when it’s creating forms, e.g.

...
<% using (Html.BeginForm()) {%>
<%: Html.ValidationSummary(true) %>
...
<%: Html.TextBoxFor(model => model.Link) %>
<%: Html.ValidationMessageFor(model => model.Link) %>
...

It’s in the ValidationSummary helper that shows object-level errors and the ValidationMessageFor helper that shows property-level errors:

clip_image004

clip_image006

There are other means of validation that you will want to investigate, like the validation attributes supported by MVC and SilverLight, but IDataErrorInfo is the one with the broadest reach. It’s also the one that’s simplest for you to check yourself if you’re not getting the automatic support you want from your GUI framework. For example, because the SaveChanges method on the context base class is virtual and because we’ve got the Object State Manager, we can check ourselves for object errors using IDataErrorInfo:

using System.Data.Objects;

namespace AdMan.Models {
  partial class sbdbEntities {
    public override int SaveChanges(SaveOptions options) {
      // Make sure we're detecting all changes. base.SaveChanges
      // does this, but that may be too late.
      DetectChanges();

      // Get all the new and updated objects
      var objectsToValidate =
      ObjectStateManager.
      GetObjectStateEntries(EntityState.Added | EntityState.Modified).
      Select(e => e.Entity).OfType<IDataErrorInfo>();

      // Check each object for errors
      foreach (var obj in objectsToValidate) {
        // Check each property
        foreach (var property in obj.GetType().GetProperties()) {
          var columnError = obj[property.Name];
          if (columnError != null) { throw new Exception(columnError); }
        }

        // Check each object
        var objectError = obj.Error;
        if (objectError != null) { throw new Exception(objectError); }
      }

      // All clear
      return base.SaveChanges(options);
    }
  }
}

Here we’re overriding the SaveChanges method we’ve been calling all this time to take advantage of the IDataErrorInfo interface. The first call to DetectChanges is to make sure we’ve gotten all the changes into the Object State Manager (only necessary if you’re using EF POCO classes). The call to the GetObjectStateEntries method on the ObjectStateManager class produces all of the added and modified objects, their state, what the old and new values are, etc. We pull off each one of the entities that implement IDataErrorInfo and call the methods to check for property and object-level errors. If we find one, we throw an exception, otherwise we let the call to SaveChanges through.

This code isn’t needed if you’re already using a GUI framework that supports IDataErrorInfo, but it’s still handy to know you can roll your own code into SaveChanges if you need to.

Where Are We?

IDataErrorInfo is the core of data validation support in GUI libraries since .NET 1.x and while there are simpler ways to do it for individual libraries, IDataErrorInfo works just fine with MVC and EF, two of the most popular GUI libraries we’ve got just now.

0 comments



Saturday, Jan 15, 2011, 10:32 PM

EF Concurrency Mode Fixed + MVC

Imagine a very simple EDM to describes web advertisements:

image

Now imagine that I’d like to build a web application to manage instances of the Ad type. If multiple people are editing ads at once, especially the same set of ads, I’m likely to run into concurrency errors. By default, EF lets the last change win.

For example, if Chris and Bill are both editing Ad.Id == 1, if Chris pushes his changes to the database first, EF will not notice that the ad has been updated underneath Bill will he saves his changes and Chris’s changes will be lost. What we really would like to happen is that, when Bill attempts to save his changes, that we check if the data has changed since we cached it so that Bill gets an error and is able to merge his changes in with Chris’s.

This style of multi-user concurrency management is called “optimistic concurrency” because it assumes few people will be changing the same data at the same time. It’s the most efficient means of concurrency management when that condition is true. Another type of concurrency management is named “pessimistic concurrency,” and is generally implemented using locks on the database, which tends to slow things down.

By default, EF provides no concurrency support; if two people push changes to the same row in the database, whoever’s change goes in last wins. This results in data loss, which in the world of data is a big, fat, no-no.

The way that EF lets you decide how a row is changed is via the Concurrency Mode property on every one of the entity’s properties in the designer. By default, the Concurrency Mode is set to “None”, which results in SQL like the following when an update is needed:

update [dbo].[Ads]
set [Title] = @0, [ImagePath] = @1, [Link] = @2, [ExpirationDate] = @3
where ([Id] = @4)

The Id column is used to select whether to perform an update, so any changes made to the underlying columns for that row are not detected and are therefore lost. The way to tell EF which columns to check is with the Concurrency Mode property set from None (the default) to Fixed on an entity’s property. For example, if you set Concurrency Model to Fixed for each of the read-write properties for our sample Ad entity, the update would look like the following:

update [dbo].[Ads]
set [Title] = @0, [ImagePath] = @1, [Link] = @2, [ExpirationDate] = @3
where ((((([Id] = @4) and ([Title] = @5)) and [ImagePath] is null)
and ([Link] = @6)) and ([ExpirationDate] = @7))

This is handy, but it also requires that we keep around an entity in memory in both its original state and its updated state for the length that the user is editing it. For desktop applications, that’s not an issue, but for stateless web pages, like MVC-based web pages, it is.

It’s for this reason that the EF team itself recommends using a special read-only column just describing the “version” of the row. Ideally, whenever any of the data in a row changes, the version is updated so that when an update happens, we can check that special column, e.g.

update [dbo].[Ads]
set [Title] = @0, [ImagePath] = @1, [Link] = @2, [ExpirationDate] = @3
where (([Id] = @4) and ([TimeStamp] = @5))

Here, the TimeStamp column is our “version” column. We can add such a column in our SQL Server database using the “timestamp” type, as shown in SQL Server Management Studio here:

clip_image001[8]

The semantics of the timestamp type are just what we want: every time a row is updated, the timestamp column is updated. To see this new column in the Entity Data Model, you’ll have to right-click on the designer surface and choose Update Model from Database, which results in the TimeStamp being added to our model:

clip_image003

The TimeStamp field will come through as type Binary, since EF4 doesn’t have direct support for it, and with a StoreGeneratedPattern of Computed (which is exactly right). To enable EF to use the new column to perform optimistic concurrency, we need only change the Concurrency Mode to Fixed.

Now, here’s a simple Edit method on our MVC controller:

// GET: /Ad/Edit/5
public ActionResult Edit(int id) {
  return View(db.Ads.Single(ad => ad.Id == id));
}

This kicks off the view, but with one key missing ingredient – the view doesn’t have the TimeStamp field in it; because it’s mapped in EF as binary data, the MVC form generator wouldn’t provide a field for it. To make sure we pass the version of the data along with the data itself, we have to add a field to our HTML form and, because we don’t want the user to see it, let alone edit it, we need to make it hidden:

<% using (Html.BeginForm()) {%>
...
<%: Html.HiddenFor(model => model.TimeStamp) %>
...
<% } %>

The Html.HiddenFor is an MVC helper that produces HTML that looks like so:

<input id="TimeStamp" name="TimeStamp" type="hidden" value="AAAAAAAAB9E=" />

Now, when we press the Save button, the SQL we saw earlier is invoked to use the ad’s unique ID as well as the version (our timestamp column). If there’s a concurrency problem, i.e. somebody else has updated the underlying row in the database since we cached our values on the HTML form, we get an exception:

clip_image004

The message is saying that no rows were updated, which happens when the timestamp of the underlying row no longer matches. To provide a more helpful message, you’ll want to catch the specific error yourself:

// POST: /Ad/Edit/
[HttpPost]
public ActionResult Edit(Ad ad) {
  try {
    if (!ModelState.IsValid) { return View(); }
    // Attach the ad to the context and let the context know it's updated
    db.Ads.Attach(ad);
    db.ObjectStateManager.ChangeObjectState(ad, EntityState.Modified);
    db.SaveChanges();
    return RedirectToAction("Index");
  }
  catch (OptimisticConcurrencyException ex) {
    ModelState.AddModelError("", "Oops! Looks like somebody beat you to it!");
    return View(ad);
  }
}

Here we’re catching the OptimisticConcurrencyException and setting our own message before sending the user back to their data for them to grab what they want and try again.

Where Are We?

EF works great with MVC, but in the case of optimistic concurrency, you’ve got to work around the stateless model of the web a little to get it working just the way you like.

0 comments



Saturday, Jan 15, 2011, 9:36 PM

Using SQL Server Profiler with EF

I’m a big fan of the SQL Server profiler for figuring out what the Entity Framework (EF) is really doing on the wire. If you’re unfamiliar with how to use the profiler, the easiest thing to do once you’ve got it started is File | New Trace. It will ask to which database you’d like to connection and then pop-up the Trace Properties window. If you click on the Events Selection tab, you can filter the events you see. For tracing EF, it doesn’t matter what type of events we see, but it does matter from whom they come. To see EF calls (or any ADO.NET calls) against your database, the easiest thing to do is to press the Column Filters button and set the ApplicationName to be like “.NET SqlClient Data Provider”:

clip_image001

When you press the Run button, you’ll see a rolling list of calls made to that instance of SQL Server from EF. Now when you run an EF program, you’ll see exactly what SQL that EF is generating for SQL Server:

clip_image002

This is a handy technique to see whether EF batches SQL statements when you call SaveChanges (EF4 doesn’t batch) or how many round-trips lazy loading will cost you (lots – prefer the Include method).

0 comments



Friday, Jan 7, 2011, 2:32 PM

Be Careful with Data Services Authentication + Batch Mode

I was doing something quite innocent the other day: I was trying to provide authentication on top of the .NET 4.0 WCF Data Services (DS) on a per method basis, e.g. let folks read all they want but stop them from writing unless they’re an authorized user. In the absence of an authorized user, I threw a DataServicesException with a 401 and the right header set to stop execution of my server-side method and communicate to the client that it should ask for a login.

In addition, on the DS client, also written in .NET 4.0, I was attempting to use batch mode to reduce the number of round trips between the client and the server.

Once I’d cleared away the other bugs in my program, it was these three things in combination that caused the trouble.

The Problem: DataServicesException + HTTP 401 + SaveChanges(Batch)

Reproducing the problem starts by turning off forms authentication in the web.config of a plain vanilla ASP.NET MVC 2 project in Visual Studio 2010, as we’re going to be building our own Basic authentication:

image

Next, bring in the Categories table from Northwind into a ADO.NET Entity Data Model:

image

The model itself doesn’t matter – we just need something to allow read-write. Now, to expose the model, add a WCF Data Service called “NorthwindService” and expose the NorthwindEntities we get from the EDMX:

public class NorthwindService : DataService<NorthwindEntities> {

  public static void InitializeService(DataServiceConfiguration config) {
    config.SetEntitySetAccessRule("Categories", EntitySetRights.All);
    config.DataServiceBehavior.MaxProtocolVersion =
      DataServiceProtocolVersion.V2;
  }
  ...
}

Notice that we’re allowing complete read/write access to categories on our service, but what we really want is to let everyone read and only allow authenticated users to write. We can do that with a change interceptor:

[ChangeInterceptor("Categories")]
public void OnChangeCategory(Category category, UpdateOperations operation) {
  // Authenticate
  string[] userpw = GetCurrentUserPassword();
  if (userpw == null ||
    !userpw[0].Equals("admin", StringComparison.CurrentCultureIgnoreCase) ||
    !userpw[1].Equals("pw")) {

    HttpContext.Current.Response.
      AddHeader("WWW-Authenticate", "Basic realm=\"Northwind\"");
    throw new DataServiceException(401, "Unauthorized");
  }
}

// Use HTTP Basic authentication
string[] GetCurrentUserPassword() {
  string authorization = HttpContext.Current.Request.Headers["Authorization"];
  if (string.IsNullOrEmpty(authorization)) { return null; }
  if (!authorization.StartsWith("Basic")) { return null; }
  byte[] base64 = Convert.FromBase64String(authorization.Substring(6));
  string[] userpw = Encoding.ASCII.GetString(base64).Split(':');
  if (userpw.Length != 2) { return null; }
  return userpw;
}

The change interceptor checks whether the client program provided a standard HTTP Basic authentication header and, if so, pulls out the admin user name/password pair. If it isn’t found, we set the “WWW-Authenticate” header and throw a DataServicesException, which will turn into an HTTP error response, letting the client know “I need some credentials, please.”

The code itself is very simplistic and if you want better code, I recommend Alex James’s most excellent blog series on Data Services and Authentication. However, it’s good enough to return a 401 Authorized HTTP error back to the client. If it’s the browser, it’ll prompt the user like so:

image

The browser isn’t a very interesting program, however, which is why I added a service reference for my new service to my plain vanilla console application and wrote this little program:

class Program {
  static void Main(string[] args) {
    var service =
      new NorthwindEntities(new Uri(@"http://localhost:14738/NorthwindService.svc"));



    service.Credentials = new NetworkCredential("admin", "pw");

    var category = new Category() { CategoryName = "My Category" };
    service.AddToCategories(category);
    //service.SaveChanges(); // works
    service.SaveChanges(SaveChangesOptions.Batch); // #fail


    Console.WriteLine(category.CategoryID);
  }
}

Here we’re setting up the credentials for when the service asks, adding a new Category and calling SaveChanges. And this is where the trouble started. Actually, this is where the trouble ended after three days of banging my head and 4 hours with the WCF Data Services team (thanks Alex, Pablo and Phani!). Anyway, we’ve got three things interacting here:

  1. The batch mode SaveChanges on the DS client which bundles your changes into a send OData round-trip for efficiency. You should use this when you can.
  2. The DataServicesException which bundles extra information about your server-side troubles into the payload of the response so that a knowledgeable client, like the .NET DS client, can pull it out for you. You should use this when you can.
  3. The HTTP authentication scheme which doesn’t fail when it doesn’t get the authentication it needs, but rather asks for the client to provide it. You should use this when you can.

Unfortunately, as of .NET 4.0 SP0, you can’t use all of these together.

What happens is that non-batch mode works just fine when our server sends back a 401 asking for login credentials, pulling the credentials out of the server reference’s Credentials property. And so does batch mode.

However, where batch mode falls down is with the extra payload data that the DataServicesExpection packs into the HTTP error resposne, which confuses it enough so that the exception isn’t handled as a request for credentials, but rather reflected back up to the client code. It’s the interaction between all three of these that causes the problem, which means that until there’s a fix in your version of .NET, you need a work-around. Luckily, you’ve got three to choose from.

Work-Around #1: Don’t Use DataServiceException

If you like, you can turn off the extra information your service endpoint is providing with the DataServiceException and just set the HTTP status, e.g.

HttpContext.Current.Response.AddHeader("WWW-Authenticate", "Basic realm=\"Northwind\"");
//throw new DataServiceException(401, "Unauthorized");
HttpContext.Current.Response.StatusCode = 401;
HttpContext.Current.Response.StatusDescription = "Unauthorized";
HttpContext.Current.Response.End();

This fix only doesn’t work with Cassini, but Cassini doesn’t work well in the face of HTTP authentication anyway, so moving to IIS7 should be one of the first things you do when facing an authentication problem.

Personally, I don’t like this work-around as it puts the onus on the service to fix a client problem and it throws away all kinds of useful information the service can provide when you’re trying to test it.

Work-Around #2: Don’t Use Batch-Mode

If you use “SaveChanges(SaveChangesOptions.None)” or “SaveChanges()” (None is the default), then you won’t be running into the batch-mode problem. I don’t like this answer, however, since batch-mode can significantly reduce network round-trips and therefore not using it decreases performance.

Work-Around #3: Pre-Populate the Authentication Header

Instead of doing the “call an endpoint,” “oops I need credentials,” “here you go” dance, if you know you’re going to need credentials (which I argue is most often the case when you’re writing OData clients), why not provide the credentials when you make the call?

var service =
  new NorthwindEntities(new Uri(@http://localhost/BatchModeBug/NorthwindService.svc));



service.SendingRequest += delegate(object sender, SendingRequestEventArgs e) {
  var userpw = "admin" + ":" + "pw";
  var base64 = Convert.ToBase64String(Encoding.ASCII.GetBytes(userpw));
  e.Request.Headers.Add("Authorization", "Basic " + base64);
};

Notice that we’re watching for the SendingRequest event on the client-side so that we can pre-populate the HTTP Authentication header so the service endpoint doesn’t have to even ask. Not only does this work around the problem but it reduces round-trips, which is a good idea even if/when batch-mode is fixed to respond properly to HTTP 401 errors.

0 comments



Wednesday, Dec 29, 2010, 9:13 PM in The Spout

Enabling the Tip Calculator in Your Brain

I can’t imagine anyone reading this blog needs to read this, but I can’t help myself.

When I was just a wee lad, probably the most valuable thing I learned was how to perform mathematical estimation, the importance of which and several techniques you can get by reading Jon Bentley’s The Back of the Envelope (this essay along with several others, are collected in his most excellent books Programming Pearls and More Programming Pearls, both of which are still relevant a decade later). Not only is estimation generally quicker than running a calculator, but even when you do run a calculator, it helps you figure out when you did it wrong, the latter of which has saved my bacon time and again.

For example, as much as I love the Windows Phone 7 marketplace and it’s quality and quantity of applications, the ones that puzzle me are the “tip calculator” apps (several!). I don’t understand why it’s worth the trouble of pulling out your phone and punching buttons when you can know the tip instantly.

For example, let’s assume the dinner bill is $37.42. If the service was bad, that’s a 10% tip (you have to tip them something ‘cuz the IRS assumes you will and taxes them accordingly – bastards). So, with a 10% tip, take the bill and move it right one decimal point: $3.74. Now, round up or down depending on how bad the service was, e.g. $3.50 or $4. Quick and easy.

Assuming the service was great, that’s a 20% tip, so double the bill and move it right one decimal point, making the math easier for yourself, e.g. $37.42 is close to $35, doubling is $70, so a $7 tip. Boom: 20% tip.

If you want to get fancy and provide a 15% tip for good but not great, then average the two numbers: ($4 + $7)/2 = $5.50. Zim zam zoom.

Honestly, as great as the apps are on your phone, tablet or BlueTooth headset (seriously), think about using the apps in your head first. Now only are they quicker and cheaper, but using them staves off dementia (which is a good thing!).

Oh, and if the tip is added as a mandatory minimum, then the additional tip is easy: $0.00. I don’t deal well with authority.

0 comments



Saturday, Dec 18, 2010, 5:02 PM

Windows Phone 7: Beating Expectations

Years ago, when I was on my T-Mobile Dash, I would purchase a new phone every quarter or so, just to see if something better had come along. Always, within a week or so, I returned it and went happily back to my T-Mobile Dash. Then came the iPhone, which I instantly fell in love with. I didn’t think I’d ever give it up. Then came the Samsung Focus, one of the first Windows Phone 7 phones and I haven’t turned my iPhone 4 back on since. It’s not all I’d hoped for, but it’s damn close!

Meeting Expectations

Let’s check my list and see how WP7 did:

Stuff I Forgot To Ask For

I believe that the universe gives you what you ask for and in this case, even if I didn’t get everything, there was even some stuff I forgot to request:

Beating Expectations

Seriously, ever day is something new and cool on this phone. I continue to get blown away by features I never thought I’d want that have really changed how I use my phone:

Where Are We?

According to my math, I got a little more half what I asked for, but true love can’t be measured in percentages. Of the features that I’m missing, only camera quality, copy-paste and Kindle are things I actually miss from my iPhone 4, and two of those are supposed to be fixed in software RSN.

On the other hand, my Samsung Focus has giving me more than a dozen things I never thought to ask fore and really use. The full calendar support, contact linking, voice dialing (with great Bluetooth support), voice searching, the auto-correct on the keyboard, the location and phone number recognition and OneNote sync’ing make this phone a delight to use every day.

0 comments



Tuesday, Dec 14, 2010, 8:36 AM

If you want something from eBay, don’t bid on it!

I’m fond of quoting my father to my sons. I have a terrible memory for these kinds of things in general, but what he says sticks with me:

I’ve learned a ton of things from my father and continue to do so, so when I wanted to win something on eBay as a Christmas present for my girlfriend/fiancé’ (what’s it called when you’re engaged to be engaged?), I knew he had the experience, so I tapped it. And here’s what he told me:

If you really want something on eBay, don’t bid on it; that only gives your competition information on how to outbid you.

Instead, set yourself some free time when the auction is going to happen and start up two browser window at the following pages:

  1. The page where the count-down timer is shown.
  2. The page where you have already entered your top bid and are poised at the Confirm Bid button.

The idea is that people’s “top bid” changes over time as the auction goes on. I know this happens to me:

“Oh, this is only worth $20 to me. Well, maybe $25. OK, $40, but that’s all. Dammit I gotta have it! Where’s the button to enter the Social Security number of my first born!?”

So, instead of putting in your top bid and walking away, which lets other folks probe your top bid with their top bid and deciding later that their top bid goes toppier, wait ‘til the last minute to put in your bid. I believe the practice is called “sniping” and there are even apps that do it, although so far, I’ve found IE and a cool hand sufficient.

Of course, the most important question is this:

“Dad, at what time in the countdown do I press the Confirm Bid button?”

“Oh, well, I do it at 4 seconds, but my computers are slow.”

What can I say; the man’s a pro.

0 comments



2605 older posts       30 newer posts