Archive for June, 2012

« Older Entries

Tight-lipped throwdown: Python vs. Coffeescript

Thursday, June 28th, 2012

TL;DR: Coffeescript, while not as polished, facilitates writing code with a similar level of expression as Python, allowing for terse yet readable source.

One of the main reasons I love Python is the level of expression, particularly as that applies to list processing and more generally functional programming. With list comprehensions and built-ins like filter and map, you can write code that almost reads word-for-word like the question it is trying to answer.

Partly for this reason, writing Javascript (or any other loop-oriented language) has grown to feel increasingly cumbersome. When I decided to take a look at Coffeescript a couple months ago, I was pleasantly surprised to learn that it too contained list comprehensions and rudimentary filtering. Combined with some of the functional tools of Javascript’s Array type, this promised to let my write lovely, expressive code in the browser. Or server side. Or whatever, Javascript is a thing now.

As part of that learning process, I decided to write a Hanging With Friends solver (finished product) to get a taste for Coffeescript. I had already written one in Python, and I wanted to see how different the two would be. You can find the full code of the core solvers here. Here are some of my thoughts:

Coffeescript list comprehensions go a long way toward cleaning up ugly sections of loop-riddled code. This remains my favorite part of CS. As an example, to create a set of sets of words of a certain length:

Python

word_sets = [filter(lambda x: len(x) == l, words) for l in lengths]

Coffeescript

word_sets = (x for x in words when x.length == l for l in lengths)

The lack of a built-in way to do filtering makes the CS version less readable, but it is still very concise. Multiple levels of nesting can also make the code harder to read, though this is a problem in Python as well. Worse though is the way different structures are expressed in Coffeescipt. For instance, in Python, you can do this:

>>> [a+b for a in range(1,4) for b in range(1,4)]
[2, 3, 4, 3, 4, 5, 4, 5, 6]
>>> [[a+b for a in range(1,4)] for b in range(1,4)]
[[2, 3, 4], [3, 4, 5], [4, 5, 6]]
>>> [[[a+b] for a in range(1,4)] for b in range(1,4)]
[[[2], [3], [4]], [[3], [4], [5]], [[4], [5], [6]]]

In Coffeescript, though, the rules are next to inscrutable.

coffee> (a+b for a in [1...4] for b in [1...4])
[ [ 2, 3, 4 ],
  [ 3, 4, 5 ],
  [ 4, 5, 6 ] ]
coffee> ((a+b for a in [1...4]) for b in [1...4])
[ [ 2, 3, 4 ],
  [ 3, 4, 5 ],
  [ 4, 5, 6 ] ]
coffee> (((a+b) for a in [1...4]) for b in [1...4])
[ [ 2, 3, 4 ],
  [ 3, 4, 5 ],
  [ 4, 5, 6 ] ]
coffee> ([a+b for a in [1...4]] for b in [1...4])
[ [ [ 2, 3, 4 ] ],
  [ [ 3, 4, 5 ] ],
  [ [ 4, 5, 6 ] ] ]
coffee> [[a+b for a in [1...4]] for b in [1...4]]
[ [ [ [Object] ],
    [ [Object] ],
    [ [Object] ] ] ]

I couldn’t figure out how to get a flat list like the first Python example above. Maybe I’m just doing it wrong. As another issue, the different format for iterating over keys in an Object is less clean than the Pythonic style of iteration and seems to afford less flexibility. Ultimately, though, list comprehensions can handle most situations that I would field with map/filter and list comprehensions in Python.

The conditional syntax of the list comprehensions certainly limits the balance between terseness and readability. For instance, it is easy to first filter then map, but it’s a good bit messier to go the other way.

n = (x / 6.0 for x in [123...234] when (x*x) % 17 == 1)
n = (y for y in ((x*x) % 17 for x in [123...132]) when y % 5 == 1)

In Python, it’s straightforward and easy to understand: just swap the order.

n = map(lambda x: x / 6.0, filter(lambda x: (x**2) % 17 == 1, range(123, 234)))
n = filter(lambda x: x % 5 == 1, map(lambda x: (x**2) % 17, range(123, 132)))

Many of the other shortcomings of Coffeescript are less a failing of the list comprehensions themselves than a lack of expressive built-ins like set which must be shimmed. Many of my most elegant lines of code do things like build a dictionary of unique sublists based on a filtered, larger list. This sort of thing is harder to do in Coffeescript.

Finally, and this is a bit nitpicky, but the order of operations on the operator aliases don’t match the expectation of natural language. It was confusing to find that is not and isnt do two different things.

Despite the focus on shortcomings, I really like Coffeescript as a language, particularly compared to raw Javascript. What are your opinions of the two languages? Have I missed any shortcomings? Am I doing it wrong?

_____
Joseph

Tags: , , ,
Posted in Dev | 8 Comments »

Improve the reliability of your PC

Tuesday, June 26th, 2012

TLDR – scale your CPU speed to match demand and your PC will be more reliable.

Microsoft Research has published a very interesting peer reviewed paper where they look at the causes of crashes in over a million consumer PCs. This is the most comprehensive investigation into crashes on home PCs that anyone has ever undertaken. The paper has a bunch of cool findings including what you can do to reduce the chances your PC will crash or die completely.

Apart from hardware completely killing itself, we have all been victims of software crashes. It is easy to blame these on the programmer but sometimes the hardware itself has done something to pull the rug out from under your program. Sometimes it is just an essay you have been working on, but every now and then it is right before your game gets saved.

Microsoft found, after studying the crash reports from more than a million PCs that there are a few things we can do to make our machines live longer. Some people will intuitively have guessed some of these conclusions but this is the first time we have had some facts to back them up.

Overclocking

When Intel or AMD make a CPU they perform a bunch of tests at the factory to choose which clock speed the CPU can reliably run at. You do not have to run your CPU at this manufacturer specced speed though. For years people have been overclocking their CPUs to run faster. When overclocking a CPU you ramp up the speed iteratively until you find a speed which seems to give you system stability. Often you will upgrade cooling or buy a bigger power supply to keep the CPU running at the higher speed. Microsoft found that over an 8 month period a CPU that is overclocked is up to 20 times more likely to have a failure than at a vendor rated speed. Microsoft don’t names names, but one CPU vendor does better when overclocked (figure it out).

Underclocking

People have been wondering if reducing the clock speed of a CPU will make it more reliable. The idea is that reducing the speed means CPUs do not get as hot and require less electricity, the reverse of overclocking. Microsoft found that running your CPU at a lower speed meant machines were up to 80% less likely to crash in the 8 month period.

The paper shows that reducing the CPU speed reduces the failure rate and suggests that you can minimize the likelihood of a CPU crash by operating at the slowest CPU speed to achieve the desired performance.

Matching CPU Speed to System Demand

As Microsoft points out, we PC folks already have a technology in our computers that can control the CPU speed called DVFS. The difficult thing is matching the speed of the CPU to the demand. If you set the speed too low you will be slowing the machine down, if you set it too high then you are needlessly taxing the CPU as well as wasting energy.

MiserWare has a consumer version of its power management software called Granola Personal which was designed to match CPU speed to system demand. We made it primarily to reduce your energy footprint but it will do exactly what Microsoft suggests, set your CPU to the lowest speed it needs to run at.

Download Granola Personal

Granola settings window

Granola is available for PCs running Windows XP-7 and Linux Ubuntu, Fedora, Debian and RHEL. By default after it is installed it will manage your CPU in “MiserWare mode” which will change the CPU speed to match demand. You can also force your CPU to its lowest speed if you wish, using the settings window which looks like the screenshot. Matching your CPU speed to demand has the bonus of saving you some money on your power bill.

If you have a large number of PCs like a university lab and do not want to have to install Granola Personal on every machine there is a simpler to version for you to use. Granola Enterprise (free to try) is designed to be rolled out on large numbers of machines and be centrally managed. You can find out more about it here.

Too Short – Want To Read More

If you find this interesting I highly recommend reading the paper. It has a wealth of cool stuff and technical details.

—
Mat

Tags: , ,
Posted in Granola, Granola Enterprise, IT Impact, MiserWare | No Comments »

boatshoes: Python utilities for building Unix daemons

Tuesday, June 19th, 2012

boatshoes is a small Python package that handles two important aspects of daemon creation in Python: daemonization and single-instance enforcement.

Fork boatshoes on github

from boatshoes.DaemonContext import DaemonContext
from boatshoes.SingleInstance import (SingleInstance,
                                      RunningInstanceError)
# Pre-fork initialization code goes here
# ...
# Time to split from the parent process
with DaemonContext(True) as dc:
    # Once you're inside the context, you are in the child process.
    #   You can open files, go crazy.
    try:
        # This call keeps an open handle
        syslog.openlog()
        # Ensure there is only one instance running
        SingleInstance("/var/run/mycooldaemon.pid")
        # Signal handling works since we're in the child process
        self.setup_signals()
        self.start()
        # Drop privileges AFTER daemonization and single-instance checking
        self.handle_priv()
    except RunningInstanceError, e:
        syslog.syslog(syslog.LOG_ERR, str(e))
        # Send specific return values to the parent process
        dc.return_value = -2
# After exiting the context, the parent has exited with the correct
#   return value, and the child continues on.

DaemonContext

There are a whole gang of modules out there for daemonization, but this one offers several improvements. It follows best practices as laid out in Advanced Programming in the Unix Environment, making it work as advertised in all Posix environments.

It also works as a context manager, maintaining a pipe that allows the child process to return an integral return code. What good is that? It means that DaemonContext daemons can play nicely with init.d and other return-value based process management tools. That means you can do more complex initialization in the child process, particularly that involves open handles which would be closed by the daemonization routine.

It can also transparently handle conditional daemonization, as from a command-line switch.

SingleInstance

SingleInstance is a RAII-style process PID lock. Like DaemonContext, it uses Advanced Programming best practices. Specifically, it guarantees a single instance by acquiring a system file lock on the pidfile and writing its PID. This lock is held until the application exits, at which point the kernel will clean up the lock.

Privileges

Weren’t there only two things boatshoes did? Privileges is the sad cousin of the other two. It’s just a couple convenience functions for handling file chown/chgrp and privilege dropping.

_____
Joseph

Tags: , , , ,
Posted in Dev | No Comments »

Compiling the mistakes of a professor and entrepreneur

Tuesday, June 19th, 2012

Thinking about your mistakes can be a disheartening exercise, an experience in self-denial, or an event from which you grow. Here’s hoping this next series of blog posts is the latter.

But let me preface this with some working assumptions. There are good mistakes and bad mistakes. And there are little mistakes and big mistakes. For example, sometimes you make a mistake in the short term that ends up working in your favor. That’s a good mistake. A bad mistake might be repeating the same mistake twice or not heeding someone’s sage advice. A little mistake might be choosing the wrong logo or mantra or something. But, honestly, none of these types of mistakes really matter.

The ones that matter are the BIG mistakes. I know, no surprise there, right? But you’re going to make every type of mistake along the way, so hopefully the big ones will be few and far between. Why? Because BIG mistakes can kill small companies. In this post, I’m going to focus on the BIG mistakes I’ve made along the way. Hopefully, you can avoid them and if not, at least you’ll know you’re not alone when making them.

When do the big mistakes matter? They matter when you don’t recover from them. The beautiful thing is I’m living proof you can recover. How do I know? Well, because I’m still here, that’s why. So, let’s see what happened and whether these things can be avoided; and perhaps most important how we managed to survive.

Mistake #1: Losing your biggest and only customer

After we were funded at Virginia Tech by Merrill Lynch to build power management software with performance guarantees for critical servers, we started a pilot with ML and our newly incorporated company. Though it was slow going, it went great. We hit all our numbers: 15% energy savings with virtually no loss in performance for tough simulation workloads running on ML servers. They ran the software on analyst machines and no one complained. Everyone was happy, what I refer to as a regular “love fest”.

Enter global economic disaster. ML basically went bankrupt by the end of 2007 and ultimately got bought by Bank of America, but before that, like animals running from a fire or rats leaving a burning ship, everyone on our project at ML got new jobs elsewhere. The project was first put on hold and then canceled entirely. We never got a cent. We were left with great results we couldn’t talk about to anyone (under NDA) and no more champions. We actually eked forward on the project for about 6 months just because the results were so compelling, but then it just fizzled.

One customer is not enough

The BIG mistake was that we put too much stake in one customer. One customer is not enough. No matter how big they are or how deep their pockets, you must branch beyond them. You must create something that lots of people will buy. I would shoot for 3-4 people that want to buy what you will make. Two is too few. But more than that and you just might be on to something that you can sell to masses of clients for hundreds of millions per year.

We designed what ML asked for (a Linux-based solution) which it turned out few really wanted. We were early as a company. We didn’t have the chutzpah to tie cost discounts to timely completion of the pilot and so (not surprisingly in hindsight) without a forcing function on ML, the pilot went on forever (mostly in delays from their side).

It’s not that they didn’t love us, we were great together. But, we started out with the leader of their R&D acquisition team (our champion) and when we got to the pilot we were assigned an intern. Yeah, that’s right, an intern from some name brand school that they hired for a semester. And guess what (another big surprise), he knew next to nothing about system administration. It wasn’t his fault, he was only in his second year undergrad or something – he may not have even been a CS major for all we knew. Also, we designed software we could install; not software to be installed by a second year undergrad with little experience. We went to great lengths to get the pilot done quickly including sending one of our technical folks to London (yeah, the one in England) just to help them through. Epic failure.

So, you must tie your pilots to milestones or create a pilot workflow that is not on your critical path. For example, our latest software is designed for anyone to do their own pilot on their own time. They get value from obtaining their free energy footprint (i.e. incentive) and we get the benefit of more people using our software and more chances at upsell opportunities to the pay-for version. A pilot now costs my company next to nothing aside from the random telcon or support email for bigger potential clients. We made the software incredibly easy to install, by anyone, even our 2nd year undergrad friend (BTW, he was a really nice guy and learned fast — the real issue was ML gave him 10 projects like ours to do in just 3 months).

Time Kills All Deals

Another mantra you should commit to memory: Time Kills All Deals. That’s always been true – especially when dealing with big businesses. Champions move around and you will lose them over and over again if your pilots take too long to convert. Keep that in mind when you start out.

So, how did we recover from the loss of our biggest and only client? The results had come back promising. So, we used them as evidence of the viability of creating intelligent power management software and we closed a small funding round with a venture firm, some angels and a state-sponsored VC. Luckily we closed that quickly and just before the big economic bust. So, we had capital to keep us afloat while we figured out what to do next.

… Windows programming … “steaming pile of crap”

It was initially a tough time for the team since we felt like we weren’t creating software people wanted. So, partially with some thought and foresight mixed in with a little bit of desperation, we decided to (gulp) create a Windows product for consumers. Now, this is a team that lived and breathed Linux. No one knew how to write Windows code and everyone dreaded it. People have referred to Windows programming as a “steaming pile of crap” and we held that opinion at the time. Frankly, it was a scary proposition.

But, as Alexander the Great said “Fortune Favors the Bold” and my co-founder and I knew we needed to get people to use our awesome software. The results were too compelling to be ignored — energy savings with no noticeable performance loss. And people were 1) screaming for more battery life from their laptops, 2) looking for ways to be green, and 3) looking for ways to cut costs in their IT infrastructure and data centers. So, we went for it. It took us 6 months to write and release the first version of Granola… but, we got 100,000 downloads in the first 100 days. Wow. We could breathe again. And that took us from castaways from the worldwide economic depression to the makers of the world’s most popular power management software, Granola.

What was Mistake #2? Failing to raise the capital. More on that in the next blog post.

Tags: , , , ,
Posted in Entreprofessor, MiserWare | No Comments »

How effective are your sustainability efforts?

Friday, June 15th, 2012

TL;DR: Turn off your computer when you aren’t using it and you can buy a goat. There’s some other stuff too, but that’s the most important part.

There are a plethora of posts about ways to improve your energy footprint in your home. The suggestions usually run from the obvious (turn stuff off) to the drastic (get a $1500 energy audit). Unfortunately, many of these articles are a little shy on the actual benefits you could expect from implementing these measures. For example, which should you do first: change your lightbulbs to CFLs or start using a programmable thermostat?

Granola Personal has always included a breakdown of your savings in three dimensions: energy, money, and carbon footprint. It also calculates a number of equivalence classes for each of these dimensions, giving you a concrete example of it’s benefits. I took a little time to do some research, and I’ve done the same thing for a number of common household sustainability suggestions.

Use a clothesline instead of a dryer (5 loads/month)

You’d save 149.5 kWh, enough energy to power a dryer, a space heater, and a big plasma screen for a day

You’d reduce your carbon footprint by 203.32 lbs of CO2, as much as 4 trees, 22 miles in a compact car, and 34 daisies

You’d save $17.94, enough cash for a monkey wrench to throw in the gears, a political bumper sticker, and a tomato from the farmers market

Swap a 60W bulb with a 13W CFL equivalent (used 10hrs/day)

You’d save 171.5 kWh, enough energy to power a dryer, an air conditioner, and a refrigerator for a day

You’d reduce your carbon footprint by 233.24 lbs of CO2, as much as 4 trees, a day of electricity for your home, and 2 miles in a compact car

You’d save $20.58, enough cash for a 16-pack of CFL bulbs, a political bumper sticker, and a tomato from the farmers market

Eliminate all of your vampire load

You’d save 574.8 kWh, enough energy to power a radio station transmitter, a water heater, and an incandescent bulb for a day

You’d reduce your carbon footprint by 781.73 lbs of CO2, as much as a small car, a tree, and a gallon of gasoline

You’d save $68.98, enough cash for a goat for lawn mowing and 2 packs of granola bars

Turn off your 100W computer when it’s unused

You’d save 584.0 kWh, enough energy to power a radio station transmitter, a water heater, and a small window unit AC for a day

You’d reduce your carbon footprint by 794.24 lbs of CO2, as much as a small car, a tree, and a day of electricity for your home

You’d save $70.08, enough cash for a goat for lawn mowing, 2 packs of granola bars, and a reusable grocery bag

Turn your thermostat down/up 10 degrees when you’re at work

You’d save 1149.6 kWh, enough energy to power an electric vehicle charger, a radio station transmitter, and an air conditioner for a day

You’d reduce your carbon footprint by 1563.46 lbs of CO2, as much as midsize car, 5 trees, and a gallon of gasoline

You’d save $137.95, enough cash for a giraffe adoption, a Kill-A-Watt power meter, and a pack of granola bars

Use a ceiling fan instead of a window unit AC while you sleep

You’d save 1389.92 kWh, enough energy to power 2 electric vehicle chargers, an electric furnace, and a big plasma screen for a day

You’d reduce your carbon footprint by 1890.29 lbs of CO2, as much as midsize car, 2 500-mile flights, and a tree

You’d save $166.79, enough cash for a giraffe adoption, a goat for lawn mowing, and a pack of granola bars

Change all your light bulbs from incandescent to CFL

You’d save 1524.37 kWh, enough energy to power a commercial heater, a dryer, and a big coffee maker for a day

You’d reduce your carbon footprint by 2073.14 lbs of CO2, as much as midsize car, a small car, and a tree

You’d save $182.92, enough cash for a solar panel, 2 packs of granola bars, and a shirt from the thrift store

So it looks like using alternative cooling/heating, upgrading your thermostat, changing your light bulbs, and (of course!) turning off your computer are all good bets. If you’re interested in doing the latter, be sure to check out Granola Enterprise. For $20, you can control up to five computers, which is, like, 25 lawn-mowing goats worth of savings.

If you’ve got any questions or comments, or if you would like to see anything else estimated, let me know in the comments.

UPDATE: Assumptions used on the calculation and references:
All savings are annualized, so when it says ‘enough to power a tanning bed for a day’ it means you’d save enough in a year to power a tanning bed running continuously for a day. When it says ‘as much [CO2] as a midsized car’, it means the amount of CO2 a midsized car would release in a year.
US National average cost/kWh of $0.12 (source)
Each kWh releases 1.31lbs of CO2 into the atmosphere (source)
Average american yearly energy consumed is 11,496kWh (source)
You can save 10% on your electric bill by adjusting your thermostat (source)
The emissions of variously sized automobiles (source)
Energy used by lighting systems is roughly 1/6th total residential energy (source)

_____
Joseph

Tags: , ,
Posted in IT Impact | No Comments »

Why isn’t every professor an entrepreneur?

Wednesday, June 13th, 2012

Professors and researchers are idea machines. We regularly attack problems so hard we refer to them as “Grand Challenges”. I’m surprised we don’t have more monuments erected to celebrate our glorious profession.

If ideas were enough, all of the following people I see regularly would have thriving companies we’ve all heard of: my brother, my dad, my in-laws, my colleagues at VT, my colleagues abroad, my barber, students that take my classes, students from various business schools, parents of my kids’ friends, random people that approach me after I give a talk, and many more.

… having a good idea is the easy part.

So, why don’t they? Because having a good idea is (frankly) the easy part. Average professors lack the skills to build a company. But I’m living proof that some professors can learn to build companies. The problem is that learning on the job is far from optimal.

Before I created my company when I first started talking to VC’s, the prevailing wisdom was that I was a typical (absent minded) professor and just about anyone would be better to run a business than me.

In fact, I had already built a business once — I had grown my research group from zero to ten people and raised enough research capital to sustain a $1.5M/year operating budget for about 7 years. I managed all types of personalities, built software, partnered with other groups and labs, wrote proposals, produced high quality papers, and my work changed the way people use computers today. I was the CEO of a small business that had international influence and was part of a multinational, billion-dollar company (my university). So, to trivialize my experiences as not even remotely similar to building a company was a bit insulting.

Early on when dealing with potential investors, my professor title was the elephant in the room.

Luckily, being a professor means having thick skin. Early on when dealing with potential investors, my professor title was the elephant in the room. Like people trying to tell you you’re crazy without saying the word “crazy”. “This is a great idea, Kirk. But, are you really sure you’re the right person to make this into a company?” Some, under the auspices of tough love, were more direct: “you should stick to your day job.”

Luckily, my experiences and my ego were enough for me to discount most of these folks. Now, to be honest they were partially right. What they should have been saying was that there are people out there who would be better at building a business from a good idea. That’s absolutely true. Having built a startup, I now admire business brilliance now as much as I admire scientific brilliance. But, don’t let anyone substitute an MBA as equivalent to business brilliance. Find people who have built businesses from nothing into something worth tens of millions of dollars at least.

… I now admire business brilliance now as much as I admire scientific brilliance.

Over the years I’ve found both those talented enough to run my business and those willing, but I’ve not found those criteria combined in one person. That means I had to convince others (my investors, my cofounder, my developers, my mentors) to take this journey with me based on a good idea and somewhat irrational optimism. I’ve made plenty of mistakes, but luckily the mistakes I’ve made we’ve been able to recover from (so far).

So, when should you go for it? Some say that when you can convince at least one other intelligent person to join you, that’s the time to start. Others say you carefully analyze 1) the market potential, 2) the people involved, and 3) the technology, in that order. My advice would be to find someone willing to buy a product that captures your technology if you build it. Try to estimate how many more people would buy what you build. If there are enough people willing to buy, then start building your products. Find someone to take this journey with you since it is long and arduous.

What did I do? Well, I’m not sure if my story is something that will convince you to go for it, or dissuade you from it. While the stories you read about information technology companies still prevail in the press, most of these “overnight successes” are about a decade in the making. So, the get-rich-quick-fantasy ends shortly after you start your company. Successful entrepreneurs are not driven by money, but by passion for seeing their ideas propagate in the marketplace. As Guy Kawasaki would say, “make meaning.”

… our first customer evaporated …

For us, under my leadership, we initially built the wrong product. We built client-server software for Linux and our first customer (Merrill Lynch) evaporated in the Wall Street economic disaster. Luckily, we recovered and quickly built a Windows freeware version (Granola) that was wildly popular. Requests for Granola Enterprise versions started generating some revenue. We then realized we could not build client-server software for the Enterprise in an emerging Cloud-based world. So, we moved to a cloud-based infrastructure and adoption is picking up rapidly with our recent free software that measures your IT energy footprint. These were not technical decisions. We were responding to the market to build products for purchase so we could grow as a company. These key changes have enabled us to sustain ourselves long enough to attract several big government contracts and a growing user base of universities and corporations.

In summary, an idea is not enough – especially for a professor. Execution is everything. When you figure out how to productize your idea for the first time, be ready to adapt. If you can bring in a business builder who believes in your idea as much as you do, you are starting from stronger footing. If you can, find customers that will buy what you are selling as soon as you make it. But, if you lack all of these and can’t see yourself pursuing this idea for 5-7 years or more, you should probably just stick to your day job.

Most of our trials and tribulations so far have been business related. Our small size and technical capability have allowed us to adapt more quickly than others in the space. But, the CEO (me) was learning on the job. This means we possibly could have avoided some of these mistakes. What’s the biggest mistake I’ve made so far? Running out of money. But that’s the topic of another blog post altogether.
_____
Kirk

Tags: , , ,
Posted in Entreprofessor, Granola, Granola Enterprise, MiserWare | 3 Comments »

We Bearly Knew Ye

Tuesday, June 12th, 2012

After a much-too-extended poll, the people have spoken and our polar bear now has a name: Norbert! You may or may not start seeing the creepy little fellow poking his nose into the corners of our website. Let us know what you think.

Tags: ,
Posted in Granola | No Comments »

Paypal Donate Link

Friday, June 8th, 2012

Several people have emailed us asking for a way to donate money to Granola using Paypal. I have resurrected a donate button we used to use in the MicroMiser days. Click the big donate button:


We will manually add your contribution to the list. Thank you all for your support!
_____
Mat

Posted in Granola | 5 Comments »

The Entreprofessor

Friday, June 8th, 2012

pro·fes·sor
A teacher of the highest rank in a college or university.

en·tre·pre·neur
A person who organizes and operates a business, taking on financial risk to do so.

entre·professor
In Portuguese, this means “between teacher”. In English, this is a portmanteau combining the sound and meaning of the words entrepreneur and professor.

The Entreprofessor: Yet Another Blog? Well, yes and no.

Those that know me know that while I am a technophile in many aspects, I’ve been reticent about the world of social media. I’m not sure why I’ve avoided writing personal anecdotes more publicly other than to provide the standard “who cares what I have to say” argument I’ve used (at least to myself) for years. Optimistically, I could argue that generally I was waiting until I had something worth saying.

I am a firm believer that advice is worth exactly what you pay for it (and I’ve never been paid for advice to my best recollection). Nonetheless, I do have a unique set of experiences that may help someone somewhere. And to this end I, begrudgingly to some extent, submit to you the reader that I am often asked to recall certain experiences over the usual set of libations at workshops, conferences and the like. Admittedly, I find these discussions many times invigorating as they bring to the surface the raw excitement I encountered when first asking these questions myself.

So, in some ways, yes, this is the beginning of yet another blog. But, in other ways it is not since the topic is generally my experience at the intersection of academia and industry. You see, I have been a professor now for over a decade and an entrepreneur for almost half that time. I’m by no means done with either career; in fact you could say I’ve really just gotten rolling. Nor may I be the best person to seek advice from — while I’ve had my share of success, others have certainly surpassed my accomplishments on both sides of the fence. Still, perhaps the most interesting aspect is that my story is not finished and without taking the time to put some of it in print, I am likely to forget many of the details.

They say smart people learn from their mistakes and smarter people learn from the mistakes of others. I’ve tried to do both. And in this way I am indebted to many that have helped me along the way. Consider this blog part therapy and recollection, and part paying forward for the advice selflessly imparted to me throughout the years.

So, what do I plan to cover? Well, I thought I would begin with answers to questions I am asked frequently and go from there. For instance, after a wave of publicity surrounding my company and our initial products, colleagues began approaching me at conferences to hear my story and to understand the basics of spinning out companies from universities. Perhaps it is just red car syndrome on my part, or perhaps there is pent up demand for a roadmap on how to take an idea from the university laboratory and create a business.

But alas, a word of caution. The opinions in my blog are mine and mine alone. They are strictly based upon my experiences and in no way should be considered as representing the official opinions of my company (MiserWare) or my university (Virginia Tech). That means others may not agree with my conclusions or advice and you would be best served to read everything out there and draw your own conclusions. This also means I’m not planning on holding anything back — which means you’ll hear the good the bad and the ugly of starting companies from universities.

Now, let’s stop talking about what this blog is and get to writing actual useful content, shall we?
_____
Kirk

Tags: ,
Posted in Entreprofessor, Granola, MiserWare | No Comments »

Herding Cats: Developing a cross-platform hybrid web app

Tuesday, June 5th, 2012

Since founding MiserWare became my full time job in 2007, our software has undergone a number of iterations. From its original incarnation as a Linux-only, server-only, TCP-communicating software stack to the latest version of Granola Enterprise, a hybrid web-app (meaning native applications that work in conjuction with a web service) available on PCs, laptops, and servers for Linux, Windows, and VMware, the architecture of our power management software has been tweaked, reconfigured, and completely rewritten more than once. In this post, I’m going to describe our evolution and some of the technology we’ve used along the way.

TL;DR: Lessons learned

  • Use a high-level language for native application development unless you absolutely cannot.
  • Use the same language for your clients and your backend as much as possible.
  • Get your logging straightened out early on.
  • Try to encourage widespread deployment of the clients by individuals.
  • Rewrite only as dictated by your requirements.
  • Don’t be afraid to rewrite if you really need to.

ServerMiser ES: The early days

Our company (and thus the first version of our software) was spawned under very specific circumstances: our research group had been approached by a major international financial organization with a serious problem in datacenter power. They were in the process of putting together a huge datacenter restructuring and consolidation program with a large number of vendors of wildly varying sizes. After performing a paid proof of concept of our technology, they asked us if we would be interested in commercializing it. Long story short, we agreed, and we set about figuring out how to integrate into the web of products in their program.

This framing of the problem, along with our backgrounds, led to some design decisions. The software would be multi-tiered client-server over TCP; the agents would be small and lightweight; though they ultimately wanted more, the majority of their servers were Linux-based, so we would focus on that. Using a subset of C++ that included classes and almost nothing else, we created this software: ServerMiser ES was born. Despite generally abstaining from the more advanced features of C++, we used boost::asio for our networking.

What have we got now?

  • ServerMiser ES: C++, boost::asio, dpkg, rpm

Pros: We knew the technology, and it fit the problem we were trying to solve.

Cons: Verbose, low-level networking, didn’t use advanced language features, small standard library

MicroMiser, and the birth of Granola

We knew early on that testing was going to be very important in supporting system software on numerous different environments, so about six months after beginning development, we carved out the core power management technology and created a standalone agent that we called MicroMiser. We ran a little promotion and got a few hundred users, uncovering a bunch of problems along the way.

A few months later, we had hired 3 new developers, for a total of 5. Still cranking on ServerMiser ES, we decided it was time to look into Windows support. After the success of MicroMiser for Linux, we thought that would be a good first step: create a Windows version of MicroMiser, along with a clever little GUI. I broke off to work on this project, while everyone else continued work on ServerMiser ES.

Since our existing codebase was written in C++, I decided to start there; after some refactoring, much of the core code was reusable, though the underlying OS hooks were significantly different, and significantly different among different versions of Windows. Around that time, I had also been reading some books about modern C++, and began integrating features using templating and exceptions. We also began using more of the boost libraries, including smart pointers and functors.

Hollis, who would later become our resident designer, helped us design a fancy little GUI. The GUI code was written first using Windows GDI code, but later was refactored to use GDI+, since it did things like support PNG and transparency. You know, the little stuff. After releasing it, our userbase of MicroMiser grew by a factor of 10, despite still requiring account signup to download. With this release we also included a tool for collecting logfiles and customer feedback easily, and they made a huge difference in the amount and quality of feedback we received while also cutting down on our support touches.

At this point, we knew we wanted to open it up to a larger audience, and based on the advice of a marketing guru who was helping us out, we also knew we wanted to do some rebranding to test some different marking language. Thus, with very little changes to the codebase, Granola was born. The one new piece was a GUI for Linux, created using Python with Glade and GTK+. We created a new website for Granola and removed the account requirement for download. In the 100 days following the release, we had over 100,000 downloads.

What have we got now?

  • ServerMiser ES: C++, boost::asio, sqlite3 with C bindings, some Python for tooling, dpkg, rpm
  • MicroMiser/Granola: C++, Win32 GDI/GDI+, VBS for some tooling, WIX, dpkg, rpm

Pros: Still knew the language, code reuse among applications. Started using some more advanced C++.

Cons: All of the cons from last time, and now a bunch of Win32 code shoehorned into classes

Granola matures, and now we need to talk to a website

After the initial explosion of Granola, we were jumping to try and keep people excited. Our green message resonated well with individuals who wanted to do their part for the environment, and we wanted to increase the visibility of that message and of the visibility of the community surrounding the software. In other words, we needed a way to talk to our website.

After looking at the C/C++ libraries for doing this, I decided it would be more flexible and interesting to embed a Python interpreter in our software, having it and the Python standard library do the heavy lifting. It was quite the journey, and I learned a lot along the way, but ultimately it did exactly what we needed.

Meanwhile, we developed a rudimentary API on the backend to support the new communication pathway. Because our site was already written in PHP, and because our needs were simple, we eschewed a web framework and instead crafted our API by hand. End result: Granola clients that could talk to our website and send savings data that would aggregate on a user’s dashboard. We began selling this as Granola Enterprise.

What have we got now?

  • ServerMiser ES: C++, boost::asio, sqlite3 with C bindings, Xerces/XML, MySQL, some Python for tooling, dpkg, rpm
  • Granola clients: C++, Win32, GDI/GDI+, VBS, boost::python, Python, WIX, dpkg, rpm
  • Granola API: PHP, MySQL

Pros: Embedded Python makes some things a whole lot easier, boost::python is easier than regular embedding

Cons: Embedding Python can be a total pain in the ass. The clients and the backend speak different languages. Because we needed cert-verified HTTPS, we couldn’t use something intuitive like Requests, and urllib2 needed some work.

Granola Enterprise and the end of ServerMiser ES

Shortly after creating the first version of Granola Enterprise and as a result of some excellent media coverage we received, we began to get inbound sales requests. After some quick initial traction, we re-evaluated our strategy and came to the conclusion that our new model – client software + web app hybrid – was the new way to go (another story for another day!). Based on this, we wanted to unite the clans and provide the functionality of ServerMiser ES in our web-based dashboard. This minimally included the ability to group machines to aggregate savings and policy information, and to schedule different power management policies by group.

On the web site, we began writing what ultimately became the Granola Dash, creating an API-backed one-page web application, still using PHP for the API and jQuery to handle the AJAXiness. We also defined a protocol for communicating the power management policies that dictated how the clients would act.

On the client side, we needed a more flexible way to handle power management in enterprise settings, so we broke the power management aspect of the software away from the communication bits and called it Granola Manager. We then created a new service, Granola Connect, to handle communication to the website. Like the GUI, it was a C++-with-embedded-Python hybrid. Most of the code was reused between the two, with the only real new code (and only platform-specific code) being the daemonization/service routines and system information gathering.

Around that time, we also began adding VMware support, which looks a lot like the Linux support with some platform-specific data gathering code. Once we were done with that, we no longer had ServerMiser ES, and from the ashes Granola Datacenter was born.

What have we got now?

  • Granola (GUI): C++, Win32, GDI/GDI+, VBS, boost::python, Python, WIX, dpkg, rpm
  • Granola Manager: C++, WIX, dpkg, rpm
  • Granola Connect: C++, Win32, boost::python, Python, JSON, WIX, dpkg, rpm
  • Granola API: PHP, MySQL, JSON
  • Granola Dash: HTML/Javascript, jQuery

Pros: Splitting the clients apart gave us better agility while sharing most code, JSON makes communication easy, eliminating ServerMiser ES means most of the code is now shared. The new Granola Manager service is simple and self-contained.

Cons: Still verbose, lengthy compile cycle to build clients, no good way to do cross-platform builds, segmentation of the developer environments

The Unification of Granola Enterprise

Energy Footprint view

Once we had all of that functionality working, we began looking at ways that we could enable what we consider our core directive: making power management as easy and smart as possible. We realized the missing link at enterprise scale was historical and aggregate data. This was reinforced by feedback we were getting from our recently hired VP Sales (who in turn got it from customers). We decided that it was time to revamp the way the core software worked.

I had also been writing an increasingly large amount of Python code, and the embedded Python in the clients was growing. I reckoned it was time to ground-up rewrite the monitoring and communication aspects of our software, this time in pure Python. Let me say, on a personal note, that this was one of the better decisions I’ve made. The resulting code is much cleaner, more readable, more maintainable, more testable, and much much MUCH shorter. The new Python clients are compiled as native executables so that the users don’t need a Python installation. They are also packaged as native packages: MSIs, RPMs, and DEBs. The core power management, still written in C++, is still cordoned off in its own executable.

We designed some data structures to hold the historical data we thought was relevant, and made sure that they would translate well between our clients (now in Python) and our dashboard (in Javascript). As a result of these structures, we created Sanka, a Python queue runner for our backend with a file-based datastore. With this functionality now in the queue runner, the PHP API became not much more than routing and database helper functions. We also added a huge amount of reporting and analytics, allowing us to get a much better handle on our funnel.

With the new historical data in place, we also took an opportunity to merge Granola Enterprise and Granola Datacenter, now a feature called Power Tuning. The clients are all exactly the same, and the license management takes place on our website. This is ultimately how we arrived at our “Measure -> Take action -> Keep saving” mantra. The newly crafted Granola Connect clients enable us to offer the Energy Footprint information for free, after which a user can upgrade their licenses as their need dictates. We also removed the login capability from Granola Personal, which means that our userbase is better segmented.

What have we got now?

  • Granola Personal (GUI): C++, Win32, GDI/GDI+, VBS
  • Granola Connect: Python, pywin32, py2exe, cxfreeze, JSON, WIX, dpkg, rpm
  • Granola Manager: C++, WIX, dpkg, rpm
  • Sanka (queue runner): Python, JSON, redis, file storage
  • Granola API: PHP, MySQL, redis
  • Granola Dash: HTML/Javascript, jQuery, some coffeescript

Pros: Writing applications in Python is easy, expressive, and maintainable. It also helps maximize code reuse among platforms. Granola Personal is much simpler without the embedded Python. The API is much simpler without having to manipulate data structures. Having the clients and the queue runner means code can be reused. Coffeescript is terse and similar to Python.

Cons: py2exe is out of development, Python Windows library bindings can be painful and/or undocumented.

Conclusion

It’s been a long journey to get where we are. Our software has come a long way, from addressing a very specific problem in the datacenter to being a full-enterprise-scale hybrid web application applicable to laptops, PCs, and servers. We’ve learned a lot along the way, and I’m sure there’s still quite a lot to learn.

In summary, here are my lessons for writing a cross-platform hybrid web app, or really any software:

  • Use a high-level language for native application development unless you absolutely cannot. Packaging and building interpreted applications is totally a thing, so don’t use that as an excuse. I suggest Python!
  • Use the same language for your clients and your backend as much as possible. Obviously, the scope of this advice is limited, and I’m not sure we’ll start seeing a huge proliferation of Javascript-only native applications, but I could be wrong. Having Python in both the clients and the back end has made my life a WHOLE lot easier.
  • Get your logging straightened out early on. Both the clients and the backend need to have really good logging, and there should be really good ways of getting ahold of the logs. Wrap that up in a feedback mechanism as well, and you can gather suggestions and bugs in one fell swoop.
  • Try to encourage widespread deployment of the clients by individuals. Testing at the scope of hundreds of thousands of differently-configured systems is a great way to find weird bugs that only arise in certain environments.
  • Rewrite only as dictated by your requirements. Though our software has undergone a good bit of churn, I feel like we have only scrapped and rewritten as has been required to support new functionality we wanted. Rewriting ground-up is very expensive in terms of time, and should be avoided as much as possible, particularly when you are in the process of creating both clients and a backend.
  • Don’t be afraid to rewrite if you really need to. As I said above, one of the better decisions I’ve made at this company was to rewrite a major chunk of our code. Rewriting is generally bad, but sometimes it’s good.

_____
Joseph

You should consider signing up for a free Granola Enterprise account and establishing your energy footprint.

Tags: , , ,
Posted in Dev, Granola Enterprise | No Comments »

« Older Entries