# Linux in the Flesh: Adventures Embedding Linux in Hardware

This is not quite a liveblog, but a set of notes about the points I found most interesting in this morning’s Linux Conf AU keynote, given by Dr. Andrew “bunnie” Huang, best known for Hacking the Xbox and developing the chumby.

The talk was a fascinating look at how complex developing embedded Linux devices is. Let’s start with the takeaways:

One of the points bunnie made was that customizing embedded devices is really a frontier at this point (though he did not use the word “frontier”). There are not a lot of folks doing it. The “Sustainability” bullet point emphasizes this – bunnie talked about how people create custom environments for the device and want updates pushed to it, but that’s difficult because the environment may mean special tweaks to the updates.

The cost point was interesting because we often see a feature and wonder why they cannot put a feature that seems low-cost to us. Here is why:

So if the Cost of Goods Sold (COGS) is $30, you want to add some margin so you can earn a living, say$15. The retail markup will be $45, so the total cost to the consumer is now$90. Then there’s a bit of rounding up to the magic price of $99. Now if you decide to add a$5 feature to it, the COGS is now $35. Your margin becomes$17.50, and the retail markup becomes $52.50. Now the total cost is$105, and gets increased to the magic price point of $149. So there is a lot of argument about *which*$5 features to add with that price point. (Also note that $5 does not seem like much, but when your total COGS is$30, you’re adding 16.67% of the cost)

The tech stack is surprisingly full of old-school standards, I was pretty excited to see MySQL in it:

One of my coworkers left the keynote looking at the challenge and wanting to take it on. I heard all the complexity bunnie told us about and realized, that’s not a stack I want to spend the time to learn, for what I would do with it. It is not where I want to spend my time…here is a slide that shows where the time was spent on the chumby device design:

The actual hardware design, encircled by a box on the slide, is 11%. And the 19% in the product and software design is iterative, some of the 11% of time in the hardware design overlapped with product and software design, because they are intimately tied together. In fact, they’re so intimately tied, that the hardware is actually “dominated by software concerns” – this is not something I was aware of, but of course once I think about it, it makes sense:

bunnie got these times by going back to e-mails, so this is more like the phases of the project. The bulk of the time is the marketing and business development plus the mass-production ramp-up. bunnie talked about the tooling, which was also quite interesting. After showing us all the steps and explaining that a tool run takes 6 weeks and can cost around $20k (I assume USD, that was not specified), this slide came up: What a startup will do is try one design, spending the time and money, and if it’s not right they’ll have to spend the time and money again, in a serial way. Apple can put down$100k and have 5 different tooling runs happen simultaneously, and they pick the best one, so at the end of 6 weeks, Apple has a great design, whereas a startup only has their first design.

And with that being said, Apple re-tooled the iPhone 3G at least 3 times. So that’s why Apple stuff looks so much nicer than what a startup can come up with – that and they have brilliant designers.

The software design is not easy, either:

The code has to be extremely optimal, binaries have to be stripped, so the UI is responsive, stable, and your gadget succeeds.

The last takeaway on that first slide I showed was “Ship or Die”. Most folks know about this concept, but this slide made it pretty clear:

As bunnie explains it, most gadget sales (with some exceptions) happen between Black Friday [the day after US Thanksgiving] and Christmas [the Dec 25th one]. If you are aiming to release in mid-November, and your deadline slips a few weeks, you are now missing your huge window of sales, and you will have to wait until next year to really sell your product.

There was much more content than I am able to put here, and I am glad I got the chance to see this keynote!