spire.io

Weekend Reader: May 18, 2012

2012-05-18T00:00:0.000Z

A good amount of links fly across our internal IRC channel and team emails, usually these links are directly related to our own development or business interests. We do manage to slip in other kinds of nerdery though, so get ready to fire up your Instapaper...

Facebook

Chances are you've already seen about a hundred Facebook IPO posts today, and maybe even got some of the details from one of the multiple Twitter accounts Mark Zuckerberg's hoodie is operating right now, so we'll keep it brief.

The shares rose 23 cents above the IPO price of $38 as of 4 p.m. in New York. Facebook sold 421.2 million shares to raise $16 billion yesterday, giving the company a $104.2 billion market value.

-Bloomberg

Don't Track Me, Bro

In less monetarily exciting social news, Twitter announced that they will be supporting the Do Not Track feature in Mozilla's Firefox browser.

Taco Conf

We had a great time at Taco Conf last weekend. So did Mikeal Rogers. We have to say that we didn't mind the air conditioning on Sunday at all though.

Teaching the User How to Learn

Rands on how designing applications should be a little more like game design (Using Portal as an example, of course.)

That’s how I want to learn. Don’t give me a book; I don’t want a lecture, and I don’t want a list of topics to memorize. Give me ample reason to memorize them and a sandbox where I can safely play. Test me when I least expect it, shock me with the unknown, but make sure you’ve given me enough understanding and practice with my tools that I have a high chance of handling the unexpected.

Miscellany

Wax seals and USB drives: The idea leaves a lot to be desired security-wise, which Bruce Schneier's commenters have no problem pointing out.

How Pixar Almost Deleted Toy Story 2: By the point where the narrator says about a year of work was erased in twenty seconds we were all feeling a little ill.

Collaborative Conferences: Farmhouse Conf and Taco Conf

2012-05-18T00:00:00.000Z

"Huh, Taco Conf?" I thought as I watched Mikeal and Max talk about a conference that revolved around riding bikes to taco stands. "That looks fun, but there's no way I can take that to our CEO." I believe this happened around the end of February. I had just started my new position at spire.io in January, and I really didn't want to seem like some kid who was going to be throwing money at something so he could ride around The Bay for a weekend. To my surprise, Diego (our CEO) leaned over my cubicle and asked me if I'd messaged Mikeal about sponsoring Taco Conf.

He got it immediately. Some of our most productive times at spire.io have been over a taco lunch or dinner as we talked about work, or anything really. When you collect a group of people who are all working on something, chances are that no conversation will be completely irrelevant to their project. Analogies tend to circle back to programming or the project at hand. People talk about other projects they're excited about - What they like. What they don't - and in the process, the rest of the party comes to take away at least a small lesson or shift their perspective to some minor degree. And isn't that what a conference is supposed to do, really? So why aren't conferences more like taco lunches?

We had been sponsoring, hosting, and participating in all sorts of events before, but Diego's go-ahead there really drove home the notion that we shouldn't be afraid to experiment and get involved in anything that seemed interesting. As I explained to a Farmhouse presenter who asked why I had attended, "It's never a bad idea to listen to smart people talk about what's exciting to them. Even if you have no knowledge of the subject matter and don't understand everything they're talking about, at the very least you'll get a sense of how they're thinking about the world and tackling problems." That logic is why I ended up attending Farmhouse Conf and Taco Conf in back-to-back weekends.

Together, they've helped me identify some of my favorite conference experiences and showed me why some could have been better. I definitely plan on trying to bring some of their successes into our own events, and think I'll probably end up looking back on the last two weekends as setting a new standard for developing the feeling of an open community over the course of an event.

This is what I thought made them successful:

Talks from people of different backgrounds/areas of expertise
Attendees from different backgrounds
Time for attendees to talk
High comfort levels between attendees

My explanation to the Farmhouse presenter is still true; the talks are valuable, but it didn't leave any room for the time in between when the audience gets together to talk about, analyze, and make sense of them. Giving experts time to talk about their work is obviously a good idea, but if the point is to take away knowledge and have truly learned something from it all, the audience needs time to chew it over and work with the material in order to internalize it. Ideally, they will have the chance to talk to someone with different experiences or perspectives. Nobody edits their opinion when they are agreeing completely with someone else, they do it when they are challenged or when they hear another idea that they like and try to incorporate it with their own. As an organizer this seems like a tricky thing to pull off and I'll probably be struggling with it for a while, but at least I've learned not to be afraid to throw in a few speakers that seem to come from out of left field.

(Photos by @jlsuttles and @maxogden)

Both Farmhouse and Taco Conf were successful because of the time they allowed attendees to congregate between talks and presented opportunities for becoming comfortable with one another. One of the things that impressed me most about Farmhouse Conf was that Shane seemed to understand that it was the conversations and the people in his backyard that were making that particular Saturday special, and he quickly adapted the conference to support that. He started expanding the time between talks and decided to borrow some time from lightning talks.

Then there was the fact that everyone watched the same talks. Some conferences have five rooms going at any time with overlapping talks, forcing people to pick and choose talks. If you've been to a conference before, chances are you've had the experience of disappointment when you saw the two most interesting talks slated for the same time. The talks are obviously the most important thing in this event structure. Attendees carry around marked up programs and hurry from one auditorium another. When there's nothing that interesting going on, many conference goers skip the talks altogether and mess around on their laptops.

All the emphasis is on the various talks and it creates a fragmented experience. When you do get around to meeting people, most of the time is wasted on summary. "I saw talks A, B, and C. They were about this, that, and the other." The listener is heavily dependent on what the speaker found important and whatever bit of analysis he or she works into his or her summary. The conversation doesn't really have much room to grow from there. However, when everyone has seen the same talk, they can launch directly into analysis.

For example, I had a very different take on the tone of Brian's talk at Farmhouse and Jason was able to persuade me that I was wrong by pointing directly to some of his statements. I don't think that would have happened had I been summarizing; it's hard to politely disagree about something as subjective as tone if you weren't there. It's a bit counter-intuitive, but by directing all the attention on the same talks the organizers actually take a good part of the burden of being interesting off the speakers and put it on the audience.

All of the talks and the conversations, the bike riding and the hacky sacking, created the distinction between going to a conference and being a part of one. Every person who was there contributed to wealth of knowledge being shared and impacted the direction of the conference in some way. That's something special that we should aim for in every event. Thank you to all the organizers and everyone who came out to both conferences!

Weekend Reader: May 11, 2012

2012-05-11T00:00:00.000Z

The last week was pretty busy for us. Those who went to Farmhouse Conf were getting readjusted to normal life, and everyone was putting the finishing touches on our Identity service for our release on Wednesday. Still, we managed to get some reading done and throw some links around on IRC.

Functional Programming

Functional Programming in C++

My pragmatic summary: A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. In a multithreaded environment, the lack of understanding and the resulting problems are greatly amplified, almost to the point of panic if you are paying attention. Programming in a functional style makes the state presented to your code explicit, which makes it much easier to reason about, and, in a completely pure system, makes thread race conditions impossible.

-John Carmack

(via Buzz Anderson)

Essential Complexity

Shell Apps and Silver Bullets

Sometimes the quest for a silver bullet comes from a good place. Technology is about removing mindless work. Great engineers go through a phase where they try to create the ultimate framework to remove their daily grind, only to realize it’s more work to maintain a gigantic framework.

The difference is shell apps come from the wrong mentality. They start from, “How do we reduce effort?” instead of “How do we deliver the best product?”

-Benjamin Sandofsky

DuckDuckHack

DuckDuckGo announced an "open source platform to create instant answer plugins" for their search engine. If you haven't noticed the instant answers feature before, "random number" is a good example of how they can be useful.

What is a Technology Company?

A reminder from Alex Payne to re-evaluate what we mean when we talk about tech startups.

You are a technology company if you are in the business of selling technology. That is to say, if your product – the thing you make money by selling – consists of applied scientific knowledge that solves concrete problems and enables other endeavors, you are a technology company.

By this definition, most of the companies that dominate the “tech blogs” are not technology companies. They’re just, well, companies. These businesses might use technology, or develop technology, or even be run by people who used to work at technology companies, but they don’t exist to create and sell technology.

Legal

Sparkfun Gets a Subpeona

Please know that subpoenas are very serious, but they are not set in stone. The law enforcement agency was simply trying to do their job and worked with us to limit the scope while getting the information that they needed. I hope you are never served a subpoena but if you are sent one, don't be caught flat footed. Please let this homepage post serve as a reason to raise your hand and ask more questions.

Twitter Stands Up For One of Its Users

Indeed, even though Twitter provided notice to the Twitter user in this particular case, and even though he was able to get an attorney to file a motion seeking to quash the subpoena, the court found that the Twitter user did not have legal “standing” to challenge the D.A.’s subpoena.

If Internet users cannot protect their own constitutional rights, the only hope is that Internet companies do so.

(via Ben Brooks)

Security

With the latest Lion security update, Mac OS X 10.7.3, Apple has accidentally turned on a debug log file outside of the encrypted area that stores the user’s password in clear text.

-ZDNet (via Ben Brooks)

Cat Feeders

This Is All Your App Is: a Collection of Tiny Details

Fair warning: this is a blog post about automated cat feeders. Sort of. But bear with me, because I'm also trying to make a point about software.

There is a certain kind of person who gravitates to posts about cat feeders and cereal as windows into software design. We just so happen to have one working here. His name is Thomas and he somehow ended up with the authority to edit and post our Weekend Reader. He says Marco Arment does some good work in this area too.

Miscellany

The case of the 500 mile email

I was working in a job running the campus email system some years ago when I got a call from the chairman of the statistics department.

"We're having a problem sending email out of the department."

"What's the problem?" I asked.

"We can't send mail more than 500 miles," the chairman explained.

I choked on my latte. "Come again?"

"We can't send mail farther than 500 miles from here," he repeated. "A little bit more, actually. Call it 520 miles. But no farther."

(via Kottke)

Creating a Serverless App

2012-05-08T00:00:00.000Z

Why Serverless?

First off, there are still servers involved in a 'serverless' application; the key difference is they are servers that are NOT maintained by the application developer.

So why would you want to have servers maintained by someone else? There are many reasons, but most of them boil down to the basic premise that setting up and maintaining servers for an app takes a disproportionate amount of time compared to the value they add. What makes an app successful or not is determined by the quality of the app itself, not by the management of a set of servers. Managing servers is simply a necessary evil in order to have a distributed application.

When a developer is creating a new web application, the best use of time is in creating the application itself. The application is what will determine the success or failure of the venture; the server's only requirement is to make sure end-users are actually able to use the application. The server's value is binary; it either works and allows the application to be judged on its merits, or it doesn't work (or works too slowly) and the application can't be fairly evaluated.

The second reason an application developer would want to have their app servers maintained by someone else is scalability. When an application is first being developed and tested, spending time on scalability is a waste of time that should be spent developing the app itself. On the other hand, if an application does become successful, it is often a scramble to redesign an app to be properly scalable. During this time, all of the new users flocking to the app will be unable to use it while the servers are overloaded.

When the servers are maintained by a (competent) third party, however, this is not a problem. The servers are already built to be scalable; with no extra work, an application developer will have scalability from inception through success and rapid growth.

Why is security so hard in a serverless environment?

None of the reasons listed above for why an application developer should want to give up maintaining their own servers are particularly groundbreaking or that profound; developers have been aware of these advantages for a while, and a number of services have sprung up to address this market. These API services aim to provide the functionality that would normally require an app developer to have their own server.

However, security is a very difficult problem to solve for these services. While it might not seem obvious at first glance, the root of the security problem is actually quite simple to explain. It boils down to the inability to grant a user of the application limited access to resources.

Take, for example, a simple chat application. I chose chat as my example because many of the API services use it as their canonical example (such as PubNub's and Firebase's) for a serverless application.

While the implementation varies slightly for each provider, the basic process is to use a simple javascript library the service provides and then use your application key in your own javascript code to enable the end-user to interact with the API.

The limitation of this model is readily apparent; since every visitor to the site receives the same application key when they download the javascript file, there is no way to distinguish users on the site; everyone has the same credentials. This has a number of consequences; it is impossible to identify who is initiating a message, there is no way to only send messages to specific users, and there is no way for a returning visitor to be able to have ownership of messages they sent previously. Even if the chat provides a way to set a name that will be displayed next to the messages, there is no way to verify that chat user 'TrustedGuy' who sent one message is the same as the 'TrustedGuy' you talked to yesterday.

The end result of these limitations is that all of these serverless application examples skirt the issue of security by granting the same, restricted, access to everyone. This works fine, as long as the only application you want to make is an anonymous chat room.

So how can you overcome this problem?

The reason the existing API services only provide for this sort of anonymous messaging is that they do not provide for any authentication service besides the global API key. There is no way for a visitor to the application to authenticate beyond proving that they have access to use the application's messaging service.

Spire.io overcomes this limitation by providing a way for the visitors to an application to register with it. When that visitor returns to the application at a later time, they can re-authenticate and maintain their same identity on subsequent visits.

How does persistent identity help with security?

Authentication alone is not enough to allow for the kind of persistent identity that was described above. The authentication process has to somehow enable access to the correct set of resources.

This is not a trivial thing to do for a general purpose authentication API; what an authenticated user has access to will vary by application.

Spire.io boasts two features that allow an application to control access to various resources in whatever way works for the application in question. The first feature is spire's use of capabilities. A capability can be thought of as a key for a specific action on a specific resource; for example, anyone who has a channel's URL and its 'publish' capability can publish to that channel. This means that giving access to any resource is as simple as giving someone the URL and the appropriate capability(s) for the action(s) that we are giving access for.

The second part of the solution is the idea of a 'profile' that each authenticated user has. This profile is simply a JSON object that is returned whenever a user authenticates with an application. Applications can set a 'default' profile, a starting JSON object that is copied to a user's profile when that user first registers with the application. Each authenticated user can also update their own profile, overwriting or adding to the JSON object. Since an application user can only update their own profile, and only store data that they already have access to, the profile provides a secure way for data to be persisted.

Using these two features together, an application can decide what resources to give access to for new users by default, and can store the access information for other resources in a user's profile at a later point, and be assured they will retain access to that resource even if they log out and log in on another computer.

How would an example application utilize these features?

See this example in action!

Since a chat application is the canonical example used by so many API services, we will use that example; however, we are going to add a secure, private, messaging component as well.

After creating an account and application with spire, the next step is to create the default channel that we will give everyone access to; in our example, we will have one main chat channel and one subscription to that channel. We then need to set the default profile to include the URLs and capabilities for our newly created channel and subscription. We will also need the default profile to include the URL and capability that will allow new channels to be created (to provide for private communication). The default profile for our application would therefore look something like this:

{
  main_channel: {
    url: "https://api.spire.io/account/Ac-hAoBuSM/channel/Ch-hBwE/",
    capabilities: {
      publish: "gMI6QCFpd1IfHFPXWb5iwEg"
    }
  },
  main_subscription: {
    url: "https://api.spire.io/account/Ac-hAoBuSM/subscription/Su-oCwE/",
    capabilities: {
      events: "FTG32k3CfJRpIHuX8NSPiA"
    }
  },
  create_channel: {
    url: "https://api.spire.io/account/Ac-hAoBuSM/application/Ap-uKoPuSM/channels",
    capabilities: {
      create: "5ZHwxx4ludf0DtMRDoqaqQ"
    }
  }
}

Now, whenever someone registers with our application, they will immediately have access to a main channel they can publish messages to and a subscription they can listen for messages on. They also will have the ability to create new channels.

In our client code, the only identifying information we need to include is the application key. The client will send this key to spire.io, and spire will return enough information to the client to allow it to either register a new user or authenticate for an existing user.

After registering as a new user, spire.io will return the newly created user resource, which will include the default profile. Using the create_channel information in that profile, the client will create a new channel; this channel will be for private messages to the newly registered user. The client will also want to create a subscription for this new channel and start listening for messages, at the same time also adding the URL and capability information for the newly created channel to the user's profile (so as to retain access when we log in again).

Now, whenever the client posts a new message to the main chat channel, the posting user can include the information required to respond securely and privately to the posting user. For example, the posted message might have a JSON body that looks like this:

{
  message: "Hey guys, this chat is cool!  Message me if you want to talk",
  private_channel: {
    url: "https://api.spire.io/account/Ac-hAoBuSM/channel/Ch-z1Ih/",
    capabilities: {
      publish: "jIJMixcc6Rzx33JGYPL9oA"
    }
  }
}

With that information, other listening clients will have everything they need to display the message and also post a private response to the messaging user's private channel. That user will either receive the private reply immediately, or it can be store and retrieved later when the user logs in again. With no independent server code, we have implemented a public chatroom that has persistent, private, and secure messaging capabilities.

Using resource capabilities along with user profiles, the possibilities for secure, serverless applications are limitless. The application has complete control over who gets access to what, and those decisions can persist through subsequent authentications. What will you build?

Hello, (Serverless) World

2012-05-07T00:00:00.000Z

Riddle Me This...

What do Instagram, World of Warcraft, and Facebook Chat all have in common?

Surprisingly, the answer to this question is causing a profound shift in software development, a shift towards serverless applications.

Whether you are sharing Lomo-fi'd pictures of your kitten, trolling Azeroth, or chatting it up at work, behind the scenes all these products rely on a "real-time messaging" engine. Normally when talking about real time, we often think about chat rooms or IMs, but it is the same technology underneath. Bits of information need to be moved quickly and reliably.

This example illustrates one of the core pillars of software development today: many different types of applications are powered by the same underlying technology (or "back-end"). Games, chat rooms, productivity tools, social networks and more – all these share the same back-end functionality in one way or another. What makes them different is what these common building blocks are shaped into.

But Real-time messaging is just ONE component service.

Even though many of the components are similar, almost all developers write them from scratch with each new project. These components are not core to one's product. When we remove these "distractions" people can focus and build something better and faster. More products, better products, more opportunities.

How do we remove these distractions?

Enter The Cloud

With the rise of cloud computing it is now possible to confidently off-load some of the costs and technical burdens in creating a software application. The most common example can be found in "Infrastructure-as-a-Service" (IaaS) providers such as Amazon, Joyent, and Rackspace, who essentially "rent" physical infrastructure. Now companies are asking themselves, "Why buy a mountain of hardware that will become obsolete in a year, when I can rent the latest and the greatest only when I need it and not a moment sooner?" And that is why IaaS market is making billions in just a few years.

And with the new Platform-as-a-Service (PaaS) market, companies will ask themselves, "Why spend resources building out back-end functionality that already exists, when I can get that same back-end technology at a fraction of the cost and time?"

This is the world of serverless applications.

Build Upwards

The idea is refreshingly simple: spend time on features, not infrastructure. The reality is that the end-user doesn't care about logistics, they only care about the finished product and service. How it happens is not important. There were World of Warcrafts and Instagrams and Facebooks before, but they all lacked a compelling experience and package. Now entrepreneurs and developers can spend resources focusing on creating something truly captivating.

This is where Spire.io comes in.

With serverless applications like Spire.io there are no complicated servers or infrastructure to fight with. They are all hosted, maintained, and scaled by Spire.io. Because we focus on back-end services we can provide customers with excellent quality and performance. Spire.io provides back-end solutions at a fraction of the price, delivered in a fraction of the time.

How does Spire.io provide these back-end solutions? How do we make sure a developer can drop a feature like real-time communication into their product? The answer is APIs (Application Programming Interface).

The Spire.io APIs are designed to be dropped into any project (whether the project is written in JavaScript, Ruby, or other languages). A few lines of code are provided where the developer can just "hook" into the service. All a developer needs to do is hook up the wiring and they have a professional grade back-end. Spire.io manages the performance and status of the system, allowing the developer to focus on building the features.

Whether it is an Instagram, a World of Warcraft, or a Facebook Chat, Spire.io makes it possible.

Weekend Reader: May 4, 2012

2012-05-4T00:00:00.000Z

Our Architecture

Our CTO Dan Yoder wrote a post detailing the architecture behind spire.io and why we chose each component. We were very pleased with its reception. The Hacker News community had some particularly good questions. Between Dan's post and the team's responses, you'll end up with a good picture of what makes spire.io tick.

The Business of APIs

It's been a busy week for Dan, he also made an appearance in BetaKit's article on the trend of businesses relying more and more heavily on APIs, and the companies that have risen up to support them.

“Each generation of APIs commoditizes an entire class of application features, raising the bar for everyone building new applications.”

The Largest Computer

Alex Pasternack at Mother Board takes us back in time to look at the SAGE project.

The machine’s specs included a number of innovations: approximately 24 68K magnetic-core memories, which worked faster and held more data than earlier technologies, and rivaled the RAM of the first personal computers; the first real-time operating system, with overlapping computing and I/O operations, capable of processing about 75,000 instructions per second; real-time transmission of data over telephone lines; the first use of CRT terminals and light pens; and redundancy and backup methods and components, which led to the highest reliability of computer systems of the day. Though the tubes were unreliable, the use of two computers at each site, with one processor kept on “hot standby” at all times, gave the system a remarkably high overall uptime, with around 99% availability, which is on par with Amazon’s EC2 servers today.

It was also a giant: with 55,000 vacuum tubes, weighing 275 tons, occupying half an acre of floor space, and using up to three megawatts of power (each came with its own generating and air conditioning plant), the AN/FSQ-7 remains the largest computer ever constructed.

Do Androids Dream of Ironic Tees?

Babak Radboy shows us what happens when spam meets art at Zazzle

There is an entirely different order of product being developed here, far beyond the outer reaches of irony. I first started seeing them in Google Image searches; the most random queries were returning pictures of t-shirts, trucker hats, and especially ties that were truly uncanny. One could not, by looking at them, decipher how they had come about, what possible thought process lay behind them, who they were for, or why anyone would want them. They had something akin to the lost-in-translation weirdness of Chinese Shanzhai culture, but what was being lost was in a language far more distant than Chinese; one got the impression the “designers” of these pieces were speaking strictly in ones and zeros. I had visions of design-bots, data mining for user patterns, instantaneously designing products based on trending search queries, generating t-shirts like predictive text and graphics through some kind of visual auto-tune. Amazingly, it turns out I am not totally wrong.

Nothing's Gonna Stop Me Now

Click the link. Chase for your dream.

Our Architecture

2012-05-01T00:00:00.000Z

People often ask us about the technologies we use to power the spire.io platform. Our architecture is a big part of what makes us unique and ensures that your applications will be reliable, performant, and secure.

There are basically five major components involved:

Load-balancer and proxy
A dispatcher
A message transport
A collection of workers
And a data store

Each of these has different requirements and thus employs different technologies. Let's start where our request processing starts - with the load balancer and proxy.

Load Balancer And Proxy

This is really two things, but, conceptually, they are really doing one job. First, we want to distribute requests evenly across our servers. Second, we want to provide quick responses in some cases: for redirects, bad requests, or obvious denial-of-service (DOS) attacks. We can also do some course-grained routing. This is particularly helpful for setting up failover or for doing rolling deploys. Finally, we terminate SSL at this layer, freeing up the dispatcher from doing routine SSL handshaking.

We run our servers on Amazon Web Services and use their elastic load balancer (which is where we terminate SSL). AWS gives us data centers around the world, which is nice (and great for load testing). However, this is an area that is evolving rapidly and we are always re-evaluating cloud providers. We also use HA Proxy, which is ideal for dealing with high-levels of concurrency, as our proxy server.

And speaking of the dispatcher ...

The Dispatcher

One of the design goals for our platform is to make persistent connections cheap. We want to be able have millions and millions of connections open simultaneously and not have to worry about it from a cost standpoint.

Keeping sessions open for a long time is useful for supporting HTTPS, because of the overhead associated with establishing an SSL session. We want clients to be able to keep a connection open so they can have the best of both worlds: performance and security. Although we terminate the SSL at the proxy layer, we still need to keep connections open on the dispatcher for long-running requests. This gives us true real-time responsiveness, even for applications with millions of clients.

Another design principle is that we never want to drop requests. So that means we have to grab them off the wire and persist them somehow as quickly as we can. We don't want to do any more than that so that we can handle as many requests and connections as possible per machine.

We wrote the dispatcher in Node for three reasons. One reason is that we already knew Javascript. Another is that it has very nice asynchronous HTTP libraries, which makes it easy for us to keep connections open. Finally, it outperformed Ruby in our load testing, another language we're quite familiar with and which features support for asynchronous HTTP, for this particular task. Of course, it's still not as fast as, say, using C, but it's simple enough that it can be easily rewritten if the current implementation becomes a bottleneck.

The Message Transport

We use Redis as our message transport. Messages are distributed across any number of Redis servers, allowing us to easily handle millions of them simultaneously. (By the way, we use lists and blocking operations rather than pub-sub.) The dispatcher constructs a task from a request and places the message containing the task into a worker queue. Which brings us to ...

A Collection Of Workers

Messages are handled by workers that specialize in performing a particular task. They perform the task and then post a message with a particular result. The dispatcher that was waiting for the specific result can then transform that into an HTTP response.

Workers are written in JRuby, which makes it easy for us to make changes quickly and spin up multiple worker threads - one for each CPU. High-volume workers may eventually be rewritten in C.

Data Store

We also use Redis as our data store. It's data structure-oriented approach provides a great deal of flexibility over other NoSQL data stores and relational databases. For example, we use SortedSets to make message retrieval (for application messages, not our internal messages) simple and fast.

Advantages

This architecture is highly elastic. It's easy to simply add servers wherever we have bottlenecks. And it's easy to add new features - we can just add new types of workers. Finally, it's easy to optimize any given component because they are decoupled from one another - all interaction is done via the message transport, which is language neutral. For example, we could rewrite the dispatcher in C or a particularly high-volume worker in Node, or even Lua, and it wouldn't affect anything else.

Security

We have put a great deal of thought into how to handle security effectively in an API that lives out in the wild and wooly World-Wide Web. In particular, we use HTTPS for all API requests and capability-security - fine-grained authorization-based keys, rather than course-grain authentication-based keys - which is both more flexible and more secure at the same time. We will be introducing other APIs that build on this approach in the near future.

Roadmap

We have a lot of cool things coming down the road, including new APIs and enhancements to our Messaging API (including Web Sockets support). Follow us on Twitter - @spireio - or subscribe to our blog to keep up-to-date.

Weekend Reader: April 27, 2012

2012-04-27T00:00:00.000Z

CISPA

In a surprising move, the vote for the Cyber Intelligence Sharing and Protection Act was moved move up a day, taking place Thursday afternoon and passing the House of Representatives.

As Wired's David Kravets explains:

The measure, which some are calling the Son of SOPA, allows internet service providers to share information with the government, including the Department of Homeland Security and the National Security Agency, about cybersecurity threats it detects on the internet. An ISP is not required to shield any personally identifying data of its customers when it believes it has detected threats, which include attack signatures, malicious code, phishing sites or botnets. In short, the measure seeks to undo privacy laws that generally forbid ISPs from disclosing customer communications with anybody else unless with a court order.

The bill immunizes ISPs from privacy lawsuits for voluntarily disclosing customer information thought to be a security threat. Internet companies are also granted anti-trust protection to immunize them against allegations of colluding on cybersecurity issues. The measure is not solely limited to cybersecurity, and includes the catchall phrase “national security” as a valid reason for turning over the data.

CISPA also allows ISPs to bypass privacy laws and share data with fellow ISPs in a bid to promptly extinguish a cyberattack.

You can find all the last minute changes to the bill at Gizmodo, where Mat Honan and Brian Barrett speculate that the scheduling change was engineered to minimize news coverage for the bill. The Gizmodo post also includes a list of who voted, so you can find and contact your representatives. We found this record of the votes, referred to us by Brad Barrish via Twitter, to be a little easier on the eyes though.

The Internet Archive

Instead of keeping track of you, the Internet Archive would like to "create a permanent copy of every Internet page ever created, as it existed at the time."

Catching up to Sci-Fi

Big-name backed Planetary Resources want to boldly lead us toward the future -- and massive profits -- by mining asteroids for precious metals and other resources like oxygen and water. We often think of precious metals as being valued for their rarity and the ease with which they can be used for vanity items, but Planetary Resources co-founder Peter Diamandis points out that having a larger supply on hand will reduce the cost of items like defibrillators and allow us "to use them in mass production, like in automotive fuel cells." The plan involves space telescopes, robotic probes, spacecraft, and a healthy amount of laser based equipment, all of which have cool names like, "Rendezvous Prospector," just as you would expect.

Hybrid Funding

Andrew Cove takes a look at the "e-paper watch" startup, Pebble, which has surpassed its $100,000 goal by over $6.5 million, and sees the future of funding.

Pebble's Kickstarter success isn't an anomoly; that they didn't raise money isn't a mistake on the part of investors. Pebble is the first example of what will soon be a common occurrence: a startup raises seed capital to build the software platform that powers a prototype physical product; it then pre-sells the first run of production goods, and uses the traction established to raise a venture round to fund the growth of the platform and the team.

(via Justin Singer)

Being Cool Post-Internet

Why the Old-School Music Snob is the Least Cool Kid on Twitter:

File-sharing had rendered us, the knowledge guardians, irrelevant. Within a few years, knowledge had ceased to confer any distinction, and hoarding it had become about as socially advantageous as stamp collecting. Thanks to the Internet, cultural knowledge was now a collective resource. Which meant that being cool was no longer about what you knew and what other people didn’t. It was about what you had to say about the things that everyone already knew about.

(via Timoni)

Paywall problems? You can circumvent them with a quick Google search.

Real-Time Messaging for the Real World: Detecting Client Disconnects

2012-04-25T08:00:00.000Z

The spire.io Messaging API makes it easy to route messages to many thousands of clients simultaneously. But how can we tell which clients are actually listening for messages?

The Problem

When one client becomes disconnected for unexpected reasons (like closing a browser window), how can the other clients detect that one has gone offline? In these cases, there is no way for the disconnecting client to alert the others that it is about to disconnect. So how do the other listeners know that the client is gone?

This problem is particularly relevant to mobile and web applications. Mobile applications often suffer from connectivity problems: changing networks, going on- and off-line, and running out of battery. Similarly, a browser running a web application can crash, or the user can navigate to a new page or close the browser entirely.

The Solution

Happily, spire.io has two unique features that combine to solve the problem of detecting disconnects.

Firstly, "Subscription Events" are created whenever a subscription is created or deleted. Secondly, subscriptions can be given an expiration time, after which they will be automatically deleted.

Subscription Events

In my last blog post, I discussed subscription events and how they partially solve the disconnection problem.

When a new client comes online and creates a new subscription to a channel, a "join" event will be sent to all listeners of that channel. The "join" event includes the subscription name and timestamp. Similarly, if a client deletes a subscription, a "part" event will be sent with the subscription name and timestamp.

However, until recently, the client had to explicitly delete the subscription in order to send part events to the other listeners. If the client becomes disconnected for unexpected reasons, it will not have a chance to delete the subscription, no part events will be sent, and other clients will not know that it has disconnected.

Expiring Subscriptions

As of version 1.1.6, you can specify an optional expiration time when creating a new subscription. If the subscription is not accessed for that amount of time, it will expire. When a subscription expires, it is automatically deleted, and a part event is created. This part event will have its reason property set to "expired", so expirations can be distinguished from normal deletions (which have reason set to "deleted").

How to use it

To use the new subscription expiration feature, just send an 'expiration' parameter (in milliseconds) when you create the subscription.

With the spire.io javascript library, the code looks like this:

// Create a new subscription with a 60 second expiration.
var subscriptionName = "my-expiring-subscription";
var subscriptionExpiration = 60000; // 60 seconds
spire.session.createSubscription({
  name: subscriptionName,
  channelNames: ["my-channel"],
  expiration: subscriptionExpiration
}, function (err, subscription) {
  if (!err) {
    // subscription is a subscription with a 60-second expiration
  }
});

This creates a subscription called "my-expiring-subscription", with a 60-second expiration. The subscription is listening on the channel called "my-channel".

Creating the subscription will send a join event that looks like this:

{
  channel_name: "my-channel",
  subscription_name: "my-expiring-subscription",
  timestamp: 1335318932874000,
  type: "join",
}

Join events get created for all new named subscriptions, not just those with an expiration time.

If we do not poll this subscription for events for over 60 seconds, it will expire, and a part event like the following will be created:

{
  channel_name: "my-channel",
  subscription_name: "my-expiring-subscription",
  timestamp: 1335318932874000,
  type: "part",
  reason: "expired"
}

If we delete the subscription manually with subscription.delete, we will get a similar part event. The only difference is that the reason will be "deleted".

{
  channel_name: "my-channel",
  subscription_name: "my-expiring-subscription",
  timestamp: 1335318932874000,
  type: "part",
  reason: "deleted"
}

Learn more

To learn more about subscription events check out my blog post about keeping track of subscriptions, or head to the spire.io api docs.

Weekend Reader: April 20, 2012

2012-04-20T00:00:00.000Z

Patents

Twitter announced their Innovators Patent Agreement, which is designed to give their employees peace of mind that their creations will not be used as weapons to stifle competition and innovation in the future. It's nice to see a large company trying to set an example for the industry in a climate of patent stockpiling.

From the release:

The IPA is a new way to do patent assignment that keeps control in the hands of engineers and designers. It is a commitment from Twitter to our employees that patents can only be used for defensive purposes. We will not use the patents from employees’ inventions in offensive litigation without their permission. What’s more, this control flows with the patents, so if we sold them to others, they could only use them as the inventor intended.

A patent is still a patent in Marco Arment's eyes, and he points out that the potential for aggressive legal action remains under the IPA.

First, it’s interesting to read their definition of “a Defensive Purpose”, which, loosely summarized, means they can use their patents to:

Countersue anyone who sues or threatens to sue Twitter for any intellectual-property claim.

Sue anyone who has sued anyone else for patent infringement in the last 10 years.

Sue anyone “to deter a patent litigation threat” against Twitter.

These leave a lot of room for interpretation.

Net Neutrality

Speaking of competition, Reed Hastings has something to say about Comcast's cap favoritism for their Xfinity service.

“The same device, the same IP address, the same wifi, the same internet connection, but totally different cap treatment,” he added. “In what way is this neutral?”

Valve is Different

Any article that begins with "It all started with Snow Crash" is going to be worth the read, and Michael Abrash's post on working at Valve is no exception.

The idea that a 10-person company of 20-somethings in Mesquite, Texas, could get its software on more computers than the largest software company in the world told him that something fundamental had changed about the nature of productivity. When he looked into the history of the organization, he found that hierarchical management had been invented for military purposes, where it was perfectly suited to getting 1,000 men to march over a hill to get shot at. When the Industrial Revolution came along, hierarchical management was again a good fit, since the objective was to treat each person as a component, doing exactly the same thing over and over.

The success of Doom made it obvious that this was no longer the case. There was now little value in doing the same thing even twice; almost all the value was in performing a valuable creative act for the first time.

(via Kottke)

$$$

Despite David Karp's previously strong anti-ad stance, Tumblr announced that they will be introducing paid ads beginning May 2nd.

Miscellany

The coolest desk ever made features a plethora of secret compartments, a drawer operated pipe organ, and a pneumatic logic board.

Weekend Reader: April 13, 2012

2012-04-13T00:00:00.000Z

Instagram

It seemed like everyone on the internet had an opinion about Facebook's acquisition of Instagram earlier this week. Between arguments over whether Facebook overpaid at $1 billion, to whether Instagram was going to get shut down or suffer in quality, cutting through the noise on Twitter was pretty difficult for the next few days. We believe the whole conversation can be summed up excellently between Om Malik's post, Here is why Facebook bought Instagram and Paul Ford's post, Facebook and Instagram: When Your Favorite App Sells Out.

JSCONF

The transcript of Brian Ford's talk "Is Node Better" is up now. He covers a lot of ground on bringing relying on reason and logic to bear solving problems and choosing languages.

Isaac Z. Schlueter has posted a lengthy, level headed response based on notes he took during the session that includes some of his own issues and concerns regarding Node

Bug Squashing

Our friends at Boundary give us a look at a particularly tricky bug

Competition

The Justice Dept. Sues Apple and Publishers Over E-Book Pricing

The publishers who have settled are required to end their e-book contracts with Apple and any other retailer with a “most favored nation clause,” which says that no other retailer can sell e-books for a lower price. For two years, the publishers are also prohibited from restricting any retailer’s ability to discount e-books.

Back to the Future

In honor of the Encyclopedia Britannica giving up its print edition, Bob Stein has shared illustrations made in 1982 for his work on what the future, Intelligent Encyclopedia would be like. The illustrations and scenarios are remarkably prescient in light of how we consume information today.

(via Kottke)

The Wire

Why You Love "The Wire," Explained in Fascinating Detail

The great thing about Lavik’s video is that it itemizes and demonstrates the creative decisions--imperceptible to the “average viewer”--that made the show excellent. But it also provides a powerful demonstration of the monumental effect that those seemingly small creative decisions have on any creative end product.

(via Dave Pell's Next Draft)

Keeping track of subscriptions

2012-04-12T08:00:00.000Z

One feature request that developers often ask for is a way to tell which clients are listening for messages on a channel.

For example, a chat application needs to know what users are in each chat room at any given point in time.

Or, to take a more complex example, say you have a distributed logging setup where log data is generated by various clients and sent to multiple servers (some store the data long-term, others parse the data in real time and send alerts if necessary, etc).

Since log data is important, we would like to know which servers are currently listening for data. Furthermore, we should be notified when servers come online or go offline.

The Spire API makes this very easy.

Each Spire channel knows which subscriptions are currently listening on that channel, and exposes the names of these channels through a channel_subscriptions resource described below. In addition, channels send special events called "joins" and "parts" whenever a new subscription is created or deleted.

Using these two mechanisms, it is a simple matter to keep track of all subscriptions listening to a particular channel.

Find all subscriptions to a channel

All channels now have a channel_subscriptions resource inside them. Making a GET request to that resource will return all the named subscriptions to the channel.

Using the javascript client, we can find all the subscriptions for the channel chan with code like the following:

// `chan` is a channel object defined elsewhere
chan.subscriptions(function (err, subs) {
  // `subs` will be a hash of subscription_name => subscription
});

This code assumes that you already have a channel object called chan.

If we don't have the channel object, but know the channel name, we can get the subscriptions for the channel with a function like this:

function getSubsForChannel (channelName, cb) {
  // First, get the channel.
  spire.channel(channelName, function (err, chan) {
    if (err) return cb(err);
    // Then get all the subscriptions.
    chan.subscriptions(function (err, subs) {
      if (err) return cb(err);
      cb(null, subs);
    });
  });
}

This function takes a channel name and callback as arguments. It fetches the channel, and then fetches the subscriptions for the channel. If there are no errors, the callback will be called with the subscription hash.

Getting back to our distributed logging example, finding all the servers listening for log data on channel "logs" is as simple as this:

getSubsForChannel("logs", function (err, subs) {
  if (!err) {
    // `subs` has all servers subscribing to log data
  }
});

Once we have the list of current subscriptions, we want to keep it up to date. We could just poll the channel subscriptions resource, but that's not very elegant. Instead, we can use subscriptions to listen for subscription creation and deletion events, just like we use subscriptions to listen for message events.

Subscription events

In addition to the usual "message" events (which correspond to messages published to a channel), there are two new kinds of events: "joins" and "parts".

A "join" event is created whenever a subscription is created to a channel. A "part" event is created whenever a subscription is deleted.

Once you have a subscription to a channel, GETting that subscription will return a hash that looks like this:

{
  joins: [],
  parts: [],
  messages: [],
  first: 12345,
  last: 12346
}

The messages array contains messages that have been published to the channel.

The joins array contains an event for each subscription that was created. It will contain the subscription name and timestamp.

Similarly, the parts array contains an event for each subscription that was deleted.

The first and last are timestamps corresponding to the first and last events returned. They are there for convenience, so you don't need to inspect the arrays to find the first and last timestamp yourself.

With the javascript client, you add listeners for joins and parts the same way you add listeners for messages:

  // `sub` is a subscription created elsewhere
  // Add a listener for published messages
  sub.addListener('messages', function (messages) {
    // `messages` is an array of message events
  });
  // Add a listener for new subscriptions
  sub.addListener('joins', function (joins) {
    // `joins` is an array of join events
  });
  // Add a listener for deleted subscriptions
  sub.addListener('parts', function (parts) {
    // `parts` is an array of part events
  });
  sub.startListening({ });

For our logging example, we want to be notified when a log server goes on or offline, so our listeners should send some type of notification.

Caveat

Currently, the only way to remove a subscription from a channel is to explicitly DELETE it.

sub.delete(function (err) {
  if (!err) {
    // sub successfully deleted
  }
});

This deletion removes the subscription from the list of current channel subscriptions, and triggers a part event.

We are working on a new feature that will allow subscriptions to timeout after a set period of time. After this feature is released, if a user disconnects from your application or navigates away from the page, Spire will delete the subscription for you and send the "part" event.

Version 1.1

2012-04-11T00:00:00.000Z

New Features

We're very pleased to announce our 1.1 release, which features:

Named subscriptions and subscription events. This makes it easy and convenient to keep track of who is participating in chat, games, or collaborative applications.
Apple Push Notifications. You can provide your APN credentials in your spire.io account settings and have notifications sent to iOS devices.
Delete And Update Individual Messages. Great for updating feeds.

Named Subscriptions

We hope you'll agree that named subscriptions and subscription events are a really elegant way to connect identity to subscriptions. Named subscriptions are just ordinary subscriptions with names associated with them. These can be whatever your application sets them to be, so it's very flexible.

You can see named subscriptions join and leave channels using subscription events, making it easy to notify participants in a channel in realtime who else is participating. You get these events the same way you get messages: simply make a GET request to a channel. Then, just check the joins and parts properties in addition to the messages property.

For more information, check out our new post detailing how to take advantage of named subscriptions and subscription events.

Apple Push Notifications

We are also pleased to announce support for Apple Push Notifications. Our mobile developers have been asking for this - now your messages can reach iOS devices while supporting the native consumer experience.

More Soon!

We've also got lots more goodies in store, including more documentation, more features, and even ... a whole new API!

Weekend Reader: April 6, 2012

2012-04-06T00:00:00.000Z

JSCONF

A few of our engineers went to JSConf this week and returned with a bunch of cool stories, some of which even involved coding. Our engineer Nick Lacasse was kind enough to link those of us at home to a few talks he liked.

Application Cache: Douchebag, presented by Jake Archibald

The Application Cache has skills we need, but if you asked him to paint your bathroom he'd somehow manage to flood your kitchen and break your TV in the process, and he wouldn't care.

While there weren't any slides available, Nick also sent in a link to their ECMA proposal after hearing a presentation on RiverTrail, which compiles javascript to parallizable GPU code.

Fellow spire.io dev Jason Campbell also posted some notes he took at NotConf the preceding day.

Codestre.am is another nifty project we heard about at JSConf that allows you share your terminal by live stream or recording. This could be a great tool for helping our non-technical staff as they take their first fledgling steps towards becoming hackers.

(More) Women in Tech

Etsy partners with Hacker School, providing a total of $50,000 in financial aid for women to join

Twenty is a small number, but it would roughly match the total number of female engineers I’ve hired in the past 17 years. Even a small change can have a large impact, given the severity of the issue.

The post is much more than a generic announcement, and is worth reading in full. It also includes information about how to apply for one of these grants yourself, so apply or forward it accordingly.

Redis

Redis Persistence Demystified

Salvatore "antirez" Sanfilippo gives a great high level overview of how Redis persistence works.

Type

Typographer and teacher, Indra Kupferschmid, recently posted a proposal for making a more accessible system for classifying typefaces with a nice bit of history and context thrown in.

The problem with research in any field is that you dive into a subject on such specialized and detailed level that you forget that your distance to the language and knowledge of the normal people gets greater and greater. It helps to step back every now and then and ask the actual user. A classification should help them to find, select and combine typefaces, and not the scholar in the first place.

Open Source

VentureBeat reports that Facebook open-sourced their mobile test suite, Ringmark.

For those that are building with the web today, it’s a major hurdle to learn native technologies like Objective-C and Java, and we hope that an improved mobile web can unlock a large contingency of developers that could, and will, be developing for mobile.

Miscellany

A certain spire.io employee likes talking at length about how we are in the golden age of text editors, and Brett Terpstra has what has to be the most comprehensive list, complete with reviews, of ios text editors in existence.

Martin Wattenberg and Fernanda Viegas have made a beautiful visualization of wind activity in the U.S. using data the updates hourly.

Weekend Reader: March 30, 2012

2012-03-30T00:00:00.000Z

Talks

Jeremy Keith makes yet another appearance on the weekend reader, this time with a post inspired by Andy Hume's SXSW talk, CSS for Grown Ups: Maturing Best Practices, wherein he provides some more resources relevant to Andy's theme of "creating design systems". Keith also provides a round up of some of the better design focused talks from South By here as well.

Video from Luke Wroblewski's keynote at DrupalCon Denver is up now. If you haven't been following along with his blog about why teams should focus on developing for mobile first, this video will do a great job of bringing you up to speed.

"7 Years of YouTube Scalability Lessons in 30 Minutes." Todd Huff recaps YouTube engineer Mike Solomon's talk from PyCon.

Open Source

Mitchell Hashimoto provides some "Lessons Learned Building Open Source"

Go version 1 was released on Wednesday.

Photo "sharing"

Quora user Changneng Chen gives an in depth answer on how photo filters, like Instagram's, work.

Rack

Puma.io

Unlike other Ruby Webservers, Puma was built for speed and concurrency. Puma is a small library that provides a very fast and concurrent HTTP 1.1 server for Ruby web applications. It is designed for running Rack apps only.

Miscellany

Kickstarter provides a follow up on the massive impact that Double Fine's fundraising success has had on the site as the 60,000 new backers that the project brought in have gone on a massive nerdy spending spree.

The NY Times takes a look at the growing interest around learning to code and the idea that reading code is the new literacy.

“Inasmuch as you need to know how to read English, you need to have some understanding of the code that builds the Web,” said Sarah Henry, 39, an investment manager who lives in Wayne, Pa. “It is fundamental to the way the world is organized and the way people think about things these days.”

Finish Weekend

2012-03-029T00:00:00.000Z

As textbook dabblers and side project enthusiasts, we know how hard it can be to get that extra burst of energy needed to get a project out the door. Thanks to those web apps that have been three quarters done for too long, we were only too happy to be involved with Collective Idea's Finish Weekend in Boston last weekend.

We didn't get to have a team member out there, but we tried to keep up from the sidelines and thought it looked like a great success. The space Thoughtbot so generously provided looked warm and inviting for all the finishers that showed up. There was food when they got hungry, and Collective Idea's own Tim Bugai finished up the iOS app for Brian Ryckbost's liquorli.st to help Michiganders find good prices on liquor when they're out and about. For our part, we learned that drinking in Michigan is complicated.

Ian Lamont, half of the team behind the mobile classifieds app Invantory, wrote a great recap of the event on the Invantory company blog. You can also read his follow up post, where he goes into more detail about exactly what he accomplished on his site redesign.

Some other notable projects were the real-time audio analyzers Alex Wiltschko brought in to work on, CodePupil - a platform for learning to code via exercises and games, and weight loss network Lose It or Lose It, who came in to polish the "Keep It Off" feature.

Thanks to Daniel at Collective Idea for doing such a great job of keeping us in the loop. This type of event is a great idea and we look forward to being involved in more. Hopefully we will be able to attend one in person soon.

Announcing JS.la

2012-03-023T00:00:00.000Z

We love JavaScript. The language itself is a pleasure to work with, and it's been amazing to watch the community grow and mature. As soon as we started talking about organizing more meet ups in LA, we knew we wanted to have a killer JS group to go to. So we were really excited when Jason said he had the opportunity to take over organizing JS.la. Carlo Flores, our head of Ops, is bringing his experience as co-organizer of LA DevOps to the table, and we think the result is going to be a group that combines solid technical talks and fun story swapping.

The group will meet the last Thursday of every other month in the Hollywood area beginning with this month's meeting at 7 P.M. on March 29th, hosted by the kind people at CityGrid. In addition to the socializing and casual JS talk you would expect, the meeting will feature a presentation by James Halliday (@substack) on breaking up big node.js apps into lots of tiny services, and an introduction to WebGL and three.js by Travis Glines (@travisglines). You can read expanded descriptions on the JS.la site.

Jason and Carlo are still looking for people interested in giving a short lightning talk. (About 5-15 minutes long.) You can contact them via the js.la twitter account: @losangelesJS

MAKE SURE TO RSVP, as space is limited

7 P.M., March 29th

CityGrid 8833 W. Sunset Boulevard West Hollywood, CA 90069

View Larger Map

Weekend Reader: March 23, 2012

2012-03-23T00:00:00.000Z

The Cloud

Amazon reacts to customer demand for hybrid cloud options by partnering with Eucalyptus to "[enable] customers migrate workloads between private clouds and Amazon Web Services."

Ops

Netflix's Adrian Cockcroft provides some insight into how Netflix operates, and expands on his notion of NoOps.

Several hundred development engineers use these tools to build code, run it in a test account in AWS, then deploy it to production themselves. They never have to have a meeting with ITops, or file a ticket asking someone from ITops to make a change to a production system, or request extra capacity in advance. They use a web based portal to deploy hundreds of new instances running their new code alongside the old code, put one "canary" instance into traffic, if it looks good the developer flips all the traffic to the new code. If there are any problems they flip the traffic back to the previous version (in seconds) and if it's all running fine, some time later the old instances are automatically removed. This is part of what we call NoOps. The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts and writing tickets to request changes for the datacenter. Now they spend seconds doing it themselves in the cloud.

John Allspaw of Etsy responds to Cockroft. while describing Etsy's process.

Ok so you've just described an engineer deploying a change to production, choosing and relying on signals and metrics to increase or decrease confidence in said change, taking action (or not) based on those signals and metrics, and then adjusting resources to make the change pseudo-permanent. Congratulations, you've just described one of the most basic patterns of modern Operations, one that aligns quite nicely with Col. Boyd's OODA loop (http://en.wikipedia.org/wiki/OODA_loop), well-known to the Operations community. No matter how many different titles you give it, no matter who that engineer reports to, this is Operations. This is the Operations that is known currently, in the field. Really, ask anyone who's been doing this at scale. Ask Theo Schlossnagle. Ask Jesse Robbins. Ask Benjamin Black. Ask Artur Bergman. Ask Jonathan Heiliger. Ask Jon Jenkins.

You have the person responsible for the code deploy the code and take responsibility for it in production. Excellent! This is how many of us have been doing it for years.

Gaming

Rob Hawkes gives us a look at the first Mozilla HTML5 games work week. It's awesome to see so much effort being put behind pushing the web platform into new areas.

While we're at it, the new demos of the Coffee Physics library might pique your interest about browser based games.

The Long Game

Matt Haughey provides video and his stage notes from his "Lessons from a 40 year old" talk at Webstock where he encourages us to put in the work ourselves and consider our sites and products with the next ten years in mind. It's a lengthy read full of poignant moments and good advice.

I also want to mention a term people in the VC-world consider derogatory, which is “lifestyle business” where someone builds a site, and brings in enough money to support themselves. Investors see that as a bad thing, and it boggles my mind sometimes when I think about the twisted world that produces that kind of idea. I've had discussions with VC investors where they hear about my business and dismiss it as "just a lifestyle business" and I've come back with "hey, it's a pretty good lifestyle, probably a better lifestyle than yours, probably a better lifestyle than if I accepted five million dollars from you."

Michael Lopp's post "Hacking is Important" paints a picture of the paradoxes that a product or company's success inevitably brings, and how a company built to last must maintain a space for tearing down what came before.

A healthy product company is, confusingly, one at odds with itself. There is a healthy part which is attempting to normalize and to create predictability, and there needs to be another part that is tasked with building something new that is going to disrupt and eventually destroy that normality.

Miscellany

Robin Sloan has created a beautiful "tap essay" on the difference between liking and loving something on the internet.

James Bamford reports that The NSA is Building the Country's Biggest Spy Center

Weekend Reader: March 16, 2012

2012-03-16T00:00:00.000Z

Designing Hypermedia APIs

Steve Klabnik shares a portion of his Designing Hypermedia APIs book on "Programming the Media Type."

Essentially, this is the only information that's published about a particular API. What media types it expects. That's a radical departure from what one is used to. However, it's essentially accurate. And, in fact, if you recall the canonical example of the web browser, you'll realize that the only thing it knows about is HTML, CSS, and a few other media types.

To build an effective client, you must only rely in the media types served, and their definition.

Web

We have been following the progress on Mozilla's BrowserID project with interest. If you haven't taken a look yet, you can get a sense of what it's about from their overview statement.

This specification, BrowserID, defines a mechanism for websites to request, from the user via her user-agent, a signed assertion of email-address ownership. Web sites can use this mechanism to register users on their first visit and log them back in on their subsequent visits. The trust path for these assertions of email-address ownership is designed to be federated and decentralized: individual domains can certify their own users.

Responsive Design

Tama Pugsley and Andrew Hovey created an awesome tool for checking your site across a variety of devices at the same time. Which is cool. And it's called The Responsinator. Which is even cooler.

How Yahoo Weaponized My Work: Former Yahoo employee Andy Baio gives a personal account of his experience watching patents come from his work.

The scary part is that even the most innocuous patent can be used to crush another’s creativity. One of the patents I co-invented is so abstract, it could not only cover Facebook’s News Feed, but virtually any activity feed. It puts into very sharp focus the trouble with software patents: Purposefully vague wording invites broad interpretation.

Of the ten patents Yahoo says Facebook is infringing on, Josh Constine at Techrunch thinks he's identified their best chance of success.

Marco Arment on the "Curator's Code."

The problems with online attribution aren’t due to a lack of syntax: they’re due to the economics and realities of online publishing.

The interest in the code itself seems to have already died, but this post should be referenced whenever there's a debate about ethically covering someone's post.

Michael Carney picks up where Tara Brown left off last week, with a positive outlook for the LA tech scene. He's also included a list of local incubators and accelerators as a resource.

Miscellany

Andrew Chen shares a story from a visit to Pixar that reminds us to take a step back and appreciate our products through the eyes of others.

REST Made Simple, Part 3

2012-03-13T08:00:00.000Z

In Part 1 and Part 2, we've introduced several key ideas:

The Web is really a massively distributed key-value store, or tuple-space.
The keys are URLs and the values are resources.
We can update the tuple-space using methods: get, put, and delete.
We use post to send resource-specific requests to a given resource.
Using the right methods makes it easier to support caching.
A given resource may have different formats, or representations.
Clients specify the representation they prefer to use.
We use headers for this so we can share keys between clients easily.

So what does having a massively distributed tuple-space buy us? Why did the architects of the Web choose this abstraction in particular?

Great For Sharing Documents!

It's easy to understand if you remember that the original intent was to share documents. So what we have is a giant, living library. Even better, it's a library where you can ask for a document in whatever format or language you prefer. (It might not be available that way, but there's at least a way to ask, and a way for the document's owner to give you what you asked for.) The owner of a document can update it, add new formats, and so on. Because the URLs are format-neutral, it's easy for people to share documents, too: they just share the URL. Finally, for documents that are really widely read, they can be served easily by intermediaries from cache.

Which brings us to hypertext. We can embed URLs in the documents themselves to link one to another. And this is what we've come to know and love as "the Web." It worked really, really well for documents.

Great For Sharing Documents Objects!

But we can go a bit further than that. In fact, what we really have is an abstraction for dealing with documents. So anything we can model in terms of documents will work just as well. If there was a programming abstraction based on documents, we could integrate all kinds of interesting things into this giant library that everyone is already using. And, in fact, there is such an abstraction: object-oriented programming.

Replace "document" with "object" above, and you've got it. In fact, I'll just make it easy for you:

... the original intent was to share objects. So what we have is a giant, living library. Even better, it's a library where you can ask for an object in whatever format or language you prefer. (It might not be available that way, but there's at least a way to ask, and a way for the object's owner to give you what you asked for.) The owner of an object can update it, add new formats, and so on. Because the URLs are format-neutral, it's easy for people to share objects, too: they just share the URL. Finally, for objects that are really widely read, they can be served easily by intermediaries from cache.

Which brings us to hypertext. We can embed URLs in the objects themselves to link one to another ...

Now We've Got it

Okay! So, the Web gives us programmers a protocol for a giant, distributed key-value store for dealing with objects! We can get, put and delete them, we can send them requests with post, we can share them using the keys, we can deal with different representations for them to help us support different kinds of clients ... if nothing else, that's pretty interesting.

Of course, we've just replaced one question with another. What is that useful for?

Application Flows

In our last installment, we said we were going to talk about hypertext. We've already mentioned that objects can reference (via hyperlinks) other objects using the keys (URLs), which is useful. And they can obviously also reference documents, too. We can mix and mingle the two.

On the face of it, this doesn't seem to be real earth-shaking. It turns out, though, that we can effectively define entire application flows in terms of links. If we think of application in terms of a state-machine, each object or document we have on the client represents a node of that state machine, with literal links to the other nodes. We follow a link and transfer the next state to the client. This is the infamous "state transfer" of the REST acronym. (We've already covered the "REpresentational" part.)

Until Next Time ...

Our giant key-value store allows us to move objects (including documents) around on the network, and to define entire application flows in terms of hyperlinks. That's pretty cool. But in our next installment we're going to spend some time looking at the challenges of this approach. In particular, why do so many programmers get frustrated with it? In the real world, we see all kinds of APIs, but rarely do they take full advantage of the architectural model that HTTP offers. Is that a mistake? Or the natural consequence of flaws in HTTP? Or something else entirely? See you then.

Weekend Reader: March 9, 2012

2012-03-09T00:00:00.000Z

We've been pretty busy with the release of 1.0 and getting ready for SXSW this week, but we still found time to get a little reading done.

SOLID Design Principles in JavaScript

Derek Greer has put together an in depth series on SOLID design principles.

Web

Jeremy Keith has a nice round up the ongoing discussion concerning vendor prefixes.

The superficial issue is that web developers have been implementing -webkit- properties without then adding the non-prefixed standardised version (and without adding the corresponding prefixes of other vendors). The more fundamental problem is that while vendor prefixes were intended to introduce experimental features until those features became standardised, the reality is that the prefixed version ends up being supported in perpetuity. Nobody is happy about this situation but that’s the unfortunate reality.

Beards

This has been a rather down week for beards, and this article in particular sparked a lively debate in the office between the bearded and the non-bearded. This is a big deal for the staff of a young tech company. Amanda Hess at GOOD understood how distraught some of us would be and posted her response at 3:00 A.M. this morning. It's pretty clear what we'll be talking about at lunch.

Los Angeles

Startup Weekend LA was a great event that we were proud to be a part of. It's always inspiring to see a plan come together, and the participants all had a great energy about them. Techzulu has posted the winning presentation from Snazzy Room on YouTube. A final congratulations is in order, as well as another "Thank you" to Coloft for hosting.

We love working out of Los Angeles and Tara Brown hits most of the reasons why in her post for Forbes. A diverse community, great investors, more city to explore... We think it's pretty great. She also has solid opinions on how we can improve and provides some good resources for getting more involved in tech here.

Speaking of startups in LA, our fellow Angelenos at Factual are having a developer contest with Box revolving around associating files with place IDs.

Other Miscellany:

When Beards gets it's own section, you know that the Miscellaneous links are going to be extra weird.

How I Helped Destroy Star Wars Galaxies - Patrick Desjardins recounts how he built is own real world empire in Star Wars Galaxies and watched it crumble.

In Which Eben Moglen Like, Legit Yells at Me for Having Facebook

So you’re using them and every time you tag anything or respond to anything or link to anything, you’re informing on your friends. You’re part of the problem, you’re not part of the answer. Why are you calling up to ask me about the problem you’re creating?

There are quite a few Instapaper fans here, so we appreciated Dan W's tongue in cheek tribute, the Instapaper Placebo bookmarklet.

I don’t need a nice mobile app for reading. I don’t need a way to remove all the clutter from the page. I don’t need an online cross-platform bookmark syncing service. I just need a way of offloading all my good intentions.

spire.io at SXSW

2012-03-7T00:00:00.000Z

The release of 1.0 has been great. We couldn't have done it without the help of our early beta members and the helpful feedback you provided, and we'd like to meet you in person. We know many of you will be descending on Austin this week, and it just so happens that two of our own developers, Matthew and Daniel, will be at SXSW as well. We'll be posting updates on their locations via Twitter during the conference to help increase your chances of spotting them in the wild around the convention center, but you'll be guaranteed to see them at some events we're involved in.

Our friends over at Rocket Space are throwing a "Greatest Hits" party at the Bing lot on Saturday and Daniel and Matthew will be there from 4 - 6 P.M. They will also be at All Girl Hack Night's Dev Brunch, of which we are proud sponsors, on Monday from 9 A.M. - 12 P.M. at Frank.

You can talk to them about any questions you have about the platform, applications you're excited to build, our newly announced pricing plan, or you can just hint that you'd like them to buy you a beer.

Hope we see you there!

Spire at All Girl Hack Night's SXSW Dev Brunch

2012-03-07T00:00:00.000Z

We are happy to be sponsoring All Girl Hack Night's Dev Brunch at SXSW this year.

We looked into their group after one of our developers told us Garann was looking for sponsors and were really impressed with the All Girl Hack Night community. Garann is a great talent and a passionate organizer with the goal of bringing more women into the industry and drawing attention to the great women developers who are already out there. We agree that this needs to happen, and after talking to her briefly, sponsoring this event was a no brainer.

We think it's great that Girl Develop It is also getting involved. They work to connect women who may just be starting out, and help them develop their technical chops in order to change the developer landscape. We feel a great opportunity for improvement lies in creating strong environments and communities (like hacking groups) that encourage women. The importance of these groups, and others like them, is especially true for individuals who are just starting out, as people encouraged early on in their development careers will be around later to help others. We love seeing people get excited about coding and we are incredibly happy to play even a small role in the process with these organizations.

The brunch is for men and women both. It should be a lot of fun, and we hope we see you there.

The brunch is on March 12, from 9-11 at Frank. Don't forget to register.

View Larger Map

Version 1.0

2012-03-07T00:00:00.000Z

After months of hard work, spire.io has reached GA!

No small amount of thanks is due to you, our customers, for all the great feedback we've gotten during our Beta. We'll take a small break, catch our breath, see what's happening on Reddit, and get cracking on the great stuff we'll be rolling out in 1.1.

A few highlights of our 1.0 release:

Documentation. Documentation is the key to any API and we take great effort to make ours as clear and complete as possible. If we've failed, we urge you to contact us directly with any questions and one of our developers will follow up as soon as humanly possible.
Pricing. We have the largest free plan in the market because we believe that we'll be successful if you're successful. Spending time optimizing your architecture or analyzing messaging prices are "rounding errors" in the process of building a great product.
Innovation. From our hypermedia (formerly known as REST) API to our application of capability security, we're proud to have a great engine under the hood. Innovation is hard and we are always looking for feedback and great ideas.

By the way, our mission of helping you build great products goes beyond our code: if we can help you by recommending a particular language or library or service, we're glad to do it.

Weekend Reader: March 2, 2012

2013-03-02T00:00:00.000Z

Node

Managing Node.js Dependencies with Shrinkwrap

... it's understood that all software changes incur some risk, and it's critical to be able to manage this risk on your own terms.

REST

WebSockets vs. REST

How do WebSockets play into REST?

Talks

The Buiness of Bookmarking by Maciej Ceglowski of Pinboard (From PDA '12)

As conference season continues, we've been quite pleased by how many presenters are doing a great job of writing up their talks. Slides are great, but many of the best presentations use slides as supplements and leave a lot of gaps for people trying to follow along after the fact.

Maciej has always been really good about sharing the details behind the business and trying to guide other businesses toward reasonable action that will keep them operating comfortably for an extended period of time.

Outgrowing the Cloud

Mike Moore's talk from this year's Ruby on Ales conference. We had some guys there and it sounds like it was a great event.

Mobile

LG reportedly building first Boot2Gecko phone as Mozilla preps app store

B2G pairs Mozilla's Gecko HTML rendering engine with the Linux kernel. The platform's application stack is built entirely with standards-based HTML and JavaScript. Hardware capabilities and other platform features are made accessible through JavaScript with a new set of APIs that Mozilla is working to standardize.

Miscellany:

Steve Moore's take down of the TSA

TSA’s de facto policy to this point has been to react to the latest thing tried by a terrorist, which is invariably something that Al Qaeda identified as a technique not addressed by current screening. While this narrows Al Qaeda’s options, their list of attack ideas remains long and they are imaginative. Therefore, if TSA continues to react to each and every new thing tried, three things are certain:
Nothing Al Qaeda tries will be caught the first time because it was designed around gaps in TSA security.
It is impossible to eliminate all gaps in airline security.
Airline security screening based on eliminating every vulnerability will therefore fail because it is impossible. But it will by necessity become increasingly onerous and invasive on the travelers.

McSweeney's noted that it took them 11 years to make a Yelp joke and only three to get to Kickstarter. This is accelerated culture, people.

Ira Glass on Storytelling

A short video by Ira Glass on how most people doing creative work start with "good taste" but lack the skills to match that taste, so there are grueling years of disappointing, but increasingly-improving work, is another reminder that failing can be either a road to success or an excuse.

Changelog: Feb 28, 2012

2012-02-28T00:00:00.000Z

We updated the spire.io API

API 2012-02-28:

API support for retrieving a subscription by name
More efficient storage and usage of subscriptions created without names
Fixed a bug in the task management system's ephemeral key generation
Updated documentation to reflect that timestamps are now in microseconds.
Fixed a security flaw in the use of secret keys for authenticating accounts.

Site 2012-02-28:

Using the newest version of spire.io.js for all javascript on the main site and the examples.
PushState changes update the title of the page appropriately.

Clever with Capabilities

2012-02-28T00:00:00.000Z

The code for this article is available on github

Spire.io APIs use something called a capability for authentication and authorization. (To learn more about capabilities, see our previous post on the topic.) Essentially, instead of giving users a session and then checking the user's permissions for each action, we give users an encrypted key that is tied specifically to an action they are allowed to take. For example, with the "create channel" capability, for the account "alice", any client that knows that capability can create as many channels for alice as they want, but can't take any other action on alice's account. This is very different than the situation where alice's session key is compromised in a typical application, and an implementation of the principle of least privilege.

This is an awesome way to manage access both for us the API provider and you the developer. Let's walk through a simple example of one way to use the capabilities provided by Spire.io to manage the access your users have to different aspects of your app. We're big fans of node.js at Spire.io, so this example will use node's Express web framework to serve up the capabilities to the client as well as host the site.

The Server

We'll start with some simple code to initialize the Spire.io library, as well as initialize variables to hold the subscription and channel resources that we'll be sharing with our clients.

We're going to have two channels, client-publishable-channel and client-subscribable-channel. I imagine you can guess what capabilities we'll be sharing with the client for each.

Below is the callback we'll pass in when we connect to Spire.io. We use helper methods from the Spire.io library to create the channel and subscription and assign their url and capabilities to the variables we initialized earlier.

Now here's the guts of our application - a very simple callback on the channel that our client will publish to, that does something we don't want clients to spoof for each other. In this case, we're just reversing whatever string we're sent.

Now we set up our Express app with a single method for discovery. We'll just give our clients enough capability to publish to one channel and listen on another. In this trivial example, everyone who loads the page will be on the same two channels. In a more serious implementation, you'd probably make someone authenticate here, and give each user their own channel for publishing, and depending on the application, either the same server channel (in the case of e.g. a chat application where everyone is in the same room) or their own read-only channel.

Finally, we start our server in a loop that makes sure all our resources have been created before launching:

The Client

Now that our client has the capabilities, it needs to use them. Below is all the JavaScript we need. It should be fairly self-explanatory if you've read the Spire.io.js README and followed the code above.

That's all there is to selectively handing out capabilities with the Spire.io API using our JavaScript library. Having this kind of granular authorization for resources opens up a huge array of message-based applications. Obviously our example is intentionally simplistic, but with only a little more complexity you can imagine using a collection of read- and write-only channels to support multiplayer gaming, for example, with a server ensuring movements and actions are legal before replaying them to clients. (As a matter of fact, we wrote that sample code back when we were alpha testing the messaging service - who knows, maybe it will see the light someday).

We're really excited about giving application developers flexibility and power to be really creative on our platform, and capability-based security is only one part of that. If this has inspired you to build a new kind of app based on real-time messaging, or if you have questions or comments, please leave them below or give us a shout on twitter @spireio.

Check out the code on github!

If you want to learn more about capabilities, read our blog post about it

Weekend Reader: Feb 24, 2012

2012-02-24T00:00:00.000Z

Web

Mozilla rolls out their brwoserID service, renamed "Persona"

Git for beginners

We've been bringing all our non technical staff up to speed on git, and this interactive cheat sheet has been helpful. (via indexzero)

Use the cheat sheet and Git Immersion to create a happy git using workforce.

Cut Throat Coding

You Are Not Ruthless Enough: Chris Parker on the hazards of an "unclear separation of responsibilities" via Buzz Andersen

“This is just temporary. I’ll clean it up in a bit.”
“It was really late in the schedule and this was the fastest thing we could do to get the feature working.”
“I tried a bunch of things and this is what seemed to work.”

DevOps/NoOps

Andrej Koelewijn on "self service architecture:"

Removing project dependencies, collaboration and coordination should be main priority for architects. Don’t try to design the whole system. Try to specify modules that can be created independently. The goal should be to get rid of large projects, by replacing everything with small projects, that don’t need to coordinate their efforts.

Phil Hollenback - Devops is Here Whether You Like it or Not

Virtualizing your neckbeard buys you a few years but that's it. It certainly doesn't improve your career skills. If I was hiring a sysadmin right now, I'd ask if he has any vps experience. If the candidate said yes, I'd ask how he managed those vpses. If his answer wasn't puppet or chef or something similar, I'd be pretty suspicious.

Oh, you want more ops debating? Chris Siebenmann has a nice response to Hollenback.

Miscellany:

This article on the man behind 5 Hour Energy came up at lunch last week, but it didn't show up in my feed until yesterday. The notion of a monk saying "I'm killing it right now" is funny enough to secure a late appearance in the Reader.

Music for Programming: A series of mixes intended for listening while programming to aid concentration and increase productivity (also compatible with other activities).

Jeff Preshing takes a Look Back at Single-Threaded CPU Performance.

James Altucher on the things that hold back his productivity.

Socal Piggies Meetup and More

2012-02-24T00:00:00.000Z

We had the pleasure of hosting SoCal Piggies' (Python Interest Group) February meetup at our office on Wednesday and we'd like to thank everyone who came out. The members were really active, and Craig Kerstiens from Heroku and Joshua Timberman from OpsCode were both nice enough to travel out here to present.

If you couldn't come out, you can listen to Craig's presentation on the best practices for app development and achieving environment parity here, and Josh's presentation providing an introduction on Chef and configuration management across all infrastrutcture here. We've linked to their slides in the descriptions so you can follow along.

We'd like to make a final thank you to Grig for organizing the even and Evite for providing the pizza and beer.

We will be hosting the PyLadies intermediate Python workshop tomorrow, Feb 25, and holding LA Hack Night every other Wednesday, starting Feb 29. We'll be alternating with the good people over at Carbon 5 in Santa Monica. You can get more details at their meetup page.

Beautiful reference docs with JSDoc and Noc

2012-02-23T08:00:00.000Z

Documentation Matters

When developers evaluate new libraries or tools, documentation is usually the first thing they look at. If your library has poor documentation, most developers move on without bothering to learn more.

It doesn't matter how great the code is, how many tests it has, or how many forks it has on GitHub. If your code has bad docs, developers are not going to use it.

As a developer, I know how important documentation is, but I still find myself neglecting it far too often. For one thing, I put off writing documentation until the very last minute. But after hours or days or weeks of hacking and bug hunting, writing documentation is usually the last thing on my mind.

Inline Documentation

So when I recently undertook a major refactoring of the Spire javascript library, I wanted to force myself to write good documentation in advance. I started investigating various tools to convert inline documentation (in the form of code comments) into nice html reference pages.

The benefit of inline documentation is two-fold. First, it allows the programmer to write the documentation at the same time as the code; no need to hastily assemble it at the last minute.

Second, keeping the documentation inside the code makes the code easier for developers to work with. There is no need to flip back and forth between the docs and the code. Furthermore, if a developer changes a class's interface, or a function's signature, the docs can easily be updated at the same time.

First attempt: Docco

When I started looking for good inline documentation tools, the first one I tried was Docco. Docco lets you write Markdown in your comments, and those comments get displayed side-by-side with the code. The Docco website itself is generated with Docco. I chose Docco because it supported a lot of languages and made very nice looking docs.

My first impressions with Docco were great. I liked being able to structure the documentation however I wanted. Since I controlled the markdown, I could put headers and subheaders wherever I wanted. I put lists, embedded code examples, and horizontal rules all over the place, and it was glorious.

But soon my refactoring project grew, and extended to multiple files and classes, and took on a hierarchy all its own. Docco doesn't really support multiple files. It creates one html page per source file, and if you want to link them, you have to put the links your markdown comments manually. I found myself spending lots of time linking different parts of the docs, and trying to make my markdown organization mirror the overall project organization.

Writing documentation had once again become a chore and a time-sink. I decided to leave Docco in search of something else.

Second attempt: JSDoc and Noc

Now that my project was large, I wanted a tool that could understand the inherent structure in my code. I didn't want to have to duplicate that same structure in my comments. I decided to try JSDoc, which is a comment format for javascript projects.

JSDoc is based on JavaDoc. There are a bunch of special tags you put in your comments to designate that a certain function is a constructor, specify what kinds of arguments the function takes, show a code example, and much more.

It took a while to learn all the tags, but once I did, I found myself eagerly documenting functions before I even wrote them! I was planning out entire classes, describing the methods, their arguments, and their return values. Once the docs were in place, writing the actual code was easy because I had already thought through what the code needed to do.

You can see an example JS file with lots of JSDoc comments here.

Soon I had all my code marked up with JSDoc. I started looking for tools to convert my JSDoc-ified project into html pages. The most popular tool is jsdoc-toolkit, but that depends Rhino, which depends on Java, and I didn't want to introduce Java into our dependencies just for documentation. I checked out a few JSDoc tools that ran on Node, and ended up settling on a great tool called Noc.

Noc runs jsdoc-toolkit in Node, rather than in Rhino. Noc is smart about my project's structure and links classes and files automatically. It supports the full suite of JSDoc tags, and can use a variety of templates. Noc comes with the CodeViewer template, which looks nice to me, but others are available.

You can check out the spire js client docs here, and the code and comments used to generate the docs on the spire.io.js github.

Web Capabilities: A Better Way To Secure Your Apps

2012-02-21T00:00:00.000Z

Capability security provides an authorization model that works like keys on a keychain. A given capability provides privileges for accessing a specific object, much like you might have a house key, a car key, and a mailbox key, each of which make it possible to access separate objects.

You can even have capabilities that provide different privileges for the same object. A valet key allows you to get into the car and start the engine, but not open the glove box.

Why Capabilities Aren't Already In Common Use

Capability security is attractive because of its simplicity. But it's never been widely adopted in modern operating systems, largely for historical reasons. We've carried that over to Web applications even though the architecture of the Web is completely different from that of an operating system. But there's nothing stopping us from using capability security in Web applications today.

A Few Definitions

A capability is an unforgeable key that references an object and a set of privileges. A Web capability, then, is an unforgeable key that references a resource and a set of methods (ex: get, put, etc.). It's actually a bit more complex than that in real life, but, conceptually, that's a reasonable starting point. Finally, a capability description is some representation of the object and privileges referenced by a (Web) capability.

Example: Get Messages

For example, suppose we have a Web capability to allow us to read messages published on the foo channel. The capability description could be a JSON object:

{
  url: "http://accounts/acme/channels/foo",
  methods: [ "GET" ]
}

When you want to GET messages on the foo channel, you must provide a matching capability. That is, the capability should reference a capability description that looks something like the one above. The capability itself can be in a header or appended as a parameter in the URL. If the capability doesn't match, you get a 401.

Advantages Of Capabilities

Capabilities offer numerous advantages over typical security models for Web applications.

They're simpler to enforce.

That simplicity means the associated code is less bug-prone. Simple things are harder to screw up, and when you screw up security, it's a different order of screw-up than most other things.

They're more efficient.

Capabilities can be made self-describing. That is, we can encrypt the description of a capability and then use that encrypted description as the actual key. To check a capability, all we have to is decrypt it and check the privileges against the request. We can do this with identity-based schemes, but not with specific authorizations. That still requires a reference to external data at some point.

They're more flexible.

We can share capabilities, much the way you can give a valet your valet key. And you can share exactly what you want to share and nothing more because capabilities are so fine-grained. This makes it possible to accommodate collaborative scenarios that would otherwise need to be anticipated in advance and handled explicitly in code.

The Principle Of Least Privilege

The possibility of sharing capabilities means that different clients of an API can be given to different capabilities. Each client can have only those capabilities that they need. This is very different from what happens with most Web APIs, where access is all-or-nothing.

With capabilities, each activity supported by a given API (reading my feed, posting an update) has distinct capabilities. I could share only those that an application absolutely needed. This is called the principle of least privilege and it's an important advantage that capabilities provide.

We see a variant of this approach with mobile applications today. But the capabilities are managed opaquely by the operating system and applications can't share capabilities with each other.

Using A Web-Capabiliity

Enough talk! Let's look at how Web capabilities can be used in practice. First, let's look at a capability can be included in an HTTP request. We'll be using CoffeeScript in the examples that follow.

In this example, we're also making use of an open source HTTP library we wrote, Shred, that runs in both the browser and in the Node.js environment, and makes it easy to make rich HTTP requests - and deal with the responses.

shred.post
  url: channel.url
  headers:
    content_type: schema["1.0"].message.mediaType
    accept: schema["1.0"].message.mediaType
    authorization: "Capability #{capabilities[channel.url].post}"
  content:
    content: "Hello, World!"
  on: 
    201: (response) ->
      // we get back the message we publish
      assert.equal response.content.data.content,"Hello, World!"

Basically, what we're doing here is sending a request that says "please post the 'Hello, World!' message to this channel." We have a channel object that provides the URL we want to use. We have a schema object that has all the media types defined for us. Finally, we have a capabilities object where we've cached capabilities we have for given URL and method combinations.

What we're most interested in here is the authorization header. We're sending a capability with the request, prefixed with the Capability scheme, that tells the server that we're authorized to do this.

Checking The Capability

So what happens on the server? A capability description consists of a resource URL and a list of methods. We could extend this further to include additional information, such as the acceptable media types, but for now let's keep things simple. First, we want to check the authorization header.

[scheme,key] = context.request.headers["authorization"]?.split(" ")

If the scheme is Capability, then we know we have a capability key. Later, we want to actually look up the key in Redis. We access Redis via store object.

if scheme == "Capability"
  hash = "#{context.account.key}.capabilities"
  store.hget hash, key, (error, capability) ->
    if error || !capability
      context.emit "unauthorized"
    else
      context.workspace.capability = 
        new Capability(JSON.parse(capability))

We lookup the key in a Redis hash, scoped by the the account key (which makes it easy for us to revoke all capabilities for an account if necessary). If we don't find it, for whatever reason, we simply return a 401. We don't want to give a malicious attacker any additional information here.

So now we have an object that describes the privileges associated with a given capability. Now, we can match it against the request:

validFor: (request) ->
  request.url == @url and request.method in @methods

If the capability description's validFor method returns true for a given task, we go ahead and process the task. If it returns false, we return a 401.

Issuing Keys

Generating keys is straightforward. When we return a resource representation, we generate the associated capability description, store it in Redis, and return the key (that is, the capability) in the header of the response.

For example, suppose you've just created a new channel. Obviously, we want to give you full permissions to do whatever you want with it. So the capability description looks like:

capability = new Capability
  url: channel.url
  methods: [ "get", "post", "delete" ]

We don't allow put here, because we there is no put operation for channels.

Generating Random Keys

Next, we generate a key. Here's our key generation code, using the now-standard Node crypto library, which in turn uses OpenSSL, and the asn1 module. We generate a string of random bytes, Base64 encode them, and then make sure we get rid of any characters that will mess with URLs, for cases where we want to use a key in a URL.

ByteEncoding = require("asn1").Ber
crypto = require("crypto")
# http://en.wikipedia.org/wiki/Base64#URL_applications
safe64 = (str) ->
  str.replace(/\\//g, "_")
    .replace(/\\+/g, "-")
    .replace(/[=\\n]/g, "")
module.exports =
  safe64: safe64
  randomGenerator: (length) ->
    () ->
      safe64( crypto.randomBytes(length).toString('base64') )

Storing Keys

We store the capability description in Redis, associated with the key:

hash = "#{account.key}.capabilities"
json = JSON.stringify(capability)
store.hset hash, json, @, (error) ->
  if (error)
    // uh-oh ...

Returning The Response

Finally, we construct our response and put any capabilities (the keys, not the descriptions) associated with the response in an HTTP response header. (Actually, at the moment, we return the capabilities with the response bodies, which makes them much less cacheable. But that's another topic.)

Less Than 100 Lines Of Code ...

And that's it. Less than 100 lines of code, and this encompasses virtually all of our API security code. Yet it's actually more secure in many respects than a more typical session-based approach that would have much more code. That's because it's usually pretty obvious what capabilities you want to grant based on the context when a resource is first created. For example, if you've created a channel, you get a capability for joining it.

But Wait, There's More ...

There's a lot more to Web capabilities than I've been able to get into in this blog post. Capabilities separate identity and authorization, which is a profound change from typical session-based Web security models. Ultimately, Web capabilities need to take into account not just methods, but possibly query parameters and media types. Self-describing keys introduce additional challenges. There are also implications for HTTP caching. Check back for follow-up blog posts, where we'll explore these issues.

In the meantime, hit us up in the comments with your thoughts. Are Web capabilities something you'd consider in your Web applications?

UPDATE: Got two great links on other similar efforts. I wasn't aware of these, but I'm looking forward to studying them and see how they've addressed some of the challenges of doing capabilities on the Web.

There's Waterken's notion of "web-keys".
Google has a secure component model for Web apps called Caja that uses object capabilities.

We'll keep adding to this list as people give us more feedback. Thanks!

If you want to see capabilities in action, check this example

Weekend Reader: Feb 17, 2012

2012-02-17T00:00:00.000Z

Security

Nadia Heninger on factorable keys.

We have been able to remotely compromise about 0.4% of all the [7 million tested] public keys used for SSL web site security. The keys we were able to compromise were generated incorrectly--using predictable "random" numbers that were sometimes repeated. There were two kinds of problems: keys that were generated with predictable randomness, and a subset of these, where the lack of randomness allows a remote attacker to efficiently factor the public key and obtain the private key.

And another good response from David Wachtfogel

Apple announced an update for iOS requiring user permission in order for apps to access the address book. This follows Arun Thampi's post which revealed that Path was uploading your address book to their servers, and led others to confirm that several other big name companies do as well.

Scaling

A detailed rundown of how Tumblr has transitioned to deal with their scaling issues. http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html

Netflix announced Servo, an open source tool for exposing and publishing application metrics in Java.

As @getsomerestbook points out, you should be using HTTP caching to make sure your API scales. Mark Nottingham has a nice tutorial to help you out.

Why Monitoring Sucks

Carlo has been getting pretty into these posts on monitoring.

Why Monitoring Sucks, or the Future is Toolbox

Why Monitoring Sucks -- For Now

Fun, Productivity, and Other Miscellany

Andy Baio (Wax Pancake) and his nephew made a site for people to make and share text adventure games. http://waxy.org/

Jake Donham's High-Assurance Web Programming with Coq Rock! is probably the funniest thing this list compiler has read all week. (via @brixen and @rbxbx)

It sounds strange to say, but we're a little proud of how few meetings we manage to have. If you're looking for a way to make meetings less painful and costly, Jeff Atwood's tips will serve you well.

37signals have a great reminder that editing yourself becomes more valuable as you approach the finish line.

POSIX close(2) is broken

Weekend Reader: Feb 10, 2012

2012-02-10T00:00:00.000Z

On the Web

Adobe proposes a nice spec for HTML5 image progress events.

Peter-Paul Koch takes an interesting position regarding what and who we call "WebKit" vendors and the reality facing practical web developers today in his response to CSS WG co-chair Daniel Glazman's recent call to action.

Working without Photoshop

With the combination of tools like livereload and css compilers like less and stylus, designer/ developer hybrids can quickly and easily sketch out basic designs and move to more stylized and polished versions with the joy knowing that they won't have to go anywhere near photoshop.

Sadly, sometimes it is unavoidable to open the beastly program. Luckily, psd.js is making a great effort to parse PSD files with JavaScript. Now all we need is a way to convert those layer styles into css rules...

Security

Chrome to disable online SSL cert revocation checks:

While the benefits of online revocation checking are hard to find, the costs are clear: online revocation checks are slow and compromise privacy. The median time for a successful OCSP check is ~300ms and the mean is nearly a second. This delays page loading and discourages sites from using HTTPS. They are also a privacy concern because the CA learns the IP address of users and which sites they're visiting.

On this basis, we're currently planning on disabling online revocation checks in a future version of Chrome. (There is a class of higher-security certificate, called an EV certificate, where we haven't made a decision about what to do yet.)

Cryptographers have succeeded in cracking encryption schemes used by a number of satellite phones.

Node

procstreams is a cool way to do UNIX scripting in node.

Lessons from the Physical World

Jim Koch of Samuel Adams on the relationship between popularity and quality. He makes some interesting examples about the quality improving aspects of scale that we often miss while looking for cost savings.

Weekend Reader: Feb 3, 2012

2012-02-03T00:00:00.000Z

LA Ruby Conf

A few of us are heading out to the LA Ruby conf, come find us if you are headed to beautiful burbank this weekend.

How I Got Massively Faster Db With Async Batching

A good article on how to get around database bottlenecks. William wrote a custom layer that would take insert and update commands from his clients, batch them into large transactions, and flush them periodically. Apparently online games use this technique as well, and he went from being db-bound to saturating network and db disk i/o with a single evented server.

Firefox Finally Adds Developer Tools

Exciting and big news from the Firefox team!

New Identified Research Reveals Engineers Far More Likely than MBAs to Build and Run Companies

My suspicions now have a basis in some facts, and there seems to be a trend where MBAs now work for engineers instead of the other way around.

Internationalization coming to JavaScript

The draft is available for review and aims to support collation (string comparison), number formatting, date and time formatting, and lets applications choose the language and tailor the functionality to their needs.

If you don't want to wait there is always Alex Sexton's new jed library.

Installing web apps

Tim Berners-Lee vision for web apps as the true write-once, run-anywhere contender.

Login & Passwords Data

Luke W. drops some numbers about how ubiquitous log-ins are today and drive home the point that many of the challenges developers face with password systems are out of their control and hinge on the users themselves. Because the human brain struggles to successfully associate random text with the appropriate object, password recovery should play an important role anywhere a user needs to create an identity online.

Information Does Not Want To Be Free

Information (content) does not want to be free. Instead, information wants to be distributed friction-free. Via brooksreview.net

On Cryptography and Dogmas

A great article on from one of the authors of Redis about hashing passwords with SHA1.

Confessions of a Late Bloomer

In a startup like spire.io where only two of us studied computer science in undergrad (if we finished at all) and some actually started our journey into tech in our late twenties, its a good reminder that "blooming late" as its own distinct set of advantages and risks which, like everything else worth a dam in this world, can be magnified by working through the 10,000 hour rule as fast, hard and honestly as possible.

I leave you with swarms of quadrotors, have a great weekend!

Weekend Reader: Jan 27, 2012

2012-01-27T00:00:00.000Z

At the end of last week a few of us headed out to SCALE and just a few days later a few more of us headed out to NodeSummit. Don't worry though, there are still a ton of links.

Performance Analysis

At Both SCALE and NodeSummit engineers from Joyent gave presentations on how to handle analyzing what your apps are doing in production:

Devops Link Pile

JavaScript/ node.js Link Pile

Don't understand why it might be a good idea to call callbacks inside of process.nextTick, check out the post over on howtonode.org on Understanding process.nextTick()

@mrwooster posted an excellent tutorial on writing a twitter bot in node.js, we might have to adapt it for our IRC bot...

Didn't get a ticket to JSConf?

O'Reilly is putting on Fluent Conference focusing on "JavaScript and beyond". If you are worried that the quality wont be up to par make sure to submit a proposal! Call for participation ends 01/31/2012...

REST Made Simple, Part 2

2012-01-23T08:00:00.000Z

In our last installment of REST Made Simple, we introduced the idea that the Web is really a giant key-value store, or tuple-space, with the keys being URLs. And that get, put, and delete are the primary operations that allow us to manipulate this tuple-space, with post being a catch-all. Finally, we talked about how having operations with well-defined semantics makes it easier to implement features like caching.

The Wild, Wild West

So far, so good. We've got clients asking servers to put things into the tuple-space or to get things out of it. And now we run into the first difficulty because we have no way to be sure that one client is putting in the same thing another client is expecting to get out. Remember, this is the Web. It's a globally distributed tuple-space. We don't want to make too many assumptions about how it's going to be used. It's the Wild, Wild West out there.

For example, consider the introduction of a new video format. During the time it takes for everyone to switch to the new format, some user-agents will support the new format and some will support only the old one. If we publish a video, we'd like to be able to support both formats. But how can we do this if each key (URL) can have only one value?

Sharing Is Caring

One approach would be to simply have two separate key-value pairs, one for the old format and one for the new one. For example, we could simply add a format parameter to the URL. Or perhaps we add an extension to the URL corresponding to the version. But what happens if we want to share the URL? We'd have to share two URLs! And then hope people used the one that matched their user-agent software.

And it gets worse. We might want to support more than just new video formats - streaming, compression, subtitles in different languages, and so on. The number of possible keys that can potentially correspond to our video will grow exponentially. In turn, this makes it impractical to share URLs without providing some sort of guide to using them.

A Rose By Any Other Name ...

Perhaps more importantly, it's really the same video. There's something troubling about a key-value store that has more than one key for the same value. What would be nice would be if we could request the value for the key in a specific format (or language or character set, and so forth), using some sort of standard specification. In other words, have just one URL, but some way for a user-agent to tell the server what format, language, character-set, and so on, that it prefers.

Content Negotiation, For The Win

And, of course, this is exactly what HTTP does. HTTP headers are used to tell the server how the client wants to see the value. (Or, in the case of put, how it is able to provide the value.) These are standardized (to satisfy the constraint of providing uniform interfaces) and completely distinct from the URL, the key. For example, in a get, the video format the client wants is provided in the accept header. Now we can share keys between clients and still give each client what it wants.

Representations And Resources

This is the distinction in REST between representations and resources. The resource is the value associated with a given key (URL). But a client must request a specific representation for that resource. This allows us to seamlessly deal with change and complexity in the real world. We can handle everything from languages to character sets to new formats, all without blinking an eye. All thanks to this separation of the resource from its representation. This strategy is known as content-negotiation, since the user-agent "negotiates" the representation with the server.

All Together Now

Let's recap. We have a tuple-space. The keys are URLs and the values are resources. We can put things in, take them out, or remove them entirely. Clients, known as user-agents, can access a representation using the key, but also must provide information about which representation(s) they're using. We don't want to include information about the representation in the key because then we end up with a bunch of keys for the same thing, which makes it hard to share keys between clients that prefer different representations.

What's In It For You?

Simple enough? This arrangement - a tuple-space that supports multiple representations for a given value - turns out to work better than any of the many similar prior attempts to provide an RPC-based model for distributed computing. It's worth studying and, arguably, worth emulating. And since HTTP is so ubiquitous, emulating it is easy: just use HTTP the way it was designed.

Same Bat Time, Same Bat Channel

In our next installment, we'll take a look at the often-overlooked importance of using hypertext.

Weekend Reader: Jan 20, 2012

2012-01-20T00:00:00.000Z

Shred Now Works in the Browser

Even though we already talked about this earlier we are still pretty excited. Shred (our node.js http wrapper) is now available in the browser via the amazing Browserify project.

Git Powerup

@boblet walks through how to boost your git workflow using aliases and some fancy log and diff work.

Javascript Link Pile

If your'e just getting started with JavaScript metafilter has a great post about it with tons of helpful links.

More advanced JavaScript coders will find the JavaScript Pattern Collection useful / fascinating, since it's a github repo you can even make additions.

The brain JavaScript library has been around a while but I ran across it again last week, could be useful where you might need to sprinkle a little machine learning on a project. It works in the browser too!

Monumental Week for the SOPA Opposition

It's really exciting that all of our efforts online this week helped shelve both SOPA and PIPA acts indefinitely. What is really staggering are the numbers behind the effort. It was a good week.

Lastly, Make Sure to Take Some Time to Slow Down

William Deresiewicz gave this great lecture at the United States Military Academy at West Point in October 2009. Lots of really salient points. In an age (and industry) which is fast, we must not forget the value of slowing down, craftsmanship, and concentration. Some things should be fast, other things may be slower but the "clarity of thought" will offset the apparent "slowness."

I used to have students who bragged to me about how fast they wrote their papers. I would tell them that the great German novelist Thomas Mann said that a writer is someone for whom writing is more difficult than it is for other people. The best writers write much more slowly than everyone else, and the better they are, the slower they write. James Joyce wrote Ulysses, the greatest novel of the 20th century, at the rate of about a hundred words a day—half the length of the selection I read you earlier from Heart of Darkness—for seven years. T. S. Eliot, one of the greatest poets our country has ever produced, wrote about 150 pages of poetry over the course of his entire 25-year career. That’s half a page a month. So it is with any other form of thought. You do your best thinking by slowing down and concentrating.

Shred v0.7: Now with browser support!

2012-01-20T08:00:00.000Z

I'm very pleased to announce the latest release of Shred, a javascript HTTP client library that supports gzip, https, redirects, and proxies.

Shred was initially written for the Node.js platform, but as of version 0.7, Shred can run in any modern browser as well!

How we did it

We used the awesome Browserify tool to package Shred and all of its dependencies into one handly javascript file. The equally-awesome http-browserify provides a wrapper around XHRs that looks just like Node's http module.

Using these tools, we got Shred working in the browser almost immediately. We only ran in to two sticking points.

The first sticking point was Shred's requiring of the zlib library, which is used to gunzip responses with a "Content-Encoding: gzip" header. This module isn't available for browsers, but luckily we don't need it because the browser transparently handles content encodings for us. That means that all responses that get to Shred in the browser are already gunzipped for us! Problem solved.

The second sticking point was https. Node has two separate modules for handling http and https requests, however the http-browserify library only provides the "http" module. Luckily, the browser handles all of our SSL difficulties for us, so the http-browserify library is perfectly capable of sending requests to sites over HTTPS. Thanks to Substack for quickly reviewing and accepting my patch that makes this possible.

How you can benefit

Now that it can run in Node and in the browser, Shred is a great library to choose when writing code that needs to run client-side and server-side.

For example, we use Shred in our javascript spire.io client library, which also runs in node and in browsers. We previously had to use different http libraries in each environment, with slightly different interfaces. We ended up with a lot of conditional logic in our code that depended on the environment it was running in.

Switching to Shred significantly simplified our code and our dependencies.

How to get it

To get Shred for the browser, head to the Shred repo on Github and download browser/shred.bundle.js (or the minified version browser/shred.bundle.min.js). You only need to source that file in a script tag in your application.