rawjeev's techposts

May 24

Fogbeam Labs: Why The "Star Trek Computer" will be Open Source and Released Under Apache License v2

May 20

Darwin’s Finches, 20th Century Business and APIs

May 16

“It’s time to stop thinking of computer programming as a specialty subject. Schools should respect it as a fundamental skill.” — Why High Schools Should Treat Computer Programming Like Algebra - Jordan Weissmann - The Atlantic (via infoneer-pulse)

(via notational)

Design for the New Normal (Revisited) | superflux -

thisisliams:

This is an absolutely fascinating look at a possible future.

(via notational)

May 06

Objects and functions, conflict without a cause? -

journo-geekery:

Via:

(via notational)

May 04

node.js - A giant step backwards?

fennbcom:

A week into my first serious real-world node.js project, I was convinced it was the biggest step backwards since the Great Leap Forward.

6 weeks on, I’m definitely not so sure.

As a quick preface, the bulk of my recent work has been largely in python, with only limited production experience in async frameworks (ie: Tornado - I’d heard enough horror stories from Twisted I’d avoided it so far).

So right of the bat with node, I was very quickly put-off by how messy very simple things got almost straight away.

Some Examples

Consider the trivial case of checking for an entry in cache then fetching it from a datastore if required.

Synchronous version:

https://gist.github.com/1124578

Fairly standard stuff, now look at a naive async version:

https://gist.github.com/1124579

Woah, now I’ve ended up with two different code-paths where my result could end up and I can’t share common code. The only way to stick to DRY principles here is to create a new function like doOperationOnThing() and pass the result in. But you’d then have to do that for every case where you wish to check the cache.

Alternately, you could create a cachedDB.getFromCacheOrDB(…) type method and provide the functionality this way, but this is really only one of many very basic programming constructs (the “if” statement in this particular case) that don’t work well with async I/O.

To my horror, it wasn’t just if statements that didn’t work well when you introduce async I/O into them, but loops as well. Any kind of loops. What kind of joke programming revolution was this?

An illustrative example:

https://gist.github.com/1124580

Caveat Emptor

There is so much wrong with this code it’s hard to explain to an async beginner where to start. The most immediate point being that this theoretical function would likely return at worst an empty array and at best a (probably incomplete) result set… in a random order.

Everything I’d heard was that node.js was that it was supposed to make it easier to write concurrent, asynchronous apps. This didn’t seem easier to me.

So what’s going wrong in the above example? As a brief explanation: For those just starting out, it’s sometimes easier to think of async I/O (or any async) operations as queueing the I/O request, where the results will arrive at a random time “later”, at which point a callback is called (passing in the results).

So what’s happening above is we’re queueing a whole bunch of requests to get blog posts out of the DB (ignoring the fact that this is a fairly inefficient way to do things in the first place), which will all execute in parallel and return sometime “later”.

Because of this, once the requests are queued (relatively instantly), the program execution reaches the return statement and bails out of the function before any of the queries have completed, leaving the result empty.

Subsequently, at some point in the future (“later”), each DB request will then return a result will be put on to the array, but this could be milliseconds, seconds or minutes later (if your DB is slow) at which point it will probably be too late and your software has very possibly already returned the result array and discarded the variable (ie: from a web point of view, you’d likely render a blank page).

Now What?

So, if if statements don’t work and loops don’t work. Why am I bothering with this?

Well, it turns out that all this pain and suffering in primitive behaviours makes for a lot of ease in genuinely hard stuff like parallel programming. Ostensibly, all this hoop jumping allows node.js (or any async environment) to deal with parallelism by largely throwing it out the window.

This is by no means new, it’s a technique that’s been used extensively for eons, great/popular examples being Twisted, EventMachine, Tornado, Nginx and lighttpd.

Thankfully, the node community is very strong & active, and everyone is running up against these sorts of problems. There are a whole spate of flow-control helper libraries, that make working around these (initially baffling) caveats a lot more manageable.

Where’s Your Head At?

Once you get your head around thinking in async terms, node.js starts to actually make a lot of sense. Javascript as a language is an excellent fit for evented I/O -  it has proper anonymous functions and closures, is highly dynamic and comes from a browser environment, where any network operation could take some time, it makes perfect sense that the language has been designed to work this way.

Possibly most importantly - node.js was built from the ground up to be async. Unlike Twisted or EventMachine, where you have to constantly find async versions of all of your drivers (ie: MySQL, memcache, etc), node.js only has async versions of it’s various I/O libraries available. You may be able to shoot yourself in the foot with a for-loop but you can’t (easily) accidentally make a blocking call and block/destroy your entire server (all too easy to do in Python or Ruby, where things are always synchronous by default).

A nice canonical example (from http://www.catonmat.net/http-proxy-in-nodejs/) is the 20-line proxy server:

https://gist.github.com/1124535

Whilst incredibly simple, this proxy is quite capable of handling very high request rates and (more importantly) concurrency, without fooling around with threads, forking, shared data structure, etc. More importantly, you could easily introduce some sort of other IO into the mix, ie: stats collection into redis, without worrying too much about breaking your concurrency capabilities.

Conclusions

So what do I think about node.js now?

If I had to implement some sort of intelligent proxy system, unifying REST API or long-polling/comet system, I’d reach for node before anything else.

These guys agree, and, consequently, have a great fork of node.js on their github that includes Ubuntu packaging configurations, along with a whole lot of other cool node-related stuff.

Would I use node.js for my next major web app?

Perhaps not quite yet. But I might give you a different answer next week.

—

Anyone got any particular love/hate moments or use-cases for node?

Getting used to the ways of node is a shift in the basic thought process to someone used to calling something, and waiting for the result. I am new to nodejs and its async culture but, building an app made it clear (I am now out to explore the full MEAN Stack).

However I still don’t like the extremes of sync or async. I think certain things are better done synchronously and certain things the other way.

The MEAN Stack: MongoDB, ExpressJS, AngularJS and Node.js

This is a guest post from Valeri Karpov, a MongoDB Hacker and co-founder of the Ascot Project. 

A few weeks ago, a friend of mine asked me for help with PostgreSQL. As someone who’s been blissfully SQL-­free for a year, I was quite curious to find out why he wasn’t just using MongoDB instead. It turns out that he thinks MongoDB is too difficult to use for a quick weekend hack, and this couldn’t be farther from the truth. I just finished my second 24 hour hackathon using Mongo and NodeJS (the FinTech Hackathon co­sponsored by 10gen) and can confidently say that there is no reason to use anything else for your next hackathon or REST API hack.

First of all, there are huge advantages to using a uniform language throughout your stack. My team uses a set of tools that we affectionately call the MEAN stack:­ MongoDB, ExpressJS, AngularJS, and Node.js. By coding with Javascript throughout, we are able to realize performance gains in both the software itself and in the productivity of our developers. With MongoDB, we can store our documents in a JSON-­like format, write JSON queries on our ExpressJS and NodeJS based server, and seamlessly pass JSON documents to our AngularJS frontend. Debugging and database administration become a lot easier when the objects stored in your database are essentially identical to the objects your client Javascript sees. Even better, somebody working on the client side can easily understand the server side code and database queries; using the same syntax and objects the whole way through frees you from having to consider multiple sets of language best practices and reduces the barrier to entry for understanding your codebase. This is especially important in a hackathon setting: the team may not have much experience working together, and with such little time to integrate all the pieces of your project, anything that makes the development process easier is gold.

Another big reason to go with MongoDB is that you can use it in the same way you would a MySQL database (at least at a high level). My team likes to describe MongoDB as a “gateway drug” for NoSQL databases 
because it is so easy to make the transition from SQL to MongoDB. I wish someone had told me this when I first starting looking into NoSQL databases, because it would have saved me a lot of headaches. Like many people, I was under the impression that CouchDB would be easier to use, while the performance improvements from MongoDB were something I could take advantage only once I had gotten my feet wet with CouchDB. Instead CouchDB ended up being much more difficult to work with than I anticipated, largely because it uses custom Map­Reduce functions to query data, rather than the more traditional SQL queries I
 was used to. When I finally switched I was surprised to find that with MongoDB I could still write queries and build indices; the only difference is that the queries are written in JSON and query a flexible NoSQL database.

As a NoSQL database, MongoDB also allows us to define our schema entirely on the code side. With a No
SQL database you’re faced with the inescapable fact that the objects in your database are stored in a format that is unusable by your front­end and vice versa. This wastes precious time and mental energy when you inevitably run into a data issue or need to do some database administration. For example, if you change your ActiveRecord schema in Ruby on Rails, you have to run the “rake” command to make sure your SQL columns stay in sync with your schemas. This is a clear violation of the age­-old programming principle D.R.Y.­ (Don’t Repeat Yourself). In contrast, MongoDB doesn’t care what format the documents in your collections take (for the most part anyway). This means that you spend a lot less time worrying about schema migrations, because adding or removing data items from your schema doesn’t really require you to do anything on the database side.

At this point I should note that to get the most out of MongoDB in your MEAN stack, you’re going to want to take advantage of MongooseJS. Mongoose is a schema and general usability tool for Node that lets you use MongoDB while being as lazy as you want. For example, with Mongoose we can define a schema in JSON:

 var UserSchema = new Mongoose.Schema({
username : { type : String, validate: /\S+/, index : { unique : true } }, profile : {
name : {
first : { type : String, default : “” } last : { type : String, default : “” }
} }
});

We can then create a model by mapping our schema to our MongoDB collection:

 var User = db.model('users', UserSchema);

For all of the Ruby on Rails + ActiveRecord fans out there, note that this User object we’ve created above now allows us easy access to the basic features you would normally associate with ActiveRecord. For example, we can do thing like:

User.findOne({ username : 'vkarpov' }, function(error, user) { /* user is either undefined or a user with username vkarpov */ });


User.findOne({ _id : req.params.id }, function(error, user) { /* user with ID defined by the hex string in req.params.id */ });

User.find({ 'profile.name.first' : 'Valeri' }, function(error, users) { /* users is a list with users with first name Valeri */ });

var user = new User({ username : 'vkarpov' });
user.save(function(error, user) { /* Saves user with default values for profile.name.first and .last into 'users' collection */ });

var user2 = new User({ username : 'v karpov' });
user2.save(function(error, user) { /* Error – regular expression validation for username failed */ });

Another powerful tool that MongoDB and MongooseJS provide is the ability to nest schemas. You’ll notice that in the User schema above we have “profile” and “name” objects that are part of a nested schema. This is a simple and useful design choice that illustrates how powerful nested schemas can be. Suppose that we want to give our user the ability to edit their first and last name, but not their username. Assuming the website has a /profile route where our user can the change first and last names, the Javascript front­end will get a JSON object as the result of a call to User.findOne on the backend. After the user modifies their profile, the front­end makes a POST request to /profile.json with the user object in JSON as the body. Now on the backend (using ExpressJS syntax) we can simply use:

function(req, res) {
user.profile = req.body.profile; user.save.function(error, user) { res.redirect('/profile'); }); } }

That’s it. Mongoose takes care of validating of the profile information, so we don’t have to change the POST /user.json route if we change the User schema, and there is no way the username field can be modified. We could do the same thing when using Ruby on Rails and ActiveRecord, but this would require having a separate Profile schema in a separate table, meaning that among other things we’d incur a performance penalty because of the extra underlying join operation.

MongoDB is the bedrock of our MEAN stack, and you should strongly consider using it for your next project. You can write your entire stack in one language, have schemas for ease of use on top of a flexible and performant NoSQL database, and nest schemas when you don’t truly need to have separate collections. All
of this adds up to you spending more of your precious hackathon hours building the other cool aspects of your product and less time figuring out how your objects translate between different levels of the stack.

By the way, MongoDB and MEAN are useful well beyond hackathons­ we use this approach for all of our commercial products, most recently The Ascot Project. Want to read more about the MEAN stack or how we use MongoDB? Check out my development blog.

Check out the Ascot Project at the next MongoDB User Group in New York City. 

(Source: mongodb)

“At best data, visualisation is expert storytelling.” — Scott Murray (Interactive data Visualisation for the Web)

May 01

[video]

Apr 27

The MySQL Band Reforms As SkySQL -

The founders of the original MySQL, the open-source database, are getting back together in a merger between Monty Program and SkySQL.

Apr 19

Stevey's Blog Rants: Execution in the Kingdom of Nouns -

In Javaland, by King Java’s royal decree, Verbs are owned by Nouns… Verbs in Javaland are responsible for all the work, but as they are held in contempt by all, no Verb is ever permitted to wander about freely. If a Verb is to be seen in public at all, it must be escorted at all times by a Noun. Of course “escort”, being a Verb itself, is hardly allowed to run around naked; one must procure a VerbEscorter to facilitate the escorting…

Apr 13

New directions in web architecture Again -

A focus on data is finishing the revolution that started with Ajax.

Apr 12

Introduction to Node and its Core Modules by aryoung -

timreynolds:

Just had to post this it’s the quickest intro to node.JS you will ever read

(Source: timreynolds, via notational)

Apr 05

how not to write technical literature

From MSDN, the definition of _bstr_t Class is:

A _bstr_t object encapsulates the BSTR data type. The class manages resource allocation and deallocation through function calls to SysAllocString and SysFreeString and other BSTR APIs when appropriate. The _bstr_t class uses reference counting to avoid excessive overhead.

—————————

Excellent work, MSDN camp. 

This is what happens when a technical writer clearly has no clue about what he is dealing with, but a deadline’s approaching fast. Sort of reminds me of third graders reading their book report about a book they haven’t read. 

It’s like asking Microsoft what a monkey is. They’ll go:

A monkey eats bananas. A monkey does not eat mud. A monkey can have monkey babies. A monkey will free up its resources automatically via system function calls. 

Thanks. That’s what a monkey does. But what the hell is it?

Luckily, a Google search on “what the fuck is a _bstr_t” still works. 

PS: It’s a C++ class wrapper to wrap VB style strings. Works like a smart pointer to (binary) strings.

(Source: joethought)

Apr 02

Stowe Boyd: Prospero's Precepts -

stoweboyd:

A collection of thoughts on belief, theory, and doubt.

Peter Sturrock via Brainpickings

1. All beliefs in whatever realm are theories at some level. (Stephen Schneider)
2. Do not condemn the judgment of another because it differs from your own. You may both be wrong. (Dandemis)
3. Read not to…