Larrikinism

GSOC 2011: ssc Week 10

Tasks completed this week:

1. Implemented checkout of an existing appliance from Studio. ssc checkout --appliance-id=APPLIANCE_ID now works.
2. Integration tests for the appliance, package and repository handlers. (Currently at about 50% test coverage).
3. Completed the refactoring and migration to the New DirectoryManager module.
4. Added the build and build_status commands

Schedule for next week:

• Improving test coverage and removing all the deprecated and unused methods.
• Writing unit tests for the helper modules.
• Writing usage examples for the app to help new users.

GSOC 2011: ssc Week 9

Tasks completed this week:

1. Replacing of ArgumentParser with Thor. I’d given a mid-week update about this [1] on the mailing list. Please check there for complete details.

2. Implementing the general commands (checkout, commit and status). As a direct consequence of using thor for the command line parsing, we get to use the invoke method to call other actions. This made writing these commands quite easy.

3. Started on refactoring the DirectoryManager module. The NewDirectoryManager is currently in use in the general commands and will gradually replace the older version which is a mess of helper functions right now. There will be separate classes for handling each of the local storage files (for software, repositories and files) deriving from a parent LocalStorageFile class which implements the generic #read and #save methods.

4. The gem is now hosted on rubygems.org . So, you can install it with gem install ssc. Use the fantastic thor generated usage instructions to help you use the app.

The tasks for next week are:

• Write integration tests
• Implement a command to get information about installed packages from the currently run system.
• Commands for Appliance building and build status reporting.

If you have any suggestions for features. Please do write in on the studio-users mailing list.

[1] http://lists.opensuse.org/opensuse-project/2011-07/msg00098.html

GSOC 2011: ssc Week 6

Most of the work this week has gone into the File Handler. Adding and removing files in appliances has been implemented although there is a lot of work left to be done to make this robust. Right now its pretty basic. However I’m going to ease up on adding functionality right now to make the app more robust and usable. The mid-term evaluations are coming up and the plan is to have a usable app that is community tested by then. Not to say that new features won’t be added over the next two weeks, but it will be a second priority. In making the app more stable, the following things will be addressed:

1. Ruby 1.8.7 (backward) compatibility
2. Extended test-suite including integration tests which require a network connection
3. Better error handling.
4. A method to roll back changes in case of failure.
5. Documentation.

That will be the focus over the next two weeks building up to the mid-term evaluation. Thank you for reading. As always you can check out the core here

GSOC 2011: ssc - Week 4

The DirectoryManager module has been polished and it now handles duplicate entries in the software and repositories files. The app now handles errors better and crashes more gracefully with meaningful error messages. As for functionality, I’ve added package banning and unbanning. Also package importing has been included in the current release.

I’ve started work on the file handling of the app. Currently it works similar to the packages or repositories with the difference that the file in question is copied over to <appliance_directory>/files. Adding and removing these files will be simple enough within the current framework. And with some work time-stamp information can be incorporated to do more efficient syncs. I hope to have that done by next week and follow up with a basic implementation of the commit command in the week after that so that we can have a first usable version of the client.

If you want to check out the application now, you can take it for a spin using the gem. Use with caution, and all that. It shouldn’t do anything catastrophic, but, I wouldn’t use it with anything other than a test appliance at this point. Download it from here. Install it with rubygems (gem install ssc-0.1.0.gem) and run ssc help to see the usage instructions.

GSOC 2011: ssc - Week 3

This week I was working on the local storage feature of the client. As proposed we were going to add and remove packages, repositories and files with a git-like workflow. i.e. ssc package add gnuchess would make a local record of the intention to add that package and on ssc commit the changes would actually be pushed.

In order to facilitate this I’m abstracting all the directory management to a DirectoryManager module. Which abstracts methods to save, read and diff from local sources. Right now the syntax I’ve chosen is YAML. For instance the software file in the appliance directory - which catalogues the changes made to the appliance’s software - stores names of installed packages. Unless the ssc command is invoked with the -r | --remote option, it will display the packages in that file. In addition when ssc package add gnuchess is run with out the -r option, it will make and addition add gnuchess to the software file. Similarly for remove. I am having some problems figuring out how to integrate patterns into this architecture so any help on that front would be appreciated.

That’s it for this week. I’m going to be working on pretty much the same thing next week. Extending and refactoring this functionality. So hopefully we will have a more usable version then.

Thank you for reading.

GSOC 2011: ssc - Week 2

In terms of user functionality, searching for software, templates, appliances and packages has been implemented this week. The actual api calls are performed by handler classes like SSC::Handler::Appliance which exposes public methods like list, show, repositories and installed_software. I’m trying to follow a template that allows any command of the type ssc <class> <action> <arguments> to map directly to Class#action(arguments) with minimum modification along the way. The command line arguments (—option value) are parsed into an options hash that keeps getting passed down the stack and used as necessary. Right now I’m not happy about exposing data that is unnecessary for certain functions via this hash but it makes sense to have the checking of data in the handler action rather than at the top level which makes that inevitable.

There’s been some abstraction and refactoring to make way for having the .sscrc file what will hold the appliance configuration in the appliance directory. Right now there isn’t a lot of validation of options and the parser merely merges options from .sscrc and the command line to pass into the various handlers.

One tiny useful takeaway from this week has been learning about the Method class. Its very useful in meta-programming. For instance when calling methods with the Class#send method it would be useful to make sure that you’re passing the correct number of arguments. The Method#arity method can be used to check the number of arguments that a method takes before passing arguments to it.

Next week I’m going to be focusing on the structure of the appliance directory and handling those commands that work locally.

GSOC 2011: ssc - Week 1

Introduction:

My proposal for a command-line client for Suse Studio has been accepted for this year’s Google Summer of Code. You can see the full proposal here. In short, the project is pretty self-evident from the title. In case you don’t know what Suse Studio is, its a web service that allows you to design custom ISOs of linux distributions. As you can imagine, designing a custom variant of a linux distro will involve a lot of configuration. This tool aims to ease the hassles involved with using the web interface to make these customisations. The most common use case for this tool as I see it will be modifying default configuration files. The tool will allow you to make all the modifications you want locally in an appliance directory and push the changes when you’re ready. If you want a more thorough view of how it will work please do read the complete proposal.

Project Details:
Now that the introduction is out of the way, some updates about my implementation of this proposal. The coding period started a couple of days back. The code is going to be hosted on gitorious, here. I’ll be pushing there almost daily and when we have a relatively stable version we’ll push to the upstream repo. The rough timeline that I hope to follow is as outlined in the proposal:

• May 24th - Jun 5th: Create the framework for the command line client. Get the option parsing working and implement some of the basic search functionality.
• Jun 6th - Jun 19th: Implement the appliance directory management features.
• Jun 20th - 26th: Release a first alpha version for testing and reviews
• Jun 27th - Jul 11th: Tidy up, refactor code. Raise test coverage. Fix Bugs.
• Jul 12th - Jul 17th: Mid-Term Review
• Jul 17th - Jul 31st: Implement the commit, update and status commands.
• Aug 1st - Aug 15th: More testing, bug fixing and community review.

High level documentation will be maintained on the openSuse wiki. Once there is substantial code, the class/method level documentation will be hosted at rdoc.info.

Work Update:

ssc uses the studio_api gem to make API calls to Studio. While trying out the gem in preparation for this project, I made a small but important(at least I think so :)) bug fix contribution to the project. It probably required someone with a crappy internet connection like mine to discover this.

So far the code written is mostly framework stuff. There’s an ArgumentParser that does the command line parsing. I’m using mocha and shoulda for testing. I decided to bump the oh-so-cool rspec/cucumber frameworks for simple test unit. I’ve used rspec in most of my projects and other than syntactical niceness I don’t see what test unit lacks. I’d like to try out that theory here.

That’s it for now I think. Not a lot, I know but hopefully, I will have more for you next time. Thanks for reading.

Bye Bye NerdTree. Hello CommandT

CommandT is an awesome vim plugin made by Wincent Colaiuta of http://wincent.com . Its been around for a while but I wasn’t able to use it earlier since it requires vim complied with ruby support which the standard Ubuntu vim package lacks. I tried compiling it from source but after a couple of hours of struggling to meet dependencies, I gave it up. Recently, I heard that v1.0 of the plugin had been released and figured the time was ripe for another attempt. Turns out what I had to do was crazy simple. On Ubuntu 10.04 just

sudo apt-get install vim-gtk

For some reason vim-ruby which is the only package you need is a virtual package included in vim-gtk, vim-nox or vim-gnome. So just install one of them, grab the vba, open it up in vim and so :so %.

This is way better than NerdTree and FuzzyFinder for file navigation for obvious reasons: It doesn’t take up screen real estate, files are fewer key taps away and its blazingly fast. Enjoy using it.

node.js+ mogodb+ websockets

Now there’s an amazingly buzz-word laden title, if I ever saw one. Anyway, if you, like me, have been hearing more and more about of these terms floating around the interwebs in additions to things like ‘coffee script’, ‘nosql’, ‘event driven programming’, ‘document-oriented databases’ etc. etc. read on for a gentle introduction to new paradigms of programming, new server architectures and new persistence layers.

I’ve always found that the best way to get started with a new technology is to start writing code using it. Of course this has to be followed by very in depth reading about the topic, but it helps if you’ve actually done something to follow along with what people are writing. So in my quest to understand node.js, mongodb and HTML5 websockets, I chose to implement a very basic EtherPad( used in typewith.me ) clone.

The components:

node.js

From what I’d heard about it, its a blazingly fast web server written in JavaScript running on Google’s V8 engine. I still haven’t gotten to a stage where I can benchmark the code myself. But the benchmarks provided here looked really exciting. Installing node was a cinch on my ubuntu system. The usual:

          $ wget http://nodejs.org/dist/node-v0.1.99.tar.gz
          $ tar xfz node-v0.1.99.tar.gz
          $ ./configure
          $ make
          $ sudo make install

Brownie points to the node team for making the make tool give pretty output.

The exciting, game-changing thing about node.js is of course Evented I/O. The presentation below is the best explanation of the concept, that I’ve come across so far.

Evented I/O based web servers, explained using bunnies View more presentations from Simon Willison.

Coffee Script

Since I was going to be writing JavaScript, I thought I’d kill two birds with one stone and look into Coffee Script which helps you avoid JS’s ugly ugly syntax. I would definitely recommend investing the extra time learning this useful tool if you’re going to be developing for node.js or any JS library for that matter. Also as a bonus you can run node.js apps directly using coffee app.js.

express

In three words this is: Sinatra for JS. Being a big fan of the simplicity and transparency that sinatra offers, I pounced on this particular framework for my app. The framework for node.js scene had erupted recently with lots of exciting developments, including: A standard base for frameworks to be built on (similar to ruby’s rack): connect. Being developed by the fabulous guys behind ExtJS (now Sencha). Connect is still however in its infancy and does not have much support.

          get('/' ->
            'Node says, Hello world!'  
          )

kiwi

Pakage management for node.js libraries based on rubygems.

          $ kiwi install express
          $ kiwi list
          $ kiwi remove express
          etc. etc.

I must add here that node.js is still at version 0.1.99 which is infancy compared to most software projects. And is by far not the only server to incorporate Event-driven I/O and to be written in JS(see the end of this article). It has however made sure that those idioms extremely popular and has fostered development along those lines in the ruby community and beyond as well. Think: EventMachine and all the em-* libraries.

mongodb

So I needed a persistence layer for the app and wanting to use a NoSQL DB. I went for mongo coz’ … well … how do you not look into something named mongo. Anyway mongo is a great place to start your jouney to the Dark Side of The Moon, the moon being the database implementations landscape.

To get started playing with mongo, sudo apt-get install mongodb. This gives you a rather limiting mongo REPL where you can try stuff out. When migrating from an RDBMS to mongo there are some changes in vocabulary that you will have to embrace. Tables are collections and rows/records are documents(hence document oriented db). I won’t go into a tutorial about mongodb as there are quite a few of those out there. But, I’ll try and cover some of the benefits.

1. Dynamic Queries like:

   db.users.find({address: /[D|d]elhi/, age: $lt 25}).sort({name: 1})

That’s right, I just queried a db with a regex. And chained queries! Stuff, I could only do with an ORM that would ineffeciently translate that into SQL.

1. Neat JSON-like language for queries called BSON

2. Embed has-many associations.

          db.users.insert({
            name: 'Ratan',
            notes: [
              {content: 'The Daily Show is awesome ...'},
              {content: 'Jon Stewart is awesome ...'}
            ]
          })

No unnecessary tables for holding strictly hierarchical data.

I’m sure there are many more but there are the ones that I encountered within 2 days of using it. Another thing that I should add here is the scalability promise of NoSQL DBs. Many users(me included) are drawn to NoSQL by its horizontal scalability. This article seems to debunk that, so i would advise further research before relying on that promise.

I’m not in the best position to debate the pros and cons of document-oriented DBs vs. the traditional RDBMS. But I do think I can propose a rule of thumb to decide between the two if your data fits well into either of the models:

 use NoSQL if (no. of objects central to your app)/(no. of small, supporting and dependent objects) >> 1  

websockets

One of the most awaited aspects of HTML5 this revolutionary protocol will make all the hackery featuring flashsockets and jssockets to create real-time web apps obsolete. Support is limited to only chrome and safari currently but that is because the spec for the protocol is not finalised yet. Firefox, waiting for the spec to be finalised, will support it in Firefox 4 and IE … well… First impressions of websockts are that its is extremely easy to use and will be one of the prime uses for node.js apps. Writing your server and client in JS gives the feeling of just on complete library connected by passing JSON messages to each other.

I used the extremely simple ws.js on the server side and jquery.ws.js on the client side. If you want a bit more power and flexibility, check out Socket.IO which comes with its own client library.

Hope this cleared the air about a couple of these web dev buzz words and encourages you to look further into it. While writing this article a tweet from DHH brought this to my attention. Clearly there is a lot out there beyond the technologies out lined here and there are just current-popular implementations of philosophies and ideas that have been around for a while.

rxvl: The making of

I’ve been meaning for some time to write my own blogging engine and now that I finally got the time I went for it. It was meant to be a learning excercise so I tried to use tools and libraries that I’d never used before. I ended up chosing Sinatra for the framework to write it in with an SQLite DB and using DataMapper as the ORM. Also, I intended to host it on Heroku. In this article I’ve described everything from setting up my development environment for sinatra to finally deploying it on Heroku. Enjoy!

Setting up

Our first goal is to get a basic, “Hello World!” Sinatra app running with passenger and apache. You most probably have apache and passenger setup, so feel free to skip the next bit. For those who don’t, you should. Setting up passenger is the easiest thing. I’ve written about it here . Set up a VirtualHost to point to a directory named public in the directory where we’re going to have our sinatra app (/path/to/app/public). Create this empty directory. No special configuration is required in the apcache config file just treat it like a normal rails app. Now in /path/to/app lets create the sinatra app in the file app.rb. For a basic Hello World! application it’ll include something like this:

          get '/' do
            "Hello World!"
          end

Next we need the rackup file, Sinatra being a rack framework. This is my config.ru:

          require 'sinatra'
          require 'app.rb'
          root_dir= File.dirname(__FILE__)
          ENV['RACK_ENV'] ||= 'development'
          set :environment, ENV['RACK_ENV'].to_sym
          set :root, root_dir
          disable :run
          run Sinatra::Application

And that’s it. Point your browser to the ServerName that you’d specified in that VirtualHost config and you should see Hello World!

DataMapper

Now lets start writing the actual app. We start as with a normal rails app with the models. As you can imagine, we’d need a post model. You would also usually use a comments model but I want to use Disqus for my commenting system so I’ll demonstrate inter-model relationships with a category model. Post and Category have a has and belongs to many relationship. I’ll show you how to do that with DataMapper. So first the migrations. Create db directory and create the following migrations.rb file:

          DataMapper.setup(:default, ENV['DATABASE_URL'] || "sqlite3:///home/sapna/src/testing/db/datastore.sqlite3")
          class Post
           include DataMapper::Resource
           property :id, Serial
           property :title, String, :required => true
           property :slug, String
           property :file_name, String, :required => true
           property :created_at, DateTime
           has n, :categorizations
           has n, :categories, :through => :categorizations
          end
          class Category
           include DataMapper::Resource
           property :id, Serial
           property :name, String
           has n, :categorizations
           has n, :posts, :through => :categorizations
          end
          class Categorization
           include DataMapper::Resource
           property :id, Serial
          
           belongs_to :category
           belongs_to :post
          end

The explanation:

• The first line defines the link to the database. I’ve used an sqlite database here. Setting up a connection to a mysql or postgres DB should be no different. the ENV[‘DATABASE_URL’] bit is for heroku. My local database is the datastore.sqlite3 file. Once this is delpoyed it’ll get the information of the DB URL from heroku. DataMapper::Resource is a module that you will have to include in all your model classes. Since the migration is the first time I’m defining the model class Post I’m including it there.

has n :categories

and the Category class would have

belongs_to :post

Extending the principle, you can see how what is shown above would work. This gives you Category.new.posts and Post.new.categories.