Perl on Dancer

Tuesday, April 22, 2014

Executing remote commands from a Dancer application

I developed an application to execute commands remotely from a Dancer application. It is not secure if you don't do it correctly but sometimes you may need it, if so, make sure the command is not enter as a parameter from the user and nobody can read your username or password to access to the server.

This is an example of how to execute a remote command using Dancer, Net::SSH2 and fork. It is NOT secure at all, it is just to have an idea and to get some lines of code to implement a function to execute a remote command.

How to send the public key to the remote servers?

To allow the remote server accept the connection, it is required this server knows and authorize the key. This is done by send the public key to the remote host execute this command:

cat ~/.ssh/id_rsa.pub | ssh $user@$server 'cat >> ~/.ssh/authorized_keys'

Or this one:

ssh-copy-id -i ~/.ssh/id_rsa.pub $user@$server

There are some comments about this code:

The password variable may be optional.
Due to access restriction to the private key I copy it to a directory where the application can read it. Make sure it is only accessible from the dancer application.
May be good idea to have a function to parse the command and make sure it does not have malicious commands such as 'rm'.
I tried this code with threads but there are modules not thread-safe so I recommend to use fork instead.

	#!/usr/bin/env perl

	use strict;
	use warnings;
	use Dancer;
	use Net::SSH2;

	sub execCommand ($$) {
	my ( $ssh2, $cmd ) = @_;

	my %args=(
	timeout => 1_000, # polling timeout
	bufsize => 10_240, # read buffer size when polling
	);

	$ssh2->blocking(1); #needed for ssh->channel
	my $chan=$ssh2->channel(); # create SSH2 channel
	if ($ssh2->error()) {
	return (undef, undef, 100);
	}

	# exec $cmd (caveat: only one command line can be executed over this channel. No "ls -l;whoami" combo. Use ssh->shell instead.
	unless ($chan->exec($cmd)) {
	return (undef, undef, 500);
	}

	# defin polling context: will poll stdout (in) and stderr (ext)
	my @poll = ( { handle => $chan, events => ['in','ext'] } );

	my %std=(); # hash of strings. store stdout/stderr results
	$ssh2->blocking( 0 ); # needed for channel->poll
	while(!$chan->eof) { # there still something to read from channel
	$ssh2->poll( $args{'timeout'}, [ @poll ] ); # if any event, it will be store into $poll;

	my( $n, $buf ); # number of bytes read (n) into buffer (buf)
	foreach my $poll ( @poll ) { # for each event
	foreach my $ev ( qw( in ext ) ) { #for each stdout/stderr
	next unless $poll->{revents}{$ev};

	#there are something to read here, into $std{$ev} hash
	if( $n = $chan->read( $buf, $args{'bufsize'}, $ev eq 'ext' ) ) { #got n byte into buf for stdout ($ev='in') or stderr ($ev='ext')
	$std{$ev}.=$buf;
	}
	} #done foreach
	}
	}
	$chan->wait_closed(); #not really needed but cleaner

	my $exit_code=$chan->exit_status();
	$chan->close(); #not really needed but cleaner

	$ssh2->blocking(1); # set it back for sanity (future calls)

	return ($std{'in'},$std{'ext'},$exit_code);
	}

	sub execute ($$$$) {
	my ($ip, $username, $password, $cmd) = @_;
	my $pid = fork();

	if ($pid) {
	# This is the parent (DANCER)
	debug "Process started with PID $pid\n";
	} elsif ( $pid == 0 ) {
	# This is the child
	my $ssh2 = Net::SSH2->new();
	$ssh2->connect( $ip ) or debug("Cannot connect to $ip");
	my $publicKeyFile = './id_rsa.pub'; # path(setting('appdir'), 'db', 'id_rsa.pub'); # I prefer to copy the public key in your app dir due to permissions issues
	my $privateKeyFile = './id_rsa'; # path(setting('appdir'), 'db', 'id_rsa'); # I prefer to copy the private key in your app dir due to permissions issues
	if ( $ssh2->auth_publickey( $username, $publicKeyFile, $privateKeyFile, $password ) ) {
	my ($stdout, $stderr, $exitcode) = execCommand($ssh2, $cmd);
	} else {
	debug "Could not authenticate to $ip with $username";
	}
	$ssh2->disconnect();
	} else {
	debug "Could not fork: $!\n";
	}
	}

	set logger => "console";
	set log => "core";
	set show_errors => 1;

	get '/uptime/:ip' => sub {

	my $username = "the username";
	my $password = "the password";

	execute(param('ip'), $username, $password, "uptime > /tmp/dancer_example.txt");

	return 'uptime is running';
	};

	dance;

	true;

view raw dancer_remote_ssh.pl hosted with ❤ by GitHub

Thursday, December 26, 2013

Nitrous.IO: Dancing on the clouds

In a previous post about Preparing the dance floor I explain how to setup Dancer in several OS such as CygWin on Windows, Linux (Ubuntu and CentOS) and Mac OS X (OS X Mavericks) but recently I found that is possible to install and use Dancer on Nitrous.io.

Why Nitrous.io?

Why not? Nitrous.io is a cloud development environment this means you can code for free using a Web Browser in your computer or tablet; or even using a SSH client from anywhere. You also have access to a Web IDE and collaborative coding.

Nitrous.io allows to code in several programming language such as Ruby on Rails, Python with Django, Node.js and Go but it does not explain how to code with Perl on Dancer.

Installing Dancer on Nitrous.io.

The installation process is similar to what was explained in the post Preparing the dance floor but there is a few minor changes.

Cpanminus cannot be installed in a share directory you so cannot use the sudo option and the installation will be done in ~/perl5. After it is installed, the directory ~/perl5/bin need to be included in the PATH variable.

	curl -L http://cpanmin.us \| perl - App::cpanminus
	echo 'export PATH=$PATH:$HOME/perl5/bin' >> ~/.bashrc
	. ~/.bashrc

	cpanm App::cpanoutdated
	cpanm App::cpanlistchanges

view raw install_cpanm_on_nitrous.io.sh hosted with ❤ by GitHub

Now you are ready to update the Perl modules but this will increase the used storage. If you have some nitro (N2O) available it is recommended to get more storage.

	# (Optional) Print a list of the outdated modules:
	cpan-outdated -p
	# (Optional) See the changes for the outdated modules:
	cpan-listchanges `cpan-outdated -p`
	# Update the outdated modules:
	cpan-outdated -p \| cpanm

view raw Update_Perl_modules.sh hosted with ❤ by GitHub

Now you are ready to install dancer.

cpanm Dancer

view raw Install_Dancer.sh hosted with ❤ by GitHub

Create the application

To create the application and start it just do the same you do in other environment.

cd workspace
dancer -a myapplication
cd myapplication
mv bin/app.pl bin/server.pl
./bin/server.pl

To view the application, in the upper left side, select Preview > Port 3000. In the new browser tab there is your new application.

To have the application on Github follow the instructions from Nitrous.io help in http://help.nitrous.io/github-add-key/.

Nitrous.io is just another good option to develop your dancer application. There are pros and cons about this, but this is a great opportunity to dance in the clouds.

Sunday, December 22, 2013

MVC: Dancing with a Model

Dancer is a RESTful framework like Sinatra but I want to use Dancer in a MVC application model because allow us to have more efficient code reusing components and separate the logic, the data management and the way the information is displayed.

What is MVC?

There are a lot of tutorials everywhere explaining what MVC is. As a quick explanation, MVC stand for Model View Controller. Model is the data, View is an interface to view the data that is changed or processed in the Controller. To know more about MVC you may read these:

The Model

This is all about this post will be: The Model. As mentioned above, the model does nothing else but data management. It does not depend of the controller or the view.

There are a lot of Perl modules to handle the data. I use Dancer::Plugin::Database which use DBI::db module, however there are others modules such as Rose::DB and Dancer::Plugin::DBIC which use DBIx::Class. I'll provide an example with Dancer::Plugin::Database and Dancer::Plugin::DBIC.

Dancer::Plugin::Database

I use Dancer::Plugin::Database for projects that do not require a complex database design. It handle the connection to the database and it support many database managers such as SQLite, MySQL, PostgresSQL, Oracle, etc... For development I use SQLite and it is the one I'll use in this post but you may consider MySQL or PostgresSQL.

First you have to do is to install the Dancer::Plugin::Database module executing this line and update the Modfile with same line. Also, make sure you have installed the module for the database manager you will use, in this case is SQLite

	cpanm Dancer::Plugin::Database
	cpanm DBD::SQLite

view raw Install_Dancer-Plugin-Database.sh hosted with ❤ by GitHub

Configure the connection to the DB. Open environments/development.yml to add these lines to the end:

	plugins:
	Database:
	driver: SQLite
	database: 'db/development.sqlite'

view raw Dancer-Plugin-Database_SQLite_Config.yml hosted with ❤ by GitHub

Remember to use space instead of tab in any YAML (.yml) file, Dancer will complain if you use tabs.

Create the directory db, where all the databases will be (just like Rails). Create the db/migrations directory to have the following SQL files to create the tables.

db/migration/001_up_teams.sql will be used to create the teams:

	BEGIN;

	CREATE DATABASE IF NOT EXISTS my_application;

	USE my_application;

	/*
	TABLE: Teams
	DESC: It is a real team such as UNIX Admins, DBAs or Human Resources.
	*/

	DROP TABLE IF EXISTS Teams;
	CREATE Teams (
	id INTEGER PRIMARY KEY NOT NULL,
	name TEXT NOT NULL,
	email TEXT,
	create_on TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP,
	update_on TIMESTAMP NULL
	);

	COMMIT;

view raw 001_up_create_DB_and_Teams.sql hosted with ❤ by GitHub

.. and db/migration/001_down_teams.sql to eliminate them:

	BEGIN;

	USE my_application;

	DROP TABLE IF EXISTS Teams;
	DROP DATABASE IF EXISTS my_application;

	COMMIT;

view raw 001_down_drop_Teams_and_DB.sql hosted with ❤ by GitHub

Same for the Users, use db/migration/002_up_users.sql to create the users:

	PRAGMA foreign_keys=ON;

	BEGIN;

	/*
	TABLE: Users
	DESC: Users to access the application.
	*/

	DROP TABLE IF EXISTS Users;
	CREATE TABLE Users (
	id INTEGER PRIMARY KEY NOT NULL AUTO_INCREMENT,
	username TEXT NOT NULL,
	password TEXT NOT NULL,
	contact_id INTEGER,
	role_id INTEGER,
	create_on TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP,
	update_on TIMESTAMP NULL,

	FOREIGN KEY(contact_id) REFERENCES Contacts(id) ON UPDATE CASCADE,
	FOREIGN KEY(role_id) REFERENCES Roles(id) ON UPDATE CASCADE
	);

	COMMIT;

view raw 002_up_create_Users.sql hosted with ❤ by GitHub

.. and db/migration/002_down_users.sql to eliminate them:

	BEGIN;

	USE my_application;

	DROP TABLE IF EXISTS Users;

	COMMIT;

view raw 002_down_drop_Users.sql hosted with ❤ by GitHub

Now, lets apply those migrations:

sqlite3 db/development.sqlite < db/migrations/001_up_teams.sql 
sqlite3 db/development.sqlite < db/migrations/002_up_users.sql

If something goes wrong you can always rollback with:

sqlite3 db/development.sqlite < db/migrations/002_down_users.sql 
sqlite3 db/development.sqlite < db/migrations/001_down_teams.sql

IMPORTANT: For every statement you do in the UP sql file, you have to undo it in the DOWN sql file and in the opposite order.

In order to test your application you need to fill the database with some data. We can do that with a seeder sql file. Create the db/development.seed.sql file with this:

	-- Adding Teams
	INSERT INTO Teams (name, email) VALUES ('App Admin', 'app_admin@company.com');
	INSERT INTO Teams (name, email) VALUES ('Finance', 'finance@company.com' );

	-- Adding Users
	INSERT INTO Users (first_name, last_name, email, team_id) VALUES ('Johandry', 'Amador', 'johandry@company.com', 1);
	INSERT INTO Users (first_name, last_name, email, team_id) VALUES ('John', 'Smith', 'johnsmith@company.com', 2);
	INSERT INTO Users (first_name, last_name, email, team_id) VALUES ('Juan', 'Smith', 'juansmith@company.com', 2);

view raw development.seed.sql hosted with ❤ by GitHub

Confirm the database content with:

$ sqlite3 development.sqlite "SELECT * FROM Users;"
1|Johandry|Amador|johandry@company.com|1|2013-12-22 07:10:30
2|John|Smith|johnsmith@company.com|2|2013-12-22 07:10:30
3|Juan|Smith|juansmith@company.com|2|2013-12-22 07:10:30
$ sqlite3 development.sqlite "SELECT * FROM Teams;"
1|App Admin|app_admin@company.com|2013-12-22 07:09:47
2|Finance|finance@company.com|2013-12-22 07:09:47

In case you want to confirm the same from the application, create this temporal route in the lib/myapplication.pm file:

	use Dancer::Plugin::Database;

	get '/testdb' => sub {
	my @teams = database->quick_select('Teams', {});
	my @users = database->quick_select('Users', {});

	template 'testdb' => { teams => \@teams, users => \@users }
	};

view raw test_rule.pl hosted with ❤ by GitHub

And the template views/testdb.tt with this:

	<div id="page">
	<table class="table">
	<thead>
	<tr>
	<th>#</th>
	<th>Name</th>
	<th>Email</th>
	</tr>
	</thead>
	<tbody>
	<% FOREACH team IN teams %>
	<tr>
	<td><% team.id %></td>
	<td><% team.name %></td>
	<td><% team.email %></td>
	</tr>
	<% END %>
	</tbody>
	</table>
	<table class="table">
	<thead>
	<tr>
	<th>#</th>
	<th>Name</th>
	<th>Email</th>
	<th>Team</th>
	</tr>
	</thead>
	<tbody>
	<% FOREACH user IN users %>
	<tr>
	<td><% user.id %></td>
	<td><% user.first_name %> <% user.last_name %></td>
	<td><% user.email %></td>
	<td><% teams.${user.team_id}.name %> (<% teams.${user.team_id}.email %>)</td>
	</tr>
	<% END %>
	</tbody>
	</table>
	</div>

view raw test_rule_template.html hosted with ❤ by GitHub

IMPORTANT: You need to switch from Template::Simple to Template::Toolkit. I'll not explain this change here but basically you need to uncomment the template_toolkit lines and comment the template simple line in config.yml.

In a next post I'll write about Dancer::Plugin::DBIC but now you have at least one model to dance with.

Friday, December 13, 2013

Preparing the dance floor

I think the Dancer tutorial is very good explaining how to install Dancer and start your first application. However there are a few things that are not there - and should not be there either - but may be important to have in every Dancer project. Let's start up an application from scratch.

I have tested this in Linux (CentOS and Ubuntu), Mac OS (OS X Mavericks) and Cygwin on Windows but I have not tested it in Windows directly. If you have, please, share your comments about it.

Install cpanminus and update your Perl modules.

cpanminus (cpanm) is much user friendly than cpan, I really recommend to install it and also other cpanm helpers such as cpan-outdated and cpan-listchanges.

In Mac OS X you have to add to the PATH variable the directory where cpanm was installed. In my Mac OS X 10.9.1 was installed in /opt/local/libexec/perl5.12/sitebin but confirm the path before do this.

	curl -L http://cpanmin.us \| perl - --sudo App::cpanminus
	# Or, update it if you already have it:
	cpanm --self-upgrade --sudo

	cpanm App::cpanoutdated
	cpanm App::cpanlistchanges

view raw Install_cpanminus.sh hosted with ❤ by GitHub

Now you have them installed it's time to update your Perl modules:

	# (Optional) Print a list of the outdated modules:
	cpan-outdated -p
	# (Optional) See the changes for the outdated modules:
	cpan-listchanges `cpan-outdated -p`
	# Update the outdated modules:
	cpan-outdated -p \| cpanm

view raw Update_Perl_modules.sh hosted with ❤ by GitHub

Install Dancer

With cpanm the Dancer installation is very easy:

cpanm Dancer

view raw Install_Dancer.sh hosted with ❤ by GitHub

You can use the same command to update Dancer, in case you already have it installed.

Create your application scaffold

First thing you need to have is a working or hosting directory. If this is your development environment, you can use the directory Documents/workspace inside your home directory (Windows, Linux or Mac OS). If this is a Testing or Production environment, you can use /opt/ to host your application.

Now it is time to create the application with the command 'dancer -a application_name'. Notice that I used application_name, but you can use a module name like Dancer::application_name.

Once the application scaffold is created go to the application directory and execute it.

$ cd /home/$USER/Documents/workspace
# Or go to /opt if this is not a Development environment
# cd /opt
$ dancer -a myapplication
+ myapplication
+ myapplication/bin
+ myapplication/bin/app.pl
+ myapplication/config.yml
+ myapplication/environments
+ myapplication/environments/development.yml
+ myapplication/environments/production.yml
+ myapplication/views
+ myapplication/views/index.tt
+ myapplication/views/layouts
+ myapplication/views/layouts/main.tt
+ myapplication/MANIFEST.SKIP
+ myapplication/lib
  myapplication/lib/
+ myapplication/lib/myapplication.pm
+ myapplication/public
+ myapplication/public/css
+ myapplication/public/css/style.css
+ myapplication/public/css/error.css
+ myapplication/public/images
+ myapplication/public/500.html
+ myapplication/public/404.html
+ myapplication/public/dispatch.fcgi
+ myapplication/public/dispatch.cgi
+ myapplication/public/javascripts
+ myapplication/public/javascripts/jquery.js
+ myapplication/t
+ myapplication/t/002_index_route.t
+ myapplication/t/001_base.t
+ myapplication/Makefile.PL
 
$ cd myapplication/
$ ./bin/app.pl

Open a web browser and go to: http://localhost:3000 to see the new application with a page very similar to a new Ruby on Rails application.

I like to rename the bin/app.pl file to bin/server.pl.

mv bin/app.pl bin/server.pl

Here is where most of the tutorials finish with the installation or setup but what about git?

Git setup

This is an important step. Even if you are the only developer of the project, you always must have a version control and repository for the application. You can setup a git server in other server but I recommend to use GitHub if you can or your company permit it.

Create an account in http://github.com and create a new repository. Choose the same name of your application, enter a description, check on 'Initialize this repository with a README' and select .... What???, there is Perl, Ruby and Rails but no Dancer!!!!, well .... select Perl in the 'Add .gitignore'. Click on 'Create Repository' and that's it.

In the next page there is a box at the right bottom corner with the title 'SSH clone URL'. Click on the 'copy to clipboard' icon or just copy that line. It will be something like this: git@github.com:your_github_user/myapplication.git.

Assuming that you already have git installed - if not, go to http://git-scm.com/downloads or google how to install git in your OS - go to the application directory to setup git. Execute these commands:

	git init
	git add .
	git commit -am "First commit"
	git remote add origin git@github.com:your_github_user/myapplication.git
	# Get the README and .gitignore files
	git pull origin master
	# Send the entire application to Github
	git push origin master

view raw First_commit.sh hosted with ❤ by GitHub

If you get an error about the public key (if this is your first time you will), then generate and add your key to github as explained here: https://help.github.com/articles/generating-ssh-keys

Now every time you made a change to the application you have to add the new files (if any), commit the changes and push them to github. Like this:

	git add .
	git commit -am "Change description"
	git push origin master

view raw Git_push.sh hosted with ❤ by GitHub

If you are working with a team or doing changes in different machines, you have to pull the latest changes from github before start working.

git pull origin master

view raw Git_pull.sh hosted with ❤ by GitHub

Or, if you want the code in a new machine or you want a new fresh copy just clone the application with:

git clone git@github.com:your_github_user/myapplication.git

view raw git_clone.sh hosted with ❤ by GitHub

To know more about git there are a lot of excellent documentation. For example:

Does Dancer have a Gemfile like Rails?

Unfortunately no, there is no such file in Dancer. However, you can make your own file, I name it Modfile and it have one module per line using the cpanm format:

#!/bin/bash

cpanm Module::Name[version]
cpanm git_repository
cpanm module.tar

Read the cpanm manual (man cpanm) to know more about this format. This is an example:

#!/bin/bash

cpanm Dancer
cpanm Dancer::Plugin::Database
cpanm Dancer::Plugin::FlashMessage
cpanm Template
cpanm Test::More                      # install Test::More
cpanm Net::SCP~1.0000                 # 1.0000 or later
cpanm Net::FTP~"&gt;= 1.0000, &lt; 2.0000"  # latest of 1.xxxx
cpanm Net::SSH2@0.9990                # specific version. same as Net::SSH2~"== 0.9990"
cpanm git://github.com/plack/Plack.git

Save the file and give it execution permission with:

chmod +x Modfile

All the Perl modules required by the application will be updated, downgraded or installed to the latest or specified version executing the Modfile file:

./Modfile

You are almost ready to start coding, let's learn some steps first and you'll be ready to dance.

Sunday, December 8, 2013

Shall we dance?

A few months ago I start working on a new Perl web application. I wanted to do it in Ruby on Rails but the client request to be in Perl for supportability reasons, this means: "If you go, will be easier and cheaper to find a Perl developer than a Ruby developer.". No matter how many reasons, pro and cons I presented, the final decision was to do the web application in Perl.

I was in the task to find a Perl web framework as good as Rails for Ruby. There are a ton of Perl web frameworks but the most famous or known and the top 3 I choose were: Catalysis. Mojolicious and Dancer. You can google and read a lot of comparisons about all of them and take your own decision about which to use for your application. I choose Dancer.

I do not want to criticize the other frameworks. Every framework or language are good but you need to choose one that fit the needs and requirements of your project. At that time I thought Catalysis would be a huge framework for this application, now I think it can fit perfectly because the application grow more than I expected or analyzed. Regarding Mojolicious, I tried once and I did not find enough documentation at that time, by the time I was comparing those 3 frameworks it continue to lack of documentation.

Dancer, as you can read in the excellent documented Dancer site, http://www.perldancer.org/, is a "simple but powerful web application framework for Perl", it was inspired in Sinatra, another Ruby web application framework. Then I read all the excellent documentation, tutorials and training; if you do it, I think you will also choose Dancer.

Dancer is a RESTful framework, just like Sinatra. I was looking something more MVC like Rails but I did not find a good, known and well documented framework in Perl. So, I give a try to REST and Dancer. If you are like me, looking for a MVC or Rails-like Perl framework, Dancer is not the answer but unfortunately there is no answer yet. If you know some framework, please, share your comments.

Even thought Dancer is well documented I had some difficulties to find well explained documents about how to use some modules or how to do some specific things or what modules were the best or recommended. So, base on that I decided to document what I have done and maybe it would be useful for you too. Also, I hope to receive feedback and we all can learn more about how to dance with Perl.

So, say no more ... Shall we dance?

Pages