What's New in WebGUI 8.0 #3: Upgrade System

Tags:

Originally posted as: What's New in WebGUI 8.0 #3: Upgrade System on blogs.perl.org.

Following The Path

If you installed WebGUI 0.9.0 back in August of 2001 (the first public release), you've had a stable upgrade path through WebGUI 7.10.8 (January 2011) and beyond. Plainblack.com has been through every upgrade for the last 10 years, a shining bastion to our upgradability.

A WebGUI 7.10 user would not even recognize a WebGUI 6.0 database, much less the database used by the 1.x series, but slowly, gradually, our upgrade system brought new features to every WebGUI site that wanted them.

The Ancient Way

Our old upgrade system was quite simple:

docs/upgrade_2.9.0-3.0.0.pl
docs/upgrade_3.0.0-3.0.1.sql
docs/upgrade_3.0.0-3.0.1.pl

Our upgrade.pl script would check for docs/upgrade_*, compare version numbers, and then execute the .sql and .pl scripts in order until there were no more upgrades left.

Because each .pl script was executed individually, there was a considerable amount of boilerplate in each script (123 lines). Because there was only one script per version, some scripts could get quite long. We had conventions to manage these limitations, but it was still a bit of a mind-twist to write an upgrade routine.

Later, when we moved to simultaneous beta and stable trees, it became even more difficult to manage these huge upgrade scripts. Collecting the new features from the beta tree to apply to the stable tree was a time-consuming manual task that some poor coder had to perform, back hunched over a dimly-lit screen in the wee hours of the night, testing and re-testing the upgrade to make sure stable lived up to its expectations.

Though our upgrade system had performed admirably, it was time for a fresh look at the problem.

The Modern Vision

The individual files for upgrades was working quite well, but didn't go far enough. Our new upgrade system has one file per upgrade step. Each sub from an old upgrade script would be one file in the new upgrade system. What's more, additional file types would be supported:

$ ls share/upgrades/7.10.4-8.0.0/
addNewAdminConsole.pl
admin_console.wgpkg
facebook_auth.sql
migrateToNewCache.pl
moveMaintenance.pl
moveRequiredProfileFields.pl

So now, instead of a single file for an upgrade, we have an entire directory. In this directory, the .pl files are scripts to be run, the .wgpkg files are WebGUI assets to add to the site, the .sql files are SQL commands to run, and any .txt files will be shown as a confirmation message to the user for gotchas like "All your users have been logged out as a result of this upgrade. Deal with it.".

So now, if you want to add your own custom upgrade routine, you just add another file to the directory which means less worrying about conflicts. When we need to build another new stable version release, we can just move the unique upgrade files from beta to the new upgrade.

The best part of the new upgrade system is how the .pl scripts are written. When you are in a .pl, you have a bunch of sugar to make the basic tasks much easier.

# Old upgrade routine. Just another day in a session
sub migrateToNewCache {
    my $session = shift;
    print "\tMigrating to new cache " unless $quiet;

    use File::Path;
    rmtree "../../lib/WebGUI/Cache";
    unlink "../../lib/WebGUI/Workflow/Activity/CleanDatabaseCache.pm";
    unlink "../../lib/WebGUI/Workflow/Activity/CleanFileCache.pm";

    my $config = $session->config;
    $config->set("cache", {
        driver              => 'FastMmap',
        expires_variance   => '0.10',
        root_dir            => '/tmp/WebGUICache',
    });

    $config->set("hotSessionFlushToDb", 600);
    $config->delete("disableCache");
    $config->delete("cacheType");
    $config->delete("fileCacheRoot");
    $config->deleteFromArray("workflowActivities/None", "WebGUI::Workflow::Activity::CleanDatabaseCache");
    $config->deleteFromArray("workflowActivities/None", "WebGUI::Workflow::Activity::CleanFileCache");

    my $db = $session->db;
    $db->write("drop table cache");
    $db->write("delete from WorkflowActivity where className in ('WebGUI::Workflow::Activity::CleanDatabaseCache','WebGUI::Workflow::Activity::CleanFileCache')");
    $db->write("delete from WorkflowActivityData where activityId in  ('pbwfactivity0000000002','pbwfactivity0000000022')");

    print "DONE!\n" unless $quiet;
}

If you're familiar with WebGUI session, this is pretty standard, but still much boilerplate and convention. The new scripts remove boilerplate and enforce what was once merely convention.

# New upgrade routine. migrateToNewCache.pl
use WebGUI::Upgrade::Script;
use Module::Find;

start_step "Migrating to new cache";

rm_lib
    findallmod('WebGUI::Cache'),
    'WebGUI::Workflow::Activity::CleanDatabaseCache',
    'WebGUI::Workflow::Activity::CleanFileCache',
;

config->set("cache", {
    'driver'            => 'FastMmap',
    'expires_variance'  => '0.10',
    'root_dir'          => '/tmp/WebGUICache',
});

config->set('hotSessionFlushToDb', 600);
config->delete('disableCache');
config->delete('cacheType');
config->delete('fileCacheRoot');
config->deleteFromArray('workflowActivities/None', 'WebGUI::Workflow::Activity::CleanDatabaseCache');
config->deleteFromArray('workflowActivities/None', 'WebGUI::Workflow::Activity::CleanFileCache');

sql 'DROP TABLE IF EXISTS cache';
sql 'DELETE FROM WorkflowActivity WHERE className in (?,?)',
    'WebGUI::Workflow::Activity::CleanDatabaseCache',
    'WebGUI::Workflow::Activity::CleanFileCache',
;
sql 'DELETE FROM WorkflowActivityData WHERE activityId IN (?,?)',
    'pbwfactivity0000000002',
    'pbwfactivity0000000022',
;

done;

The first thing we do in our new upgrade script is use WebGUI::Upgrade::Script. Now, instead of using the session for everything, we have subs imported for various tasks. This means that many times we can run an entire upgrade script without opening a WebGUI session, or creating a version tag unnecessarily.

If we do need a session, or a version tag, they will be automatically assigned relevant information describing what we're doing. When we're done, they will be automatically cleaned up and committed. What once was done with boilerplate, and subject to random deletion or subversion, is now enforced policy.

In all other respects, a WebGUI upgrade script is a Perl script. You can add modules, write subroutines, and do anything necessary to move WebGUI into the future.

The Internet is always evolving. With the WebGUI 8 upgrade system, we've made it easier to evolve with it.

Stay tuned for next time where I'll show off our CHI-based caching system.

What's New in WebGUI 8 #2: Auth Improvements

Tags:

Originally posted as: What's New in WebGUI 8 #2: Auth Improvements on blogs.perl.org.

Auth changes

WebGUI::Auth was developed as part of 6.0.0, back in 2003. To put that into perspective:

  • Auth predates Facebook, which was founded in 2004.
  • Since Auth, there have been two Summer Olympiads.
  • Auth was written when I was still in college.

Since then, it has not fundamentally changed, though everything about the Internet surely has.

Continue reading What's New in WebGUI 8 #2: Auth Improvements...

What's new in WebGUI 8.0 #1 - PSGI/Plack

Tags:

Originally posted as: What's new in WebGUI 8.0 #1 - PSGI/Plack on blogs.perl.org.

Overview

On September 9, 2009, WebGUI 8 was announced. Since it was the first major API change since WebGUI 7, we started planning WebGUI 8 with high expectations. Over the course of the last 16 months:

  • We've adopted PSGI/Plack as our platform, enabling us to work in any HTTP environment.
  • We've made massive changes to the Asset system to make them easier to build.
  • We've reworked Auth to make it easier to add Twitter and Facebook authentication for your users.
  • We've rebuilt the upgrade system to be easier for developers and system administrators.
  • There is a new caching system enabling memcached and local memory caches.
  • The entire content management interface was rebuilt from scratch with the latest buzzword-worthy technologies.
  • Assets are now based on Moose, the new object-oriented system for Perl.
  • We've created WebGUI::FormBuilder, a way to make introspect-able forms.
  • We're migrating to InnoDB to take advantage of transactions and foreign key constraints.
  • We're making a brand-new WRE based on Nginx, and Server::Starter for more efficient resource usage and no-downtime restarts.

And these are just the major features. We've cleaned up and straightened out a lot of the crooked parts of programming in WebGUI, and we have plans to do even more. We've greatly improved the flexibility of WebGUI without compromising any more backward compatibility.

Ambition is the first step

Unfortunately, our ambitious goals have taken longer than we thought. We had promised a WebGUI 8.0 Beta in January, and I have to say that isn't possible. The way things look right now, WebGUI 8.0 will be ready this summer.

So, as atonement and apology, I offer this series of bi-weekly blog posts about all the new stuff in WebGUI 8.

PSGI / Plack

The most exciting change in WebGUI 8 is the move to PSGI using Plack. WebGUI is no longer just an Apache/mod_perl application, though it can certainly still run under Apache/mod_perl. Now, WebGUI can also run under Starman, a high-performance web server specifically for PSGI applications. WebGUI can run inside POE. WebGUI can once again run as CGI. We are no longer tied to one way of deploying WebGUI, and that greater flexibility will allow for better performance for your specific need.

#!/usr/bin/env perl
# webgui.cgi - Run WebGUI as a CGI application
use lib 'lib';
use Plack::Loader;
use Plack::Handler::CGI;
$ENV{WEBGUI_CONFIG} = "www.example.com.conf";
my $app = Plack::Util::load_psgi("share/site.psgi");
Plack::Handler::CGI->new->run($app);

Since WebGUI is just a PSGI application, it can also run alongside any other PSGI applications. WebGUI and Catalyst can work together on the same site.

For example, here's WebGUI and Pod::Browser running together. Pod::Browser takes /doc and serves the WebGUI POD, every other request goes to WebGUI.

# pod.psgi
use lib 'lib';
use Plack::Builder;
use Plack::Util;

BEGIN {
    $ENV{POD_BROWSER_CONFIG} = "pod_browser.json";
}
use Pod::Browser;
use Catalyst::Engine::PSGI;
Pod::Browser->setup_engine('PSGI');

$ENV{WEBGUI_CONFIG} = "core.conf";
use WebGUI;

builder {
    mount "/doc" => sub { Pod::Browser->run(@_); };
    mount "/" => Plack::Util::load_psgi("share/site.psgi");
};

# pod_browser.json
{
    "Controller::Root" : {
        "self" : 1,
        "inc" : 1,
        "namespaces" : [ "WebGUI*" ]
    }
}

For developers, Plack opens up new ways of building WebGUI sites. Instead of WebGUI::URL handlers, now we use Plack middleware to handle such tasks as:

  • Opening the WebGUI::Session
  • Handling /extras and /uploads
  • Showing the maintenance page, if necessary
  • Handling the wgaccess files for uploads permissions
  • Showing debugging information and performance indicators

By using more flexible tools, we can write more reusable code. You can write a Plack middleware for WebGUI that will work in your other Plack applications, and WebGUI can take advantage of the middleware out there on CPAN.

Plack middleware currently only replaces URL handlers, but someday we can replace WebGUI::Content handlers as well. WebGUI's pieces can be mixed and matched to create your own applications without using parts of WebGUI that you don't need.

By switching to PSGI/Plack, we've made WebGUI easier to deploy and easier to develop.

Next time, I'll talk about the improvements to the WebGUI Auth API.