Thursday, September 18, 2008

XPlanner Authentication with NTLM

I spent most of the day setting up XPlanner and the documentation seemed to be lacking in a few spots. I'll go over the caveats I stumbled on.

The XPlanner install was happening on the lone linux box (Fedora 8) on an NT domain. The box was almost completely empty so I had to install java first. I chose to install the jsdk 1.4.2 even though that version's is EOL'd. XPlanner hasn't been updated in a while (since 2006) and I didn't want to mess w/ a new version of the sdk.

I upacked and installed the "standalone" version of XPlanner 0.7b7. So far so good. I was immediately able to use the default username/password to login.

The hardest part was figuring out how to authenticate based on our NT credentials. Most of the configuration occurs in the xplanner-0.7b7-standalone/webapps/ROOT/WEB-INF/classes/ file.

Here's how I was able to get the NT authentication to work. In the file, find the authentication strings:

# XPlanner security configuration

# NTLM login module[2][2].name=NTLM[2].option.userIdCaseSensitive=false[2].option.domain=DOMAIN[2].option.controller=CONTROLLER

To use the NTLM module, comment out the first set to security strings, uncomment the NTLM strings, and change the NTLM array index to 0. Here's how my file looked after I finished.

# XPlanner security configuration

# NTLM login module[0][0].name=NTLM[0].option.userIdCaseSensitive=false[0].option.domain=DOMAIN[0].option.controller=CONTROLLER

The NTLM module uses the local DB as a fall back, so any logins that you creaed locally that aren't in the Active Directory should still work.

Tuesday, September 9, 2008

JPEG Marker codes

I'm doing a bit of JPEG manipulation for a project and it took me longer than I would have liked to find a consise table of JPEG segment markers or frame markers or whatever you want to call them. Here's a C# enum with all the values. Hopefully that will save some typing for you.

        public enum JpegMarkers
            // Start of Frame markers, non-differential, Huffman coding
            HuffBaselineDCT = 0xFFC0,
            HuffExtSequentialDCT = 0xFFC1,
            HuffProgressiveDCT = 0xFFC2,
            HuffLosslessSeq = 0xFFC3,

            // Start of Frame markers, differential, Huffman coding
            HuffDiffSequentialDCT = 0xFFC5,
            HuffDiffProgressiveDCT = 0xFFC6,
            HuffDiffLosslessSeq = 0xFFC7,

            // Start of Frame markers, non-differential, arithmetic coding
            ArthBaselineDCT = 0xFFC8,
            ArthExtSequentialDCT = 0xFFC9,
            ArthProgressiveDCT = 0xFFCA,
            ArthLosslessSeq = 0xFFCB,

            // Start of Frame markers, differential, arithmetic coding
            ArthDiffSequentialDCT = 0xFFCD,
            ArthDiffProgressiveDCT = 0xFFCE,
            ArthDiffLosslessSeq = 0xFFCF,

            // Huffman table spec
            HuffmanTableDef = 0xFFC4,

            // Arithmetic table spec
            ArithmeticTableDef = 0xFFCC,

            // Restart Interval termination
            RestartIntervalStart = 0xFFD0,
            RestartIntervalEnd = 0xFFD7,

            // Other markers
            StartOfImage = 0xFFD8,
            EndOfImage = 0xFFD9,
            StartOfScan = 0xFFDA,
            QuantTableDef = 0xFFDB,
            NumberOfLinesDef = 0xFFDC,
            RestartIntervalDef = 0xFFDD,
            HierarchProgressionDef = 0xFFDE,
            ExpandRefComponents = 0xFFDF,

            // App segments
            App0 = 0xFFE0,
            App1 = 0xFFE1,
            App2 = 0xFFE2,
            App3 = 0xFFE3,
            App4 = 0xFFE4,
            App5 = 0xFFE5,
            App6 = 0xFFE6,
            App7 = 0xFFE7,
            App8 = 0xFFE8,
            App9 = 0xFFE9,
            App10 = 0xFFEA,
            App11 = 0xFFEB,
            App12 = 0xFFEC,
            App13 = 0xFFED,
            App14 = 0xFFEE,
            App15 = 0xFFEF,

            // Jpeg Extensions
            JpegExt0 = 0xFFF0,
            JpegExt1 = 0xFFF1,
            JpegExt2 = 0xFFF2,
            JpegExt3 = 0xFFF3,
            JpegExt4 = 0xFFF4,
            JpegExt5 = 0xFFF5,
            JpegExt6 = 0xFFF6,
            JpegExt7 = 0xFFF7,
            JpegExt8 = 0xFFF8,
            JpegExt9 = 0xFFF9,
            JpegExtA = 0xFFFA,
            JpegExtB = 0xFFFB,
            JpegExtC = 0xFFFC,
            JpegExtD = 0xFFFD,

            // Comments
            Comment = 0xFFFE,

            // Reserved
            ArithTemp = 0xFF01,
            ReservedStart = 0xFF02,
            ReservedEnd = 0xFFBF

Tuesday, June 3, 2008

eApps and yum

One of my new projects is hosted on eApps. While they've been great so far, I spent some time trying to get some real developer tools on my box using yum. The first problem was that php was failing with the following error:

PHP Warning: Module 'modulename' already loaded in Unknown on line 0

As it turns out that means that the xls package was included in the binary, and also being declared as a dynamic package. So, to fix this I commented out the following line in the /etc/php.d/xsl.ini:


Now on to yum itself. Doing any kind of yum update resulted in the following error:

Error: Missing Dependency: glibc-common = 2.3.4-2.36 is needed by package glibc-dummy-centos-4

After doing a bit of research, I found that the dummy-centos-4 package isn't really necessary if you're going to be upgrading your gcc libraries anyways so away it goes.

yum remove glibc-dummy-centos-4

After that I ran a full yum upgrade which worked just fine.

Friday, May 2, 2008

Stripping Accennts From Text With PHP

Most of the PHP string functions don't really work with UTF-8 encoded strings ( I found this post on the Symfony forums that had a solution.

$text = iconv('UTF-8', 'ASCII//TRANSLIT', $text);

Done and done.

Monday, April 14, 2008

SQL Join with count

For the real estate site I've been working on, I needed to get a list of real estate offices and how many active listings they had on the MySQL DB we are using. I had a table of offices and a table of listings. I needed each office represented even if that office had no active listings. It seemed like an easy application of the left join command. Not quite as easy as I thought. Here's my first attempt:

FROM offices LEFT JOIN listings ON ( = listings.office_id)
listings.status = 1

However, this only returned office/count pair where the count was greater than 0. It seemed like the WHERE condition wasn't being applied correctly, like it was being applied AFTER the tables were joined. I wanted the WHERE condition to be applied to the listings table and then have the results joined to the offices table. So, here's the solution I found: in situations like this, the WHERE clause needed to be added to the ON clause in the LEFT JOIN like this:

FROM offices LEFT JOIN listings ON ( = listings.office_id AND listings.status = 1)

Works like a charm.

Friday, April 4, 2008

Dynamic Credentials in Symfony

For a long time now I've been using dynamic credentials in Symfony as laid out in this code snippit:

// to put in the actions.class.php file
function getCredential()
$this->post = $this->_retrievePost(); // retrieving the object based on the request parameters
if ($this->getUser()->isOwnerOf($this->post))

// the hijack is over, let the normal flow continue:
return parent::getCredential();

However, it seems like this isn't really the proper way to add dynamic credentials. The getCredential() method is called by the Symfony FilterChain to discover what credentials that module requires, not to add credentials to the user.

For me, the problem cam to a head when I was trying to show a link if a user had a particular credential. The problem was that the page with the link was not a secure page. Since the page was not secute (is_secure: off in the config/security.yml file) the getCretential() method was never called.

The solution is to use the preExecute() method instead of getCredential()

// to put in the actions.class.php file
function preExecute()
$this->post = $this->_retrievePost(); // retrieving the object based on the request parameters
if ($this->getUser()->isOwnerOf($this->post))

// the hijack is over, let the normal flow continue:
return parent::getCredential();

Thursday, March 6, 2008

Accent mode in Emacs 22

For a long time I used the emacs iso-accents-mode for my editing in Spanish. Looks like this has finally been removed from the standard emacs distros, so I had to figure out whatever they replaced it with. Here's what I (eventually) found: To get the same functionality as the old iso-accents-mode, you need to change the input mode. The official doc can be found here, but here are the basic commands.

M-x list-input-methods

C-x RETURN C-\ method RETURN
This is how to select the input method.

Toggles the input mode on/off. Pretty useful.

The mode I use is latin-1-prefix. It should be familiar to the iso-accents-mode followers.

Loading NetFlix Prize Data into MySQL

I may be a year late, but I decided to enter into the NetFlix Prize contest the other day. I've already got MySQL (Lampp) running on my box (Fedora 8) so I decided to load the data into my DB to do some basic analytics. There's some good info out there, but nothing that encompassed all the steps I took, so I'm going to repeat my process here.

1. Enter the Contest and download the dataset. Nothing too complicated here, you just need a unique team name and an email address.

2. Get Ruby where it needs to be. For this I used yum to install both the ruby and ruby-devel packages.

2a. Then you need to download and install Ruby/MySQL. Here are the basic steps from

tar zxvf mysql-ruby-2.7.tar.gz
cd mysql-ruby-2.7
ruby extconf.rb --with-mysql-config
ruby test.rb [hostname] [username] [dbpassword]
sudo make install

2b. After that is all set and you can connect to the MySQL w/ the test.rb script, download the ruby DBI from here:

tar zxvf dbi-0.1.1.tar.gz
ruby setup.rb config --with=dbi,dbd_mysql
ruby setup.rb setup
sudo ruby setup.rb install

In the config command, I chose to only include the MySQL packages of the DBI. I think it's generally better to only include the packages you want as opposed to always including everything. If you don't specify which DB packages you want, the installer will assume you want everything. If you don't have all of the required development packages already on your machine (ie oracle and sybase) you'll get into errors when you run the setup command.

3. Now find a ruby script to load the data. I used this one from juretta:

# Created by Stefan Saasen on 2007-04-24.
# Copyright (c) 2007. All rights reserved.
require 'dbi'

# Configuration
DB_DATABASE = 'netflix'
DB_USER = 'user'
DB_PASSWORD = '******'

# Directory with netflix dataset
NETFLIX_DOWNLOAD_DIR = File.join("/sata_ext/net_flix")

NETFLIX_MOVIE_TITLES = File.join(NETFLIX_DOWNLOAD_DIR, "movie_titles.txt")

# Profile...
start =

# Connect to the MySQL database
DBI.connect("DBI:Mysql:#{DB_DATABASE}", DB_USER, DB_PASSWORD) do |dbh|
# create the necessary tables
stmts =";").delete_if{|stmt| stmt.strip.empty?}
stmts.each do |stmt|
# Execute the current SQL statement

# 1. Insert movie titles into movie_titles table using simple INSERT STATEMENTS
sql = "INSERT INTO movie_titles (id, year_of_release, title) VALUES(?,?,?)"
# It would be even more efficient to use INSERT statements that
# use VALUES syntax which can insert multiple rows.
# e.g. INSERT INTO movie_titles (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
dbh.prepare(sql) do |sth| do |f|
f.each do |line|
# Execute the prepared INSERT Statement
id, year_of_release, title = line.split(",")
sth.execute id.to_i, year_of_release.strip, title.strip

# 2. Insert rating data from training_set using the fast LOAD DATA INFILE syntax
# The filename contains the movie id!
"IGNORE 1 LINES (customer_id, rating, date) SET movie_id = ?;"
dbh.prepare(sql) do |sth|
Dir[NETFLIX_TRAINING_SET + "/*.txt"].each do |trs_file|
puts "Importing #{trs_file}..."
if trs_file =~ /([0-9]+)\.txt$/
movie_id = $1.dup.to_i
raise "Missing movie_id (file: #{trs_file})"
# Execute the prepared statement using the .txt file and the movie_id
sth.execute File.expand_path(trs_file), movie_id

# Add index for certain columns
#puts "Creating index on the ratings table - may take a while"
#["CREATE INDEX mid_idx ON ratings (...)", "CREATE INDEX cid_idx ON ..."].each{|stmt|}
puts "Import successfully finished!\n"
puts "The netflix data import took #{sprintf("%0.2f", ( - start)/60.0)} minutes"

USE netflix;

DROP TABLE IF EXISTS `movie_titles`;
CREATE TABLE `movie_titles` (
`id` int(11) NOT NULL,
`year_of_release` YEAR(4) default NULL,
`title` VARCHAR(250),

CREATE TABLE `ratings` (
`id` int(11) NOT NULL auto_increment,
`movie_id` int(11) NOT NULL,
`customer_id` int(11) NOT NULL,
`date` date default NULL,
`rating` int(1),

The last part there are the SQL commands to create the tables needed for this script.

I loaded all of the data onto my machine (Dual Intel 2.13 GHz w/ 1GB RAM) in about 13 minutes.