Category: Development


Review of AssemblySys dataServices

On a large data migration project that I am currently spearheading, we have a large installed userbase of over 2 million users running on a social networking engine. The schema has been redesigned from scratch, and code is being written to match the new schema, using the all-powerful MySQL database as the system to manage all that data.

Since this social network is global, we need good and reliable location information. The current location model is flawed and full of holes, so we have chosen AssemblySys‘ data to replace it.

We are not using AssemblySys’ schema, as we’ve rolled our own. I’ve designed our new schema to be hierarchial in nature, treating all locations on the planet as ‘nodes’ with a tree relationship, with “Earth” being the parent of all nodes. This model allows us to account for all countries and their idiosyncratic ways they divy up their adminstrative divisions, which to say the least varies a lot.

Currently AssemblySys does not have strong support for postal codes, and only about 5 countries use postal codes anyway. However, I was able to secure zip codes from a different vendor and graft them in to our location model.

The AssemblySys location database is quite through and complete, with accurate geodata for the cities. In fact, it is so complete it even lists some towns that don’t show up on Google Maps! I verified that some of these obscurities I found do, in fact, exist.

And I uncovered a good bit of curious geographical trivia, like the fact that there are 5 towns in Kentucky called “Boston”. Must be a nightmare for the Post Office there! I also found there is a town called “Philadelphia” in South Africa! At first, I thought these must be errors, but I verified that these obscure towns do indeed exist.

Next came the task of transforming their location data to our model. This is  where I had the most problems, because their data is not arranged in the nice, clean, hierarchical fashion our model is. In fact, it’s laid out in a very cumbersome fashion requiring a number of sub-keys to cull out the proper hierarchy.

To their credit, though, AssemblySys was quick to respond to my questions about how to access their data and shot back examples that was very helpful with the effort. But I felt their model was way too complicated than it needed to be, and perhaps could have used a bit more normalization. But I was able to do the transform after a few days of wrestling with it.

Overall, I am pleased with the quality of the AssemblySys product. I am not happy with their schema layout and the rather obtuse and complicated queries to cull out the structure. However, perhaps most users will use their database as is and perhaps it works better in that context, though the queries can get quite cumbersome from my estimation. The service is good, though completely email-based. The price is reasonable and the data is accurate.


KDE Konsole Backgrounds and ssh

If you are a GUI-oriented person, you need not read this. But if you are like me, you make heavy use of the console. If you are managing many machines as well as your own Linux workstation, it’s VERY important to know where your console session is.

Too many times in the past I had wanted to bring down my workstation, and would type “shutdown” or “reboot” in the console window, only to find out to my horrors that the console was really a remote session to one of my web servers serving up hundreds of web sites.

Whoops!

Well, that prompted me into developing a solution where I can tell at a glance where I happened to be logged in. This way,  I wouldn’t be in danger of issuing dangerous commands on the wrong server. And if you are working for someone else, it also keeps you from being FIRED!

I use KDE to do my development and administration, and I have fallen in love with Konsole. Konsole, despite its quirks, has a lot of nice features that makes it a shoe-in for what I am talking about.

My approach now is to create login bash scripts to begin a session with whatever machine I need to ssh into, and have that script also do something nice to the Konsole background in the process.

I also develop a lot of websites, as well as other things. Sometimes, it’s helpful to change the background when I go into emacs so that I have the proper contrast for syntax coloring, etc. Same approach works there as well.

kschemaset_konsoles.png The secret to my machinations? A little script I wrote called kschemaset. It does all the “magic” in  resetting the Konsole background, and is actually derived from a similar script that didn’t do everything I needed. But in the fine tradition of opensource, I grabbed it and enhanced it over time. It currently does require you to either create images or acquire images for your backgrounds, but eventually I wish  to create a more comprehensive opensource package that will do all that magic for you automatically. But for now, it’s a lot of fun to come up with a cute background that represents the server you are working on! I recommend choosing either very light pastel colors or very dark colors with low contrast, because you want to be able to read the text without killing your eyes.

You don’t have to create any artwork at all, actually, and many may elect to do it that way. It’s all up to how you want to proceed.

The first step is to install the kschemaset script somewhere where it can execute. There are actually other related scripts involved, and they are all available from here. Typically, I create a local bin directory to my login account and alter the .bashrc file to add it to the PATH environment variable. Since you are obviously well adept at such things if you are interested, I won’t bother with holding your hand here.

The next step involved is to create Konsole schemes to reflect the servers you wish to work on. All kschemaset does is invoke a schema for the duration of the session, and restore the old schema when the session is concluded. Eventually, not only do I want to automate the creation of new Konsole schemas, but also the images used for the backgrounds. But for now you can do this by hand — or automate these steps yourself. If you do, let me know so I don’t wind up reinventing your wheel!

Konsole has one annoying problem that seems not to have been fixed in recent times — when you add a new schema, any Konsoles that were open prior to the addition tend to get “confused” and may start displaying the wrong schema. The workaroud for this is to either to manually select the schema for those Konsoles out of sync, or to restart them. Hopefully this problem will be fixed soon.

Now, you simply create scripts based on kschemaset that will launch your new sessions, and change not only the background, but the tab text as well, on the fly, and to reset everything to the pre-existing schema and tab text when you’re done. I’ve even did this with emacs to give me a flat-black background to do my editing on.  The possibilities are endless.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
# Set the schema of the currently-running Konsole
# Based on konsoledcopschema
 
 
# So running script can know this is where it's running
# (to enable scripts to enable kschemaset via recursion)
export kschemaset=1
 
. kconfuns
getKonsoleInfo
getAppInfo ${*}
shift
 
if [[ ! ${appSchema} ]] ; then
    echo "(schema $appName not found)"
fi
 
export -f ksetSchema getKonsoleInfo getAppInfo
 
if [ "${inKonsole}" == "1" ] && [[ ${appSchema} ]] ; then
    # dcop $konsole konsole reparseConfiguration
    dcop $konsole $session setSchema "${appSchema}"
    dcop $konsole $session setSchema "${appSchema}" # testing -- may not have gone through the first time
    if [ -n "${appName}" ] ; then
        dcop $konsole $session renameSession "${origSession}: ${appName}"
    fi
 
    if [ -n "${*}" ] ; then # run the command and reset to prveious schema.
        ${*}
        dcop $konsole $session setSchema "${origSchema}"
        dcop $konsole $session renameSession "${origSession}"
    fi
else
    kschemaset=2
    ${*}
fi

And now for the next script:

[-]?Download kconfuns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/bin/bash
# Konsole functions. Include with ". kconfuns"
 
kloc=~/.kde/share/apps/konsole/
 
getKonsoleInfo()
{
    # Make sure we're in Konsole
    if [ -n "${KONSOLE_DCOP}" ] ; then
        export inKonsole='1'
        export konsole=`echo $KONSOLE_DCOP | cut -d\( -f 2 | cut -d, -f1`
        export session=`dcop $konsole konsole currentSession`
        export origSchema=`dcop $konsole $session schema`
        export origSession=`dcop $konsole $session sessionName`
    else
        export inKonsole='0'
    fi
}
 
getAppInfo()
{
    export appName="${1}"
    export appSchema=`ls -1 $kloc | egrep "${appName}\."`
}
 
ksetSchema()
{
    dcop $konsole $session setSchema $1.schema
}
 
ksetSessionName()
{
    dcop $konsole $session renameSession "$*"
}

And here is an example of lauching an ssh session using kschemaset:

[-]?Download polaris
1
2
3
#!/bin/bash
# Log on to the Polaris Web Server
kschemaset Polaris  ssh -t youraccount@yourserver.yourdomain.com $*

And it’s that simple!