September 11, 2007

Automation of Nimblegen pair and gff extraction from gridded tif file

This is my “first” ruby script after hello world, so i’m sure there’s probably a better way to code it.

The process of extracting the pair files and gff file generation using the NimbleScan GUI can be tedious and time consuming if you have more than one design file. For NimbleGen’s new HD2 arrays, there are 12 blocks x 2 colors = 24 images per genome wide set to process. For repetative and limited tasks, it is much easier to run NimbleScan2.3 via command line and automatically match up the corresponding ndf and pos files to generate the corresponding pair and gff files.

The script is run within the folder which contains the “gridded” image files. Unfortunately, the automatic alignment of the grid to the spots is not optimized such that manual adjustment is still required after the “auto-align” to ensure proper placement of the grid. However, once the image is gridded and saved, NimbleScan can do the processing without the GUI.

The argument to the script is pair, gff, or both depending upon which files you want to generate. Note that gff requires the pair files, so most of the time you just run both.

bin is the NimbleScan binary executable

designpath is where you have the design files stored

designfile is the common prefix to the ndf and pos files… you may have to edit this if yours is different

maxthread is the max number of concurrent threads to run.. if you only have 1 or 2 cpus, then decrease this number to avoid thrashing

Note: the image files are named something_Anumber_something.tif the script keys off of the _Anumber_ and uses the number to obtain the correct ndf or pos file. A01 refers to design file 01, A02 -> designfile 02 and so on.

The image files must also have something_635_ and something_532_ in order to match the red and green channels from the same array block.

———————————————————

#!/usr/bin/ruby

# $Id: nimblescan.rb,v 1.2 2007/09/11 17:16:53 kching Exp $
#
#
#

# puts “hello ruby\n”

# get list of tif files

bin = ‘NimbleScan’

designpath = “/var/NimbleScan2/Design/”
designfile = “2007-06-19_HG18_Economy_”
#NimbleScan -xprobes -a -S -o ./PairData/ -n /var/NimbleScan2/Design/2007-06-19_HG18_Economy_01.ndf -i rdh_161_ng120987_A01_H3K4me3_hESC_090607_5
32_grid.tif

paircmd = ” -xprobes -a -S -n ”
gffcmd = ” gff -n -R -x ”
#NimbleScan gff -n -R -x -o ./GFF/ -c rdh_161_ng120987_A01_H3K4me3_hESC_090607 -p /var/NimbleScan2/Design/2007-06-19_HG18_Economy_01.pos ./PairDa
ta/rdh_161_ng120987_A01_H3K4me3_hESC_090607_635_grid.pair ./PairData/rdh_161_ng120987_A01_H3K4me3_hESC_090607_532_grid.pair

ARGV.each{ |option|
#puts option + “\n”
if(option=~/gff/i)
$command=’gff’
elsif(option=~/pair/i)
$command=’pair’
elsif(option=~/both/i)
$command=’both’
end
}

if(! (defined? $command) )
puts “usage script < gff | pair | both >\n”
exit
end

ls = `ls -1 *_635_*.tif`
ls = ls.chomp
files = ls.split(/\n/)

#puts ls

$maxthread = 5;

$threads = []

files.each{ |file|
# generate the 532 file
file532 = file.sub(/_635_/, ‘_532_’)
file=~/_A(\d\d)_/
blocknum = $1
#puts “block “+blocknum+”\n”
#puts “name ” + file + “\n”
#puts “new name ” + file532 + “\n”

if($command == ‘pair’ || $command == ‘both’ )
cmd = bin + paircmd + designpath + designfile + blocknum + “.ndf” + ” -i ” + file
$threads << Thread.new(cmd) { |mycmd|
puts cmd + “\n”
system(cmd);
}
cmd = bin + paircmd + designpath + designfile + blocknum + “.ndf” + ” -i ” + file532
$threads << Thread.new(cmd) { |mycmd|
puts cmd + “\n”
system(cmd);
}
end

threadlist = Thread.list;
while(threadlist.length >= $maxthread)
threadlist = Thread.list;
puts “Threadcount ” + threadlist.length.to_s + ” join thread\n”
sleep(10);
$threads[0].join;
end
}

$threads.each { |mythread| mythread.join }

ls = `ls -1 *_635_*.pair`
ls = ls.chomp
files = ls.split(/\n/)

files.each{ |file|
file532 = file.sub(/_635_/, ‘_532_’)
file=~/_A(\d\d)_/
blocknum = $1

file=~/(.+)_635_grid\.pair/
rootname = $1

if($command == ‘gff’ || $command == ‘both’ )

cmd = bin + gffcmd + ” -c “+ rootname + ” -p ” + designpath + designfile + blocknum + “.pos ” + file + ” ” + file532
$threads << Thread.new(cmd) { |mycmd|
puts cmd + “\n”
system(cmd);
}

end

threadlist = Thread.list;
while(threadlist.length >= $maxthread)
threadlist = Thread.list;
puts “Threadcount “+ threadlist.length.to_s + ” join thread\n”
sleep(10);
$threads[0].join;
end
}

$threads.each { |mythread| mythread.join }

ruby script to autogenerate pair and gff files

5 Comments

  1. through above script and scanning NimbleScan-2.3 source I figured how to grid images and retrieve data files and QC files. You say “Unfortunately, the automatic alignment of the grid to the spots is not optimized such that manual adjustment is still required ….”. How do you go about this? Visual inspection of the gridded images allows me to see whether fiducials are properly aligned. But other than that? I don’t feel like going over all 380K, or 2.1M probes to check alignment. What do you mean by ‘manual adjustment’? How do you spot improper auto alignment?
    At the moment I plan to do the alignment phase automatically, save alignment QC data (‘qcmetrics’ or ‘autoqc’ as output file), and check if these QC data are “not normal” compared to other alignment QC data.

    Best, Ludo

    Comment by ludo — April 15, 2008 @ 6:24 am

  2. Hi Ludo,

    what I meant about the manual alignment was that the white reference spots on the array
    were not automatically aligned to the the grid by the software. People in the lab did not feel
    it did an adequate job, just for these reference spots not every spot on the array. Once you
    fix the reference spots, the other spots are also shifted accordingly.
    Like after auto-alignment, there would be a slight pixel shift on the bottom corner. Thus they would auto-align.. then tweak
    by hand and save the grid. The script would automate the process from there.

    -keith

    Comment by kching — April 15, 2008 @ 10:00 am

  3. Hi Keith,

    I guess by reference spots you mean the spots in the feducials? But how do you fix them? I’ve tried manual alignment, but never got that working (following the Users Guide, I can select maybe two corners but then I can’t select any more: despite all pointing and clicking I do). If I auto-align first and then select manual align I loose the grid I’ve just aligned and I’m back at the manual align problem. I don’t see any way to change anything in the grid location after auto-aligning by hand.

    By the way, I’m using NimbleScan-2.3 (I’ve tried 2.4 but that seem awfully buggy).

    Ludo

    Comment by ludo — April 16, 2008 @ 11:24 pm

  4. Hi Ludo,

    I’m not sure what version of NimbleScan we had. I’ve since left the lab. But I know that
    you can set the feducials by hand and adjust the grid slightly.. maybe try control-drag
    or alt-drag.. I can ask someone in the lab who’s done it..

    -keith

    Comment by kching — April 17, 2008 @ 7:42 pm

  5. Ah, found it. If the cursor is in ’select mode’ I can left-click the 4 corner-cells from the 4 corner-feducials (these cells have a different color from the rest of the feducial cells). Then I can drag it anywhere and the grid will adjust accordingly. After selecting a corner cell you can also move it using the arrow keys.
    NimbleScan-2.3 (and 2.4) display 2 values (alignment and uniformity) that are supposed to give an indication regarding the quality of the alignment. When you move the corner cell around these values are updated on the fly. The one image I’ve been playing with to figure this out indeed seemed to have room for a tiny adjustment given these values. I have a hard time though deciding which position visually gives the best alignment of the grid. I also don’t know what this ‘alignment score’ and the ‘uniformity score’ exactly compute. And I don’t know whether to base my decision on the position of the grid, or on the positions of the cells inside the grid (which align with the pixels of the tif image). I’ll discuss and show all this with our microarray facility, who will do all this feature extraction annyway.

    Keith, thanks for pointing me in the right direction.

    Ludo

    Comment by ludo — April 18, 2008 @ 7:51 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.