Progress Reports
This page is a work in progress, the formatting is all over the place, should be fixed soon.
Week 1
Monday
• Day was spent at WASP
• Saw the Cray XT3
Tuesday
• Meeting with supervisors
• Sorted out access card, keys.
Week 2
Monday
• Got familiar with the drishti environment.
• Downloaded copies of the relevant simulation data from cognac
• Imported a .raw file into drishti which was created from code (simple cube)
Tuesday
• Got my PATH variables set up on Cognac.
• Created first build of project using Cognac.
• Created Sourceforge.net project (owls-h1)
• Ran code several times, making some improvements
• 4th Attempt provided some output. NOTE: Smoothing kernel was set to 1 for simplicity.
Wednesday
• Smoothing kernel now working correctly
• Ran several more simulations on cognac
• Created code to generate histograms
• Plotted histograms of mass, density and smoothing
The scale was difficult to get correct for the mass and the density because of the range and the fact that a high percentage of the masses were very low values.
• Graphed the smoothing kernel function
• Having a few problems with importing data into drishti. I am writing the data in a float format, however, when I try to load the data into drishti specifying float, the output is not as expected. When loading in the data and specifying integer values the visualization looks correct. This is probably a problem with the drishti menu.
• The menu problem was fixed by the developer of drishti, but still having problems with the volume.
• Started reading about OpenMP and MPI.
Thursday
• Reading about OpenMP
• Data in raw file is now scaled out so that it is visible in drishti
• Produced several high quality renders of the data.
• Simulation wraps around to enable tiling of the simulation
• Fixed a slight bug in the smoothing kernel that was skipping over some points.
Code should have been } else if (ratio <= 1) {
instead of
} else if (ratio = 1) {
Searched the smoothing data, with the old code, 34 points were being ignored.
Didn't make too much of a difference and the points shouldn't have been ignored, this fix just made sure that they weren't.
• Ran a few benchmarks on autoparallelised OpenMP code, nothing to report yet.
Friday
• Added code to calculate the HI Gas Density.
• Made several bug fixes
• Code was still running at the end of the day, not sure if I have it working correctly yet.
• Ran code on cognac
Week 3
Monday
• Having problems with the HI gas density calculations, changed some of the code and it is running at the moment. I think the problem is, when performing the following calculation in my code:
I am calculating the first part so that HIdensity[i][j][k] += HImass[ip] / (smooth[ip]*smooth[ip]*smooth[ip].
This means that the first part of the HIdensity calculation is the sum of the densities at every point from the data set that is within the maximum radius of that voxel. This calculation should not be the problem, because the smoothing length is never 0.
The problem is in the next part of the calculation:
My original approach was simply HIdensity[i][j][k] *= ((HImass/ density[i][j][k]) * kernel
This approach is not working because the HImass values can be 0 which resets the density. I need to look into this calculation more.
• Started looking at compiling hdf5 code on cognac with little success. I don't think my compiler options are correct at the moment, have emailed my supervisor and I am waiting on a reply.
Tuesday
• Moving towards HI gas density
• Rewrote the code to calculate the first part of the gas density, ran the code and got the HI density that isn't scaled or smoothed correctly yet, but shows that the first part of code is indeed working.
• Added in the other code for scaling and smoothing, running at the moment, waiting on results.
Wednesday
• Checked output from the previous day's job, was having problems with float NaN and inf values. Checked my code and realised there were several points where a divide by zero was occurring.
• Seemed to have fixed the problem, ran the code again and the histogram and distribution of data looked correct.
• Running the code another time so that I can check the output of the HIdensity in the .raw file.
• Was ill, so I worked from home.
• Got output from HIdensity calculations, looks correct. Emailed the results to both of my supervisors.
• Cleaned up sections of my code
• Signed up for an OpenMP forum account
• Read about OpenMP, will try parallelizing some of my code and benchmarking tomorrow.
Holidays
Had time off over the christmas holidays, started back on the 6th Jan
Week 4
Wednesday
Started producing cubic images from drishti. Drishti doesn't support the output of .tga files which are necessary for the cube2dome program to convert the cubic images into a fisheye projection.
The command that I use to convert these images is:
sudo sips -s format tga *.png
Then I rename the file extensions to .tga using Automator in osx.
The next step is to convert the cubic images into a set of frames using the cube2dome program.
The following command is what I am using at the moment:
cube2dome -n 1 -vt 90 -w 1024 %c_hidensity%d.tga
Installed the ffmpeg binary on my laptop so that I could convert image files to video.
I then have to convert the images back into png format to work with ffmpeg and rename the file extensions.
sudo sips -s format png *.tga
The last step is to convert the images into a video file using ffmpeg
ffmpeg -f image2 -i c_hidensity%d.png -r 25 -s 1024x1024 output_video.avi
The above command was changed slightly to include the -sameq command which keeps the video quality the same as the input images
The full command is now:
ffmpeg -f image2 -i c_hidensity%d.png -sameq -r 25 -s 1024x1024 output_video.avi* First output video has been created and is viewable here - http://www.youtube.com/watch?v=Ppa5El4-8b0 as you can see there are some problems with the transparency differences between several images. I am unsure if this is because of the image conversions or is a different problem. This problem is also visible in the individual fisheye frames as well.* Still working on the problem with differing transparencies of images.* Figured out how to batch render in drishti, also to change the naming convention from image1.png image2.png to image0001.png image0002.png the following filename is used:
imagename$4f.png* Downloaded graphic converter to speed up the image conversion process.* Created a cubic image set of a longer video (221 frames)* Successfully researched how to tile volumes in Drishti
Thursday
• After being frustrated by the number of steps required to convert the .png cubic images to a fisheye projection video, I experimented with automator tasks and while they did work, most of the time they were slow, buggy and didn't show any kind of output such as percent completed.
• Created a shell script to automate the conversion process. Fixed a few bugs with it. Also might be able to update part of it to avoid some of the extra steps, but here it is in its current form:
cd ~/HIDensity/Converted
#convert to tga
sudo sips -s format tga *.png
#rename files
for f in *.png; do
base=`basename $f .png`
mv $f $base.tga
done
echo "Converted to tga..."
#convert to fisheye
cube2dome -n 1 -vt 90 -w 1024 %c_image%04d.tga
echo "cube2dome finished processing..."
#Delete everything but the converted files
shopt -s extglob
sudo echo !(*c_*) | xargs rm
echo "Deleted extra files..."
#convert back to png
sudo sips -s format png *.tga
#rename files
for f in *.tga; do
base=`basename $f .tga`
mv $f $base.png
done
echo "converted back to png..."
#combine into a video file
ffmpeg -f image2 -i c_image%04d.png -sameq -r 25 -s 1024x1024 output_video.avi
echo "done!"
• Another problem that I have to work on is that when the volume is being loaded into drishti, I have to change my scaling from 0..1 to 0.. 1 - 1/nvol. This problem leads to seams when tiling volumes.
• Still getting the transparency / lighting bug even with shadows and lighting turned off. Email has been sent to Ajay regarding a possible bug with Drishti.
• Tried to fix the seams problem by scaling the volume by n / n-1
Did not work as expected because drishti translates by relative coordinates. So if two bricks have a gap between them and are positioned at (0,0,0) and (1,0,0) and are scaled by (n/n-1, n/n-1, n/n-1), the translation changes so that they still have a gap. The only possible solution to this problem is not a nice solution but should work. Instead of scaling the volumes which does nothing to the end product, change the translation so that the volumes match up. Because a translation of 1 translates by the entire volume, a translation of 1 - (1/n) moves the volume by one voxel.
The problem with this solution is that it has to be applied to every translation. For example the original volume is located at (0,0,0), the second volume at (0.99609375, 0, 0) the third volume is located at (1.9921875, 0, 0) and so on. Although this solution is not optimal, it may be my only option and will only have to be done once per tiled volume.
Friday
• Progress report at AARC building
Week 5
Monday
• Had a positive result with the fisheye projection, no difference in lighting between the images for an axis aligned camera translation but there appears to be image stitching errors. Will have to run more tests.
• Higher resolution simulations of both the total Hydrogen densities and the H1 gas densities look much clearer and better. The higher resolution volumes are 512x512x512 and are ~540MB. I do want to try an even higher resolution such as 1024x1024x1024 but this size resolution has a total file size of about 4.3GB which means I would need double that to run my program. Not sure if this will cause problems but I estimate the time of execution would leap to more than 20 hours.
• TODO - Post side by side image comparisons of different resolutions. UPDATE - In Tomorrow's report.
• Created better graph representations of the smoothing kernel function
• Creating a fisheye projection appear to work now. Not sure really what changed apart from turning off depthcues.
• TODO - Post links to images / video of fisheye projections
Tuesday
• More pretty pictures. This one is showing the difference in resolution and size between a 256^3 and 512^3 volume.
• STILL having intermittent problems with seams / stitching / transparencies from cubemaps exported from Drishti.
• Created a command line python tool to take the 6 image faces and combine them into a skybox / cubemap format.
• When calling the utility the format is as follows
%c_image0001.png for images named in the convention f_image0001.png, l_image0001.png etc
Source Code:
import Image
import sys, os
list = {0:"f", 1:"l", 2:"r", 3:"b", 4:"d", 5:"t"}
filenames=[]
for infile in (sys.argv[1:]):
for a in range(0, 6):
filenames.append(infile.replace('%c', list[a]))
#open images
frontimage = Image.open(filenames[0])
leftimage = Image.open(filenames[1])
rightimage = Image.open(filenames[2])
backimage = Image.open(filenames[3])
downimage = Image.open(filenames[4])
topimage = Image.open(filenames[5])
#get dimensions of image
dimensions = frontimage.size
#Create new image + past old images into it
newimage = Image.new(frontimage.mode,(dimensions[0]*4, dimensions[1]*3), "white")
newimage.paste(frontimage, frontimage.size)
newimage.paste(leftimage , (0, frontimage.size[1]))
newimage.paste(topimage , (frontimage.size[0], 0))
newimage.paste(downimage , (frontimage.size[0], frontimage.size[1]*2))
newimage.paste(rightimage, (frontimage.size[0]*2, frontimage.size[1]))
newimage.paste(backimage , (frontimage.size[0]*3, frontimage.size[1]))
newimage.show()
• Python seems to be very easy to use for image operations.
• Code isn't perfect, but it gets the job done.
• Sample Output:
• Working on getting other values output to a volume. At the moment I am trying to create a volume that shows the velocity of points instead of the densities. This will mean that I can have multiple volumes loaded in Drishti, showing total gas density, h1 gas density, and velocity. This requires me to add code to my existing source code. About half way through at the moment. Hopefully should have some results by tomorrow.
• UPDATE - Got first results back. Structure looks slightly different and looks quite interesting.
Here is a picture of the volume showing velocity.
Wednesday
Wednesday
• New version of Drishti has been released which supposedly fixes the previous problems with lighting etc. In the new version lighting and depth cues work with cubic image creation.
• Fisheye projections work. The next step is to create spherical projections that will allow the viewer to look in any direction in the movie. I will be using the tool cube2sphere written by paul bourke to covert the raw cube face images (f_image0001.tga, r_image0001.tga) etc into a single spherical projection frame. Apart from the difference to cube2dome, cube2sphere doesn't process multiple frames at a time only a single image. I will have to create a script that will feed the program the separate image faces for each frame.
• Size of projection for the WASP dome is 4K.
• Here is an example of what the cube2sphere program does.
• Should have some early results by the end of the day / tomorrow.
• UPDATE: here is the preliminary script used to feed the filenames to the cube2sphere program
import os
filelist=[]
for filename in os.listdir("/Users/dan/HIDensity/Converted/"):
basename, extension = filename.split('.')
if extension == "tga":
basetype, generalname = filename.split('_')
if basetype == "f": #so we only get one list
start = "%c_"
generalname = start + generalname
filelist.append(generalname)
for a in filelist: #call the conversion
os.system("cube2sphere " + a)
• UPDATE - The following is an updated version of the above code that prints its progress. The script has to be in the same folder that it is operating on. Should probably also fix the hardcoded directory, but this is ok for now.
import os
filelist=[]
count = 0
totalcount=0
for filename in os.listdir("/Users/dan/HIDensity/Converted/"):
print "Processing file " + str(count)
basename, extension = filename.split('.')
if extension == "tga":
basetype, generalname = filename.split('_')
if basetype == "f": #so we only get one list
start = "%c_"
generalname = start + generalname
filelist.append(generalname)
count = count + 1
print "Found " + str(count) + " files"
totalcount = count
count = 0
for a in filelist: #call the conversion
count = count + 1
print "Converting file " + str(count) + " of " + str(totalcount)
os.system("cube2sphere " + "-w 4096 -h 2048 " + a)
• and here is the modified bash script that takes care of all the conversions automatically
• Haven't tried this script in it's entirety yet so I am testing it now.
cd ~/HIDensity/Converted
#convert to tga
sudo sips -s format tga *.png
#rename files
for f in *.png; do
base=`basename $f .png`
mv $f $base.tga
done
echo "Converted to tga..."
#convert to spherical projection
python2.6 ../convert2sphere.py
echo "cube2dome finished processing..."
#Delete everything but the converted files
shopt -s extglob
sudo echo !(*c_*) | xargs rm
echo "Deleted extra files..."
#convert back to png
sudo sips -s format png *.tga
#rename files
for f in *.tga; do
base=`basename $f .tga`
mv $f $base.png
done
echo "converted back to png..."
#combine into a video file
#ffmpeg -f image2 -i c_image%04d.png -sameq -r 25 -s 4096x4096 output_video.avi
echo "done!"
Thursday
Thursday
• Might be an issue with calculating the HI densities. The following equation is what I have been using:
• The E^3 might not have to be there given that the smoothing kernel has this term as well. I am currently calculating a new volume with the following formula
• Here are the results of the most recent calculation. Hasn't changed the smoothing factor much but it has changed the density of the volume.
• Have to start thinking about creating an version with timesteps. Will probably have to use some sort of copy script for Cognac.
• There are ~900 different timesteps.
A single 256^3 volume takes approx 64MB on disk.
Therefore the total output at 256x256x256 is ~55-60GiB
A single 512^3 volume takes approx 512MB on disk.
Therefore the total output at 512x512x512 is ~460GiB
This means that the animation will have to be batch converted and converted to a movie file in parts to avoid using terabytes of disk space consisting of image cube faces.
• Look into h264 movie compression. Can the movies just be joined simply like avi files?
E.g. cat movie1.mov movie2.mov > newmovie.mov
• UPDATE: cat movie1.mov movie2.mov > newmovie.mov does not work for h264 files. Might be another way to do this maybe with ffmpeg
• The original formula that I had was correct and I have changed my code to reflect this.
• Need to think more about the animation stuff, will update more tomorrow.
Friday
Friday
Ok, the plan for calculating the animation is as follows.
• There are ~950 hdf5 files stored on pbstore.
• I will create a script that spawns copyq scripts
• Each copyq script will copy an hdf5 file from pbstore then spawn a script in the normal queue that will perform the volumetric calculations
• Once this script has completed, it will copy the result file back to pbstore.
• Deletion of the hdf5 file will not be necessary because it will be stored in JOBFS
• The animation will probably be from a fixed point with a rotation about half way through (paused) and then continue the animation from the new viewpoint. (Possibly a fly through?)
Potential Problems:
• Copying files, I can only copy some at a time. The copyq might be smart enough to schedule this correctly but I will have to ask about this so I don't end up filling up Cognac with hdf5 files.
• Importing into drishti - don't really want to import 950 separate raw files into drishti and convert them by hand. Should be a command line way to do this.
• Although drishti does support importing a time series, I will have to check how robust this feature is and how it will handle huge amounts of data.
• Rendering in drishti - is it possible to load in a volume, set the camera position and rotation and render a frame using only command line arguments?
• Storage requirements - Need to store in excess of 500GiB of data. External hard drives are usually too slow, might have to consider using an internal hard drive and swapping it in / out of computers that I will be using. Hard drive will need to be about 1TiB at least.
• Don't even want to think about the size of exporting these as a spherical projection at the moment.
I am sure that there will be other problems, and I will add them as I think of them.
For now, here is a new picture representing the total gas densities, because I realized that I didn't have a single image that summed up exactly what I was doing.
• EDIT - Windows and OSX are messing with my head. Windows uses the old 'standard' way of reporting file sizes E.g. 1024MB = 1GB.
• As of Snow Leopard, OSX has changed to the metric system of reporting file sizes. E.g. 1000MB = 1GB
• In the new system, GB suffix should indicate 1000MB and GiB should indicate 1024MB, but both systems still use GB instead of what they should be. Very frustrating.
Week 6
Monday
Monday
• Nothing visual to show for today. I have been working on my final report working my way towards my first draft.
• Started messing around with POVray, batch rendering the volume on Cognac. Haven't looked into it too much.
• Have a potential fix for my potential file copying problems. By monitoring how many jobs I have on the queue at any one time, I can limit the amount of jobs and therefore how much storage space I am using on Cognac. Checking when a job finishes and starting a new one automatically should be able to avoid problems. There will be more on my methods etc. when I actually get to this problem.
Tuesday
Tuesday
• Saw my supervisor Alan Duffy, turns out I haven't been getting emails from him because my university decided to stealthily change the email addresses we are using and not send anything to the old addresses.
• In terms of copying files
Trial version can convert no more than 30000 symbols.
Get the full featured version!
Get the full featured version!
