Introduction to NeuroStack
February 15, 2022Information
Introduction video to NeuroStack, a tool designed to aid neuroimaging researchers working with AWS cloud computing.
ID7455
To CiteDCA Citation Guide
- 00:00NeuroStack is an AWS CloudFormation-based
- 00:03tool designed to aid neuroimaging researchers
- 00:05working with AWS cloud computing.
- 00:08It is ideal for neuroimaging researchers
- 00:10transitioning their workflows to AWS or
- 00:13anyone working with neuroimaging at scale.
- 00:15By installing NeuroStack
- 00:16into your AWS account,
- 00:18you construct a pipeline of AWS
- 00:20resources specialized for neuroimaging.
- 00:23NeuroStack is built with the NITRC
- 00:25computational environment, or NITRC-CE,
- 00:27allowing users access to all of the
- 00:29programs pre-installed in the NITRC-CE.
- 00:32In this tutorial, we'll go over how to
- 00:35install NeuroStack into your AWS account,
- 00:37and two examples of using NeuroStack
- 00:39using two different software packages,
- 00:41AFNI and FreeSurfer.
- 00:43To begin using NeuroStack,
- 00:45first sign into your AWS account.
- 00:48Navigate to the NeuroStack page on
- 00:51the NITRC website and go to Downloads.
- 00:53The download is a URL that will bring you
- 00:56to the AWS CloudFormation stack page.
- 01:02Once on the Stack Creation page,
- 01:04everything you need is already preset.
- 01:06Just continue choosing next.
- 01:11On the final page, check the box to
- 01:14accept that CloudFormation will create
- 01:16IAM resources before creating the stack.
- 01:19NeuroStack will take several minutes
- 01:21to set up your AWS resources.
- 01:23When it says "Create Complete", it's finished.
- 01:27Let's run a simple job through
- 01:29NeuroStack to see how it works.
- 01:30We're going to convert an MRI file
- 01:32from one file type to another.
- 01:33We're going to use an AFNI program to convert
- 01:36from a NIFTI filetype to an AFNI filetype.
- 01:39The job should only take a few
- 01:40minutes to run, so you can see the
- 01:42process from beginning to end.
- 01:43I've downloaded the template
- 01:45script and a test imaging file from
- 01:47the NeuroStack GitHub Repo.
- 01:48I'll navigate to the AWS S3 console where I
- 01:51can see that NeuroStack has configured 3 S3
- 01:53storage buckets and input,
- 01:55output, and script buckets.
- 02:00I'll upload the script to
- 02:02the NeuroStack script S3 bucket.
- 02:09Let's take a look at the script.
- 02:11The top section contains commands that
- 02:13aren't intended to be altered by the user.
- 02:15The bottom section is meant
- 02:16to be modified by the user.
- 02:18There are three basic parts to the section:
- 02:21copying data from our input S3 bucket,
- 02:24performing an operation on the data,
- 02:26in this case using an AFNI command
- 02:28to convert file types, and copying our
- 02:30processed data to our output S3 bucket.
- 02:33An AFNI file consists of two
- 02:35files a .BRICK and a .HEAD.
- 02:36So we're actually copying 2 files here.
- 02:40The variable aSub is defined as the
- 02:42filename before the first period.
- 02:44So in this example we have
- 02:47a test file called test,
- 02:49which means that aSub
- 02:50will take the value "test".
- 02:54I'll upload the test imaging file to
- 02:56the NeuroStack input bucket. When I
- 02:58upload a file into the input bucket,
- 03:00it will immediately begin processing the file
- 03:02according to my script in the script bucket.
- 03:05It will take several minutes
- 03:07for the job to complete.
- 03:08When it has finished successfully,
- 03:09we will see our processed data in
- 03:11the NeuroStack output S3 bucket.
- 03:14In the meantime,
- 03:15let's watch our job status.
- 03:16We can do that by navigating to
- 03:18the AWS batch console.
- 03:19The job will move from "Submitted" to
- 03:22"Running". When it has finished running,
- 03:24it will either move to "Succeeded" or "Failed".
- 03:35Great, we can see that our job succeeded.
- 03:38We can see more details about
- 03:39our job by clicking on it.
- 03:41And we can see that the job's total
- 03:43running time was 12 seconds,
- 03:44which is the computation
- 03:45time that we're charged for.
- 03:47We can also look at the log of
- 03:49our job under Log Stream Name.
- 03:57And when we navigate to our output
- 03:59S3 bucket, we can see that our
- 04:00two processed files are there.
- 04:04In this example,
- 04:05we've uploaded a single subject file,
- 04:07but we can upload multiple files at
- 04:09a time for simultaneous processing
- 04:11by uploading the files either
- 04:13manually or through the AWS CLI.
- 04:15If you have many files to process,
- 04:17the CLI will be more efficient.
- 04:20Let's use NeuroStack to process
- 04:22the subject through FreeSurfer.
- 04:23To do this, we'll need to change our
- 04:26script in the user modified section.
- 04:28I have the FreeSurfer setup commands.
- 04:30I copied the data from our input S3 bucket.
- 04:33Freesurfer requires a license file to run,
- 04:36so I'll copy my license file as well.
- 04:39I'll then do recon-all, the command
- 04:41to process through FreeSurfer, and once
- 04:43that process is done I'll copy my
- 04:46output files to my output S3 storage.
- 04:48Now that we've modified our script,
- 04:50we can upload our script and the
- 04:52FreeSurfer license file to the
- 04:54script bucket and our test data to
- 04:56our input bucket. Again, by uploading
- 04:58to our input bucket,
- 04:59we started our process running.
- 05:02FreeSurfer will take several hours to run.
- 05:11It's the next day and I can see our
- 05:14FreeSurfer output in the output S3 bucket.
- 05:16I can go to the Batch dashboard and see
- 05:18that FreeSurfer took six hours to run.
- 05:21We hope you will find NeuroStack useful
- 05:23for your neuroimaging work on AWS.
- 05:25If you have questions about
- 05:26how to use NeuroStack,
- 05:27please visit the NeuroStack Forum
- 05:29on the NITRC website. Thank you.