Question

Using AMI with StarCluster - error authenticating

0

Entering edit mode

spollack • 0

@spollack-12387

Last seen 6.2 years ago

Biostatistics and Computational Biology…

Dear Bioconductor,

I am attempting to follow the instructions here https://www.bioconductor.org/help/bioconductor-cloud-ami/#configuring_starcluster to configure a StarCluster for running the Bioconductor AMI as a cluster.

But I am getting an error when I try to start a cluster. The error is a Python stack trace which seems to indicate AWS cloud refused to authenticate my key. I tried creating a key with "starcluster createkey" and the same error turned up.

(I couldn't create a key, so I tried using my regular AWS key and neither seems to work.)

Is this a StarCluster problem? Should I send them a question? Am I think only one it happens to? Do you still recommend using StarCluster? Is it still an active project? (The fact that I had to install Python 2.7 to build it is not encouraging.) Is there a newer better way to do this?

thanks,
- Sam

AMI • 2.5k views

ADD COMMENT • link updated 7.8 years ago by Dan Tenenbaum ★ 8.2k • written 7.8 years ago by spollack • 0

score 0 · Answer 1 · 2017-02-17

0

Entering edit mode

nitesh.turaga ▴ 140

@niteshturaga-8667

Last seen 3.0 years ago

United States

Hi Sam,

I suspect a configuration error, could you post the stack trace or your starcluster config file (without AWS secret/access keys).

ADD COMMENT • link 7.8 years ago nitesh.turaga ▴ 140

0

Entering edit mode

Sorry, just saw this. I got the AMI to work with the latest StarCluster version 0.95.6 yesterday.

But now I have a problem with connecting to Rstudio Server.

I can connect to rstudio server on a slave node just fine, but I can't connect to it on the master node. On the master node, I get the login screen and I enter ubuntu/bioc and it spins its wheel for a bit, then it says "Studio initialization error" and "Unable to connect to service."

So I logged into the master node and tried to restart. But I am getting an error with the /etc/rstudio/rserver.conf file. When I do "rstudio-server verify-installation" it says:

"Error reading /etc/rstudio/rserver.conf: option 'path' cannot be specified more than once"

The issue is that there are multiple identical lines in /etc/rstudio/rserver.conf that look like this:

rsession-path=/usr/local/bin/setup_r_mpi.sh

(It looks like every time I attempt to connect, a few more of these lines show up.)

Do you have any insight on this?

thanks,
- Sam

ADD REPLY • link 7.8 years ago spollack • 0

score 0 · Answer 2 · 2017-02-17

0

Entering edit mode

Dan Tenenbaum ★ 8.2k

@dan-tenenbaum-4256

Last seen 6 months ago

United States

StarCluster is a little old. Look info CfnCluster: https://github.com/awslabs/cfncluster .

ADD COMMENT • link 7.8 years ago Dan Tenenbaum ★ 8.2k

0

Entering edit mode

Thanks for the advice. I tried to build a cluster with CfnCluster and the Bioconductor AMI but I never did get it to work. My best attempt was a config file with this:

master_instance_type = t2.large
slave_instance_type = t2.large
master_root_volume_size = 500
slave_root_volume_size = 500
base_os = ubuntu1404
custom_ami = ami-abd0b3bc

But something is still generating an error when the master node is instantiated. CfnCluster fills in values to a four-thousand-line CloudFormation template. Somewhere in that template is the reason my master node is not instantiating. I sure would like to get it to work, though, because it looks like StarCluster is kind of moribund.

ADD REPLY • link 7.8 years ago spollack • 0

0

Entering edit mode

Since the last time I contributed to this thread, I've become aware of AWS Batch. You might want to look at that.

Whatever you do, I'd focus on keeping things simple. If you try to get CfnCluster to work, I'd start by just trying to get a simple cluster to work (with a basic AMI like one of the Alestic Ubuntu 16.04 AMIs). Then maybe install Docker on the AMI and run your job as a simple batch command using the Bioconductor Docker containers.

ADD REPLY • link 7.8 years ago Dan Tenenbaum ★ 8.2k

0

Entering edit mode

OK, thanks, good idea (Bioconductor Docker), I'll try that.
- S

ADD REPLY • link 7.8 years ago spollack • 0

0

Entering edit mode

We are finding that cfncluster creation of clusters in EC2 is extremely slow, relative to the solution available a few years through starcluster. Any other approaches in use in our community?

ADD REPLY • link 7.8 years ago Vincent J. Carey, Jr. 6.7k