Question

diffHic presplit_map.py pysam.SamtoolsError

0

Entering edit mode

shopnil99 • 0

@shopnil99-20966

Last seen 5.4 years ago

Hi, I am running the presplit_map.py script on a SLURM cluster on a small dataset (MboI restriction enzyme used).

here's the command I am running:

python presplit_map.py -G ~/../Bowtie2Index/hg19 -1 ~/../fastq/HIC003_R1.fastq.gz -2 ~/../fastq/HIC003_R2.fastq.gz --sig GATCGATC -o HIC003.bam

I am using python-2.7.10, cutadapt-1.8.3 and bowtie2/2.2.4 versions

Once the script starts running, a temporary folder is generated and a HIC003_R1.fastq.bam file is created, after which the job fails and gives me the following error:

 [E::hts_open] fail to open file './tmpnA4PdG/sorted.bam'
Traceback (most recent call last):
  File "presplit_map.py", line 242, in <module>
    pysam.sort("-o", bsorted, "-n", outbam)
  File "/gpfs0/export/opt/anaconda-2.3.0/lib/python2.7/site-packages/pysam/__init__.py", line 79, in __call__
    (retval, "\n".join(stderr)))
pysam.SamtoolsError: 'csamtools returned with error 1: [bam_sort_core] fail to open file ./tmpnA4PdG/sorted.bam\n'

I tried googling this error message and did not find any good solutions. Any idea what may be going wrong in this case?

thanks, - Iftekhar

diffHic pysam.SamtoolsError pysam.sort • 1.2k views

ADD COMMENT • link 5.4 years ago shopnil99 • 0

score 1 · Accepted Answer · 2019-07-15

1

Entering edit mode

Aaron Lun ★ 28k

@alun

Last seen 19 minutes ago

The city by the bay

I'm a bit bemused as well. Let's try to localize the error:

Is HIC003_R1.fastq.bam a valid BAM file? Try inspecting it with samtools.
If you opened up the Python interpreter, what happens when you do:

import pysam
tmpdir="tmpnA4PdG"
outbam=os.path.join(tmpdir, "HIC003_R1.fastq.bam")
bsorted=os.path.join(tmpdir, "sorted.bam")
pysam.sort("-o", bsorted, "-n", outbam)

My suspicion is that you need a newer version of pysam. I vaguely recall that pysam switched the samtools backend some time ago, and an old version of pysam might be treating the output BAM file path as the prefix instead of the full file name.

ADD COMMENT • link 5.4 years ago Aaron Lun ★ 28k

0

Entering edit mode

Hi Aaron,

The Bam file HIC003_R1.fastq.bam looks ok:

M00336:181:000000000-A29H6:1:1101:17139:1645 2129 chr17 62341747 42 80M20H * 0 0 HGFHHGCFDAGBHGG2HGFHHHFFHFHGFACGGFFDGGHHHHHHHGHHFGG AS:i:-8 XN:i:0 XM:i:2 XO:i:0 XG:i:0 NM:i:2 MD:Z:0A50C2 M00336:181:000000000-A29H6:1:1101:14932:1669 81 chr1 245889622 42 8H92M * 0 0 11E?1GFHGBEGHBHHHGHHEGGGGGFCHGHGECEGHEGHF2HGHFHHHGEGGGGGGGGFFFFFFF>ABAA AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM: M00336:181:000000000-A29H6:1:1101:14932:1669 2129 chr20 683971 1 8M92H * 0 0 CAC M00336:181:000000000-A29H6:1:1101:14615:1679 81 chr7 17169791 42 38H62M * 0 0 A@DBA?AA3 AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:62 YT:Z:UU M00336:181:000000000-A29H6:1:1101:14615:1679 2129 chr7 17176069 42 38M62H * 0 0 0 NM:i:0 MD:Z:38 YT:Z:UU M00336:181:000000000-A29H6:1:1101:13929:1684 81 chr20 11892330 42 63H37M * 0 0 0 NM:i:0 MD:Z:37 YT:Z:UU M00336:181:000000000-A29H6:1:1101:13929:1684 2113 chr6 51812440 42 37H63M * 0 0 G>BGGGHHGH AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:63 YT:Z:UU M00336:181:000000000-A29H6:1:1101:15728:1726 65 chr6 90191921 42 70M30H * 0 0 FGFFFGBF1GFHBFFH/BEABEEGC AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:70 YT:Z:UU

I ran the following script on the Python interpreter:

import os import pysam tmpdir="tmpnA4PdG" outbam=os.path.join(tmpdir, "HIC003_R1.fastq.bam") bsorted=os.path.join(tmpdir, "sorted.bam") pysam.sort("-o", bsorted, "-n", outbam)

And ended up getting a similar error message as before:

[E::htsopen] fail to open file 'tmpnA4PdG/sorted.bam' Traceback (most recent call last): File "pysamtroubleshoot.py", line 6, in <module> pysam.sort("-o", bsorted, "-n", outbam) File "/gpfs0/export/opt/anaconda-2.3.0/lib/python2.7/site-packages/pysam/init.py", line 79, in call (retval, "\n".join(stderr))) pysam.SamtoolsError: 'csamtools returned with error 1: [bamsortcore] fail to open file tmpnA4PdG/sorted.bam\n'

I am using pysam 0.8.4 with python-2.7.10 in our cluster. I also tried using python-3.6.5 which comes with pysam 0.15.1, but it gives me a different error message and no bam files are created in this case:

Traceback (most recent call last): File "presplitmap.py", line 116, in <module> raise SystemError("cutadapt failed to trim reads\n"+cuterr) TypeError: must be str, not bytes

I ended up switching the cutadapt version from cutadapt-1.8.3 to cutadapt-2.3 and it looks like that solved the issue.

thank you, - Iftekhar

ADD REPLY • link 5.4 years ago shopnil99 • 0

score 1 · Accepted Answer · 2019-07-16

I originally got this error message while running presplit_map.py using python-2.7.10 (with pysam 0.8.4), cutadapt-1.8.3 and bowtie2-2.2.4

[E::htsopen] fail to open file './tmpnA4PdG/sorted.bam' Traceback (most recent call last): File "presplitmap.py", line 242, in <module> pysam.sort("-o", bsorted, "-n", outbam) File "/gpfs0/export/opt/anaconda-2.3.0/lib/python2.7/site-packages/pysam/init.py", line 79, in call (retval, "\n".join(stderr))) pysam.SamtoolsError: 'csamtools returned with error 1: [bamsortcore] fail to open file ./tmpnA4PdG/sorted.bam\n'

Once I switched to python-3.6.5 (with pysam 0.15.1), cutadapt- 1.8.3 and bowtie2-2.2.4 I got another error:

Traceback (most recent call last): File "presplitmap.py", line 116, in <module> raise SystemError("cutadapt failed to trim reads\n"+cuterr) TypeError: must be str, not bytes

Finally, switching to python-3.6.5 (pysam 0.15.1) and cutadapt-2.3 from python-2.7.10(pysam 0.8.4) and cutadapt-1.8.3 solved the issue.