Hi,

Such big files are insane!

Ok, so I have access to a server with a lot of memory (384GB) so that should be enough. However I started the download of the uniprot_trembl.fasta.gz file (using wget from the Unix command line) and I see an ETA of more than 5h! The download speed I get is only 2 MB/s which is very slow. We have fast internet here at my institution (e.g. I easily see download speeds of 30 MB/s or more when downloading from other places) so it seems that the bottleneck is on the uniprot FTP server side.

We cannot exclude that the file actually contains embedded \0 bytes. Maybe somehow they got introduced when the file was compressed, or the file got corrupted during your download. Do the Uniprot people provide md5sums somewhere for their files so we can check them? This is pretty standard practice for institutions that provide big files for download.

Right now the AA alphabet is not enforced for AAStringSet objects so the readAAStringSet() function will accept any byte value, even the \0 byte. \0 bytes in the file are treated like any other byte value so they would end up in the AAStringSet object. Here is how such an object can be created:

library(Biostrings)
nul <- as(as(raw(1), "XRaw"), "AAString")
aa <- AAStringSet(c(rep(nul, 4), AAString("PPPLK")))

Note that, strictly speaking, there is nothing wrong with the object itself in the sense that you can do most of the usual operations on it:

length(aa)
# [1] 1

width(aa)
# [1] 9

alphabetFrequency(aa)
#      A R N D C Q E G H I L K M F P S T W Y V U O B J Z X * - + . other
# [1,] 0 0 0 0 0 0 0 0 0 0 1 1 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0     4

as(matchPattern("PLK", aa[[1]]), "IRanges")
# IRanges object with 1 range and 0 metadata columns:
#           start       end     width
#       <integer> <integer> <integer>
#   [1]         6         8         3

countPattern(nul, aa[[1]])
# [1] 4

It's just that it cannot be displayed or coerced to an ordinary character vector (the show() method for these objects actually calls as.character() on the parts of the object that are displayed):

as.character(aa)
# Error in .Call2("new_CHARACTER_from_XStringSet", x, xs_dec_lkup(x), PACKAGE = "Biostrings") :
#   embedded nul in string: '\0\0\0PPPLK'

aa
#   A AAStringSet instance of length 1
#     width seq
# Error in XVector:::extract_character_from_XRaw_by_ranges(x, start, width,  : 
#   embedded nul in string: '\0\0\0PPPLK'

If we trim the first 4 bytes, then the object can be displayed:

subseq(aa, start=4)
#   A AAStringSet instance of length 1
#     width seq
# [1]     5 PPPLK

FWIW you can replace the nulls with the letter of your choice with:

chartr(nul, "x", aa)
#   A AAStringSet instance of length 1
#     width seq
# [1]     8 xxxPPPLK

OK so that was only to show you that AAStringSet objects (like BStringSet objects) are actually allowed to contain embedded nuls.

Just to discard the possibility that these nul bytes are an artefact of the compression/decompression mechanism, do you think you can uncompress the file and try to reproduce the error on the uncompressed file? I know that decompressing such a big file is going to require a lot of resources and might take a long time but you seem to have access to some powerful hardware.

Thanks,

H.