Hi there

I recently tried to install Rsubread on Cygwin. I did that a few months ago successfully and now I tried it with the same steps as last time (I wrote quite detailed instructions last time). Of course, the machine is different - last time I had Win7, R 3.1.3 installed in CYGWIN_NT-6.1 and RSubread 1.16.1. But anyway - I expect it to work:

The current setup is a Win 2012 R2 server, R 3.2.4 installed in CYGWIN_NT-6.3 and RSubread 1.20.6. The problem is the following:

> biocLite('Rsubread')
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.2 (BiocInstaller 1.20.3), R 3.2.4 Revised (2016-03-16
  r70336).
Installing package(s) ‘Rsubread’
trying URL 'https://bioconductor.org/packages/3.2/bioc/src/contrib/Rsubread_1.20.6.tar.gz'
Content type 'application/x-gzip' length 9247359 bytes (8.8 MB)
==================================================
downloaded 8.8 MB
* installing *source* package ‘Rsubread’ ...
@@@@@ The operating system is CYGWIN_NT-6.3.
** libs
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D MACOS  -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c HelperFunctions.c -o HelperFunctions.o
HelperFunctions.c:30:24: fatal error: sys/sysctl.h: No such file or directory
compilation terminated.
make: *** [/usr/lib/R/etc/Makeconf:134: HelperFunctions.o] Error 1
ERROR: compilation failed for package ‘Rsubread’
* removing ‘/usr/lib/R/site-library/Rsubread’

I'm not sure, but sysctl.h should actually not even be included on Cygwin. The following is the include section in HelperFunctions.c:

#include <ctype.h>
#include <string.h>
#include <assert.h>

#ifdef MACOS

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <sys/sysctl.h>
#include <net/if.h>
#include <net/if_dl.h>
#include <netinet/in.h>
#include <arpa/inet.h>

#else

#include <sys/ioctl.h>
#include <net/if.h>
#include <unistd.h>
#include <netinet/in.h>
#endif

#include "subread.h"
#include "gene-algorithms.h"
#include "HelperFunctions.h"

As far as I understand it, sysctl.h should only be included if MACOS is defined - which was indeed the case (see above). However, that would mean that it used Makevars.MacOS or Makevars.SunOS... Why? It does recognize the Cygwin. Or do I miss something? Could that be fixed?

(note aside - what would be very obscure though if it could influence it: I access the machine with Microsoft Remote Desktop from a Mac... The old system ran instead on a Virtualbox - also on the Mac)

Thanks and best regards,

Marc

> sessionInfo()
R version 3.2.4 Revised (2016-03-16 r70336)
Platform: x86_64-unknown-cygwin (64-bit)
Running under: CYGWIN_NT-6.3 WINDOWS-P8HLEBT 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64 Cygwin

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

Ok - I know why it uses MACOS - the fall-through in "configure" is MacOS:

#!/bin/sh
#cp src/Makevars.in src/Makevars
UNAME=`uname`
echo @@@@@ The operating system is $UNAME.
version_no=`cat DESCRIPTION |grep Version|cut -f2 -d' '`
echo SUBREAD_VER = \"\\\"Rsubread $version_no\\\"\" > src/Makevars
if test  $UNAME = "Linux"
then
    # This is a Linux machine!
    cat src/Makevars.Linux >> src/Makevars
elif test $UNAME = "FreeBSD"
then
    cat src/Makevars.FreeBSD >> src/Makevars
elif test $UNAME = "SunOS"
then
    cat src/Makevars.SunOS >> src/Makevars
else
    cat src/Makevars.MacOS >> src/Makevars
fi

so adding something like

elif [[ $UNAME == CYGWIN* ]]
then
    cat src/Makevars.Linux >> src/Makevars

works to avoid the error above.

New problem:

> install.packages("/home/Administrator/Rsubread_1.20.6_fixed.tar.gz")
Installing package into ‘/usr/lib/R/site-library’
(as ‘lib’ is unspecified)
inferring 'repos = NULL' from 'pkgs'
* installing *source* package ‘Rsubread’ ...
@@@@@ The operating system is CYGWIN_NT-6.3.
** libs
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c HelperFunctions.c -o HelperFunctions.o
HelperFunctions.c: In function ‘mac_str’:
HelperFunctions.c:825:16: warning: implicit declaration of function ‘socket’ [-Wimplicit-function-declaration]
     int sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
                ^
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c R_wrapper.c -o R_wrapper.o
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c SNPCalling.c -o SNPCalling.o
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c aligner.c -o aligner.o
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c atgcContent.c -o atgcContent.o
gcc -I/usr/lib/R/include -DNDEBUG     -DMAKE_FOR_EXON -D SUBREAD_VERSION="\"Rsubread 1.20.6\""   -ggdb -O2 -pipe -Wimplicit-function-declaration -std=gnu99 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/build=/usr/src/debug/R-3.2.4-1 -fdebug-prefix-map=/cygdrive/e/cyg_pub/devel/R/R-3.2.4-1.x86_64/src/R-revised=/usr/src/debug/R-3.2.4-1  -c build-sam-index.c -o build-sam-index.o
In file included from build-sam-index.c:20:0:
build-sam-index.c: In function ‘transfer_SAM_to_position_table’:
build-sam-index.c:369:6: error: request for member ‘_flags’ in something not a structure or union
   if(feof(input_file.input_fp))break;
      ^
make: *** [/usr/lib/R/etc/Makeconf:134: build-sam-index.o] Error 1
ERROR: compilation failed for package ‘Rsubread’

Hm - replacing

if(feof(input_file.input_fp))break;

in build-sam-index.c with

if(feof((FILE *)input_file.input_fp))break;

brought me now through the compilation.

Tested it briefly - it works, but it's very slow compared to the old version of Rsubread (1.16). It seems not to be able to use several CPUs at once - might be due to on-the-fly decompression (in the past it decompressed the file first and then aligned all the reads) - ?

The old version (1.6) used spin lock to synchronize multiple threads whereas the new versions use pthread_mutex lock. We made this change because spin lock does not work on computers with only one CPU.

Mapping speed is not affected by the choice of locks in our tests. But we did not test it on Cygwin. Maybe pthread_mutex does not work well with Cygwin? Or maybe it is because of your Windows OS upgrade and/or Cygwin upgrade?

Thanks for the hint - I will have a look whether I find a solution for this. Searching for pthread mutex and Cygwin looks like there were several issues in the past. I will post an update if I can solve it in some way.

Ok - I tried now to fall back to pthread_spin_XXX instead of pthread_mutex_XXX. I did this with once with erasing #define USE_POSIX_MUTEX_LOCK and another time with #undef USE_POSIX_MUTEX_LOCK just before the if statement for the switch (in subread.h). Assuming that this is enough (is it?), there is no change - Rsubread still uses only one core (by the way - both interfaces should be supported by Cygwin).

Aside - I tested Rsubread 1.16.1 now on this machine as well to verify that it's related to the new version of Rsubread itself. The old version does - with both, the linux and the macos makevars. It uses as many cores as specified. Any other idea?
 

In the newer version of Rsubread, we implement a more efficient approach to read in reads from a fastq file. Only one file pointer is generated for a fastq file and all threads share this file pointer.

In Rsubread 1.16.1, multiple file pointers were generated for the same fastq file and each thread had its own file pointer. This is a less efficient approach due to the overhead cost of managing multiple pointers.

Although you observed that the older version used more CPU resources, it actually used more time to read in the reads.

So the difference you observed is indeed due to the change in Rsubread and this is irrespective of whether you used spin lock or mutex. We suggest you always using the latest version.

We used Cygwin v10.0 and Windows 10 to investigate the differences between Rsubread versions.