I am attempting to update Bioconductor within my RStudio environment. When I do so, it gives the following warning message at the end:

1. In install.packages(update[instlib == l, "Package"], l, repos = repos,  :
2.   installation of package ‘coin’ had non-zero exit status

I am running coin v 1.1-3. 

I downloaded the new version of coin from RForge, attempted to install it from source and get the following:

1. * installing *source* package ‘coin’ ...
2. ** package ‘coin’ successfully unpacked and MD5 sums checked
3. ** libs
4. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c Classes.c -o Classes.o
5. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c Helpers.c -o Helpers.o
6. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c LinearStatistic.c -o LinearStatistic.o
7. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c StreitbergRoehmel.c -o StreitbergRoehmel.o
8. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c coin-init.c -o coin-init.o
9. clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include   -fPIC  -Wall -g -O2  -c vandeWiel.c -o vandeWiel.o
10. clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o coin.so Classes.o Helpers.o LinearStatistic.o StreitbergRoehmel.o coin-init.o vandeWiel.o -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0 -L/usr/local/gfortran/lib -lgfortran -lquadmath -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
11. ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
12. ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
13. ld: library not found for -lgfortran
14. clang: error: linker command failed with exit code 1 (use -v to see invocation)
15. make: *** [coin.so] Error 1
16. ERROR: compilation failed for package ‘coin’
17. * removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/coin’
18. * restoring previous ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/coin’
19. Warning in install.packages :
20.   installation of package ‘/Users/vicki/Downloads/coin’ had non-zero exit status

Here is the sessionInfo:

1. sessionInfo()
2. R version 3.4.0 (2017-04-21)
3. Platform: x86_64-apple-darwin15.6.0 (64-bit)
4. Running under: macOS Sierra 10.12.5
5.  
6. Matrix products: default
7. BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
8. LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
9.  
10. locale:
11. [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
12.  
13. attached base packages:
14. [1] stats     graphics  grDevices utils     datasets  methods   base    
15.  
16. other attached packages:
17. [1] BiocInstaller_1.26.0
18.  
19. loaded via a namespace (and not attached):
20. [1] compiler_3.4.0 tools_3.4.0  

What am I doing wrong? Please let me know if there is other information you need.

It looks like you're missing the fortran libraries needed to compile this package.  I don't have a Mac to test this with, but I would download and install GFortran 6.1 from https://gcc.gnu.org/wiki/GFortranBinaries#MacOS 

A direct link to the version recommended for R-3.4.0 is http://coudert.name/software/gfortran-6.1-ElCapitan.dmg

R has a number of dependencies that are sometimes difficult to satisfy on macOS. If possible, you might consider installing R using homebrew (https://brew.sh/) by first installing homebrew following the instructions on the site and then doing the following from terminal to install R.

brew update
brew upgrade
brew tap homebrew/science
brew install r

This will install R and all dependencies (e.g. GFortran), RStudio will be able to find and use them, and then you will be able to install coin using the standard procedure.