Hi all, 

I came here to the support site prior to bothering bioc-devel, but searching for "circular dependencies" cf. https://github.com/Bioconductor/Contributions/blob/master/CONTRIBUTING.md#submitting-related-packages did not surface any answers that seemed relevant.  This seems like it might be a recurrent issue, though, so if anyone can document best practices for [the following, it would be great]:

1) providing a Data package (let's call it MTseekerData, because that's its name) holding data structures where

2) the S4 class data structures are generated from a package (let's call it MTseeker, because that's its name), and 

3) the package (MTseeker) requires the MTseekerData package and the latter must be submitted as a GitHub issue first, but the package wants to autoload the MTseeker software package as part of its installation (even though I don't import it, to try and avoid this), and thus without some finagling, the full battery of examples, vignette steps, and tests cannot run for the MTseeker software package. 

OK, now off to packages@bioconductor.org and bioc-devel, but hopefully this question can avoid other people doing the same.

Any answers here are much appreciated, and I'll add responses from Lori and/or bioc-devel as they come in.

Thanks all. 

[edited for grammar, edits in brackets]

From bioc-devel, specifically Michael Lawrence and Valerie Obenchain (thanks to both of you! (Note that "SPB" refers to the Single Package Builder setup that allows for the scale of contributions that BioC receives, by decoupling what used to be a nightly mega-build.)  I imagine Lori is pretty busy at the moment but I will append whatever ends up being the definitive answer. So far these are working OK, although I may need to rename the original submission ticket to reflect the new Suggests: / Depends: structure following the below. 

From Michael:

[Suggest] the data package from the software package. This might require some manual intervention on the build system side to ensure that the software package is installed before passing full check, so that the data package can be installed/checked and finally the software package checked. I guess a note about this could be added to the page at the cited URL. 

From Valerie:

[T]he usual case is the data package 'Suggests' the software and the software 'Depends' or 'Imports' the data but this dependency could be reversed. When submitting a combo of packages to the SPB for review, the package that 'Suggests' is submitted first followed by the package that 'Depends'. More on that here:

https://github.com/Bioconductor/Contributions#submitting-related-packages

One additional step taken to transition the packages to the build system after approval is to add a .BBSoptions file in the package that 'Suggests'. In this example, the .BBSoptions would be in the data package and have this one line:

ForceInstall: TRUE

[Also] the .BBSoptions file should be located at the 'top' level of the package, e.g., MTseekerData/.BBSoptions. You might as well add that now instead of waiting until the packages are approved. I think the SPB was modified to be aware of the .BBSoptions file so it may require it too (Lori would know). 

If the SPB does require .BBSoptions we should document that in https://github.com/Bioconductor/Contributions/blob/master/CONTRIBUTING.md

The Single Package Builder  (SPB) automatically installs each package before the build and check stage as part of the process.  Each issue that goes through the SPB for new submits temporarily gets its own directory to use for Rlib, installing itself and referencing any additional dependencies -  This is why we document that packages with additional or supporting packages should be built under the same issue as then they will be able to find each other (avoiding to wait for a package to be accepted, moved to the official repository, and build and check cleanly on the daily builder to be found by BiocManager::install which could be a few days).  We support the 'suggest / depends' circular dependency. In most cases (tho not always) it is a data package suggesting the software package and the software package depending on the data package.  The weaker dependency (the package that suggest the other package - most often the data package) should be submitted first.  The packages will undergo a round of failures and the follow is expected and the general workflow on SPB when dealing with circular dependency (We will update this into the submission documentation so it is more widely known)

 1.   Package A will be submitted and fail because package B is not available (but it will be installed in the Rlibs directory for the issue)

  2.  Package B is submitted and fails because package A wasn't install/built/checked properly (but also installs)
  3.  The maintainer should do an arbitrary version bump on package A to kick off a new build that should now pass 
  4.  Do any additional debugging on package A ensure a clean build and check.
  5.  The maintainer should do an arbitrary version bump on package B, only after package A is building and checking cleanly,
  6.  Review process continues and packages are accepted

Hope this helps and clarifies the submission process.  

The same failure schema will happen on the daily builder when first accepted and added.  The maintainers should approximately a week to let circular dependencies clear up on the daily builders before reaching out on support stie or mailing list.