Justin, Thanks for your patience. I've fixed this and pushed an update to Bioconductor, which should be live tomorrow. Thanks again, Kipper On Thu 19 Jun 2014 06:41:04 PM EDT, Justin Meskas wrote: > Hi Kipper, > > Thank you for your quick reply. I had previously tried flowClean on uncompensated and untransformed data, and the same error was given. > > Thanks, > Justin > > ________________________________________ > From: Kipper Fletez-Brant [cafletezbrant at gmail.com] > Sent: June 19, 2014 3:26 PM > To: Justin Meskas > Cc: Ryan Brinkman; bioconductor at r-project.org > Subject: Re: flowClean > > Hi Justin, > > I'm cc'ing the Bioc list so that others having this problem might see this. > > I have seen this error before. While it will take some time to diagnose, one comment I have is that we developed this algorithm to operate on uncompensated data. Try using flowClean before you do any other processing, and let me know how that works. > > Thanks, > Kipper > > On Jun 19, 2014 5:42 PM, "Justin Meskas" <jmeskas at="" bccrc.ca<mailto:jmeskas="" at="" bccrc.ca="">> wrote: > Hello Kipper, > > I am in Dr. Ryan Brinkman's lab and I was trying to use your flowClean package. I have downloaded version 1.0.0 from Bioconductor and I received a similar error for many of my FCS files. I have attached one FCS file and the code I used to get the error. The error was the following: > > Error in if (x == 0) { : missing value where TRUE/FALSE needed > > I do not know if there is a problem with my data, or if I am using the data incorrectly with flowClean, but if there is any help you could provide me that would be greatly appreciated. > > Thank you, > Justin

Hi Kipper, Sorry to keep emailing you, but I had another question about flowClean. I have been noticing that the clean function seems to label the first compartment for removal every time. This seems odd to me. I attached two figures. The figure called "A..." looks like most other figures, where the first compartment is labelled for removal. And the other figure, called "B...", is my unique case where, I believe anyway, the first compartment should be removed, but not the second. Are all these files somehow accidentally removing the first compartment? Or do you think it is the case that all these files have bad data at the beginning? Thank you, Justin ________________________________________ From: Kipper Fletez-Brant [cafletezbrant@gmail.com] Sent: June 23, 2014 4:38 PM To: Justin Meskas Cc: Ryan Brinkman; bioconductor at r-project.org Subject: Re: flowClean Great, thanks for the update! On Mon 23 Jun 2014 05:26:21 PM EDT, Justin Meskas wrote: > Hi Kipper, > > I downloaded your update and I ran "clean" on all my data for my current project and everything came out with no errors. Thank you very much for your quick fix. > > Justin > > ________________________________________ > From: Kipper Fletez-Brant [cafletezbrant at gmail.com] > Sent: June 21, 2014 6:07 AM > To: Justin Meskas > Cc: Ryan Brinkman; bioconductor at r-project.org > Subject: Re: flowClean > > Justin, > > Thanks for your patience. I've fixed this and pushed an update to > Bioconductor, which should be live tomorrow. > > Thanks again, > Kipper > > On Thu 19 Jun 2014 06:41:04 PM EDT, Justin Meskas wrote: >> Hi Kipper, >> >> Thank you for your quick reply. I had previously tried flowClean on uncompensated and untransformed data, and the same error was given. >> >> Thanks, >> Justin >> >> ________________________________________ >> From: Kipper Fletez-Brant [cafletezbrant at gmail.com] >> Sent: June 19, 2014 3:26 PM >> To: Justin Meskas >> Cc: Ryan Brinkman; bioconductor at r-project.org >> Subject: Re: flowClean >> >> Hi Justin, >> >> I'm cc'ing the Bioc list so that others having this problem might see this. >> >> I have seen this error before. While it will take some time to diagnose, one comment I have is that we developed this algorithm to operate on uncompensated data. Try using flowClean before you do any other processing, and let me know how that works. >> >> Thanks, >> Kipper >> >> On Jun 19, 2014 5:42 PM, "Justin Meskas" <jmeskas at="" bccrc.ca<mailto:jmeskas="" at="" bccrc.ca="">> wrote: >> Hello Kipper, >> >> I am in Dr. Ryan Brinkman's lab and I was trying to use your flowClean package. I have downloaded version 1.0.0 from Bioconductor and I received a similar error for many of my FCS files. I have attached one FCS file and the code I used to get the error. The error was the following: >> >> Error in if (x == 0) { : missing value where TRUE/FALSE needed >> >> I do not know if there is a problem with my data, or if I am using the data incorrectly with flowClean, but if there is any help you could provide me that would be greatly appreciated. >> >> Thank you, >> Justin -------------- next part -------------- A non-text attachment was scrubbed... Name: A.100.500.clr_percent_plot.png Type: image/png Size: 125773 bytes Desc: A.100.500.clr_percent_plot.png URL: <https: stat.ethz.ch="" pipermail="" bioconductor="" attachments="" 20140624="" 2b955730="" attachment.png=""> -------------- next part -------------- A non-text attachment was scrubbed... Name: B.100.500.clr_percent_plot.png Type: image/png Size: 107146 bytes Desc: B.100.500.clr_percent_plot.png URL: <https: stat.ethz.ch="" pipermail="" bioconductor="" attachments="" 20140624="" 2b955730="" attachment-0001.png="">

I also appreciate all the work, but let me play the role of skeptic for a moment. First, your identification of samples where the first bin shouldn't be removed is inherently subjective. I'm guessing that you're looking at the CLR plot (and maybe even the fluorescence over time histograms) and thinking that the difference between the first bins and the rest is not very strong. However, I don't think we have the resolution to do this by eye. That slice of data is very small on these plots, so it's hard to visually do this kind of comparison. It's very rough,particularly for the fluorescence over time plots. Kipper can look at the changepoint algorithm to see why it's picking up this first bin, but I suspect what he'll find is that it's just above the threshold for flagging the bin. This gets to the second point... there will never be "perfect" flagging for any tool like this. In my eyes, I'm willing to sacrifice specificity (flagging one bin that's borderline) for sensitivity (catching as many problems as possible) because the data files we collect are huge, and we typically don't need every last event to have to power to do a good analysis. In other words, the loss of 1% of events by flagging one bin incorrectly is worth it in my eyes, if we catch all of those other aberrant regions of collection. There are so many more bad regions in problematic files that the tool is still - overall - helping us do better analyses, even if a small proportion of the data is excluded over an abundance of caution. Third, the tool is providing guidance. Ultimately, you can choose not to gate out that first bin that you don't believe is "bad." Just plot the good/bad event parameter against time. So, while I want us to offer the best tool possible, I don't want us to get hung up in an impossible optimization loop. You can imagine that tweaking the flagging parameters will solve the "problem" of flagging the first bin, but introduce new problems where we miss bins that should be flagged. Also, I'm not convinced yet that the first bin shouldn't be flagged in those datafiles... After all, one of the most common regions for problems in collection is at the beginning of the file. We see larger abnormalities routinely in that region. I would much prefer - absent more evidence that the cleaner is being too aggressive - that we wait and see about tweaking any parameters. > Kipper Fletez-Brant <mailto:cafletezbrant at="" gmail.com=""> > June 25, 2014 7:27 PM > Wow. Justin, thank for thoroughly researching this problem. I'll > hopefully have an answer for you in the next day or two. > > Kipper > > > > Justin Meskas <mailto:jmeskas at="" bccrc.ca=""> > June 25, 2014 7:09 PM > Hello Kipper and Pratip, > > Thank you for your explanations. After looking at my data more > closely, I found that about half of the cases where flowClean was > removing only the first compartment were consistent with the shape of > the data. The other half of these files seemed to just remove the > first compartment randomly. I have created an R source code file you > can use to replicate this result. I have put it into a .tar.gz file > and will transfer it to you from my google drive in a follow up email. > Please do not redistribute the data. Inside the .tar.gz file there is > a folder called Figures that can be regenerated using the code. The > figures in Figures/Clean show the output of flowClean, while > Figures/CleanTest show plots of Marker Vs Time that I created using > plotDens from flowDensity. (I am using these Marker vs Time plots to > judge if a certain section of the data should be removed or not.) > > Files "SPLN_L000030297_P3_090.fcs" and "SPLN_L000031107_P3_141.fcs" > show when flowClean has removed the first compartment when I believe > it should not of been. The other 5 FCS files show cases where > flowClean seems to also give poor results (The other > non-first-compartment-removed files all looked good). In my opinion, > flowClean should be removing, from the following files, the following > sections: > > SPLN_L000018651_Size_113 - 0-5% marks > SPLN_L000018653_Size_115 - 0-5% and 75-80% marks > SPLN_L000018656_Size_118 - 0-5% marks > SPLN_L000019881_Size_148 - 0-5% and 20-25% marks > SPLN_L000028450_P3_054 - 0-5%, 12-17% and 55-60% marks > SPLN_L000030297_P3_090 - Nothing > SPLN_L000031107_P3_141 - Nothing > > For SPLN_L000018653_Size_115, SPLN_L000018656_Size_118 and > SPLN_L000028450_P3_054 there seems to be certain locations where only > one marker is having a problem and it is not removed. Is it the case > that flowClean does not consider 1 marker problems to be substantial > enough to remove? > > Any insight you might have on any of these problems would be greatly > appreciated. Thank you very much, > > Justin > > P.S. I have made the code, hopefully, easy enough to use so all you > have to do is change the working directory to the folder that the > files have been extracted to. Let me know if there are any problems > with the code. > > ________________________________________ > From: Pratip K. Chattopadhyay [pchattop at mail.nih.gov] > Sent: June 25, 2014 7:07 AM > To: Kipper Fletez-Brant > Cc: Justin Meskas; Ryan Brinkman; bioconductor at r-project.org > Subject: Re: flowClean > > There are probably a couple of factors at work here... > > The HTS is more likely to exhibit anomalies early in collection for > various reasons... The pressure in the system may still be building > up, the cells are settled in the bottom of the well and so more events > go through at once, clogs/debris from previous wells/runs may > dislodge. In principle, the system is engineered to avoid these > issues, but in practice, I often (but not always) see anomalies at the > beginning of the collection. Interestingly, on days/runs where there > aren't many bad regions flagged, the early regions also look good. > This inspires confidence that the algorithm is detecting true problems > and doesn't have some systematic problem. > > The second factor - relevant to the case where you felt the first > events weren't too bad - is guilt by association. Kipper has built in > a little buffer to take out some bins that neighbor trouble spots, > just to keep things as clean as possible. > > Best, Pratip > > [cid:part1.07090309.09090509 at mail.nih.gov] > Kipper Fletez-Brant<mailto:cafletezbrant at="" gmail.com=""> > June 25, 2014 8:56 AM > Hi Justin, > > We (Pratip and I) think it may likely be your data - we have observed > that the early time points of collection in a flow run tend to have > the most errors. Pratip can speak a little more to the technical > causes of this. We appreciate your comments and look forward to the > results of your tests. > > Kipper > > > Hi Kipper, > > On second thought, I think it is my data. I just checked a few files > and they seem to be consistent with only removing the first > compartment. I will run some tests tomorrow to validate this. Sorry > for the emails. > > Thanks, > Justin > > ________________________________________ > From: Justin Meskas > Sent: June 24, 2014 4:31 PM > To: Kipper Fletez-Brant > Cc: Ryan Brinkman; > bioconductor at r-project.org<mailto:bioconductor at="" r-project.org=""> > Subject: RE: flowClean > > Hi Kipper, > > Sorry to keep emailing you, but I had another question about > flowClean. I have been noticing that the clean function seems to label > the first compartment for removal every time. This seems odd to me. I > attached two figures. The figure called "A..." looks like most other > figures, where the first compartment is labelled for removal. And the > other figure, called "B...", is my unique case where, I believe > anyway, the first compartment should be removed, but not the second. > Are all these files somehow accidentally removing the first > compartment? Or do you think it is the case that all these files have > bad data at the beginning? > > Thank you, > Justin > Kipper Fletez-Brant <mailto:cafletezbrant at="" gmail.com=""> > June 25, 2014 8:56 AM > Hi Justin, > > We (Pratip and I) think it may likely be your data - we have observed > that the early time points of collection in a flow run tend to have > the most errors. Pratip can speak a little more to the technical > causes of this. We appreciate your comments and look forward to the > results of your tests. > > Kipper > > >