rhdf5 read/write concurrency surprise
0
0
Entering edit mode
@brad-friedman-6302
Last seen 10.2 years ago
An open rhdf5 handle becomes corrupted when a another process writes to the hdf5 file. This example requires you to start two different R processes, a writer and a reader ## Create empty HDF5 file in the writer process > library(rhdf5) > hf <- "x.hdf5" > h5createFile(hf) [1] TRUE > h5dump(hf) named list() ## Then, in the reader process open a handle and dump the file > library(rhdf5) > hf <- "x.hdf5" > fid <- H5Fopen(hf) > h5dump(fid) named list() ## Now leave the rhdf5 handle open in the reader and go back to the ## writer process and write a data set > h5write(1, hf, "foo") > h5dump(hf) $foo [1] 1 ## Now go back to the reader and try to read it: > h5dump(fid) named list() ## That is not right---it doesn't reflect the change. ## Maybe the handle is bad. Try to read it using the filename instead > h5dump(hf) named list() ## still can't see it. Try a new rhdf5 handle > fid2 <- H5Fopen(hf) > h5dump(fid2) named list() ## Still can't see it. Turns out if I close all the open rhdf5 handles ## I can see it. > H5Fclose(fid) > H5Fclose(fid2) > h5dump(hf) $foo [1] 1 A workaround for this is that whenever the file is modified by the writer process, the reader process has to make sure to close all open handles for the file and then reopen fresh ones. Another workaround is to never explicitly open handles with H5Fopen, and to only use rhdf's interface that accepts a file name instead of an open HDF5 handle. > sessionInfo() R version 3.1.0 Patched (2014-05-17 r65643) Platform: x86_64-apple-darwin10.8.0 (64-bit) locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] rhdf5_2.9.1 loaded via a namespace (and not attached): [1] zlibbioc_1.10.0 [[alternative HTML version deleted]]
GO rhdf5 PROcess GO rhdf5 PROcess • 1.3k views
ADD COMMENT

Login before adding your answer.

Traffic: 627 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6