EDC5133I No space left on device during Dependency build
We have a few components under source control. We created two build definitions that use the same language definitions/translators but a different build subset (in both cases we had to keep the subsets small since we were getting IOExceptions if the subset contained more than a few dozen source members). The problem is that the first build worked but the second one is failing fairly consistently.
Here's an excerpt from the build log:
* [(internal) logpublishertask] com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device.
* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager$StreamLengthUtility.run(ContentManager.java:263)
* [(internal) logpublishertask] at com.ibm.team.repository.client.internal.ContentManager.storeContent(ContentManager.java:403)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.storeContent(ContentPublisher.java:180)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.ContentPublisher.getFileContent(ContentPublisher.java:94)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractFilePublisher.initializeContribution(AbstractFilePublisher.java:49)
* [(internal) logpublishertask] at com.ibm.team.build.internal.publishing.AbstractContributionPublisher.publish(AbstractContributionPublisher.java:120)
* [(internal) logpublishertask] at com.ibm.team.build.internal.ant.AbstractContentPublisherTask.updateBuildResult(AbstractContentPublisherTask.java:174)
* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractPublisherTask.doExecute(AbstractPublisherTask.java:105)
* [(internal) logpublishertask] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:666)
Snipped the rest of the stack trace. After that we see the following:
* [antz:compile] /var/jazz502/build/MSP_PLEVEL/macrodefs.xml:18: com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0040E An I/O error occurred while the stream was being preprocessed. More information: EDC5133I No space left on device..
* IKJ56246I FILE SYSADATA NOT ALLOCATED, FILE IN USE
And so on (i.e. same error for the rest of the source members in the subset).
Is there a way to find out what device it's talking about?
Some more details about this issue. I narrowed it down to a single program that seems to be the one that RTC is struggling with. It's about 12K lines long (100Kb in size). The impact analysis for it contains 123 files it depends on. The buildable files list that gets generated for it is 105K in size. Are we hitting some size limit that RTC doesn't seem to be able to handle? If so, this could be a showstopper for us as we have a lot of "large" programs that could cause a similar problem for RTC.
|
Accepted answer
I wonder if it is /tmp that has the issue. USS services seem to be not so good to put a good message out when it has a space issue. Is there anything in the system log? Also I see your /u/ has no space. I presume this is because your user set up a mount point at /u. Is there a chance that the userid running the build has not got a directory set up?
Alex Akilov selected this answer as the correct answer
Comments
Alex Akilov
commented Apr 15 '15, 4:38 p.m.
Looks like you guys were right. Our /tmp was in memory structure for performance which is why it was allocated so small (10M). We increased it to about 32M with the ability to grow 4 times by 4M each time and that seems to have alleviated the problem. Still, would be good to be able to override the location of the scratchpad. As I indicated, setting the tmpdir value in bfagent.conf and the gateway shell script didn't seem to help. Please provide an enhancement request, if you think this would be useful.
|
4 other answers
Ok, then I suspect that is the issue. That is not a lot of space, especially when you have a large file with a lot of dependencies. In the same information screen it ill tell you the data set being used. Is there any way you can extend that? For example, our build directory has:
Block size . . . . : 1024 Total blocks . . . : 720000 Available blocks . : 531056 Blocks in use . . : 188944 Which is 100 times bigger than yours. Remember all of the build files, metadata etc are loaded into that directory prior the build. Comments
Alex Akilov
commented Mar 24 '15, 5:59 p.m.
Thanks Liam, that's what I suspected but I needed ammunition to justify my request to increase that storage :-). Working late I see :-).
Alex Akilov
commented Mar 25 '15, 2:44 p.m.
Well, sad news. We increased the space to 180K blocks (176K available) but still failed with the no space left on device. Any other ideas? |
Have you tried to expand tmp ? It looks like this is happening when publishing a log, and there's a conversion of SYSPRINT to a file and I believe it happens in tha java tmp folder (which is probably /tmp).
your tmp looked 99% full. |
So now I see that Liam & I seem to think the same thing ...
Comments
Alex Akilov
commented Mar 30 '15, 5:53 p.m.
We initially suspected tmpdir and tried to change the tmpdir setting in bfagent.conf. Not sure if that works or not since the product documentation skips that entry but documents all of the others. I think we also tried to set a build property -Djava.io.tmpdir=<new temp folder> but that didn't work either. I'm not sure if there is a way to set a different temp folder for RTC to use rather than increasing the system allocation for temp (which would require sys admin involvement, approvals, etc. and, therefore, not ideal). As far as a home folder for the build user (Liam's suggestion e.g. /u/blduser), in our shop service accounts do not get a home folder (again, a policy that may be difficult to fight/change). If there is a way to make all RTC related output go to the same mount point where the load directory is (/var/jazz502 in our case) that would be ideal for us. |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.
Comments
Have you checked the disk size? That is what the error message clearly hints to.
I'm not sure which folder it's complaining about. Here's the output of df -kP (load directory is in /var/jazz502 which has 46% available). Granted, some of these allocations are low and we've asked for additional space but it would help if the error message was a little more helpful in telling us where is the space that we're running out of (e.g. is it tmp space? work/load directory? What file was it trying to write to? etc.).