- Problems when update doesn't complete (exited during reindexing)
- Problems when update doesn't complete (out of space error during update process) Informational/Recommendations Ticket
Though the causes and symptoms are different, it seems that both of these issues are rooted in the DMC not handling updates in an atomic fashion, such that they can be cleanly rolled back if any part of the process does not succeed.
Regarding the issue of disk space and other practical concerns with large numbers of updates on top of a baseline, I did some analysis of the changes that occurred with a single update pack. All of the changes were within the “WEB-INF” folder of the DMC installation.
The “config.dmpcf” file gets a new version number a couple of new entries, growing by 300-350 bytes (generally) for each update. This growth has negligible effects, even over a large number of updates.
The index is rebuilt. While this may cause some growth due to new content in updates, this is difficult to predict and in some cases will even result in a reduction in size.
The TOC is rebuilt. This seems more likely to cause some increase in size with the addition of new content, as the XML structure is somewhat more straightforward than the structure of the index. For both this and the index, I’d need to do analysis over a larger number of updates to provide meaningful estimates.
A log file is generated for the update process. In the update I ran for my example, this was about 45k. Over the course of 80 updates, for example, this could result in some 3.5MB of log data. Perhaps a post-update process could either purge logs after a certain period of time or number of subsequent updates, or compress the logs to save space (as they would likely compress very well, given the highly-redundant nature of most log data).
The actual content volume, as a single ZIP file, is added to the installation. This is a somewhat inescapable part of the growth. Since the ZIP format compresses each file in the archive independently, there would likely be little benefit in combining multiple update volumes in some way, unless the updates frequently replace content with newer versions, in which case some sort of process to “repack” the content of multiple updates into a consolidated form might provide some space savings, as well as some possible overall performance improvements.
by Single-Sourcing Solutions
We have discovered that updating to a new revision of the software will have no effect. We have submitted the update/corruption issue as bug with PTC against the most current 6.0 M030 version. The case is open and under investigation at PTC.
We don't have solution to change behavior of the software. We have not yet isolated where this is coming from, but have an idea about the area that is responsible. We are continuing to delve deeper to see if there is a way to write code to change this behavior with the caveat that writing custom code is opening pandora's box. We are continuing to isolate to find exact source to see what is possible.
With respect to the users, we believe it will be best to advise them about best practices and behaviors which we are defining currently (i.e., recommendations about how long updates take and warnings about not closing the application during the update process, etc.)
by Single-Sourcing Solutions
This is our final report on what to do in this situation.
Error encountered when shutting down during the update process.
The Arbortext Digital Media Consumer (DMC) tool is very easy to close when the user is done with it. The user need only use any of the methods typical of Windows programs, such clicking on the "X" button in the upper-right corner of the window or pressing Alt-F4. No prompts to confirm the user's intent are presented and the application quickly disappears. An analysis of the program uncovered that the DMC does not discriminate as to what stage it is at in the update process to close out. The shutdown will occur even if it is in the midst of an important process, such as applying update to its content.
In many of the circumstances, this is not a problem, since the DMC will just restart the update process the next time the user requests it to do so. However, testing and analysis has shown that if the user picks the wrong moment to close the DMC, the indexes that make the DMC's search functions possible can be corrupted. This affects not only the DMC's search functions, but also the ability to perform any future updates, since the updates will not succeed if they cannot modify the corrupt indexes.
In some cases, the DMC has been observed to produce error messages following index corruption. However, in other cases, there are no error messages. Further update attempts simply hang and never complete.
The Arbortext Digital Media Consumer (DMC) application does not adequately protect itself from corruption that may occur in the case where the user shuts down the application at a critical point during the update process. More specifically, the indexes that power the search functions of the application can be left in a damaged state that prevents the proper functioning of the search feature as well as prohibiting any further updates to the DMC content. There does not appear to be a completely reliable way to recover these indexes to a usable state short of reinstalling the latest DMC baseline and reapplying all updates.
Attempts to quantify the likelihood of indexes being corrupted in this way are fraught with complications due to the wide range of variables that affect the timing of the process, such as computer speed and memory, other loads upon the computer's resources, network speed and the size of update packs. Additionally, a lack of timely and precise notifications about the state of the update process makes collection of meaningful data nearly impossible.
In our testing and analysis, the error was inconsistent in reproduction as it appears to be a timing issue and it is imposable to know with certainty what stage of the process you are in. However, when the error does occur, the failure is catastrophic.
The DMC does not provide a function for repairing or rebuilding these indexes, so the only known resolution for such an occurrence is to uninstall and reinstall the DMC baseline and reapply all updates.
Given the severity of the consequences and the ease with which the user could inadvertently cause them, we strongly recommend creating a simple warning prompt for the user when the user requests that the application close.
Further measures worth consideration include efforts to make the update process less obtrusive for the user. Our primary recommendation here is to create a simple Java wrapper for the DMC update functionality such that it could be invoked in an unattended manner, thus allowing updates to be scheduled to run automatically at a time that is convenient for the user, such as overnight.
An additional measure might be to add a component to the DMC that would automatically synchronize with the configured update server, downloading update packs to a local cache in an unobtrusive manner, perhaps employing bandwidth throttling techniques to ensure that foreground task performance is not significantly impacted. This would leave the user with all the same flexibility for the timing of update application without forcing them to wait through the download process on an interactive basis. In conjunction with the "scheduled
update" feature, above, this would allow the update process to be completed at the user's convenience without concerns about network availability.
A bug report has been filed with PTC, the creators of the DMC product, regarding this issue. There is no indication as to when, or if, PTC will be able to further resolve this issue.