Eclipse Community Forums: OHF » RetrieveDocumentSet

Help

Home

Home » Archived » OHF » RetrieveDocumentSet - Serious Memory Problem

Show: Today's Messages :: Show Polls :: Message Navigator

RetrieveDocumentSet - Serious Memory Problem [message #48107]

Wed, 17 September 2008 09:24

Stefan S.

Messages: 22
Registered: July 2009

Junior Member

Hi Everybody!

Recently - during developing the ITI-43 RetrieveDocumentSet - I have
stumbled accross a serious problem.

Please correct me, if I am wrong but the correct workflow is to create a
new RetrieveDocumentSetRepsonseType, loop over all documents that must be
returned and creating a DocumentResponseType by setting basic attributes
as well as the actual document every time.

I achieve this by doing the following (just the relevant lines... *g*):

RetrieveDocumentSetResponseType retrieveDocumentSetResponse =
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createRetrieveDocumentSetResponseType();

DocumentResponseType documentResponse =
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createDocumentResponseType();

//Set "easy" attributes
documentResponse.setRepositoryUniqueId(drt.getRepositoryUniq ueId());
documentResponse.setDocumentUniqueId(drt.getDocumentUniqueId ());
documentResponse.setHomeCommunityId(drt.getHomeCommunityId() );
documentResponse.setMimeType(dr.getMimeType());

//Set the "tricky" document content part
byte[] fileBytes = this.getBytesFromFile(document);
documentResponse.setDocument(fileBytes);

As you can see, the document (its content) is set using a byte array.
This tends to be a serious issue as you run out of memory in no time!

So I think the only solution to this problem is to leave setting the
document as a byte array beside, generate an OMElement from the rest of
the model and set the attachments lateron - directly within the OMElement.
(Of course the approach should be a streaming solution, because otherwise
OMElement is very likely to run out of memory, too... ;)

Bu my problem is, that I have no clue on how to achive this.

I already tried setting the attachment "manually", which looks something
like:

FileDataSource fds = new FileDataSource(document);
DataHandler dh = new DataHandler(fds);

MessageContext inMessageContext =
MessageContext.getCurrentMessageContext();
OperationContext operationContext =
inMessageContext.getOperationContext();
MessageContext outMessageContext =
operationContext.getMessageContext(WSDLConstants.MESSAGE_LAB EL_OUT_VALUE);
String attachmentID = outMessageContext.addAttachment(dh);

Sadly this does not work, because then the response does not contain a
"Document" Tag (Part), that would hold the actual content of the retrieved
document.

Can anyone provide me some help, hints - or better - has anyone stumbled
across the same problem and can provide a solution? (God, I hope it is so!
*g*)

Thanks in Advance for your time and your answers!

Greetings
Stefan

Report message to a moderator

Re: RetrieveDocumentSet - Serious Memory Problem [message #48140 is a reply to message #48107]

Wed, 17 September 2008 18:06

Matthew Davis

Messages: 269
Registered: July 2009

Senior Member

Hi Stefan,

Yes, this is a very good observation - setting the bytes of the document
into the EMF-generated model would not be advisable. Probably the best
way is as you described, using a DataHandler to add the content directly
to the OM data structure after the EMF->OM transformation. If you're
worried about creating the <Document> element (which, in all honesty,
wouldn't be too difficult), you can also add a single byte into
documentResponse.setDocument(byte[]) and then replace it with the
'correct' DataHandler after the OM transformation.

It adds additional "moving parts" (points of failure), but the number of
lines of code to handle this wouldn't be too bad.

The other way you mentioned likely won't work in an MTOM/XOP-based
environment, for the reason specified.

-Matt

Stefan S. wrote:
> Hi Everybody!
>
> Recently - during developing the ITI-43 RetrieveDocumentSet - I have
> stumbled accross a serious problem.
>
> Please correct me, if I am wrong but the correct workflow is to create a
> new RetrieveDocumentSetRepsonseType, loop over all documents that must
> be returned and creating a DocumentResponseType by setting basic
> attributes as well as the actual document every time.
>
> I achieve this by doing the following (just the relevant lines... *g*):
> RetrieveDocumentSetResponseType retrieveDocumentSetResponse =
> org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createRetrieveDocumentSetResponseType();
>
>
> DocumentResponseType documentResponse =
> org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createDocumentResponseType();
>
>
> //Set "easy" attributes
>
> documentResponse.setRepositoryUniqueId(drt.getRepositoryUniq ueId());
> documentResponse.setDocumentUniqueId(drt.getDocumentUniqueId ());
> documentResponse.setHomeCommunityId(drt.getHomeCommunityId() );
> documentResponse.setMimeType(dr.getMimeType());
>
> //Set the "tricky" document content part
> byte[] fileBytes = this.getBytesFromFile(document);
> documentResponse.setDocument(fileBytes);
>
> As you can see, the document (its content) is set using a byte array.
> This tends to be a serious issue as you run out of memory in no time!
>
> So I think the only solution to this problem is to leave setting the
> document as a byte array beside, generate an OMElement from the rest of
> the model and set the attachments lateron - directly within the
> OMElement. (Of course the approach should be a streaming solution,
> because otherwise OMElement is very likely to run out of memory, too... ;)
>
> Bu my problem is, that I have no clue on how to achive this.
> I already tried setting the attachment "manually", which looks something
> like:
>
> FileDataSource fds = new FileDataSource(document);
> DataHandler dh = new DataHandler(fds);
>
> MessageContext inMessageContext =
> MessageContext.getCurrentMessageContext();
> OperationContext operationContext =
> inMessageContext.getOperationContext(); MessageContext outMessageContext
> =
> operationContext.getMessageContext(WSDLConstants.MESSAGE_LAB EL_OUT_VALUE);
> String attachmentID = outMessageContext.addAttachment(dh);
>
> Sadly this does not work, because then the response does not contain a
> "Document" Tag (Part), that would hold the actual content of the
> retrieved document.
>
> Can anyone provide me some help, hints - or better - has anyone stumbled
> across the same problem and can provide a solution? (God, I hope it is
> so! *g*)
>
> Thanks in Advance for your time and your answers!
>
> Greetings
> Stefan
>

Report message to a moderator

Re: RetrieveDocumentSet - Serious Memory Problem [message #48300 is a reply to message #48140]

Thu, 18 September 2008 11:10

Stefan S.

Messages: 22
Registered: July 2009

Junior Member

Hi Matt!

Thanks again for your answer. During the last night I have created a
solution to this problem, which I want to share with you - or to be
general with all Newsgroup member. (Maybe someone can profit from reuse -
or whatever! *g*)

By the way... Your "phrase" (which, in all honesty, wouldn't be too
difficult) was very nice to read. Next time please do not hesitate to tell
me, that I am dramatizing, because actually it is very, very, very simple
to add attachments to an existing OMElement. But afterwards you are always
cleverer! ;)

So here is "my solution":

Having extracted all documentUIDs from the RetrieveDocumentSet Request, I
start building the RetrieveDocumentSet Reponse.

To cut things short, it is something like this:

RetrieveDocumentSetResponseType retrieveDocumentSetResponse =
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createRetrieveDocumentSetResponseType();

RegistryResponseType rr =
org.eclipse.ohf.ihe.common.ebxml._3._0.rs.RegistryFactory.eI NSTANCE.createRegistryResponseType();
//For now set the status to success - may be overwritten a few lines later
rr.setStatus(Constants.XDS_SUCCESS_RESPONSE_MESSAGE);
retrieveDocumentSetResponse.setRegistryResponse(rr);

<For all DocumentRequestTypes, found in the RetrieveDocumentSet-Request>

DocumentResponseType documentResponse =
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createDocumentResponseType();

//Set "easy" attributes
documentResponse.setRepositoryUniqueId(drt.getRepositoryUniq ueId());
documentResponse.setDocumentUniqueId(drt.getDocumentUniqueId ());
documentResponse.setHomeCommunityId(drt.getHomeCommunityId() );
documentResponse.setMimeType(drt.getMimeType());

//Attach the documentResponse to the RetrieveDocumentSetResponse
retrieveDocumentSetResponse.getDocumentResponse().add(docume ntResponse);

</End For>

Then I convert this EMF Model to an OMElement.
(Note, that the actual documents - the attachments - are still missing.)

For adding the collected attachments - which are uniquely identified by
their documentUID I created a simple method (posted below)

/**
* This method adds 0..n attachments directly - and in a streaming
way - to an existing RetrieveDocumentSet Response. 
*
* A normal RetrieveDocumentSetResponse - with missing document
(attachments) - looks like :
*
* <pre>
* <retrieve:RetrieveDocumentSetResponse
xmlns:retrieve="urn:ihe:iti:xds-b:2007"
xmlns:rs="urn:oasis:names:tc:ebxml-regrep:xsd:rs:3.0">
* <rs:RegistryResponse
status="urn:oasis:names:tc:ebxml-regrep:ResponseStatusType:Success " />
* <retrieve:DocumentResponse>

* <retrieve:RepositoryUniqueId>1.19.6.24.109.42. 1.1</retrieve:RepositoryUniqueId>

* <retrieve:DocumentUniqueId>158.226.202.44.1007 </retrieve:DocumentUniqueId>
* <retrieve:mimeType>text/plain</retrieve :mimeType>
* </retrieve:DocumentResponse>
* <retrieve:DocumentResponse>

* <retrieve:RepositoryUniqueId>1.19.6.24.109.42. 1.1</retrieve:RepositoryUniqueId>

* <retrieve:DocumentUniqueId>158.226.202.44.1008 </retrieve:DocumentUniqueId>
* <retrieve:mimeType>text/plain</retrieve :mimeType>
* </retrieve:DocumentResponse>
* </retrieve:RetrieveDocumentSetResponse>
* </pre>
*
* But we need it to look like the one below. 
* Notice, that it now contains the requested documents (attachments).
 
*
* <pre>
* <retrieve:RetrieveDocumentSetResponse
xmlns:retrieve="urn:ihe:iti:xds-b:2007"
xmlns:rs="urn:oasis:names:tc:ebxml-regrep:xsd:rs:3.0">
* <rs:RegistryResponse
status="urn:oasis:names:tc:ebxml-regrep:ResponseStatusType:Success " />
* <retrieve:DocumentResponse>

* <retrieve:RepositoryUniqueId>1.19.6.24.109.42. 1.1</retrieve:RepositoryUniqueId>

* <retrieve:DocumentUniqueId>158.226.202.44.1007 </retrieve:DocumentUniqueId>
* <retrieve:mimeType>text/plain</retrieve :mimeType>

* <retrieve:Document>VGhpcyBpcyBteSBkb2N1bWVudC4 KCkl0IGlzIGdyZWF0IQoK</Document> 
* </retrieve:DocumentResponse>
* <retrieve:DocumentResponse>

* <retrieve:RepositoryUniqueId>1.19.6.24.109.42. 1.1</retrieve:RepositoryUniqueId>

* <retrieve:DocumentUniqueId>158.226.202.44.1008 </retrieve:DocumentUniqueId>
* <retrieve:mimeType>text/plain</retrieve :mimeType>

* <retrieve:Document>VGhpcyBpcyBteSBzZWNvbmQgZG9 jdW1lbnQuCgpJdCBpcyBncmVhdCEKCg==</Document> 
* </retrieve:DocumentResponse>
* </retrieve:RetrieveDocumentSetResponse>
* </pre>
*
* This method achives this goal - in a streaming way. 
*
* @param response A complete RetrieveDocumentSet Repsonse. (Complete,
except for the attachments - documents)
* @param attachments A collection containing all attachments that have
to be added to the RetrieveDocumentSet Reponse. Every attachment
* is identified by the unique document ID.
*
* @return True if all steps finished successfully. False in case of an
error.
*
* @see OMElement
* @see OMText
* @see HashMap
* @see DataHandler
*
*/
private boolean addAttachmentsToResponse(OMElement response,
HashMap<String, File> attachments) {

logger.debug("entering addAttachmentsToResponse");

//1.) Create an iterator for navigating over all Child Elements with the
name <DocumentResponse>.
// (So to say, an iterator over all requested documents.)
Iterator<OMElement> it = response.getChildrenWithName(new
QName(Constants.IHE_NAMESPACE_URI,
Constants.RETRIEVE_DOCUMENT_SET_RESPONSE_CHILD_NAME));
OMFactory omFactory = OMAbstractFactory.getOMFactory();

try {
while(it.hasNext()) {

OMElement documentResponse = it.next();

//Retrieve the current documentUID from the OMElement
(documentResponse)
//This documentUID is needed to uniquely identify the correct
attachment file (document) that has to be added to the reponse
String documentUID = null;
if (documentResponse.getChildrenWithName(new
QName(Constants.IHE_NAMESPACE_URI,
Constants.RETRIEVE_DOCUMENT_SET_RESPONSE_DOCUMENT_UID_NAME)) .hasNext()) {
documentUID = ((OMElement)documentResponse.getChildrenWithName(new
QName(Constants.IHE_NAMESPACE_URI,
Constants.RETRIEVE_DOCUMENT_SET_RESPONSE_DOCUMENT_UID_NAME)) .next()).getText();
}

if (documentUID == null || "".equalsIgnoreCase(documentUID)) {
logger.error("Unable to retrieve the document UID. Pasting the
attachment is being skipped.");
continue;
}
else {
//Create a new XML-Element, named <Document> with the defined IHE
Namespace and the defined retrieveDocumentSet Namespace Praefix
OMElement documentElement =
omFactory.createOMElement(Constants.RETRIEVE_DOCUMENT_SET_RE SPONSE_DOCUMENT_NAME,
new OMNamespaceImpl(Constants.IHE_NAMESPACE_URI,
Constants.RETRIEVE_DOCUMENT_SET_RESPONSE_NAMESPACE_PRAEFIX)) ;

//Create a datahandler for the document, associated with this
documentUID
DataHandler dataHandler = new javax.activation.DataHandler(new
FileDataSource(attachments.get(documentUID)));

//Create a new OMText node with the above DataHandler and set
optimized to true in order to support streaming
OMText textData = omFactory.createOMText(dataHandler, true);

//Attach the text node to the documentElement and the
documentElement to the documentResponse
documentElement.addChild(textData);
documentResponse.addChild(documentElement);
}

}

return true;

}
//Just for security. Actually this should never happen. ;)
catch (OutOfMemoryError oomer) {
logger.fatal(oomer.getMessage());
}

return false;

}

So thats all! ;)
To quote Matt I must HONESTLY admit, that it was not difficult by any
means.

I do hope that someone can take benefit from the posted code. (If you need
further information, please do not hesitate to contact me...)

So long, Greetings and Have a nice day!
Stefan

Matthew Davis wrote:

> Hi Stefan,

> Yes, this is a very good observation - setting the bytes of the document
> into the EMF-generated model would not be advisable. Probably the best
> way is as you described, using a DataHandler to add the content directly
> to the OM data structure after the EMF->OM transformation. If you're
> worried about creating the <Document> element (which, in all honesty,
> wouldn't be too difficult), you can also add a single byte into
> documentResponse.setDocument(byte[]) and then replace it with the
> 'correct' DataHandler after the OM transformation.

> It adds additional "moving parts" (points of failure), but the number of
> lines of code to handle this wouldn't be too bad.

> The other way you mentioned likely won't work in an MTOM/XOP-based
> environment, for the reason specified.

> -Matt

> Stefan S. wrote:
>> Hi Everybody!
>>
>> Recently - during developing the ITI-43 RetrieveDocumentSet - I have
>> stumbled accross a serious problem.
>>
>> Please correct me, if I am wrong but the correct workflow is to create a
>> new RetrieveDocumentSetRepsonseType, loop over all documents that must
>> be returned and creating a DocumentResponseType by setting basic
>> attributes as well as the actual document every time.
>>
>> I achieve this by doing the following (just the relevant lines... *g*):
>> RetrieveDocumentSetResponseType retrieveDocumentSetResponse =
>>
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createRetrieveDocumentSetResponseType();
>>
>>
>> DocumentResponseType documentResponse =
>>
org.eclipse.ohf.ihe.xds.consumer.retrieve.RetrieveFactory.eI NSTANCE.createDocumentResponseType();
>>
>>
>> //Set "easy" attributes
>>
>> documentResponse.setRepositoryUniqueId(drt.getRepositoryUniq ueId());
>> documentResponse.setDocumentUniqueId(drt.getDocumentUniqueId ());
>> documentResponse.setHomeCommunityId(drt.getHomeCommunityId() );
>> documentResponse.setMimeType(dr.getMimeType());
>>
>> //Set the "tricky" document content part
>> byte[] fileBytes = this.getBytesFromFile(document);
>> documentResponse.setDocument(fileBytes);
>>
>> As you can see, the document (its content) is set using a byte array.
>> This tends to be a serious issue as you run out of memory in no time!
>>
>> So I think the only solution to this problem is to leave setting the
>> document as a byte array beside, generate an OMElement from the rest of
>> the model and set the attachments lateron - directly within the
>> OMElement. (Of course the approach should be a streaming solution,
>> because otherwise OMElement is very likely to run out of memory, too... ;)
>>
>> Bu my problem is, that I have no clue on how to achive this.
>> I already tried setting the attachment "manually", which looks something
>> like:
>>
>> FileDataSource fds = new FileDataSource(document);
>> DataHandler dh = new DataHandler(fds);
>>
>> MessageContext inMessageContext =
>> MessageContext.getCurrentMessageContext();
>> OperationContext operationContext =
>> inMessageContext.getOperationContext(); MessageContext outMessageContext
>> =
>> operationContext.getMessageContext(WSDLConstants.MESSAGE_LAB EL_OUT_VALUE);
>> String attachmentID = outMessageContext.addAttachment(dh);
>>
>> Sadly this does not work, because then the response does not contain a
>> "Document" Tag (Part), that would hold the actual content of the
>> retrieved document.
>>
>> Can anyone provide me some help, hints - or better - has anyone stumbled
>> across the same problem and can provide a solution? (God, I hope it is
>> so! *g*)
>>
>> Thanks in Advance for your time and your answers!
>>
>> Greetings
>> Stefan
>>

Report message to a moderator