how to upload large file more then 2 gb using jersey?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

how to upload large file more then 2 gb using jersey?

Vipin.nemade
Hi Every Buddy,
I am developing some storage domain application in which I am able to less then 2 gb file. But my project requirement is upload large size file. Please help me to achieve this.

Thanks and Regards
Vipin nemade
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Pavel Bucek-2
Hello,

can you please share more details? How are you processing incoming data?

InputStream should be practically infinite, if you are not
caching/buffering it, so something like code above should work:

         @POST
         public void post(InputStream in) throws IOException {
             int read;
             final byte[] data = new byte[2048]; // you might choose
different bugger size.
             while ((read = in.read(data)) != -1) {
                     // handle read data
             }
         }

Pavel

On 2/14/12 11:49 AM, Vipin.nemade wrote:

> Hi Every Buddy,
> I am developing some storage domain application in which I am able to less
> then 2 gb file. But my project requirement is upload large size file. Please
> help me to achieve this.
>
> Thanks and Regards
> Vipin nemade
>
> --
> View this message in context: http://jersey.576304.n2.nabble.com/how-to-upload-large-file-more-then-2-gb-using-jersey-tp7283481p7283481.html
> Sent from the Jersey mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Vipin.nemade
Hi Pavel,

I am using jersey multipart jar to upload a file.my server is crashing when file 2gb or more then 2gb.

my code of rest webservice:-



@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(@FormDataParam('file')InputStream uploadedInputStream,@FormDataParam('file') FormDataContentDisposition fileDetail,@HeaderParam("Content-Length") int length) {
               
println("Content-Lenght:"+""+length)

                       if(uploadedInputStream != null)
                        println("contain stream")
               
                        if(fileDetail != null)
                        println("contain file :"+fileDetail.getFileName())
                        String name = getFileName(fileDetail.getFileName())
                        println("File Name is:"+"  "+name)
                        String uploadedFileLocation = "D://uploaded/"+name;
                       
                       
                        writeToFile(uploadedInputStream,uploadedFileLocation);
                        println("Test3")
                        String output = "File uploaded to : " + uploadedFileLocation;
                        return Response.status(200).entity("uploadFile size"+" "+length+"bytes"+"      "+"Uploaded file name : " +name).build();
         

        }

        // save uploaded file to new location
        private void writeToFile(InputStream uploadedInputStream,String uploadedFileLocation) {
 
                try {
                        OutputStream out = new FileOutputStream(new File(uploadedFileLocation));
                        int read = 0;
                        byte[] bytes = new byte[1024];
 
                        out = new FileOutputStream(new File(uploadedFileLocation));
                        while ((read = uploadedInputStream.read(bytes)) != -1) {
                                out.write(bytes,0,read);
                        }
                        out.flush();
                        out.close();
                } catch (IOException e) {
 
                        e.printStackTrace();
                }

        }
Please help me to achive more then 2 gb file.
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Cowtowncoder
On Tue, Feb 14, 2012 at 10:02 PM, Vipin.nemade <[hidden email]> wrote:
> Hi Pavel,
>
> I am using jersey multipart jar to upload a file.my server is crashing when
> file 2gb or more then 2gb.

Crashing with... ? Can you share the stack trace; otherwise it is
impossible to speculate on what might be happening.

-+ Tatu +-
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Vipin.nemade
before reaching to my server side code it is getting crashed.
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Vipin.nemade
stack trace showing nothing............during large file upload during server crash.
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Cowtowncoder
In reply to this post by Vipin.nemade
On Tue, Feb 14, 2012 at 10:48 PM, Vipin.nemade <[hidden email]> wrote:
> before reaching to my server side code it is getting crashed.

Then you need to go and figure out the stack trace container provides
or something -- there is nothing to work on without that piece of
information. Maybe it does not even get to Jersey, who knows?

-+ Tatu +-
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Cowtowncoder
In reply to this post by Vipin.nemade
Then you need to figure out to get more information.

Good luck!

-+ Tatu +-

On Tue, Feb 14, 2012 at 11:10 PM, Vipin.nemade <[hidden email]> wrote:
> stack trace showing nothing............during large file upload during server
> crash.
>
> --
> View this message in context: http://jersey.576304.n2.nabble.com/how-to-upload-large-file-more-then-2-gb-using-jersey-tp7283481p7286756.html
> Sent from the Jersey mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

adnan.maks
I also have a requirement to upload multiple large files using jersey. So, I chose multipart file transfer. However, I am getting an exception at the server end. I think it is because of the jersey container trying to load the entire multipart file contents into MimeMultipart object that the OOM error is thrown. I want a methodology with which I can save the files to some temporary directory(do I have to implement a MessageReader/Writer?) and not loaded into Java memory.



My server code looks like below -

@POST
@Consumes(MediaType.MULTIPART_FORM_DATA)
@Produces(MediaType.APPLICATION_XML)
@Path("multipartTransfer")
public EventStatus fileTransferInviteMultiPart(MimeMultipart data) throws IOException, MessagingException, ClassNotFoundException {

--some InputSTream code goes here

}

My client code looks like below -


FormDataMultiPart formData = new FormDataMultiPart();
                if (veryLargeFile != null) {
                        char[] metadataBytes = new char[100];
                        metadataBytes = "this is file metadata".toCharArray();
                        formData.bodyPart(new FormDataBodyPart("file1_metadata", metadataBytes,
                               MediaType.MULTIPART_FORM_DATA_TYPE));
                        formData.bodyPart(new FormDataBodyPart("file1", veryLargeFile,
                               MediaType.APPLICATION_OCTET_STREAM_TYPE));
                }


String xmlString = service.path("rest").path("fileTransfer").path("multipartTransfer").type(MediaType.MULTIPART_FORM_DATA).accept(MediaType.APPLICATION_XML).post(String.class,formData);





Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Noah
In reply to this post by Vipin.nemade
What server are you using? It looks like in Glassfish there's a server-config property for Grizzly that establishes a max-post-size-bytes which by default is 2097152. If you are using Grizzly/Glassfish you are probably running into that.

You can use:

asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.max-post-size-bytes=new_value

to adjust this for the default non-SSL HTTP listener.

It doesn't look like this property is exposed in the GF Admin UI.

-Noah

On Feb 15, 2012, at 1:02 AM, Vipin.nemade wrote:

> Hi Pavel,
>
> I am using jersey multipart jar to upload a file.my server is crashing when
> file 2gb or more then 2gb.
>
> my code of rest webservice:-
>
>
>
> @POST
> @Consumes(MediaType.MULTIPART_FORM_DATA)
> public Response uploadFile(@FormDataParam('file')InputStream
> uploadedInputStream,@FormDataParam('file') FormDataContentDisposition
> fileDetail,@HeaderParam("Content-Length") int length) {
>
> println("Content-Lenght:"+""+length)
>
>                       if(uploadedInputStream != null)
> println("contain stream")
>
> if(fileDetail != null)
> println("contain file :"+fileDetail.getFileName())
> String name = getFileName(fileDetail.getFileName())
> println("File Name is:"+"  "+name)
> String uploadedFileLocation = "D://uploaded/"+name;
>
>
> writeToFile(uploadedInputStream,uploadedFileLocation);
> println("Test3")
> String output = "File uploaded to : " + uploadedFileLocation;
> return Response.status(200).entity("uploadFile size"+" "+length+"bytes"+"    
> "+"Uploaded file name : " +name).build();
>
>
> }
>
> // save uploaded file to new location
> private void writeToFile(InputStream uploadedInputStream,String
> uploadedFileLocation) {
>
> try {
> OutputStream out = new FileOutputStream(new File(uploadedFileLocation));
> int read = 0;
> byte[] bytes = new byte[1024];
>
> out = new FileOutputStream(new File(uploadedFileLocation));
> while ((read = uploadedInputStream.read(bytes)) != -1) {
> out.write(bytes,0,read);
> }
> out.flush();
> out.close();
> } catch (IOException e) {
>
> e.printStackTrace();
> }
>
> }
> Please help me to achive more then 2 gb file.
>
> --
> View this message in context: http://jersey.576304.n2.nabble.com/how-to-upload-large-file-more-then-2-gb-using-jersey-tp7283481p7286671.html
> Sent from the Jersey mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

Noah
Actually this is exposed in the Admin UI but you have to go to Configurations -> server-config -> Network Config -> Network Listeners -> http-listener-1 and then click on the HTTP tab and you will see it in there.

-Noah

On Feb 29, 2012, at 8:38 PM, Noah White wrote:

> What server are you using? It looks like in Glassfish there's a server-config property for Grizzly that establishes a max-post-size-bytes which by default is 2097152. If you are using Grizzly/Glassfish you are probably running into that.
>
> You can use:
>
> asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.max-post-size-bytes=new_value
>
> to adjust this for the default non-SSL HTTP listener.
>
> It doesn't look like this property is exposed in the GF Admin UI.
>
> -Noah
>
> On Feb 15, 2012, at 1:02 AM, Vipin.nemade wrote:
>
>> Hi Pavel,
>>
>> I am using jersey multipart jar to upload a file.my server is crashing when
>> file 2gb or more then 2gb.
>>
>> my code of rest webservice:-
>>
>>
>>
>> @POST
>> @Consumes(MediaType.MULTIPART_FORM_DATA)
>> public Response uploadFile(@FormDataParam('file')InputStream
>> uploadedInputStream,@FormDataParam('file') FormDataContentDisposition
>> fileDetail,@HeaderParam("Content-Length") int length) {
>>
>> println("Content-Lenght:"+""+length)
>>
>>                      if(uploadedInputStream != null)
>> println("contain stream")
>>
>> if(fileDetail != null)
>> println("contain file :"+fileDetail.getFileName())
>> String name = getFileName(fileDetail.getFileName())
>> println("File Name is:"+"  "+name)
>> String uploadedFileLocation = "D://uploaded/"+name;
>>
>>
>> writeToFile(uploadedInputStream,uploadedFileLocation);
>> println("Test3")
>> String output = "File uploaded to : " + uploadedFileLocation;
>> return Response.status(200).entity("uploadFile size"+" "+length+"bytes"+"    
>> "+"Uploaded file name : " +name).build();
>>
>>
>> }
>>
>> // save uploaded file to new location
>> private void writeToFile(InputStream uploadedInputStream,String
>> uploadedFileLocation) {
>>
>> try {
>> OutputStream out = new FileOutputStream(new File(uploadedFileLocation));
>> int read = 0;
>> byte[] bytes = new byte[1024];
>>
>> out = new FileOutputStream(new File(uploadedFileLocation));
>> while ((read = uploadedInputStream.read(bytes)) != -1) {
>> out.write(bytes,0,read);
>> }
>> out.flush();
>> out.close();
>> } catch (IOException e) {
>>
>> e.printStackTrace();
>> }
>>
>> }
>> Please help me to achive more then 2 gb file.
>>
>> --
>> View this message in context: http://jersey.576304.n2.nabble.com/how-to-upload-large-file-more-then-2-gb-using-jersey-tp7283481p7286671.html
>> Sent from the Jersey mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

adnan.maks
Thanks for the glassfish info Noah. I found there was a jira issue for the problem I mentioned which Paul seems to have fixed my using mimepull.jar and I was using the MimeMultiPart of mail.jar. I could circumvent it by having mimepull.jar in the classpath and using FormDataMultiPart.

-Adnan
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

adnan.maks
In reply to this post by Noah
Whatever server you are using, could you plz try doing this at the client end  because I noticed HTTP seems to have a limitation of 2 GB in streaming as well (I am not sure of this limit though) and chunking helped transfer even 4 GB files(The downside could be that http does not support resuming of aborted downloads and for large files this could be a concern for the amount of time the http connection needs to be alive and without any aborts due to network issues)).

Client client = Client.create(config);
                client.setChunkedEncodingSize(1024*1024);


- This worked for me even for streaming and multipart file transfers. Plz do this bit and then as Noah says if you are using GlassFish, plz follow what he is saying.

-Adnan
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

WOOZYANGEL
This post has NOT been accepted by the mailing list yet.
In reply to this post by Vipin.nemade
THX
@@HeaderParam("Content-Length") long length
I'VE solved the problem of uploaded file's size limited!
Reply | Threaded
Open this post in threaded view
|

Re: how to upload large file more then 2 gb using jersey?

adnan.maks
This post has NOT been accepted by the mailing list yet.
Hi WOOZYANGEL ,

Could you plz elaborate as to how you solved it so it will update our knowledgebase.

- Adnan



On Tue, Jul 31, 2012 at 10:22 AM, WOOZYANGEL [via Jersey] <[hidden email]> wrote:
THX
@@HeaderParam("Content-Length") long length
I'VE solved the problem of uploaded file's size limited!


If you reply to this email, your message will be added to the discussion below:
http://jersey.576304.n2.nabble.com/how-to-upload-large-file-more-then-2-gb-using-jersey-tp7283481p7579849.html
To unsubscribe from how to upload large file more then 2 gb using jersey?, click here.
NAML



--
Thanks and Regards,
Adnan, Mohammed
Bangalore
cell:+91 9886529874