GZIPContentEncodingFilter client and server side

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

GZIPContentEncodingFilter client and server side

Christopher Piggott
I have a file let's call it "xyz.dat" that I want to send, so on the
client side I'm doing this:

        Client c = Client.create();
        c.addFilter(new HTTPBasicAuthFilter("xxxx", "xxxx"));
        WebResource r = c.resource(uri);

        ClientResponse response = r.accept("application/somethingf")
                .post(ClientResponse.class, new File(sourceFile));



Pretty cool.  Now I want to see if it can send it with content encoding of gzip:

        c.addFilter(new GZIPContentEncodingFilter(true));

Seems to do something, but it's not good:

        Exception in thread "main"
com.sun.jersey.api.client.ClientHandlerException: java.io.IOException:
insufficient data written

Do I need to do something else, like explicitly tell it to set the
Content-Encoding header prior to making the request?

Related question... does my Resource need to be aware of this at all?
My resource method is:

        @POST
        @Consumes("application/something")
        public Response receiveFile(inputStream stream) {
           // do something with the stream
        }


for example, do I need to check Content-Encoding in my resource
method, and do something like stream = new GZIPInputStream(stream); ?
Jersey handles so many of these things transparently that I thought
this would be one of them.


--C

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: GZIPContentEncodingFilter client and server side

Paul Sandoz
Administrator
Hi,

I think i may know what is going on. GZIP encoding changes the size of  
the content that is sent and thus the Content-Length header.

The MessageBodyWriter for File knows what the size of the file is and  
thus the HttpURLConnection client is doing the following:

   uc.setFixedLengthStreamingMode((int)size);

but when the GZIP filter is present that is obviously not correct and  
thus the HttpURLConnection is complaining that the output stream is  
closed before all the bytes it expects have been written.

A work around is to do:

        ClientResponse response = r.accept("application/somethingf")
                .post(ClientResponse.class, new  
BufferedInputStream(new FileInputStream(sourceFile)));

Can you log an issue?

I think the solution would be for the GZIPContentEncodingFilter to  
declare it is modifying the size of the request entity such that this  
can be detected by the URL connection handler.


On the server-side Jersey will support GZIP if you register the  
following:

   https://jersey.dev.java.net/nonav/apidocs/latest/jersey/com/sun/jersey/api/container/filter/GZIPContentEncodingFilter.html

Paul.

On Jun 16, 2010, at 11:50 PM, Christopher Piggott wrote:

> I have a file let's call it "xyz.dat" that I want to send, so on the
> client side I'm doing this:
>
>        Client c = Client.create();
>        c.addFilter(new HTTPBasicAuthFilter("xxxx", "xxxx"));
>        WebResource r = c.resource(uri);
>
>        ClientResponse response = r.accept("application/somethingf")
>                .post(ClientResponse.class, new File(sourceFile));
>
>
>
> Pretty cool.  Now I want to see if it can send it with content  
> encoding of gzip:
>
>        c.addFilter(new GZIPContentEncodingFilter(true));
>
> Seems to do something, but it's not good:
>
>        Exception in thread "main"
> com.sun.jersey.api.client.ClientHandlerException: java.io.IOException:
> insufficient data written
>
> Do I need to do something else, like explicitly tell it to set the
> Content-Encoding header prior to making the request?
>
> Related question... does my Resource need to be aware of this at all?
> My resource method is:
>
>        @POST
>        @Consumes("application/something")
>        public Response receiveFile(inputStream stream) {
>           // do something with the stream
>        }
>
>
> for example, do I need to check Content-Encoding in my resource
> method, and do something like stream = new GZIPInputStream(stream); ?
> Jersey handles so many of these things transparently that I thought
> this would be one of them.
>
>
> --C
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: GZIPContentEncodingFilter client and server side

Christopher Piggott
Nuts, I didn't think of that.  When I do .post(ClientResponse.class,
new File(something));  the client library must be clever enough to
relize that I'm sending it a File and send the correct content-length.
 I would assume that if I .post an InputStream, for example, that it
doesn't set that header at all ...

What's correct behavior if you're using Content-Encoding: gzip ...
should Content-Length reflect the size of the uncompressed document?

>  uc.setFixedLengthStreamingMode((int)size);

OK, so I get the file size myself and force its length?

I'm still concerned about one thing ... isn't Content-Length part of
what makes persistent HTTP/1.1 connections work correctly?  Is that
going to be screwed up now?

> Can you log an issue?

Absolutely.  Will do.

--Chris




On Thu, Jun 17, 2010 at 8:46 AM, Paul Sandoz <[hidden email]> wrote:

> Hi,
>
> I think i may know what is going on. GZIP encoding changes the size of the
> content that is sent and thus the Content-Length header.
>
> The MessageBodyWriter for File knows what the size of the file is and thus
> the HttpURLConnection client is doing the following:
>
>  uc.setFixedLengthStreamingMode((int)size);
>
> but when the GZIP filter is present that is obviously not correct and thus
> the HttpURLConnection is complaining that the output stream is closed before
> all the bytes it expects have been written.
>
> A work around is to do:
>
>       ClientResponse response = r.accept("application/somethingf")
>               .post(ClientResponse.class, new BufferedInputStream(new
> FileInputStream(sourceFile)));
>
> Can you log an issue?
>
> I think the solution would be for the GZIPContentEncodingFilter to declare
> it is modifying the size of the request entity such that this can be
> detected by the URL connection handler.
>
>
> On the server-side Jersey will support GZIP if you register the following:
>
>  https://jersey.dev.java.net/nonav/apidocs/latest/jersey/com/sun/jersey/api/container/filter/GZIPContentEncodingFilter.html
>
> Paul.
>
> On Jun 16, 2010, at 11:50 PM, Christopher Piggott wrote:
>
>> I have a file let's call it "xyz.dat" that I want to send, so on the
>> client side I'm doing this:
>>
>>       Client c = Client.create();
>>       c.addFilter(new HTTPBasicAuthFilter("xxxx", "xxxx"));
>>       WebResource r = c.resource(uri);
>>
>>       ClientResponse response = r.accept("application/somethingf")
>>               .post(ClientResponse.class, new File(sourceFile));
>>
>>
>>
>> Pretty cool.  Now I want to see if it can send it with content encoding of
>> gzip:
>>
>>       c.addFilter(new GZIPContentEncodingFilter(true));
>>
>> Seems to do something, but it's not good:
>>
>>       Exception in thread "main"
>> com.sun.jersey.api.client.ClientHandlerException: java.io.IOException:
>> insufficient data written
>>
>> Do I need to do something else, like explicitly tell it to set the
>> Content-Encoding header prior to making the request?
>>
>> Related question... does my Resource need to be aware of this at all?
>> My resource method is:
>>
>>       @POST
>>       @Consumes("application/something")
>>       public Response receiveFile(inputStream stream) {
>>          // do something with the stream
>>       }
>>
>>
>> for example, do I need to check Content-Encoding in my resource
>> method, and do something like stream = new GZIPInputStream(stream); ?
>> Jersey handles so many of these things transparently that I thought
>> this would be one of them.
>>
>>
>> --C
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: GZIPContentEncodingFilter client and server side

Paul Sandoz
Administrator
On Jun 17, 2010, at 5:36 PM, Christopher Piggott wrote:
> Nuts, I didn't think of that.  When I do .post(ClientResponse.class,
> new File(something));  the client library must be clever enough to
> relize that I'm sending it a File and send the correct content-length.
> I would assume that if I .post an InputStream, for example, that it
> doesn't set that header at all ...
>

That depends on the underlying HTTP library. The point is if Jersey  
does not know what the length it does not tell the underlying HTTP  
library so it can optimize. The HTTP library might buffer the request  
to determine the length, or it may use chunked encoding to avoid  
buffering, or if it is told the length it can avoid buffering or  
chunked encoding.


> What's correct behavior if you're using Content-Encoding: gzip ...
> should Content-Length reflect the size of the uncompressed document?
>

No, it should reflect the size in bytes of the request entity.


>>  uc.setFixedLengthStreamingMode((int)size);
>
> OK, so I get the file size myself and force its length?
>

No, for the moment do not use File and use InputStream as a workaround  
or another workaround is to override Jersey's File message body writer  
(see end of email for a modified version you can use and register).



> I'm still concerned about one thing ... isn't Content-Length part of
> what makes persistent HTTP/1.1 connections work correctly?  Is that
> going to be screwed up now?
>

No. That is HTTP keep-alive. It works with Content-Length or chunked  
encoding.


>> Can you log an issue?
>
> Absolutely.  Will do.
>

Thanks!
Paul.


@Produces({"application/octet-stream", "*/*"})
@Consumes({"application/octet-stream", "*/*"})
public final class FileProvider extends  
AbstractMessageReaderWriterProvider<File> {

     public boolean isReadable(Class<?> type, Type genericType,  
Annotation[] annotations, MediaType mediaType) {
         return File.class == type;
     }

     public File readFrom(
             Class<File> type,
             Type genericType,
             Annotation annotations[],
             MediaType mediaType,
             MultivaluedMap<String, String> httpHeaders,
             InputStream entityStream) throws IOException {
         File f = File.createTempFile("rep","tmp");
         OutputStream out = new BufferedOutputStream(new  
FileOutputStream(f));
         try {
             writeTo(entityStream, out);
         } finally {
             out.close();
         }
         return f;
     }

     public boolean isWriteable(Class<?> type, Type genericType,  
Annotation[] annotations, MediaType mediaType) {
         return File.class.isAssignableFrom(type);
     }

     public void writeTo(
             File t,
             Class<?> type,
             Type genericType,
             Annotation annotations[],
             MediaType mediaType,
             MultivaluedMap<String, Object> httpHeaders,
             OutputStream entityStream) throws IOException {
         InputStream in = new BufferedInputStream(new  
FileInputStream(t), ReaderWriter.BUFFER_SIZE);
         try {
             writeTo(in, entityStream);
         } finally {
             in.close();
         }
     }

     @Override
     public long getSize(File t, Class<?> type, Type genericType,  
Annotation[] annotations, MediaType mediaType) {
         return -1;
     }
}

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]