New TIdHTTP hoNoReadMultipartMIME flag

A new hoNoReadMultipartMIME flag has been added to the TIdHTTP.HTTPOptions property. The purpose of this flag is to specify whether TIdHTTP should read the body content of “multipart/…” responses, such as “multipart/x-mixed-replace” or “multipart/byteranges”, into the target TStream or to exit immediately and let the caller read the content manually instead. By default, this flag is disabled to preserve existing behavior to have TIdHTTP read the content into the target TStream until the server closes the connection.

The primary motivation for this is to handle “multipart/x-mixed-replace” server pushes, such as from webcams. TIdHTTP does not natively support server pushes (and still does not), so the previous workaround to process the push events in real-time was to use TIdEventStream as the target TStream with a OnWrite event handler assigned, or to implement a custom TStream class that overrides the virtual Write() method. Neither approach is very intuitive.

Now, if you enable the hoNoReadMultipartMIME flag, and then any “multipart/…” response is received (see UPDATE further below), you can read the MIME data directly from the TIdHTTP.IOHandler as needed, such as by using TIdTCPStream to assign TIdHTTP as the SourceStream for TIdMessageDecoderMIME and then call the decoder’s ReadHeader() and ReadBody() methods in a loop until the server stops sending pushes by sending a terminating MIME boundary, or closes the connection. For example:

var
  Boundary, Line: string;
  TCPStream: TIdTCPStream;
  Decoder: TIdMessageDecoder;
  MsgEnd: Boolean;
  BodyStream: TStream;
begin
  ...
  HTTP.HTTPOptions := HTTP.HTTPOptions + [hoNoReadMultipartMIME];
  HTTP.Get(...);

  if IsHeaderMediaType(HTTP.Response.ContentType, 'multipart') then
  begin
    Boundary := ExtractHeaderSubItem(HTTP.Response.ContentType, 'boundary', QuoteHTTP);
    repeat
      Line := HTTP.IOHandler.ReadLn;
    until (Line = ('--' + Boundary)) or (Line = ('--' + Boundary + '--'));

    TCPStream := TIdTCPStream.Create(HTTP);
    try
      Decoder := TIdMessageDecoderMIME.Create(nil);
      try
        MsgEnd := False;
        repeat
          TIdMessageDecoderMIME(Decoder).SourceStream := TCPStream;
          TIdMessageDecoderMIME(Decoder).FreeSourceStream := False;
          TIdMessageDecoderMIME(Decoder).MIMEBoundary := Boundary;

          Decoder.ReadHeader;
          case Decoder.PartType of
            mcptText: begin
              BodyStream := TMemoryStream.Create;
              try
                NewDecoder := Decoder.ReadBody(BodyStream, MsgEnd);
                try

                finally
                  Decoder.Free;
                  Decoder := NewDecoder;
                end;
              finally
                BodyStream.Free;
              end;
            end;

            mcptAttachment: begin
              BodyStream := TMemoryStream.Create;
              try
                NewDecoder := Decoder.ReadBody(BodyStream, MsgEnd);
                try

                finally
                  Decoder.Free;
                  Decoder := NewDecoder;
                end;
              finally
                BodyStream.Free;
              end;
            end;

            mcptIgnore: begin
              FreeAndNil(Decoder);
              Decoder := TIdMessageDecoderMIME.Create(nil);
            end;

            mcptEOF: begin
              FreeAndNil(Decoder);
              MsgEnd := True;
            end;
          end;
        until (Decoder = nil) or MsgEnd;
      finally
        Decoder.Free;
      end;
    finally
      TCPStream.Free;
    end;
  end;

  ...
end;

An added benefit of this solution is that you can use the TIdHTTP.OnHeadersAvailable event to set the hoNoReadMultipartMIME flag on a per-response basis after examining the HTTP response headers before the response body content is read. That way, you can dynamically decide whether to enable the flag on certain “multipart/…” types, like “multipart/x-mixed-replace”, or to disable it on other “multipart/…” types, like “multipart/byteranges”, depending on how and when you want to parse the MIME data. – in real-time while the response is still being sent, or from a local TStream after the response has been completed. For example:

procedure TMyForm.HTTPHeadersAvailable(Sender: TObject; AHeaders: TIdHeaderList; var VContinue: Boolean);
begin
  if IsHeaderMediaType(AHeaders.Values['Content-Type'], 'multipart/x-mixed-replace') then
    HTTP.HTTPOptions := HTTP.HTTPOptions + [hoNoReadMultipartMIME]
  else
    HTTP.HTTPOptions := HTTP.HTTPOptions - [hoNoReadMultipartMIME];
  VContinue := True;
end;

...

var
  Resp: TStream;
begin
  ...
  Resp := TMemoryStream.Create;
  try
    HTTP.Get(..., Resp);
    if IsHeaderMediaType(HTTP.Response.ContentType, 'multipart/x-mixed-replace') then
    begin

    end else
    begin

    end;
  finally
    Resp.Free;
  end;
  ...
end;

UPDATE March 24, 2021: in earlier versions of TIdHTTP, the TCP connection would be closed when TIdHTTP is done processing a response if an HTTP keep-alive is not being used.  When the hoNoReadMultipartMIME flag is enabled and an HTTP keep-alive is not used, this would cause a premature closure of the connection before user code can read the MIME data from the IOHandler.  As of GitHub checkin 0F1D4047, this has now been corrected, so that the connection is no longer closed if the hoNoReadMultipartMIME flag is enabled and the response is a “multipart/…” type, regardless of whether or not an HTTP keep-alive is used.