GET, POST and fetching HTTP with HttpWebRequest

Once upon a time there was a HttpWebRequest.  He tried to GET some HTML from YouTube to feed his web browser. HttpWebRequest had default settings and only the URL was unique, so YouTube said - Go request yourself, Bad Request 400!

I have done some basic investigation and found out that usually it is important to provide the Referrer field. So when I added that field  (request.Referrer = url;) - the request worked fine for the moment...

But later I decided that I would like to POST some data to the site...I set the request method to POST (request.Method = "POST";), set content type to "multipart/form-data", specified multipart boundary and updated ContentLength (request.ContentLength = xxx;). I opened the request stream (request.GetRequestStream();) and started writing POST form data to it (and calling stream.Flush() from time to time).

After I sent the request I needed the response, so I called request.GetResponse() and read out all data from the stream it returned. (BTW - do not forget to close streams :) )

I noticed that the data is not correct, or at least some part of it is missing (the one at the beginning), so I proceeded with the investigation.

I found out that the header Expect is set to 100-continue, and the web site was against that approach. So I needed to turn it off - I used such setting : request.ServicePoint.Expect100Continue = false;

Usually I also set the UserAgent field and Accept field.