Faking browser client in HttpWebRequest

Introduce your request to server as a browser request

Using HttpWebRequest to fetch some data in your code from web is the easiest way to do it. However some web locations have restrictions to all only requests which come from browsers.

Performing a normal HttpWebRequest will return 403 status code (HTTP 403 - Wikipedia) which means forbidden access. The following example where W3C validator is invoked, response will be returned with 403 status code as this web server will deny any request which is not coming from web browser.

    class Program
    {
        static void Main(string[] args)
        {
            string urlToCheck = "";
            HttpWebRequest request = HttpWebRequest.Create(string.Concat("http://validator.w3.org/check?uri=", HttpUtility.UrlEncode(urlToCheck))) as HttpWebRequest;
            request.Method = "GET";
            Console.WriteLine(request.RequestUri.AbsoluteUri);
            WebResponse response = request.GetResponse();
            foreach (string key in response.Headers.AllKeys)
            {
                Console.WriteLine(string.Concat( key," = ", response.Headers[key]));
            }
            Console.ReadLine();
        }
    }
    

Code above will throw System.Net.WebException with message "The remote server returned an error: (403) Forbidden." The reason for this is in W3C server which blocks all requests which are not coming from web browser application.

In order to get response, we need to fake browser so that web server thinks that we are accessing with our request from a browser.

This can be achieved by simple adding of browser signature headers in a request. One of values is agent signature. To get this value you can write a simple JavaScript in HTML page and open in any browser.

document.write(navigator.userAgent);
    

You can use string which you get when you open your HTML page with this line of script and make you request with this value as UserAgent property value in your request.

static void Main(string[] args)
{
string urlToCheck = "";
HttpWebRequest request = HttpWebRequest.Create(string.Concat("http://validator.w3.org/check?uri=", HttpUtility.UrlEncode(urlToCheck))) as HttpWebRequest;
request.Method = "GET";
            /* Sart browser signature */
            request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
            request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
            request.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us,en;q=0.5");
            /* Sart browser signature */

            Console.WriteLine(request.RequestUri.AbsoluteUri);
            WebResponse response = request.GetResponse();
            foreach (string key in response.Headers.AllKeys)
            {
                Console.WriteLine(string.Concat( key," = ", response.Headers[key]));
            }
            Console.ReadLine();
        }
    

This code will manage to get response from W3C validator webpage and you get list of response headers printed out in console

Fake Browser Request

References

Disclaimer

Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.


About the author

DEJAN STOJANOVIC

Dejan is a passionate Software Architect/Developer. He is highly experienced in .NET programming platform including ASP.NET MVC and WebApi. He likes working on new technologies and exciting challenging projects

CONNECT WITH DEJAN  Loginlinkedin Logintwitter Logingoogleplus Logingoogleplus

JavaScript

read more

SQL/T-SQL

read more

Umbraco CMS

read more

PowerShell

read more

Comments for this article