WebClient Object

Long time since the last post...

I've been working on a project that interfaces with numerous HTTP servers to update the firmware on the hosting machine. I opted to use the WebClient object, mostly because I wasn't sure what the best method would be but figured using WebBrowsers would just be wasteful, and ran into a few problems that were tedious enough to make me want to post them on here.

First thing's first, some servers respond back with an "Expect" error, equating to an exception from the WebClient. The fix:

Code:

System.Net.ServicePointManager.Expect100Continue = false;

Secondly, logging into a website. The WebClient object doesn't handle cookies for you and that sucks, a lot. The easiest way to fix that is by creating a new class, inheriting the WebClient class and overriding the GetWebRequest method.

Code:

class WebClientEx : WebClient
{
private CookieContainer m_container = new CookieContainer();

protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = m_container;
}
return request;
}

Credit for this goes to http://codex.bradrowley.net/

Ok, that gets you cookies but submitting to a form wasn't all that easy...

Code:

// Create the POST form parameters
NameValueCollection nvc = new NameValueCollection();
nvc.Add("authUser", username);
nvc.Add("authPass", password);

// Pause until m_client is available
while (m_client.IsBusy)
{
}

result = System.Text.ASCIIEncoding.ASCII.GetString(
m_client.UploadValues("http://...", "POST", nvc));

And finally, the last thing I had to figure out how to do was upload a file, via a form. Fortunately, I was able to find a piece of code from madmik3 that did the trick.

Code:

// Majority of this function from http://www.codeproject.com/KB/cs/uploadfileex.aspx
public string FormUploadFile(string uploadfile, string url, string fileFormName, string contenttype, NameValueCollection querystring)
{
if ((fileFormName == null) ||
(fileFormName.Length == 0))
{
fileFormName = "file";
}

if ((contenttype == null) ||
(contenttype.Length == 0))
{
contenttype = "application/octet-stream";
}

string postdata;
postdata = "?";
if (querystring != null)
{
foreach (string key in querystring.Keys)
{
postdata += key + "=" + querystring.Get(key) + "&";
}
}
Uri uri = new Uri(url + postdata);


string boundary = "----------" + DateTime.Now.Ticks.ToString("x");
HttpWebRequest webrequest = this.GetWebRequest(uri) as HttpWebRequest; // Changed by Austin 5/22/09 - To account for cookies
webrequest.ContentType = "multipart/form-data; boundary=" + boundary;
webrequest.Method = "POST";


// Build up the post message header
StringBuilder sb = new StringBuilder();
sb.Append("--");
sb.Append(boundary);
sb.Append("\r\n");
sb.Append("Content-Disposition: form-data; name=\"");
sb.Append(fileFormName);
sb.Append("\"; filename=\"");
sb.Append(Path.GetFileName(uploadfile));
sb.Append("\"");
sb.Append("\r\n");
sb.Append("Content-Type: ");
sb.Append(contenttype);
sb.Append("\r\n");
sb.Append("\r\n");

string postHeader = sb.ToString();
byte[] postHeaderBytes = Encoding.UTF8.GetBytes(postHeader);

// Build the trailing boundary string as a byte array
// ensuring the boundary appears on a line by itself
byte[] boundaryBytes =
Encoding.ASCII.GetBytes("\r\n--" + boundary + "\r\n");

FileStream fileStream = new FileStream(uploadfile,
FileMode.Open, FileAccess.Read);
long length = postHeaderBytes.Length + fileStream.Length +
boundaryBytes.Length;
webrequest.ContentLength = length;

Stream requestStream = webrequest.GetRequestStream();

// Write out our post header
requestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);

// Write out the file contents
byte[] buffer = new Byte[checked((uint)Math.Min(4096,
(int)fileStream.Length))];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
requestStream.Write(buffer, 0, bytesRead);

// Write out the trailing boundary
requestStream.Write(boundaryBytes, 0, boundaryBytes.Length);
WebResponse responce = webrequest.GetResponse();
Stream s = responce.GetResponseStream();
StreamReader sr = new StreamReader(s);

return sr.ReadToEnd();
}

Stripping HTML from text in SQL Server 2005

Today I came across a problem where a field in a SQL server 2005 database contained encoded HTML (for example, it looked like "<div>Blah blah"). However, as this field was required in a SQL server report, this HTML needed to either be removed or formatted. Since SQL reports don't have a built-in function for this, and trying to create a function for it using their Excel-style expressions didn't sound like much fun, the only option left was to trim it in the SQL code.

Normally, I'd turn to a regular expression to handle this, however in SQL server, to use regular expressions you need to mess about with COM to get a VBScript interface etc, and I'm lazy, so I wanted a simpler way. A bit of research turned up an article at http://blog.sqlauthority.com/ which appeared to do exactly what I needed using just a normal user-defined function, and as luck would have it, it had only been written 11 days ago, so my timing was pretty good ;) Of course, it wasn't exactly what I needed, but it was pretty close, and after a few changes (mostly copy-paste code to handle each HTML entity that was likely to appear), I had a lovely new function which did exactly what I needed it to do. The performance of it is probably pretty dire, but that's acceptable in this case.

Here's the code, should anyone need it:

ALTER FUNCTION [dbo].[udf_StripHTML]
(
@HTMLText varchar(MAX)
)
RETURNS varchar(MAX)
AS
BEGIN
DECLARE @Start  int
DECLARE @End    int
DECLARE @Length int

-- Replace the HTML entity & with the '&' character (this needs to be done first, as
-- '&' might be double encoded as '&')
SET @Start = CHARINDEX('&', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, '&')
SET @Start = CHARINDEX('&', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1
END

-- Replace the HTML entity &lt; with the '<' character
SET @Start = CHARINDEX('&lt;', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, '<')
SET @Start = CHARINDEX('&lt;', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1
END

-- Replace the HTML entity &gt; with the '>' character
SET @Start = CHARINDEX('&gt;', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, '>')
SET @Start = CHARINDEX('&gt;', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1
END

-- Replace the HTML entity &amp; with the '&' character
SET @Start = CHARINDEX('&amp;amp;', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, '&')
SET @Start = CHARINDEX('&amp;amp;', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1
END

-- Replace the HTML entity &nbsp; with the ' ' character
SET @Start = CHARINDEX('&nbsp;', @HTMLText)
SET @End = @Start + 5
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, ' ')
SET @Start = CHARINDEX('&nbsp;', @HTMLText)
SET @End = @Start + 5
SET @Length = (@End - @Start) + 1
END

-- Replace any <br> tags with a newline
SET @Start = CHARINDEX('<br>', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, CHAR(13) + CHAR(10))
SET @Start = CHARINDEX('<br>', @HTMLText)
SET @End = @Start + 3
SET @Length = (@End - @Start) + 1
END

-- Replace any <br/> tags with a newline
SET @Start = CHARINDEX('<br/>', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, 'CHAR(13) + CHAR(10)')
SET @Start = CHARINDEX('<br/>', @HTMLText)
SET @End = @Start + 4
SET @Length = (@End - @Start) + 1
END

-- Replace any <br /> tags with a newline
SET @Start = CHARINDEX('<br />', @HTMLText)
SET @End = @Start + 5
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, 'CHAR(13) + CHAR(10)')
SET @Start = CHARINDEX('<br />', @HTMLText)
SET @End = @Start + 5
SET @Length = (@End - @Start) + 1
END

-- Remove anything between <whatever> tags
SET @Start = CHARINDEX('<', @HTMLText)
SET @End = CHARINDEX('>', @HTMLText, CHARINDEX('<', @HTMLText))
SET @Length = (@End - @Start) + 1

WHILE (@Start > 0 AND @End > 0 AND @Length > 0) BEGIN
SET @HTMLText = STUFF(@HTMLText, @Start, @Length, '')
SET @Start = CHARINDEX('<', @HTMLText)
SET @End = CHARINDEX('>', @HTMLText, CHARINDEX('<', @HTMLText))
SET @Length = (@End - @Start) + 1
END

RETURN LTRIM(RTRIM(@HTMLText))

END

Of course, most of the credit for this goes to Pinal Dave of SQLAuthority.com, I just hacked his code around a bit to suit my particular needs.

Note the formatting is going a bit awry, mostly due to HTML entities in the code. I'll fix this when I get a chance