winhttp 网页抓取，window.onload 接收信息不完整。

cmcboy · 发表于 2015-5-14 16:43

求助，通过学习论坛中的网页抓取，给我的工作带来了很大帮助，但近几天有个网页，一直无法提取。现象如下：
1、通过IE可以正常浏览到数据29232字节，但通过VBA 只能提取到688字节
ie浏览器通过fiddler4 的回应报文如下：后面还有很多，没有全部粘贴
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<div id="slloading" style="font-size:12px;">正在查询中...</div>

<script language="JavaScript">
function delloading(){
document.all["slloading"].outerHTML="";
}
window.onload=delloading;
</script>

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=GBK">
<link rel="stylesheet" type="text/css" href="/css/bims.css">
<title>XXXXXXXXXX</title>
</head>
<body style="padding:10px;">
<table border="0" style="width:100%;">
<tr>
      <td colspan="29"><b>XXXXXXXX</b></td>
</tr>
<tr>
      <td colspan="2">
         <table class="tableform" border="0" cellpadding="0" cellspacing="0" style="width:100%;">
            <tr>
                  <td>
                     <table class="tableform" border="0" cellpadding="4" cellspacing="1" style="width:100%;">
                        <tr>
                              <td colspan="5">
                                 <table height="20" style="width:100%;">
                                    <tr>
                                          <td>总条数:8301条
                                          </td>
                                          <td>
                                             1 <a class='page_a_link' href='query_user_info_computer_e.jsp?

2、通过vba ，fiddler4 回应
0296
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<div id="slloading" style="font-size:12px;">正在查询中...</div>

<script language="JavaScript">
function delloading(){
document.all["slloading"].outerHTML="";
}
window.onload=delloading;
</script>
0002

0000
3、VBA查看响应报文如下：
HTTP/1.1 200 OK
Connection: close
Date: Thu, 14 May 2015 08:38:21 GMT
Transfer-Encoding: chunked
Content-Type: text/html;charset=GBK
X-Powered-By: Servlet/2.5 JSP/2.1

各位大侠，请帮助一下。谢谢。

liucqa · 发表于 2015-5-14 18:17

你用fiddler看看数据在哪，有些数据不在一个链接里面

cmcboy · 发表于 2015-5-15 15:39

liucqa 发表于 2015-5-14 18:17
你用fiddler看看数据在哪，有些数据不在一个链接里面

我用FIDDLER 抓包，相关的信息就在这个页面下面，我对html 代码不是特别懂，我理解是用浏览器进行加载时，使用函数window.onload 加载分包接入到的信息（返回的包是transfer-encoding:chunked)的包。winhttp 接受到包含js代码的包是否自动运行呀？

liucqa · 发表于 2015-5-15 16:38

cmcboy 发表于 2015-5-15 15:39
我用FIDDLER 抓包，相关的信息就在这个页面下面，我对html 代码不是特别懂，我理解是用浏览器进行加载时， ...

看textview

VBA万岁 · 发表于 2015-5-15 16:59

方便上传你的代码码？

cmcboy · 发表于 2015-5-15 18:02

liucqa 发表于 2015-5-15 16:38
看textview

看fiddler 中的textview 显示如下。
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<div id="slloading" style="font-size:12px;">正在查询中...</div>

<script language="JavaScript">
function delloading(){
document.all["slloading"].outerHTML="";
}
window.onload=delloading;
</script>

cmcboy · 发表于 2015-5-15 18:08

VBA万岁发表于 2015-5-15 16:59
方便上传你的代码码？

Sub 添加新的端口()
Dim objwinhttp As Object
Dim objxmlhttp As Object
Dim doc As MSHTML.HTMLDocument
Dim aar() As String
Dim filter As String
Dim fileToOpen
Dim URL As String
Dim userhd() As String
Dim newdslamid As String
Dim Regex As Object
Set Regex = CreateObject("VBScript.RegExp")
Dim matchs As Object, match As Object
Dim x As String
Set objwinhttp = CreateObject("WinHttp.WinHttpRequest.5.1")
Set objxmlhttp = CreateObject("Msxml2.ServerXMLHTTP.6.0")
mname = ActiveWorkbook.Name
UserName = Sheets("main").Cells(2, 2)
pass = Sheets("main").Cells(3, 2)
bname = Sheets("main").Cells(5, 2)
ds = Sheets("main").Cells(9, 2)
accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*"
useragent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; Tablet PC 2.0)"
contype = "application/x-www-form-urlencoded"
alive = "Keep -Alive"
leg = 24
URL = "http://xxx.xxx.xxx.xxx:9009"
referer = "http://xxx.xxx.xxx.xxx:9009/login.jsp"
Call firstgetData(objwinhttp, URL)
If objstatus <> "200" Then
End
End If
URL = "http://xxx.xxx.xxx.xxx:9009/auth.jsp"
referer = "http://xxx.xxx.xxx.xxx:9009/login.jsp"
data = "login=" & UserName & "&passwd=" & pass
leg = Len(data)
Call postData(objwinhttp, URL)
If objstatus <> "302" Then
End
End If
URL = "http://xxx.xxx.xxx.xxx:9009/new_left.jsp?topid=40"
referer = "http://xxx.xxx.xxx.xxx:9009/new_top.jsp"
leg = 0
Call getData(objwinhttp, URL)
If objstatus <> "200" Then
End
End If
newusername = Sheets("edit").Cells(2, 2)
URL = "http://xxx.xxx.xxx.xxx:9009/4/query_user_info_computer_e.jsp?userid=&loginname=" & newusername & "bindingphone=&usertype=&username=&branch=&state=&isbinding=&realport=&atu_c=&vp=&vc=&switch=&su=&ip=&equipsource=&discnt=&ratiotype=&displaycon=userid%7Eusername%7Estate%7Ebranch%7Eisbinding%7Eatu_c%7Eusertype%7Esplitter%7Eswitch%7Edslamip%7Edslamid%7E"
referer = "http://xxx.xxx.xxx.xxx:9009/4/query_user_info_computer_e_toolbar.jsp"
Call getDataxml(objxmlhttp, URL)
Sheets("temp").Cells(1, 1) = getrptext
以下省略
Private Sub firstgetData(objwinhttp As Object, URL As String)
With objwinhttp
.Option(6) = False '不重定向
.Open "get", URL, False '获取第一个动态的POST参数，一般是用GET提交得到，这一点通过抓包软件可以确认
.setRequestHeader "Connection", "Keep-Alive"
.send
.WaitForResponse
getrptext = .responseText
Cookie = .getResponseHeader("Set-Cookie")
Cookie = Split(Cookie, ";")(0)
objstatus = .Status
End With
End Sub
Private Sub getData(objwinhttp As Object, URL As String)
With objwinhttp
.Option(6) = False '不重定向
.Open "get", URL, False '获取第一个动态的POST参数，一般是用GET提交得到，这一点通过抓包软件可以确认
.setRequestHeader "Accept", accept
If referer <> "" Then
.setRequestHeader "Referer", referer
End If
.setRequestHeader "User-Agent", useragent
.setRequestHeader "Content-Type", contype
.setRequestHeader "Connection", alive
If leg <> 0 Then
.setRequestHeader "Content-Length", leg
End If
'.setRequestHeader "Cookie", Cookie
.send
.WaitForResponse
getrptext = .responseText
objstatus = .Status
End With
End Sub
Private Sub getDatays(objwinhttp As Object, URL As String)
With objwinhttp
.Option(6) = False
'.Option(WinHttpRequestOption_EnableHttp1_1) = False
.Open "GET", URL, False '获取第一个动态的POST参数，一般是用GET提交得到，这一点通过抓包软件可以确认"
.setRequestHeader "Accept", "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*"
.setRequestHeader "Referer", "http://123.81.251.164:7001/4/query_user_info_computer_e_toolbar.jsp"
.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; .NET CLR 1.1.4322)"
'.setRequestHeader "Accept-Encoding", "gzip, deflate"
.setRequestHeader "Connection", "close"
.setRequestHeader "Cookie", Cookie
.setRequestHeader "Host", "123.81.251.164:7001"
.send
.WaitForResponse
getrptext = .responseText
objstatus = .Status
End With
End Sub
Private Sub getDataxml(objxmlhttp As Object, URL As String)
With objxmlhttp
'.Option(6) = False
.Open "GET", URL, False '获取第一个动态的POST参数，一般是用GET提交得到，这一点通过抓包软件可以确认"
.setRequestHeader "Accept", "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*"
.setRequestHeader "Referer", "http://123.81.251.164:7001/4/query_user_info_computer_e_toolbar.jsp"
.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; .NET CLR 1.1.4322)"
.setRequestHeader "Accept-Encoding", "gzip, deflate"
.setRequestHeader "Connection", "Keep-Alive"
.setRequestHeader "Cookie", Cookie
'.setRequestHeader "Hose", "123.81.251.164:7001"
.send
'.WaitForResponse
MsgBox .readyState
If .readyState = 3 Then
getrptext = .responseBody
End If
Debug.Print getrpbody
objstatus = .Status
End With
End Sub
Private Sub postData(objwinhttp As Object, URL As String)
With objwinhttp
.Option(6) = False '不重定向
.Open "post", URL, False '获取第一个动态的POST参数，一般是用GET提交得到，这一点通过抓包软件可以确认
.setRequestHeader "Accept", accept
If referer <> "" Then
.setRequestHeader "Referer", referer
End If
.setRequestHeader "User-Agent", useragent
.setRequestHeader "Content-Type", contype
.setRequestHeader "Connection", alive
If leg <> 0 Then
.setRequestHeader "Content-Length", leg
End If
'.setRequestHeader "Cookie", Cookie
.send (data)
.WaitForResponse
getrptext = .responseText
objstatus = .Status
End With
End Sub

复制代码

liucqa · 发表于 2015-5-15 20:50

cmcboy 发表于 2015-5-15 18:02
看fiddler 中的textview 显示如下。

数据在别的链接吧

		自动登录	找回密码
密码			免费注册

[求助] winhttp 网页抓取，window.onload 接收信息不完整。