[Winpcap-users] Is an NDIS 5 PacketReceive routine guaranteed to be handed the *full* packet on a system using NDIS 5?

[Guy Harris]

In NPF_tap(), the BPF filter routine is called only with data from the header buffer (HeaderBuffer) if the difference between LookaheadBuffer and HeaderBuffer is equal to HeaderBufferSize (this appears to assume that the packet will be contained in a single buffer in this case, and that the header buffer contains the entire packet), and, otherwise, called with data from the header buffer and the lookahead buffer.

This requires that all the data being checked by the filter be in either the header buffer or the lookahead buffer.

Packet.dll sets the current lookahead size to the maximum lookahead size, presumably to try to ensure that all the data *is* available.

The MSDN documentation for OID_GEN_MAXIMUM_LOOKAHEAD:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff569599(v=vs.85).aspx

says:

NDIS 6.0 and later miniport drivers do not receive this OID request. NDIS handles this OID with a cached value that miniport drivers supply during initialization.

Upper-layer drivers examine lookahead data to determine whether a packet that is associated with the lookahead data is intended for one or more of their clients.

If the underlying driver supports multipacket receive indications, bound protocol drivers are given full net packets on every indication. Consequently, this value is identical to that returned for OID_GEN_RECEIVE_BLOCK_SIZE.

and the documentation for OID_GEN_CURRENT_LOOKAHEAD:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff569574(v=vs.85).aspx

says:

NDIS handles query and unsuccessful set requests for NDIS 6.0 and later miniport drivers. NDIS obtains the information from the miniport driver during initialization and miniport adapter restart. However, NDIS sends valid set requests to the miniport driver.

For a query, NDIS returns the largest lookahead size from among all the bindings. A protocol driver can set a suggested value for the number of bytes to be used in its binding; however, the underlying miniport driver is never required to limit its indications to the value set.

If the underlying driver supports multipacket receive indications, bound protocol drivers are given full net packets on every indication. Consequently, this value is identical to that returned for OID_GEN_RECEIVE_BLOCK_SIZE.

Does setting OID_GEN_CURRENT_LOOKAHEAD to OID_GEN_MAXIMUM_LOOKAHEAD guarantee, in all NDIS 6 systems, that the entire packet will be available to the PacketReceive routine? Somebody reported a problem with WinPcap 4.1.3 on Windows 10:

https://github.com/the-tcpdump-group/libpcap/issues/455

in which they see traffic from port 80 to the capturing machine if they *don't* have a capture filter of "tcp port 80", but, on Windows 10, but not on Windows 7, *don't* see the traffic if they do have that filter. One possible cause could be that, in Windows 10, the NPF NDIS 5 driver's ReceivePacket routine isn't getting the TCP header, in which case the filter would reject the packet.

The person who filed the bug said, of the device on which he's capturing, "It's an Ethernet Network, but it's over a Open VPN connection." I presume he means OpenVPN:

https://openvpn.net

but I'll ask.

[食肉大灰兔V5]

Hi Guy,

Post by Guy Harris
In NPF_tap(), the BPF filter routine is called only with data from the
header buffer (HeaderBuffer) if the difference between LookaheadBuffer and
HeaderBuffer is equal to HeaderBufferSize (this appears to assume that the
packet will be contained in a single buffer in this case, and that the
header buffer contains the entire packet), and, otherwise, called with data
from the header buffer and the lookahead buffer.

In fact, I didn't quite understand this part when I am doing the porting
job to NDIS 6. AFAIK, like for an Ethernet packet, HeaderBuffer is the
Ethernet header data, and LookaheadBuffer is the data beginning from IP
header. If PacketSize is less than or equal to the given
LookaheadBufferSize, the lookahead buffer contains the entire packet (see
https://msdn.microsoft.com/en-us/library/windows/hardware/ff563246(v=vs.85).aspx).
I don't know why there is a compare between LookaheadBuffer - HeaderBuffer
with HeaderBufferSize. HeaderBuffer can't contain the entire packet (as
there won't be a packet with only Ethernet header and have no content).

Post by Guy Harris
This requires that all the data being checked by the filter be in either
the header buffer or the lookahead buffer.
Packet.dll sets the current lookahead size to the maximum lookahead size,
presumably to try to ensure that all the data *is* available.
https://msdn.microsoft.com/en-us/library/windows/hardware/ff569599(v=vs.85).aspx
NDIS 6.0 and later miniport drivers do not receive this OID
request. NDIS handles this OID with a cached value that miniport drivers
supply during initialization.
Upper-layer drivers examine lookahead data to determine whether a
packet that is associated with the lookahead data is intended for one or
more of their clients.
If the underlying driver supports multipacket receive indications,
bound protocol drivers are given full net packets on every indication.
Consequently, this value is identical to that returned for
OID_GEN_RECEIVE_BLOCK_SIZE.
https://msdn.microsoft.com/en-us/library/windows/hardware/ff569574(v=vs.85).aspx
NDIS handles query and unsuccessful set requests for NDIS 6.0 and
later miniport drivers. NDIS obtains the information from the miniport
driver during initialization and miniport adapter restart. However, NDIS
sends valid set requests to the miniport driver.
For a query, NDIS returns the largest lookahead size from among
all the bindings. A protocol driver can set a suggested value for the
number of bytes to be used in its binding; however, the underlying miniport
driver is never required to limit its indications to the value set.
If the underlying driver supports multipacket receive indications,
bound protocol drivers are given full net packets on every indication.
Consequently, this value is identical to that returned for
OID_GEN_RECEIVE_BLOCK_SIZE.
Does setting OID_GEN_CURRENT_LOOKAHEAD to OID_GEN_MAXIMUM_LOOKAHEAD
guarantee, in all NDIS 6 systems, that the entire packet will be available
to the PacketReceive routine? Somebody reported a problem with WinPcap

I think the caller of ProtocolReceive routine only guarantees that the
HeaderBuffer and LookaheadBuffer together cover till the TCP header. So
it's enough to make a decision for a BPF filter.

Post by Guy Harris
https://github.com/the-tcpdump-group/libpcap/issues/455
in which they see traffic from port 80 to the capturing machine if they
*don't* have a capture filter of "tcp port 80", but, on Windows 10, but not
on Windows 7, *don't* see the traffic if they do have that filter. One
possible cause could be that, in Windows 10, the NPF NDIS 5 driver's
ReceivePacket routine isn't getting the TCP header, in which case the
filter would reject the packet.

I think this is possible. Maybe he should try Npcap to see what happens. As
Npcap doesn't handle a packet in multiple buffers (as a NDIS 6 filter will
always get a whole packet one time).

Post by Guy Harris
The person who filed the bug said, of the device on which he's capturing,
"It's an Ethernet Network, but it's over a Open VPN connection." I presume
https://openvpn.net
but I'll ask.

Cheers,
Yang

[Guy Harris]

Post by Guy Harris
In NPF_tap(), the BPF filter routine is called only with data from the header buffer (HeaderBuffer) if the difference between LookaheadBuffer and HeaderBuffer is equal to HeaderBufferSize (this appears to assume that the packet will be contained in a single buffer in this case, and that the header buffer contains the entire packet), and, otherwise, called with data from the header buffer and the lookahead buffer.

In fact, I didn't quite understand this part when I am doing the porting job to NDIS 6. AFAIK, like for an Ethernet packet, HeaderBuffer is the Ethernet header data, and LookaheadBuffer is the data beginning from IP header. If PacketSize is less than or equal to the given LookaheadBufferSize, the lookahead buffer contains the entire packet (see https://msdn.microsoft.com/en-us/library/windows/hardware/ff563246(v=vs.85).aspx).

That's the NDIS 5 version of the documentation for the PacketReceive function. It would be an ugly incompatibility if NDIS 6's NDIS 5 shims didn't preserve that behavior, but it's possible that it does.

Apparently OpenVPN has both NDIS 5 and NDIS 6 drivers:

https://community.openvpn.net/openvpn/wiki/CodeRepositories

I'm guessing that the NDIS 6 driver is being used on his Windows 10 system; perhaps it does something that the NDIS 6 -> NDIS 5 shim doesn't handle correctly.

I don't know why there is a compare between LookaheadBuffer - HeaderBuffer with HeaderBufferSize. HeaderBuffer can't contain the entire packet (as there won't be a packet with only Ethernet header and have no content).

I think what it's doing is clearer if it's written as

if((PUCHAR)LookaheadBuffer != ((PUCHAR)HeaderBuffer)+HeaderBufferSize)

which tests whether the lookahead buffer begins immediately after the end of the header buffer, i.e. that there's really only one buffer; the comment before it indicates that:

//
//Check if the lookahead buffer follows the mac header.
//If the data follow the header (i.e. there is only a buffer) a normal bpf_filter() is
//executed on the packet.
//Otherwise if there are 2 separate buffers (this could be the case of LAN emulation or
//things like this) bpf_filter_with_2_buffers() is executed.
//

I think the caller of ProtocolReceive routine only guarantees that the HeaderBuffer and LookaheadBuffer together cover till the TCP header.

Meaning that they will cover the TCP header, or that they only guarantee that the stuff *before* the TCP header is provided?

So it's enough to make a decision for a BPF filter.

...that's not enough for a BPF filter such as "tcp port 80" that tests fields in the TCP header.

If it's the former, even that's not enough for BPF filters that explicitly test past the end of the TCP header, which *is* possible, and even supported by the libpcap filter language by using the general expression mechanism. But that wouldn't cause a problem with "tcp port 80".

I suspect that the NDIS 6 -> NDIS 5 shim in Windows 10 might not be handling packets provided by the OpenVPN driver - perhaps that driver is providing packets in a form that the shim can't properly map to something to hand to an NDIS 5 protocol driver like the WinPcap driver; Npcap is an NDIS 6 driver, so the shim isn't in the way.

[食肉大灰兔V5]

Hi Guy,

Post by Guy Harris

Post by 食肉大灰兔V5

Post by Guy Harris
In NPF_tap(), the BPF filter routine is called only with data from the

header buffer (HeaderBuffer) if the difference between LookaheadBuffer and
HeaderBuffer is equal to HeaderBufferSize (this appears to assume that the
packet will be contained in a single buffer in this case, and that the
header buffer contains the entire packet), and, otherwise, called with data
from the header buffer and the lookahead buffer.

Post by 食肉大灰兔V5
In fact, I didn't quite understand this part when I am doing the porting

job to NDIS 6. AFAIK, like for an Ethernet packet, HeaderBuffer is the
Ethernet header data, and LookaheadBuffer is the data beginning from IP
header. If PacketSize is less than or equal to the given
LookaheadBufferSize, the lookahead buffer contains the entire packet (see
https://msdn.microsoft.com/en-us/library/windows/hardware/ff563246(v=vs.85).aspx
).
That's the NDIS 5 version of the documentation for the PacketReceive
function. It would be an ugly incompatibility if NDIS 6's NDIS 5 shims
didn't preserve that behavior, but it's possible that it does.

OK. I thought LookaheadBuffer will always start from the end of
HeaderBuffer before this post. I looked twice at the MSDN and found
Microsoft never promised this assumption in NDIS 5, so it's reasonable for
WinPcap to split this into 2 conditions. Although for NDIS 6, **usually**
MAC, IP and TCP headers are carried in the same buffer (I confirmed this in
Npcap, also see the figure in: http://codemachine.com/article_ndis6nbls.html),
but Microsoft never guarantees it.

Post by Guy Harris
https://community.openvpn.net/openvpn/wiki/CodeRepositories
I'm guessing that the NDIS 6 driver is being used on his Windows 10
system; perhaps it does something that the NDIS 6 -> NDIS 5 shim doesn't
handle correctly.

I have a glance at the repo, and I think the NDIS 6 version OpenVPN driver
is a NDIS 6 miniport. A miniport is really far away with a protocol (NDIS 5
or 6) (see:
http://stackoverflow.com/questions/18257048/ndis-filter-driver-filterreceivenetbufferlists-handler-isnt-called?noredirect=1#comment27011779_18257048).
So it's weird that OpenVPN could affect the WinPcap driver. A miniport
should be transparent to a protocol.

Post by Guy Harris

Post by 食肉大灰兔V5
I don't know why there is a compare between LookaheadBuffer -

HeaderBuffer with HeaderBufferSize. HeaderBuffer can't contain the entire
packet (as there won't be a packet with only Ethernet header and have no
content).
I think what it's doing is clearer if it's written as
if((PUCHAR)LookaheadBuffer !=
((PUCHAR)HeaderBuffer)+HeaderBufferSize)
which tests whether the lookahead buffer begins immediately after the end
of the header buffer, i.e. that there's really only one buffer; the comment
//
//Check if the lookahead buffer follows the mac header.
//If the data follow the header (i.e. there is only a buffer) a
normal bpf_filter() is
//executed on the packet.
//Otherwise if there are 2 separate buffers (this could be the
case of LAN emulation or
//things like this) bpf_filter_with_2_buffers() is executed.
//

Post by 食肉大灰兔V5
I think the caller of ProtocolReceive routine only guarantees that the

HeaderBuffer and LookaheadBuffer together cover till the TCP header.
Meaning that they will cover the TCP header, or that they only guarantee
that the stuff *before* the TCP header is provided?

Sorry for my english, which I mean NDIS will guarantee the TCP header is
contained in LookaheadBuffer (see:
http://www.verycomputer.com/5_8593f09f8b078987_1.htm)

Post by Guy Harris

Post by 食肉大灰兔V5
So it's enough to make a decision for a BPF filter.

...that's not enough for a BPF filter such as "tcp port 80" that tests
fields in the TCP header.
If it's the former, even that's not enough for BPF filters that explicitly
test past the end of the TCP header, which *is* possible, and even
supported by the libpcap filter language by using the general expression
mechanism. But that wouldn't cause a problem with "tcp port 80".

You are right, if an application layer filter (like HTTP or FTP) is set, I
don't think the current WinPcap implementation will function well, as I
didn't see any Microsoft document said they will promise the length of
LookaheadBuffer up to application layer. So
WinPcap's PacketTransferDataComplete should also do some kind of BPF filter
work if NPF_Tap's filter fails with inadequate buffer length.

Post by Guy Harris
I suspect that the NDIS 6 -> NDIS 5 shim in Windows 10 might not be
handling packets provided by the OpenVPN driver - perhaps that driver is
providing packets in a form that the shim can't properly map to something
to hand to an NDIS 5 protocol driver like the WinPcap driver; Npcap is an
NDIS 6 driver, so the shim isn't in the way.

Possible.


Cheers,
Yang