Welcome, Guest. Please login or register.


Author Topic: Wiznet 5100/5300 /etc and Enterprise (Read 23781 times)

Offline Prodatron

  • EP user
  • *
  • Posts: 256
  • Country: de
  • Back on the Z80
    • http://www.symbos.de
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #75 on: 2015.July.05. 21:52:57 »
Hmm, that sounds odd for me. Because, what happens if you already read the packet info byte from RX buffer describing the data size, but meanwhile new packet arrives? The old packet info cannot be modified as it's already read ... That's why I thought, the new packet is stored then with its own packet-info word in the RX buffer. What I also suspected that the TCP aligned mode is just for that you mentioned: then it's one stream without any packet info in TCP mode.

Ah OK, I got your point now, but I'm afraid it's not so OK with the w5300! Just checked the "w5100 vs w5300" spec from Wiznet. It seems w5100 works like you mentioned, but w5300 does not, and it has packet info even in TCP mode (not just in UDP, though it's much more simple, only a word for packet size in TCP mode). Unless if you set TCP aligned mode. I think the difference is because w5300 being so much 16 bit organized (and maybe w5100 is not? I don't know). Thus, if TCP data would be one stream with w5300 as you mentioned then there will be problem that an odd sized packet followed by a newer packet you wouldn't know that a byte should be skipped because it's just for the word alignment "dummy" byte. That is exactly what is TCP aligned mode is for: according the specification if you are SURE that only even sized packets received (that's something I find funny, because how you can be sure ...), you can set TCP align mode, and indeed, then it will work as you mentioned: a single stream without header in  TCP mode. Just because of the facts above, I doubt it's useful with that constraint we can't be sure about in a general TCP/IP solution at least (I can imagine that with a specific protocol a designer can make sure about this ...). Or it's also possible that I misunderstood something, of course.

It seems it's kinda handy to read that w5100 vs w5300 comparsion sheet from Wiznet. For me it does not mean too much (as I don't know w5100) but for you, it can be useful.

Ops, ok, I also had a look in the W5300 manual again, and yes, there is this word in front of every package.
Ok, TCP aligned mode doesn't sound good, as it is always possible to receive packages with an odd number of bytes (best example: Telnet, where you sometimes send one single bytes in one whole package).
Because of the 16bit issue this length word in front of each package makes sense!
So my next idea is to store the word in front of the package in your socket handler. When the app only requests a smaller amount of bytes you update the remaining length in your handler. If you reach the end of the package you fetch the next length word and so on. This requires a little bit more code than the "stream" solution of the W5100 (yes, here everything is just completely byte-based!), but it should still work fine!
And now I also understand how they manage any odd package sizes without problems in the RX buffer when still having this 16bit stuff. That's really interesting!

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #76 on: 2015.July.05. 23:26:51 »
Yes. What I don't understand too much still, what exactly RECV command does, if internal (not transparent to the programmer) counters/pointers for buffers are maintained anyway on each FIFO reg access? According to spec, RECV command notifies that the host (we) received the data packet. Hmm.

"After memory copy, the host should inform the copy completion of data to W5300 by RECV command."

This sentence does not tell: after what kind of copy? A whole packet? Or every packets can be found in RX buffer? And why?

Offline Prodatron

  • EP user
  • *
  • Posts: 256
  • Country: de
  • Back on the Z80
    • http://www.symbos.de
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #77 on: 2015.July.05. 23:33:15 »
Yes. What I don't understand too much still, what exactly RECV command does, if internal (not transparent to the programmer) counters/pointers for buffers are maintained anyway on each FIFO reg access? According to spec, RECV command notifies that the host (we) received the data packet. Hmm.

"After memory copy, the host should inform the copy completion of data to W5300 by RECV command."

This sentence does not tell: after what kind of copy? A whole packet? Or every packets can be found in RX buffer? And why?
For the W5100 it only tells, that (after we copied a special amount of bytes from the buffer) we updated the pointers. It seems, that the W5100 doesn't poll the pointers but requires this command to know, that the pointers have been modified (which makes sense). I hope or guess, that it's the same for the W5300?

Offline BruceTanner

  • EP lover
  • *
  • Posts: 607
  • Country: gb
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #78 on: 2015.July.05. 23:52:18 »
To the user at least it is not a circular buffer - incoming data just goes further and further up the buffer until it runs out. Maybe RECV resets the pointers back to the beginning. Or maybe I am being symplistic - if I was doing it I'd make the buffers circular so the pointers wrapped round the start and maybe that is what happens internally. I wonder what happens if a new packet comes in between you reading the last byte and issuing the RECV?

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #79 on: 2015.July.06. 00:13:41 »
To the user at least it is not a circular buffer - incoming data just goes further and further up the buffer until it runs out. Maybe RECV resets the pointers back to the beginning. Or maybe I am being symplistic - if I was doing it I'd make the buffers circular so the pointers wrapped round the start and maybe that is what happens internally.

I don't think RECV would "reset" the internal pointers. In the specification code examples uses RECV after "reading" one packet with size signaled by the packet-info. Thus there can be other packets in the RX buffer, I think in this case it wouldn't make any sense to reset. I would expect the buffers are some kind of ring buffers by the way as the specification does not write anything other just always things like "you can't receive more" or "transmit more" than the got/free size, and with the "moving further and further" statement it means some kind of circular/ring buffer works as an FIFO. So my best guess is like your, the internal FIFO pointer "wraps around" at the configured buffer size.

Quote
I wonder what happens if a new packet comes in between you reading the last byte and issuing the RECV?

Well yes, that is my question too :) In the examples, specification always show code fragments to read a packet than issuing a RECV command, but in this way, it seems like it would be not needed, because I can't see its purpose, if copying was already done anyway ... Hmm, maybe the memory inside the RX buffer is not freed without a RECV? I mean the RX data size is well modified after each FIFO RX access, but still there is a hidden counter inside to show the free RX size (for receiving new packet by w5300) and it's updated only by the RECV command with the counter value of number of access times of the FIFO RX. Or such. I have not so much better idea, what it is used for. What I mean here: if you have a small RX buffer configured and it's full with a packet, then w5300 can't receive (no free buffer). When reading RX FIFO register for the whole packet still won't allow for w5300 to receive, but at the time of a RECV command the internal counter is updated, thus w5300 can receive as seeing some free RX memory at last. Just guessing ... It would make sense too (for me) to update the internal counter stuff after each RX FIFO read as it would result in greater receiving raw performance (ie a smaller packet would already fit after reading some bytes but not the whole packet yet). But it's possible that it would make too complicated to try to receive after each FIFO access and not only after RECV.

Well even I have the idea (I can't say if it's true) that w5300 designers didn't want to do serious checks, and after the first access of RX FIFO register w5300 is forbidden to receive new packet till the host is done with the FIFO access, signalled by the RECV. So in this way there is no "concurrent" accesses (which always makes things more dangerous / complicated), ie reading by host, writing by w5300 receiver part. It does not mean that there can be only one packet in the RX buffer, I meant that _maybe_ receiving is not possible during copy, but it is possible for any amount of data/packets (till the size of RX buffer, of course) if host signalled the copy to be ready, and doing something different.
« Last Edit: 2015.July.06. 00:16:48 by lgb »

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #80 on: 2015.July.06. 08:59:50 »
That is important: if RECV command is really like that (blocking receiving between first RX FIFO reg access and RECV command) then it's problematic if we only fetch few bytes from w5300 on application's needs, as it blocks the receiving! So then we "should" fetch the whole packet and send RECV A.S.A.P. However if I think correctly, TCP on w5300 is also packet based (ie packet info on every packet) and since w5300 does not allow IP level fragmentation, the max packet size won't be bigger than something like the max MTU can be used on an ethernet network, well about 1500 bytes, minus the protocol overhead.

Offline BruceTanner

  • EP lover
  • *
  • Posts: 607
  • Country: gb
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #81 on: 2015.July.06. 10:29:37 »
Yes and a bit of a pain if you are using all 8 sockets: I do not have the RAM to have 8x1500 byte buffers. I was planning on using just socket 0 for the EXOS device so that would be ok, but later with EXDOS integration I was thinking one socket per mapped network resource. I will do some experiments as soon as I can to see if this really is the case!

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #82 on: 2015.July.06. 10:46:06 »
Yes and a bit of a pain if you are using all 8 sockets: I do not have the RAM to have 8x1500 byte buffers. I was planning on using just socket 0 for the EXOS device so that would be ok, but later with EXDOS integration I was thinking one socket per mapped network resource. I will do some experiments as soon as I can to see if this really is the case!

Yes, that's not a little amount of RAM :( Ok, you can say - in theory - that it's not problem that w5300 can't receive more packets on that socket, as the app is not interested in more bytes yet, so the reason not all bytes are read. However, if app is really slow, when it asks for, the window size tcp information is zero for the network peer and it's kinda slow to "recover" from the stalled connection (even when the app now NEEDs data!) at tcp/ip level then. The specification is a bit messy here, mentioning, that you can force the window size update with an empty packet sending. But things goes quite complicated then, at least at the first glance for me ...

Offline BruceTanner

  • EP lover
  • *
  • Posts: 607
  • Country: gb
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #83 on: 2015.July.06. 10:56:20 »
But things goes quite complicated then, at least at the first glance for me ...
Yes, I did not go beyond a brief glance at that point! :???:

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #84 on: 2015.July.06. 11:01:10 »
But I have to admit that I am a bit lame with EXOS. Since the goal would be also having a proper EXOS integration of network resources (well, I hope so, at least). That is, for example, to able to open an EXOS channel which is a tcp connection for real, so even a BASIC program can do a POP3 connection then to check for mails, or whatever. But then here, I am a bit lost: what happens if app for an open channel like this wants to read X bytes, but only fewer is ready (in the RX buffer). Should we return with less byte read, or wait for further packets? Can EXOS provide the notion of "blocking" and "non blocking" I/O for example?

Also, without a hooked interrupt to check all sockets by EXOS, if a call is blocked because waiting for data, other connections are stalled, as not handled. Ok, EXOS is not multitasking, so you can say, it's not a problem, but then the connected peers through the network may have problem with interpreting the situation as stalled connections because of the receiving window size information is not so much updated and or no ACK is sent back (but hmm I guess ACK is sent by w5300, however I am not sure if it only happens after reading packet and issuing that RECV command!).
« Last Edit: 2015.July.06. 11:12:17 by lgb »

Offline BruceTanner

  • EP lover
  • *
  • Posts: 607
  • Country: gb
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #85 on: 2015.July.06. 12:18:54 »
My initial goal with EXOS integration is for network LOAD/SAVE in BASIC to work (and other applications of course) and I was thinking first of LOAD "ftp:blahblah" etc. (or LOAD "http:blahblah" as we have discussed before). Here the protocol on top of TCP/IP is FTP and the EXOS FTP: device will handle that.

What you are talking about is the EXOS app implementing a different protocol itself on top of TCP/IP, eg POP3 in your example, and I have not given much thought to that, but it needs to be possible. eg open a channel to "NET:192.168.0.20:1001" (can you have two : s and a filename that long?). But at least in this case the application is network-specific so it can work around whatever EXOS's blocking-or-not behaviour is. EXOS has a read status call which says whether a character is ready or not (shame it doesn't say how many are ready! :oops: ) and I would say block read should return whatever is available, possibly less than the user asked for. Surely protocols are designed so the "reader" knows how many bytes he is supposed to read? In the case of POP3 the responses from the server are single text lines which you could read character-by-character, or as a block returning however many are available, and then when a mail message itself is downloaded you already know how many to expect, so you loop doing block reads until you have the message, presumably with a timeout. Avoiding a waiting-for-network loop within EXOS seems like a good idea if at all possible!

Offline Prodatron

  • EP user
  • *
  • Posts: 256
  • Country: de
  • Back on the Z80
    • http://www.symbos.de
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #86 on: 2015.July.06. 12:45:52 »
Yes, that's not a little amount of RAM :( Ok, you can say - in theory - that it's not problem that w5300 can't receive more packets on that socket, as the app is not interested in more bytes yet, so the reason not all bytes are read.
I still wonder what happens, if you read less bytes from the RX_FIFO than stated in the last PACKET_INFO and then send a RECEIVE command. Later you can still read the remaining bytes of the current packet (and send RECEIVE again), as you still know how much bytes are left. It would be strange, if this shouldn't work, as for the W5300 these are only bytes in its memory and nothing else. In this case you don't need these 8x1500 byte buffers on EP side. But who knows... :ds_icon_frown:

The specification is a bit messy here, mentioning, that you can force the window size update with an empty packet sending.
I just read that, and this seems to be a solution for such a problem that I sometimes have with the W5100 - the connection is sometimes stalled when receiving a lot of incoming data! But I think it's not possible to send an empty package? In the example the dummy data still has a size of 1 byte. Couldn't this cause issues with some protocols?

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #87 on: 2015.July.06. 13:08:10 »
My initial goal with EXOS integration is for network LOAD/SAVE in BASIC to work (and other applications of course) and I was thinking first of LOAD "ftp:blahblah" etc. (or LOAD "http:blahblah" as we have discussed before). Here the protocol on top of TCP/IP is FTP and the EXOS FTP: device will handle that.

I see, but I guess then you an even open a channel and use that for reading, if it's usuable with LOAD too. FTP is bit more problematic too, as also a firewall administrator (among many other things) I can say, that FTP is a stone aged horrible protocol, it needs another tcp connection for data, and it can be passive / active FTP which also means the "direction" (who is the listener, I mean). Not counting ASCII/BINARY transfer etc (but I guess binary is OK if you can do that only). I try to avoid FTP if it's possible, eg via HTTP you can download anything too (or even upload, but that's some kind of other stuff, namely the DAV protocol over HTTP). But I can understand of course that many people would be happy with FTP as many sites provides FTP download, which can be used with EP directly then.

Quote
What you are talking about is the EXOS app implementing a different protocol itself on top of TCP/IP, eg POP3 in your example, and I have not given much thought to that, but it needs to be possible. eg open a channel to "NET:192.168.0.20:1001" (can you have two : s and a filename that long?). But at least in this case the application is network-specific so it can work

Btw, this is what I've already talked about: that's EXOS filenames are limited in size, which can be problematic, maybe not here, but even with FTP and/or HTTP. That's why I've suggested the "network resource declaration" with an EXOS command and you can only refer it when opening a channel. Of course, if name is suitable (eg: short enough, and does not contain character(s) forbidden by EXOS) the direct naming (what you've wrote as an example with FTP) is more than OK.

Quote
whatever EXOS's blocking-or-not behaviour is. EXOS has a read status call which says whether a character is ready or not (shame it doesn't say how many are ready! :oops: ) and I would say block read should return whatever is available, possibly less than the user asked for. Surely protocols are designed so the "reader" knows how many bytes he is supposed to read? In the case of POP3 the responses from the server are single text lines which you could read character-by-character, or as a block returning however many are available, and then when a mail message itself is downloaded you already know how many to expect, so you loop doing block reads until you have the message, presumably with a timeout. Avoiding a waiting-for-network loop within EXOS seems like a good idea if at all possible!

I think POP3 is line oriented protocol more or less as iRC is too. What a simple app can do, is try to read "some bytes" and check if it's line terminated. If not, try to read more bytes, by appending the buffer. If line termination is found, then the line can be processed, and the possible left bytes are used again to try the first line terminator, or waiting for more bytes, etc.
Otherwise POP3 is not so hard, there can be an app over plain TCP/IP stream for basic mail client (IMAP at the other hand is more complicated, but it also gives the possibility of mail folders, with POP3 there is not such a thing). Also sending mail is not so complicated, if TLS etc stuff is not needed, the SMTP authentication with the PLAIN scheme can be done (but it requires base64 encoding).

Just, if you think about a simple app: it should poll the network for incoming data, but it has other tasks, ie user keyboard handling, screen updating etc. So what I would do, is checking if there is any data (so EXOS can do this) then I try to read data with size of the maximal possible (for our buffer in EP RAM, I mean). That's my problem, that an EXOS read function requested for n bytes can return with less bytes. Or if it can (it can, if I remember exactly) is it a good strategy, would it work with LOAD etc, or it should wait a bit for more data? Because the "common behaviour" with disk files that EXOS will return "n" bytes if "n" bytes are asked (unless if EOF is got of course ...). So I am not sure if applications are ready for this behaviour, that it will be much more common that read return with less bytes than eg with disk files. Of course a network-aware app (eg POP3 client ...) will know this, but we would like that existing software can be adopted easily just with opening let's say NET:1.2.3.4:110 instead of a name for a file on the disk. And it even includes eg BASIC's LOAD if it can handle this situation well enough, or if it gets less bytes than it requested then it's interpreted as end-of-file, since basically - AFAIK - this can be only the case if end of file is hit meanwhile if you work with disk files.

Offline Zozosoft

  • Global Moderator
  • EP addict
  • *
  • Posts: 14710
  • Country: hu
    • http://enterprise.iko.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #88 on: 2015.July.06. 13:23:46 »
EXOS has a read status call which says whether a character is ready or not (shame it doesn't say how many are ready! :oops: )
What do you think about EXOS 10? It is can be used for network files?

Offline lgb

  • EP addict
  • *
  • Posts: 3563
  • Country: hu
  • æðsta yfirmaður
    • http://lgb.hu/
Re: Wiznet 5100/5300 /etc and Enterprise
« Reply #89 on: 2015.July.06. 13:33:57 »
What do you think about EXOS 10? It is can be used for network files?

Hmm, do you mean EXOS 9, or really 10? But the problem - I think - is not exactly this, but the fact that - as far as I can see - with disk files EXOS will return less bytes than asked on reading only in case of EOF (so no more bytes). Thus, maybe many EXOS apps can think that less bytes read than requested is the sign of end-of-file, so game over. With network channels the situation is not that (so not compatible) as EXOS call for a network channel can return less bytes than requested because of no data is sent by the network peer. But in this case this does not mean end-of-file, but the app should re-try later if more data is received meanwhile.

So basically me problem: EXOS apps maybe used to deal with "real files" behind the notion of "channels", but in our case network channel is a bit different as it's not an "already ready" stream of data on the disk, but received on reading, etc, in "real time". The problem that we can ask (maybe with EXOS 9) if there is any byte to read or not, but we don't know how much is that "any". Also, if EXOS 9 returned with "no characters" then what an app can think: "end of file" (closed connection like stuff) or just try to re-ask later. Maybe network channels are more like KEYBOARD: EXOS device than "real" files! And that is what EXOS apps are maybe not ready. For newly written apps it may be not a problem.
« Last Edit: 2015.July.06. 13:38:07 by lgb »