TheBackShed.com

Forum Index : Microcontroller and PC projects : Pi-cromite: Next steps

Page 1 of 2

Author

Message

matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8597

Posted: 02:28am 11 May 2017

Copy link to clipboard

Print this post

Time for a new thread and some interesting new functionality

5.3.a24

2017-05-11_123907_mmbasic.zip

This now allows you to program a web server directly in Basic

To use this new version you must start by deleting the current options file as there is a new parameter that must be initialised.

rm .options

There is a new option command to use

OPTION SOCKET nnnnn

This defaults to 80 but can be anything between 1 and 65535.

With MMBasic running, if you try and access the Pi on that socket with no program running you will get a web page saying "no data"

Then you can access the socket from Basic code by simply opening it

OPEN "SOCKET" as #n

You can then use INPUT$ and PRINT to access the socket as though it was a serial port. EOF and LOC also work as expected as shown in the simple demo program below

Dim crlf$=Chr$(13)+Chr$(10)
Open "socket" As #2
Do
Do While Eof(2) 'wait for a request
Loop
Pause 100
'print the request - normally we would parse it
Print loc(2),"Characters received"
Do While Not Eof(2)
Print Input$(1,2);
Loop
'
Print #2,"HTTP/1.0 200 OK"+crlf$+"Server:CPi"+crlf$;
Print #2,"Content-type:text/html"+crlf$+crlf$;
Print #2,"<html><head><title>My test</title></head>";
Print #2,"<body>The body</p>"+Time$+"</body></html>"+crlf$;
Transmit 'needed to close the socket properly
Loop

For socket output the PRINT statement buffers the output in the Pi. The TRANSMIT command is required to send the data and close the client connection

Note that in the example I am current using HTTP 1.0 which does not need a content length. This is tested on Chrome and works well. It doesn't seem to work on Edge. The default page uses HTTP 1.1 and does work on Edge.

I haven't yet implemented a Basic interrupt when characters have been received - this will be next. I'll also look at automatically creating a HTTP 1.1 header including content length. During development I have experienced occasional issues where the program isn't closing the socket properly when exiting leaving it unusable. I hope this isn't the case in the attached. However, in case of problems close and restart MMBasic and if that doesn't work change the socket used with the OPTION command.Edited by matherp 2017-05-12

lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3019

Posted: 03:40am 11 May 2017

Copy link to clipboard

Print this post

Thanks very much for this--a very useful feature given the power of the pi.

PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed

matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8597

Posted: 04:10am 11 May 2017

Copy link to clipboard

Print this post

5.3.a25

2017-05-11_144732_mmbasic.zip

That was easier than I thought thanks to the elegant way Geoff's code works

This version allows an interrupt to be called when a HTTP request arrives and will also optionally automatically construct a HTTP 1.1 header with the correct content length

The syntax of the OPEN command is:

OPEN "SOCKET [,interrupt] [,number chars]" AS #n

Where "interrupt" is the name of the MMbasic subroutine that will handle the HTTP request and "number chars" is the number of characters that need to be received in the request before the interrupt is triggered (exactly the same as for a serial port)

The TRANSMIT command now has an optional parameter

TRANSMIT [HTTP 1.1]

If this is included MMBasic will automatically create a 1.1 header including a correct content-length. This is tested on Chrome and Edge

Example program showing both the new features:

Dim crlf$=Chr$(13)+Chr$(10)
Open "socket,myint,100" As #2
Do
Loop
Sub myint
'print the request - normally we would parse it
Do While Not Eof(2)
Print Input$(1,2);
Loop
'
Print #2,"<html><head><title>My test</title></head>";
Print #2,"<body>The body</p>"+Time$+"</body></html>"+crlf$;
Transmit HTTP 1.1'create a header, transmit the data & close the socket properly
End Sub

Edited by matherp 2017-05-12

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 05:04am 11 May 2017

Copy link to clipboard

Print this post

Now that the content length is known would it not be better to just drop the HTTP 1.0?
Or make the HTTP 1.1 the default?
What happens when 'keep-alive' is requested?

Another interesting thing to consider is supporting 'Web-Sockets'.
This will then allow streaming info which can be used to make 'live' updating websites without the need to refresh the page.
It is supported by all major browser (http://caniuse.com/#feat=websockets )

Especially for monitoring sensors it would be great.

For easy to make webpages it would also be necessary to be able to request static content, like CSS, Script files and images (stored on SD).
Maybe the 'Transmit' command could offer that.
Like Transmit "d:\images\tiger.png" or Transmit "d:\assets\mystyle.css" etc.
Edited by MicroBlocks 2017-05-12

Microblocks. Build with logic.

matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8597

Posted: 05:26am 11 May 2017

Copy link to clipboard

Print this post

Quote Now that the content length is known would it not be better to just drop the HTTP 1.0?
Or make the HTTP 1.1 the default?

The default is no header i.e. make your own. Adding HTTP 1.1 to the transmit creates a 1.1 header otherwise nothing.

Quote Another interesting thing to consider is supporting 'Web-Sockets'.

Give us a chance

Quote For easy to make webpages it would also be necessary to be able to request static content, like CSS, Script files and images (stored on SD).

I'm very much an html beginner - how do they transmit? Point me at some "simple" references and I'll have a look.

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 09:28am 11 May 2017

Copy link to clipboard

Print this post

Web Sockets is not that much different. It is just data going back and forth.

A request for a static file is very similar to a request of a html page.
A html page can be static also!

If you have chrome then opening a developer tools screen (Ctrl+shift+I) will give you all the info you'll need.
Go to the network tab and investigate how a request is made from the browser to the server.
As an example the style sheet used by thebackshed.com is found on http://www.thebackshed.com/windmill/style3.css

The request from the browser would have that URL as well as headers like:
Accept:text/css,*/*;q=0.1

All request for static files are GET.
In your example above you already print the request header so pointing your browser to the Pi and request an image would give you all the info that needs to be parsed.

The easiest way would be to parse the URL and check the extension. If it is .css it is a stylesheet file, if it is .js it is a javascript file, if it is a .png it is a picture, etc.
The simplest would probably be to treat everything as a static file except a certain extension. It is done a similar way on most web servers. Each extension can have its own 'handler'. The default handler is just serving a static file. Take for example a website running on IIS and running ASP.NET. A file with a known ASP.NET extension will invoke the ASP.NET handler which does all the work. Extensions like .asp, .aspx, .ashx etc are all ASP.NET extension and the parser will forward the request to the ASP.NET handler. Everything else goes to the default handler.
My suggestion is to make the TRANSMIT command the default handler.

In the Pi-cromites case it can be done in MMBasic.
It would be best if static files would be served by a single command. Like the transmit command you already have. It would be too much work for MMBasic to do all the serving of pages as you probably only want to handle certain requests.

For example you could program in MMBasic that if the URL ends with .bhtml then it would programmatically build a page, like in your example above.
The test would be easy:
[code]
'get the URL from the request
IF right$(MyURL,6) = ".bhtml" then
CALL ServeActivePage
ELSE
TRANSMIT MyURL
ENDIF
[/code]
TRANSMIT would need to open the file pointed at with the MyURL, get its length, set the right content-headers and send the content of the file. It is very similar as sending html.

One of the great things about browsers is that you have a complete visualization and programming environment ready to use. Javascript and CSS can make webpages dynamic. A lot can be done from a static html file, some css and javascript. For instance
a complete page can be build using html and css. Then using javascript you can send a request for only the data, then parse that data and display it. This would mean that the data coming from the Pi can be very simple. Programming that needs to be done in MMBasic would be minimal.
I do this daily as it is my main job and if you can get the Pi-cromite to serve static files i can assure you it will open a huge amount of possibilities.
Edited by MicroBlocks 2017-05-12

Microblocks. Build with logic.

lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3019

Posted: 10:58am 11 May 2017

Copy link to clipboard

Print this post

MicroBlocks--this is very interesting. Can you provide a barebones example of javascript requesting data--for instance, as a trivial case, the date?

How do you control the timing of the requests from the browser side?

Lance

PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 11:27am 11 May 2017

Copy link to clipboard

Print this post

Lance,

This would be the barebones of it:
[code]
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
alert(xhr.responseText);
}
}
xhr.open('GET', 'http://example.com', true);
xhr.send(null);
[/code]

Timing is easy. You can have a timer or an interval.
[code]
function myFunction() {
alert('Hello');
}
window.setTimeout(myFunction, 1000); //Invokes myFunction after 1000 ms.
[/code]
[code]
window.setInterval(myFunction, 5000); //Invokes myFunction every 5 seconds.
[/code]

The XMLHTTPRequest is normally not blocking, so when using a timer you need to make sure you not do a request while another request is still in progress. The easiest way to do that is to first set a timeout and then in the function itself set a timeout again when the request is finished (xhr.readyStat == 4).

Microblocks. Build with logic.

matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8597

Posted: 11:37am 11 May 2017

Copy link to clipboard

Print this post

Quote TRANSMIT would need to open the file pointed at with the MyURL, get its length, set the right content-headers and send the content of the file. It is very similar as sending html.

This is the bit I can't find any useful information on.

Google for example "http header for sending favicon.ico" and I get nothing that shows the simple header information I need to send or how to package the image data. Do I just append the raw binary data after the header? Does it need some sort of wrapper?

I can dump the request from the client easily

"GET /favicon.ico HTTP/1.1"

but I don't know and can't find what to send back. Same issue for CSS files and other image files. HTML files are easy - the issue there is developing some syntax to merge MMBasic variables into the static file Edited by matherp 2017-05-12

lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3019

Posted: 01:29pm 11 May 2017

Copy link to clipboard

Print this post

matherp said

Quote the issue there is developing some syntax to merge MMBasic variables into the static file

My method may be hardheaded, but I embed markers in my html text ("~" followed by two digits, e.g., "~01", "~43"), and when spinning through the html output, substitute in the values of the basic variables which correspond ("~01" for date, "~02" for time).

There are no doubt more elegant ways.

Lance

PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed

panky

Guru

Joined: 02/10/2012
Location: Australia
Posts: 1098

Posted: 02:20pm 11 May 2017

Copy link to clipboard

Print this post

Peter,

RFO Basic! has a full suite of html and tcpip socket functions so the info on the forum may be helpfull for you.

Doug.

... almost all of the Maximites, the MicromMites, the MM Extremes, the ArmMites, the PicoMite and loving it!

damos
Regular Member

Joined: 15/04/2016
Location: Australia
Posts: 56

Posted: 06:37pm 11 May 2017

Copy link to clipboard

Print this post

I think JSON notation is the best option. Javascript works natively with it and most frameworks (eg ASP MVC) process it seamlessly for you. For example a JSON object can be created like this:

var myObj = { "name$":"John", "age":31, "city$":"New York" };

This can be sent by GET or POST and at the other end the variable name$, age and city$ automatically populated. I tend to use JQUERY to make things browser independent, although the underlying code it pretty much the same.

$.ajax({
url: 'SetTemperature',
type: "get",
datatype: "json",
cache: false,
data: {
temp: 25
},
success: function (data) {
....
}
});

On the web server side you could decode this and call this function:

function SetTemperature(temp)

The return value is returned as data, which can be a class, although Micromite is not quite up to that yet. It would be nice if it could handle structs, and it would be good enough for most situations.

This means all the UI work could be done in the browser on the PI and BASIC is just used for the processing and I/O side.

The biggest requirement is a JSON parser and I think these are available already (try Github). A JSON parser and encoder would be really handy in BASIC as it gives us the ability to handle complex data like structs just as strings, or even as a lightweight database. A really useful example is settings or config files where the JSON is very human readable, yet you can extract value very easily with a parser.

Another alternative that is become very popular approach is angular.js. This basically involves data binding between values of the web page and the BASIC variables.

TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5915

Posted: 07:08pm 11 May 2017

Copy link to clipboard

Print this post

The pace of your work is frightening me Peter.

My WEB programming is worse than primitive but I have something working without much trouble.

I based this crude demo on the syntax of an old data acquisition module that went up in smoke last year.
In the original, I have the main web pages on my server and only fetched the variable data frame form the module. Yes, I still use frames.

Normally you would parse the incoming request and simply send any static pages including images without any modification.
The CGI page is passed for any %xx. the xx tells us the variable to substitute into the file and if you want to have a %, simply place %% in the source.

The MMBasic code

Quote DIM crlf$=CHR$(13)+CHR$(10)
DIM vardata$(10)

OPEN "socket,myint,100" AS #2
DO
LOOP

SUB myint
'print the request - normally we would parse it
DO WHILE NOT EOF(2)
PRINT INPUT$(1,2);
LOOP
'
fn$ = "data.cgi"
' cgi
FOR n = 1 TO 10 ' create some dummy data
vardata$(n) = STR$(RND()*10,2,2)
NEXT n

OPEN fn$ FOR INPUT AS #3
DO WHILE NOT EOF(#3)
x$ = INPUT$(1,#3)
IF x$ = "%" THEN ' we have data to insert
x$ = INPUT$(1,#3)
IF x$ <> "%" THEN ' nope, just a %
x2$ = INPUT$(1,#3)
variable = VAL(x$)*10+VAL(x2$)
x$=vardata$(variable)
ENDIF
ENDIF
PRINT #2,x$;
PRINT x$;
LOOP
PRINT
Transmit HTTP 1.1 'create a header, transmit the data & close
CLOSE #3
END SUB

The data.cgi file which needs to be placed in the same folder as mmbasic.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta http-equiv="Content-Language" content="en-au">
<title>VK7RTV Data</title>
<meta http-equiv="Refresh" content="20">
</head>
<body>
<table width="100%%" height="95"><tr><td width="160" height="90">Status:<br> %06 </td>
<td width="160" height="90">Rx signal strength:<br>%02 dBuV</td>
<td width="160" height="90">Battery Volts:<br>%03 Volts</td>
<td width="160" height="90">Heatsink Temperature:<br>%04 degrees</td>
<td width="160" height="90">Audio level:<br>%05 mV</td>
</tr>
</table>
</body>
</html>

The refresh is set to 20 seconds here but in real life it was 3 seconds.
You would normally not have the print to console statements in the code.

It ran, updating every 20 seconds, with any issues.
Jim

Edited by TassyJim 2017-05-13

VK7JH
MMedit MMBasic Help

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 07:14pm 11 May 2017

Copy link to clipboard

Print this post

JSON is terrible to make or parse from MMBasic.
Just plain CSV formatted data is much easier and on the browser side also easy to use.
Added bonus is that the data size is a lot smaller.

@Matherp, it might be handy to install a TCP/IP sniffer like Wireshark.
You can then see exactly what is being send/received.

Basically any content is send with a content type and length followed by a dump of the data.

CSS
[code]
Accept-Ranges:bytes
Connection:keep-alive
Content-Length:2646
Content-Type:text/css
Date:Fri, 12 May 2017 05:08:40 GMT
ETag:"e62a8134b7d01:0"
Last-Modified:Sun, 05 Jul 2015 15:08:52 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:ASP.NET
[/code]

Javascript
[code]
Accept-Ranges:bytes
Content-Length:22657
Content-Type:application/x-javascript
Date:Fri, 12 May 2017 05:08:39 GMT
ETag:"e92f2382f4ecd1:0"
Last-Modified:Tue, 19 Jun 2012 11:34:37 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:ASP.NET
[/code]

Image (GIF)
[code]
Accept-Ranges:bytes
Connection:keep-alive
Content-Length:2332
Content-Type:image/gif
Date:Fri, 12 May 2017 05:08:41 GMT
ETag:"02cee7bd83ccb1:0"
Last-Modified:Mon, 16 Aug 2010 00:18:00 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:ASP.NET
[/code]

Image (PNG)
[code]
Accept-Ranges:bytes
Connection:keep-alive
Content-Length:491
Content-Type:image/png
Date:Fri, 12 May 2017 05:12:31 GMT
ETag:"0cfef9b032cb1:0"
Last-Modified:Tue, 03 Aug 2010 02:10:00 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:ASP.NET
[/code]

etc...

Microblocks. Build with logic.

TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5915

Posted: 07:14pm 11 May 2017

Copy link to clipboard

Print this post

One comment.
If you are reading GET requests and opening files based on them, you will need very robust error checking. trying to open a non existent file will crash the program. I will consider having an array with the valid file names in it and only serve ones that are in the list.

It's a nasty world we are connecting to...

Jim

VK7JH
MMedit MMBasic Help

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 07:29pm 11 May 2017

Copy link to clipboard

Print this post

A 'trick' that can help is to first check the referer header that is send from the browser. If it is not from a known page then reject the request.
Files can be checked for existence and a 404 error can be send if it does not exist or a 403 access denied if the referer is unknown.

Microblocks. Build with logic.

damos
Regular Member

Joined: 15/04/2016
Location: Australia
Posts: 56

Posted: 08:17pm 11 May 2017

Copy link to clipboard

Print this post

MicroBlocks said JSON is terrible to make or parse from MMBasic.

This is why it is nice to put it in C. For example, this library is lighweight:
JSON Parser

This takes the pain away. It could just drop the results into an array of strings. It is small enough for the Pi version but probably too large for even Plus.

RonnS
Senior Member

Joined: 16/07/2015
Location: Germany
Posts: 120

Posted: 09:46pm 11 May 2017

Copy link to clipboard

Print this post

hello,..

many thanks for the examples, i got Jim's example to run but i need some times to realize that I must use the browser and connect with 127.0.0.1

Thank you Jim ( and all others), this is a good basis to understand what all is meant , i still have a lot to learn

regards RonEdited by RonnS 2017-05-13

lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3019

Posted: 01:56am 12 May 2017

Copy link to clipboard

Print this post

Jim's practice is exactly mine, except that I use "~01" instead of his "%01", and I use Peter's SendText sub from a prior html example to buffer the output, which speeds the process.

Lance

PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed

MicroBlocks

Guru

Joined: 12/05/2012
Location: Thailand
Posts: 2209

Posted: 02:50am 12 May 2017

Copy link to clipboard

Print this post

I use this:
[code]
Temperature: {Temperature}
Humidity: {Humidity}
[/code]
Then i replace them with the values.
Advantage is that you can use the name of the variable so that it is easier when lots of values are used.
Also adding a line will not disturb the order.
I does work best when you have a replace command though.

Microblocks. Build with logic.

Page 1 of 2

Print this page