 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : The CMM2 needs a VULCAN workalike!!!
| Author | Message | ||||
| Paul_L Guru  Joined: 03/03/2016 Location: United StatesPosts: 769 |
No it's not a game and it has nothing to do with volcanoes or blowtorches or Star Trek! It's an ISAM RDBMS and the CMM2 can run it easily! In 1971 Fred Thompson and Jack Hatfield, while working at Jet Propulsion Laboratories, expanded RETRIEVE into JPLDIS to manage a database of electronic calculators and placed it in the public domain. I didn't notice, I was busy trying to work the bugs out of the B747 Delco INS system at the time. Cecil Wayne Ratliff had previously worked for the Martin Marietta Corporation where he wrote the Viking lander support software as a member of the NASA Viking program flight team. Then he went to work for Jet Propulsion Laboratories, and, in order to provide statistics for the office football pool, he shrank JPLDIS down so it would run on his kit built IMSAI 8080 and, being a Star Trek fan, called it Vulcan. Vulcan never did manage the football pool, but Wayne was flabbergasted when Vulcan became an extraordinarily popular and widely accepted application when marketed under the name dBase by Ashton Tate. It became so popular in small business applications because it fit nicely on a 160 K floppy disk, ran on any 8-bit micro with 16 KB of RAM and a clock running at less than 1 MHz, read the structure of a data table from the table itself, formatted fixed length data records into field variables, maintained multiple indexes of the data records, and had a BASIC like interpreter which could provide I/O screens, indexes and filters. MMbasic can easily do all of this running on the $100 cigar box sized CMM2. This little monster has 100 times the processing power and 40 times the RAM of those early micros, can display text on any VGA monitor and it can write text files on SD cards. There currently is no viable, stand alone, programmable database application. except those that have to run on top of a program stack which includes an operating system, a database backend, and an application development system. Every business owner would be a candidate user. Spreadsheet programs and flat file things like Access can keep data but, without a programming language, they have little ability to filter or modify it. An MMbasic ISAM RDBMS system on the CMM2 would be an instantaneous starting standalone system running on inexpensive hardware without an operating system. It might become as popular as VULCAN was after it was renamed dBase. I think a small group of us geezers should write it! It will not be a trivial application, but it is definitely doable. I will not be terribly useful to this project ... I'm way to old. But I will attempt to begin throwing together a specification for the system. Initially we will have to select a method for storing the structure of the data tables and indexes outside of the program, and for generating the indexes within the program code. If you're interested in helping me then please post here. Paul in NY |
||||
twofingers Guru Joined: 02/06/2014 Location: GermanyPosts: 1676 |
Hi Paul, I had the same idea for a long time - a DBase library for CMM2. A computer is only fully grown up when it has a DBM system, IMHO.  I've worked a lot with Clipper. But I feel - just like you - too old for a project like this - sorry! I hope there are others who also support the idea. Best regards Michael causality ≠ correlation ≠ coincidence |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
FIRST THOUGHTS ABOUT THE ISAM RDBMSs The words table, record, and field take on special meanings in the context of a RDBMS.They are all said to be persistent, that is, they exist before the program which created them runs, and they continue to exist after the program that created them terminates. A data table is a file which resides on a disk and consists of one or more records. A data table record is a string of characters of some definite length, which can be read in it's entirety with a single procedure step, and contains one or more fields of characters. Each field is a string of characters which can be parsed out of the record and assigned to string, integer, or floating point ram resident variables. It may be necessary to convert the type of the variable from string to either integer or floating-point during this assignment. The RDBMS program procedures which create, read and modify the records and their fields must therefore either be hard coded with the structure of each record or must be able to read the structure of a data table from either the table itself or from an associated structure file. If an RDBMS's procedures or hardcoded to work with a specific record structure then that program will only be able to access that one single data table. Hard coding the structure of the data tables records into an RDBMS is a very bad, restrictive idea. If an RDBMS's procedures are capable of reading the structure of the data table from either the table itself, or from an associated structure file then that program should be able to access any data table. Before an RDBMS is able to read the first data record of a table it must do certain things: - It must read the data records structure from a file. - It must assign a string variable memory space to contain an entire record. - It must map the locations and types of the individual field variables in this string. - It can then copy an entire unparsed record string into the reserved string space in RAM. FIRST QUESTIONS Should we store the structure of the record as a header in the table data file, or should we store it in an associated structure file? Since an up to date index will usually be open and since we have 5 MB of RAM available on the Waveshare board should we hold the entire key field and the position in the file for all of the records in a memory array and adjust the index as individual records are modified? Paul in NY |
||||
| lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3472 |
Is there any reason not to use the xBase format, open source, derived from the dBase format (or perhaps copied--the name "dBase" may not be used)? PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
| damos Regular Member  Joined: 15/04/2016 Location: AustraliaPosts: 75 |
What you are really wanting is persistence with the ability to execute queries of some sort. The easiest way would be to add SQLite as a module. This is as much as is available on bigger systems and ticks the box, and is the most compatible with the way the Maximite works. Since the development of ORM systems I hate talking directly to a database. The .Net one I use (Entity Framework) completely isolates you from the database, so the database can be anything from an XML file to a large database cluster. You cannot tell from the code and you don't care (apart for optimizing if you get performance issues). .Net uses LINQ which is a language written to provide this middle layer, so you can query, delete update etc, except all you do is update your local objects and do a Save and it then translates it into SQL and updates the database for you. The cool thing is that you don't actually need a database at all as you can construct them in code or read and write from the file system using XML or JSON serializers. The issue with Maximite is that it is not object oriented. Without that, it is really hard to do anything like ORM as there are no objects. This means that you are forced into the old fashioned CRUD (create, read, update, delete) every time you want a bit of data. In EF is looks like this: Get<People>().Where(w => w.Name == "Fred").OrderBy(o => o.Phone).Take(10).ToList() If we use SQL it just becomes select top 10 * from People where Name='Fred' order by Phone For Maximite this will probably return an 2D array and you need to know the index of each column in the array. This a big step forward. In the language you could just add a command: SQL(query$) and a function SQL(query$) |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
@damos --- One of us is missing something here. SQlite is a great little cross platform package. According to sqlite.org it will run on Android, *BSD, iOS, Linux, Mac, Solaris, VxWorks, and Windows (Win32, WinCE, WinRT). Unfortunately, the CMM2 or any other maxi-mite don't use any of these operating systems. MMBasic runs on the bare metal. I always thought that IBM gave SQL a very well chosen name ... Structured Query Language. It's great at querying a database to produce a listing of all bald male people between the age of 20 and 40 living in zip codes beginning with 127. Attempting to display a logical i/o screen for one record and check the validity of anything you type in is a task for CRUD, an old fashioned procedural language like BASIC, and especially (horrors) COBOL running under CICS. I don't understand how SQlite could do this job. Paul in NY |
||||
| damos Regular Member Joined: 15/04/2016 Location: AustraliaPosts: 75 |
It is an open source project. The idea was that you take the core engine (which I believe is written in C) and add it to the system either directly or as an add-in module. It really depends on how light it is, but CMM2 has a fair bit of room to play with. Assuming it is small enough that this can be done, your database is just a file on the SD card. You should be able to use tools on other platforms to create the database and populate it. Then the CMM2 only needs to implement one command and one function (actually just one function is required) to implement it. Assuming you have a table Person with definition: string FirstName string LastName int Age string Sex data = SQL("select * from Person where FirstName='Geoff') returns a 2-dimensional string array: [ ["Geoff","Smith","27","M"] ["Geoff","Jones","54","M"] ] This is pretty compatible with BASIC. You still may need to convert ages to integers, but it gives you a powerful way to return a recordset that works with a procedural language. To add a record you can use the command SQL("insert into Person values ('Mary','Black','35','F')") To delete all the Geoffs: SQL("delete from Person where Firstname='Geoff'") To update: SQL("update Person set FirstName='Greg' where FirstName='Geoff'") |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
The xbase .dbf file format has been around, in various incarnations, since Wayne Ratliff started developing VULCAN in 1978. It was a very simple file consisting of repeated fixed length records that could be broken down into data fields. The .dbf file's benefit is that it includes a variable length header which provided the following: - the version of dBase that created the file - the date of last update - YYMMDD - the number of records in the file - the number of bytes in the header - the number of bytes in each record - space for nine flags - variable length field descriptor array consisting of - - the field name - - the field type (C|D|F|L|M|N) - - space for four flags - - field length - - field decimal count - - default work area number - - space for 12 flags On command the dBase interpreter would open the .dbf file, read the header, map a buffer area in ROM which would contain and parse one record, copy the first record in the file into the buffer area, and wait for the next command. The .dbf file's problem is that it includes a header. If the header became corrupted the file would be unreadable. Because of the variable length of the header a manual reconstruction of the header by inspecting the actual records was almost impossible. However, it is possible to create a fixed record length data file which is easily parsable visually if you include delimiters between the fields and records. The information contained in the header would be written in a separate structure file with multiple copies. If a structure file was damaged there would usually be a second backup available. If all copies of the structure file were destroyed it would be possible to rebuild it from a visual inspection of the data file. Wayne Ratliff decided not to use separate multiple header files because of the slow speed and restricted size of the available 160 KB floppy disks. Today that restriction no longer exists. I think it might be better to switch to creating multiple copies of separate structure files and then create a data file which just contained data fields, records and delimiters. The programming changes needed to update a data table and structure file instead of a .dbf file will be trivial. For example, adding a record to each requires the following steps, if there is no index: - .dbf file - - write the formatted record to the end of the .dbf file - - re-write the .dbf header to update the number of records - - close the .dbf to update the disk cluster map - separate data table and structure file - - write the formatted record to the end of the data table - - re-write the structure file(s) to update the number of records - - close the data table and structure files to update the disk cluster map I, personally, prefer the use of the separate data table and structure files. What do you think? Paul in NY |
||||
| LouisG Senior Member  Joined: 19/03/2016 Location: AustraliaPosts: 129 |
Hi Paul_L It is an interesting idea. Could you give us some examples of possible applications for a CMM2 data base black box? Also what would be its advantages over present day techniques? I think some market research would be in order before diving into software. ... Louis - |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
Hi Louis Wayne Ratliff's original reason for writing VULCAN was to keep track of he JPL football pool. The possible applications for an ISAM include address lists, retail order status, mailing list maintenance, parts database, and, my favorite, a list of the registered identity of every track greyhound in the U.S. National Greyhound Association registry that I wrote back in 1995. You name it, an ISAM will keep track of it. The big advantage is the presence of the programming language which permits you to filter, organize and report the contents any way you want. Spreadsheets can keep the data but they can't select it and print it out any which way. I guess the market research is already under way. Look at the responses to this thread. - Michael thinks it's a good idea but he's too old to help. - Lizby wonders why I don't use the dBase file format. - Damos likes SQlite, a brilliant C language public domain library written by Dwayne Richard Hipp, PhD, Duke University, who gives the library away for free from his home in Charlotte NC, but SQlite runs on top of an operating system, (it doesn't actually know how to write to a disk and keep the FAT current), which the CMM2 doesn't have. - Your post in which you want examples of possible applications and suggest market research. It doesn't look like very many people see a need for it and are interested in using it. I certainly am not going to bother writing it if nobody wants or needs it. No matter, the CMM2 will do a great job of teaching youngsters how to fire bazookas at boogymen and androids. Paul in NY |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4139 |
Both would need to open, read, write, close, delete etc files. Neither needs an OS to do that, though it would be handy if an OS were there. If SQLite fits it does a very great deal straight away rather than needing a re-implementation of dBase or its like. SQL isn't known for being friendly towards a small-machine-slow-disk-I/O environment. Probably comes down to who wants anything & can/does put in the effort. John |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
@JohnS ---> I guess you're comparing something like SQlite to my VULCAN workalike. I am not trying to duplicate DB2 running on multiple terminals. I think we should create a cheap, single purpose, stand alone, instant on, ISAM DBMS in its own box. It's tentative features will be as follows: - there is no complicated supporting OS - the SD card contains the program and the data - the structure of the data should be kept in a separate file, not the data file - the data file itself should incorporate delimiters making the structure obvious - if a structure file is missing the data should be recoverable - never deletes files - on startup the program displays the first data record - a one line text menu then lets you choose among - - Add Edit Next Previous Browse Search Delete 1-set format 2-new format eXit - implements interactive text i/o screens using format files - not code - reports text data to formatted text files using format files - not code - data can be imported from a structured text file using format files - not code - removing the SD card should be detected by the program - the SD card files can be copied while it is removed - with only one keyboard you don't need to lock records - transaction roll-back or roll-forward is not supported If anyone has any suggestions about additional functionality please let me know. Paul in NY |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4139 |
I got the basics of that and was wanting to mention that SQLite also (as I understand it) doesn't need an OS. The SD card would of course (I expect) similarly contain the program and data (well, unless in the flash but can't quite see why it would be there). Most of the rest of the list would be things internal to SQLite that you can find out about if interested but otherwise it's just software that is used and it works without needing to know that stuff. I've no idea to what extent SQLite makes sense on a smallish device such as the CMM2 - e.g. it may not fit for all I know - but it seems worth investigating if anyone wants data handling facilities other than RYO (roll your own). John Edited 2020-07-23 20:02 by JohnS |
||||
| zeitfest Guru Joined: 31/07/2019 Location: AustraliaPosts: 621 |
The fundamental question of databases is "How much data are you prepared to lose ?" People get upset if their bank account figures are stuffed up. Almost as upset when they are forced to use crap software when there are literally thousands of solid and tested programs already available. On the other hand, the data might just be play data from a pretend store, or marketing tripe that has been carefully generated from thin air or vanity. In that case when there is a crash and the data is lost, you can say "Oh, but it's only supposed to be a play system, and it's sooooo cuuuuuttttte....", and do more programming. Any business worth two bits is going to use an existing package that is known to be working well. |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
Hi Stuart, I AGREE COMPLETELY! That's exactly what made dBase such a high volume program. It was written by one competent programmer. Ashton Tate marketed it well. It attracted competition - Clipper - Fox. So, Computer Associates bought Clipper and Microsoft bought Fox. But then Clipper and Visual Fox Pro were very small parts of CA and Microsoft and they didn't pay attention to them, sales dropped off, and they abandoned both products. Ashton Tate was mismanaged and dBase dropped off in popularity and lost its vision. That left small businesses with Access - a non programable flat file database, and Excel and clones - a spreadsheet, neither of which will do the job. They will keep the data, but they will not extract data in any flexible way. Meanwhile - people wrote work-alikes for IBM's Structured Query Language (mainframers pronounced it SQUEEL). But SQL is NOT A DATABASE, it is a convenient front end query tool which enables you to walk up to a terminal and ask DB2 to create a result table with a compound command something like this SELECT table.fieldname1, table.fieldname2, DATEDIFF(id,start,end) FROM table INNER JOIN (SELECT table.field FROM table) ORDER BY table.field ASC, table.field2 ASC SQL is a fast way of getting information out of a database, but it is NOT a database, it is exactly what the name says it is -- a Structured Query Language. It requires the support of a backend database, like DB2 or Oracle, running on top of an operating system, like VM-VS, Unix, Linux or Windows. So, after the demise of dBase and FoxPro there is NO EXISTING PACKAGE of a database backend together with a programming language to build interactive forms and reports. Access and Excel don't cut the mustard. That's why we need to develop a usable ISAM DBMS program. Geoff has provided the commands to do the job -- opening a file for RANDOM access and extracting sub-strings (fields) from longer strings. We have to figure out how to map the field data to i/o screen designs and to report designs, and how to maintain multiple real time running indexes on combinations of fields. The CMM2 finally has enough power, memory and disk space to do all of this. Paul in NY |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4139 |
Just to mention, direct from https://sqlite.org/index.html "SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world." i.e. it's the query language and database. John |
||||
 Print this page |
 |
The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2025 |
